Reviewerly/exHarmony huggingface dataset:
Deep learning based real-time tourist spots detection and recognition mechanism

More and more information on tourist spots is being represented as pictures rather than text. Consequently, tourists who are interested in a specific attraction shown in pictures may have no idea how to perform a text search to get more information about the interesting tourist spots. In the view of this problem and to enhance the competitiveness of the tourism market, this research proposes an innovative tourist spot identification mechanism, which is based on deep learning-based object detection technology, for real-time detection and identification of tourist spots by taking pictures on location or retrieving images from the Internet. This research establishes a tourist spot recognition system, which is a You Only Look Once version 3 model built in Tensorflow AI framework, and is used to identify tourist attractions by taking pictures with a smartphone's camera. To verify the possibility, a set of tourist spots in Hsinchu City, Taiwan is taken as an example. Currently, the tourist spot recognition system of this research can identify 28 tourist spots in Hsinchu. In addition to the attraction recognition feature, tourists can further use this tourist spot recognition system to obtain more information about 77 tourist spots from the Hsinchu City Government Information Open Data Platform, and then make dynamic travel itinerary planning and Google MAP navigation. Compared with other deep learning models using Faster region-convolutional neural networks or Single-Shot Multibox Detector algorithms for the same data set, the recognition time by the models using You Only Look Once version 3, Faster region-convolutional neural networks, and Single-Shot Multibox Detector algorithms are respectively 4.5, 5, and 9 s, and the mean average precision for each when IoU = 0.6 is 88.63%, 85%, and 43.19%, respectively. The performance experimental results of this research show the model using the You Only Look Once version 3 algorithm is more efficient and precise than the models using the Faster region-convolutional neural networks or the Single-Shot Multibox Detector algorithms, where You Only Look Once version 3 and Single-Shot Multibox Detector are one-stage learning architectures with efficient features, and Faster region-convolutional neural networks is a two-stage learning architecture with precise features.
Generative Adversarial Network for Text-to-Face Synthesis and Manipulation

Over the past few years, several studies have been conducted on text-to-image synthesis techniques, which transfer input textual descriptions into realistic images. However, facial image synthesis and manipulation from input sentences have not been widely explored due to the lack of datasets. My research interests center around the development of multi-modality technology and facial image generation with Generative Adversarial Networks. Towards that end, we propose an approach for facial image generation and manipulation from text descriptions. We also introduce the first Text-to-Face synthesis dataset with large-scale facial attributes. In this extended abstract, we first present the existing condition and further direction of my Ph.D. research that I have followed during the first year. Then, we introduce the proposed method (accepted by IEEE FG2021), annotated novel dataset and experimental results. Finally, the future outlook on other challenges, proposed dataset and expected impact are discussed. Codes and paper lists studied in text-to-image synthesis are summarized on https://github.com/Yutong-Zhou-cv/Awesome-Text-to-Image.
Blockbuster: A New Perspective on Popularity-bias in Recommender Systems

Collaborative filtering algorithms unwittingly produce ranked lists where a few popular items are recommended too frequently while the remaining vast amount of items get not deserved attention, also referred to as the popularity bias problem. Nevertheless, when investigating popularity bias issues in recommendations, the literature commonly estimates the popularity of the items only based on the number of ratings they have received by disregarding their magnitudes. In this study, we evaluate this problem from a different perspective by evaluating the popularity and liking-degree of the items at the same time. To this end, we first develop a method describing the blockbuster items that are both popular and strongly desired by users and propose a helpful metric that measures the potential biases of collaborative filtering algorithms towards such blockbuster items in their produced ranking-based recommendations correctly. The conducted broad set of experiments demonstrate that two popular real-world datasets are highly imbalanced towards blockbuster items. Also, most prominent collaborative algorithms, especially matrix factorization-based SVD and SVD++, lead to an undesirable bias in their recommendations towards blockbuster items.
Database-adaptive Re-ranking for Enhancing Cross-modal Image Retrieval

We propose an approach that enhances arbitrary existing cross-modal image retrieval performance. Most of the cross-modal image retrieval methods mainly focus on direct computation of similarities between a text query and candidate images in an accurate way. However, their retrieval performance is affected by the ambiguity of text queries and the bias of target databases (DBs). Dealing with ambiguous text queries and DBs with bias will lead to accurate cross-modal image retrieval in real-world applications. A DB-adaptive re-ranking method using modality-driven spaces, which can extend arbitrary cross-modal image retrieval methods for enhancing their performance, is proposed in this paper. The proposed method includes two approaches: "DB-adaptive re-ranking'' and "modality-driven clue information extraction''. Our method estimates clue information that can effectively clarify the desired image from the whole set of a target DB and then receives user's feedback for the estimated information. Furthermore, our method extracts more detailed information of a query text and a target DB by focusing on modality-driven spaces, and it enables more accurate re-ranking. Our method allows users to reach their desired single image by just answering questions. Experimental results using MSCOCO, Visual Genome and newly introduced datasets including images with a particular object show that the proposed method can enhance the performance of state-of-the-art cross-modal image retrieval methods.
Learning Future-Aware Correlation Filters for Efficient UAV Tracking

In recent years, discriminative correlation filter (DCF)-based trackers have made considerable progress and drawn widespread attention in the unmanned aerial vehicle (UAV) tracking community. Most existing trackers collect historical information, e.g., training samples, previous filters, and response maps, to promote their discrimination and robustness. Under UAV-specific tracking challenges, e.g., fast motion and view change, variations of both the target and its environment in the new frame are unpredictable. Interfered by future unknown environments, trackers that trained with historical information may be confused by the new context, resulting in tracking failure. In this paper, we propose a novel future-aware correlation filter tracker, i.e., FACF. The proposed method aims at effectively utilizing context information in the new frame for better discriminative and robust abilities, which consists of two stages: future state awareness and future context awareness. In the former stage, an effective time series forecast method is employed to reason a coarse position of the target, which is the reference for obtaining a context patch in the new frame. In the latter stage, we firstly obtain the single context patch with an efficient target-aware method. Then, we train a filter with the future context information in order to perform robust tracking. Extensive experimental results obtained from three UAV benchmarks, i.e., UAV123_10fps, DTB70, and UAVTrack112, demonstrate the effectiveness and robustness of the proposed tracker. Our tracker has comparable performance with other state-of-the-art trackers while running at ∼49 FPS on a single CPU.
Robotic Tool Tracking Under Partially Visible Kinematic Chain: A Unified Approach

Anytime a robot manipulator is controlled via visual feedback, the transformation between the robot and camera frame must be known. However, in the case where cameras can only capture a portion of the robot manipulator in order to better perceive the environment being interacted with, there is greater sensitivity to errors in calibration of the base-to-camera transform. A secondary source of uncertainty during robotic control are inaccuracies in joint angle measurements which can be caused by biases in positioning and complex transmission effects such as backlash and cable stretch. In this work, we bring together these two sets of unknown parameters into a unified problem formulation when the kinematic chain is partially visible in the camera view. We prove that these parameters are nonidentifiable implying that explicit estimation of them is infeasible. To overcome this, we derive a smaller set of parameters we call lumped error since it lumps together the errors of calibration and joint angle measurements. A particle filter method is presented and tested in simulation and on two real world robots to estimate the lumped error and show the efficiency of this parameter reduction.
A Data-Driven Personalized Lighting Recommender System

Recommender systems attempt to identify and recommend the most preferable item (product-service) to individual users. These systems predict user interest in items based on related items, users, and the interactions between items and users. We aim to build an auto-routine and color scheme recommender system for home-based smart lighting that leverages a wealth of historical data and machine learning methods. We utilize an unsupervised method to recommend a routine for smart lighting. Moreover, by analyzing users' daily logs, geographical location, temporal and usage information, we understand user preferences and predict their preferred light colors. To do so, users are clustered based on their geographical information and usage distribution. We then build and train a predictive model within each cluster and aggregate the results. Results indicate that models based on similar users increases the prediction accuracy, with and without prior knowledge about user preferences.
A multiclass hybrid approach to estimating software vulnerability vectors and severity score

Classifying detected software vulnerabilities is an important process. However, the metric values of security vectors are manually determined by humans, which takes time and may introduce errors stemming from human nature. These metrics are important because of their role in the calculation of vulnerability severity. It is necessary to use machine learning algorithms and data mining techniques to improve the quality and speed of vulnerability analysis and discovery processes. However, studies in this area are still limited. In this study, vulnerability vectors were estimated using the natural language processing techniques bag of words, term frequency–inverse document frequency, and n-gram for feature extraction together with various multiclass classification algorithms, namely Naïve Bayes, decision tree, k-nearest neighbors, multilayer perceptron, and random forest. Our experiments using a large public dataset facilitate assessment and provide a standard-compliant prediction model for classifying software vulnerability vectors. The results show that the joint use of different techniques and classification algorithms is a promising solution to a multi-probability and difficult-to-predict problem. In addition, our study fills an important gap in its field in terms of the size of the dataset used and because it covers a vulnerability scoring system version that has not yet been extensively studied.
Heuristic-Assisted BERT for Twitter Sentiment Analysis

The identification of opinions and sentiments from tweets is termed as “Twitter Sentiment Analysis (TSA)”. The major process of TSA is to determine the sentiment or polarity of the tweet and then classifying them into a negative or positive tweet. There are several methods introduced for carrying out TSA, however, it remains to be challenging due to slang words, modern accents, grammatical and spelling mistakes, and other issues that could not be solved by existing techniques. This work develops a novel customized BERT-oriented sentiment classification that encompasses two main phases: pre-processing and tokenization, and a “Customized Bidirectional Encoder Representations from Transformers (BERT)”-based classification. At first, the gathered raw tweets are pre-processed under stop-word removal, stemming and blank space removal. After pre-processing, the semantic words are obtained, from which the meaningful words (tokens) are extracted in the tokenization phase. Consequently, these extracted tokens are classified via optimized BERT, where biases and weight are tuned optimally by Particle-Assisted Circle Updating Position (PA-CUP). Moreover, the maximal sequence length of the BERT encoder is updated using standard PA-CUP. Finally, the performance analysis is carried out to substantiate the enhancement of the proposed model.
Continual Model-Based Reinforcement Learning with Hypernetworks

Effective planning in model-based reinforcement learning (MBRL) and model-predictive control (MPC) relies on the accuracy of the learned dynamics model. In many instances of MBRL and MPC, this model is assumed to be stationary and is periodically re-trained from scratch on state transition experience collected from the beginning of environment interactions. This implies that the time required to train the dynamics model - and the pause required between plan executions - grows linearly with the size of the collected experience. We argue that this is too slow for lifelong robot learning and propose HyperCRL, a method that continually learns the encountered dynamics in a sequence of tasks using task-conditional hypernetworks. Our method has three main attributes: first, it includes dynamics learning sessions that do not revisit training data from previous tasks, so it only needs to store the most recent fixed-size portion of the state transition experience; second, it uses fixed-capacity hypernetworks to represent non-stationary and task-aware dynamics; third, it outperforms existing continual learning alternatives that rely on fixed-capacity networks, and does competitively with baselines that remember an ever increasing coreset of past experience. We show that HyperCRL is effective in continual model-based reinforcement learning in robot locomotion and manipulation scenarios, such as tasks involving pushing and door opening. Our project website with videos is at this link http://rvl.cs.toronto.edu/blog/2020/hypercrl/
Service-oriented architecture for Internet of Things: A semantic approach

The Internet of Things (IoT) aids an interconnection between systems, humans, and services to develop computation-intensive autonomous applications. Service-oriented architecture (SOA) concepts have been used as defacto software architecture to develop IoT-based systems. Though SOA facilitates various benefits, several challenges exist for integration with IoT-based systems, including configurability, interoperability, and manageability. The ontology-based approach provides a semantically enriched and rigorous platform for interoperating different heterogeneous IoT-based systems and applications. The semantic modelling research has focused on IoT resource management without concerning the method of accessing and utilizing IoT information. In this context, an IoT-based large-scale SOA (IoT-LSS) ontology is proposed that supports interoperability, heterogeneity, flexibility, manageability, extendibility, scalability. This ontology integrates the concept of large-scale SOA (LSS) with semantic sensor network (SSN) ontology. Clinical decision support system (CDSS) ontology has considered as a case study to illustrate the proposed ontology. A detailed comparative analysis between the proposed system and several existing IoT-based service ontologies has been performed based on different characteristics. This analysis shows that IoT-LSS supports dynamic features and exposes to service with its different mechanisms. This article presents an ontological framework for the IoT service-based environment to efficiently use for service composition mechanisms in the large service domain.
Graph-Based Surgical Instrument Adaptive Segmentation via Domain-Common Knowledge

Unsupervised domain adaptation (UDA), aiming to adapt the model to an unseen domain without annotations, has drawn sustained attention in surgical instrument segmentation. Existing UDA methods neglect the domain-common knowledge of two datasets, thus failing to grasp the inter-category relationship in the target domain and leading to poor performance. To address these issues, we propose a graph-based unsupervised domain adaptation framework, named Interactive Graph Network (IGNet), to effectively adapt a model to an unlabeled new domain in surgical instrument segmentation tasks. In detail, the Domain-common Prototype Constructor (DPC) is first advanced to adaptively aggregate the feature map into domain-common prototypes using the probability mixture model, and construct a prototypical graph to interact the information among prototypes from the global perspective. In this way, DPC can grasp the co-occurrent and long-range relationship for both domains. To further narrow down the domain gap, we design a Domain-common Knowledge Incorporator (DKI) to guide the evolution of feature maps towards domain-common direction via a common-knowledge guidance graph and category-attentive graph reasoning. At last, the Cross-category Mismatch Estimator (CME) is developed to evaluate the category-level alignment from a graph perspective and assign each pixel with different adversarial weights, so as to refine the feature distribution alignment. The extensive experiments on three types of tasks demonstrate the feasibility and superiority of IGNet compared with other state-of-the-art methods. Furthermore, ablation studies verify the effectiveness of each component of IGNet. The source code is available at https://github.com/CityU-AIM-Group/Prototypical-Graph-DA.
ABPNet: Adaptive Background Modeling for Generalized Few Shot Segmentation

Existing Few Shot Segmentation (FS-Seg) methods mostly study a restricted setting where only foreground and background are required to be discriminated and fall short at discriminating multiple classes. In this paper, we focus on a challenging but more practical variant: Generalized Few Shot Segmentation (GFS-Seg), where all SEEN and UNSEEN classes are segmented simultaneously. Previous methods treat the background as a regular class, leading to difficulty in differentiating UNSEEN classes from it at the test stage. To address this issue, we propose Adaptive Background Modeling and Prototype Query Network (ABPNet), in which the background is formulated as the complement of the set of interested classes. With the help of the attention mechanism and a novel meta-training strategy, it learns an effective set difference function that predicts task-specific background adaptively. Furthermore, we design a Prototype Querying (PQ) module that effectively transfers the learned knowledge to UNSEEN classes with a neural dictionary. Experimental results demonstrate that ABPNet significantly outperforms the state-of-the-art method CAPL on PASCAL-5i and COCO-20i, especially on UNSEEN classes. Also, without retraining, ABPNet can generalize well to FS-Seg.
Latent Memory-augmented Graph Transformer for Visual Storytelling

Visual storytelling aims to automatically generate a human-like short story given an image stream. Most existing works utilize either scene-level or object-level representations, neglecting the interaction among objects in each image and the sequential dependency between consecutive images. In this paper, we present a novel Latent Memory-augmented Graph Transformer~(LMGT ), a Transformer based framework for visual story generation. LMGT directly inherits the merits from the Transformer, which is further enhanced with two carefully designed components, i.e., a graph encoding module and a latent memory unit. Specifically, the graph encoding module exploits the semantic relationships among image regions and attentively aggregates critical visual features based on the parsed scene graphs. Furthermore, to better preserve inter-sentence coherence and topic consistency, we introduce an augmented latent memory unit that learns and records highly summarized latent information as the story line from the image stream and the sentence history. Experimental results on three widely-used datasets demonstrate the superior performance of LMGT over the state-of-the-art methods.
On security properties of all-or-nothing transforms

All-or-nothing transforms have been defined as bijective mappings on all s-tuples over a specified finite alphabet. These mappings are required to satisfy certain “perfect security” conditions specified using entropies of the probability distribution defined on the input s-tuples. Alternatively, purely combinatorial definitions of AONTs have been given, which involve certain kinds of “unbiased arrays”. However, the combinatorial definition makes no reference to probability definitions. In this paper, we examine the security provided by AONTs that satisfy the combinatorial definition. The security of the AONT can depend on the underlying probability distribution of the s-tuples. We show that perfect security is obtained from an AONT if and only if the input s-tuples are equiprobable. However, in the case where the input s-tuples are not equiprobable, we still achieve a weaker security guarantee. We also consider the use of randomized AONTs to provide perfect security for a smaller number of inputs, even when those inputs are not equiprobable.
Bridging the Gap between Low-Light Scenes: Bilevel Learning for Fast Adaptation

Brightening low-light images of diverse scenes is a challenging but widely concerned task in the multimedia community. Convolutional Neural Networks (CNNs) based approaches mostly acquire the enhanced model by learning the data distribution from the specific scenes. However, these works present poor adaptability (even fail) when meeting real-world scenarios that never encountered before. To conquer it, we develop a novel bilevel learning scheme for fast adaptation to bridge the gap between low-light scenes. Concretely, we construct a Retinex-induced encoder-decoder with an adaptive denoising mechanism, aiming at covering more practical cases. Different from existing works that directly learn model parameters by using the massive data, we provide a new hyperparameter optimization perspective to formulate a bilevel learning scheme towards general low-light scenarios. This scheme depicts the latent correspondence (i.e., scene-irrelevant encoder) and the respective characteristic (i.e., scene-specific decoder) among different data distributions. Due to the expensive inner optimization, estimating the hyper-parameter gradient exactly can be prohibitive, we develop an approximate hyper-parameter gradient method by introducing the one-step forward approximation and finite difference approximation to ensure the high-efficient inference. Extensive experiments are conducted to reveal our superiority against other state-of-the-art methods. A series of analytical experiments are also executed to verify our effectiveness.