Reviewerly/exHarmony huggingface dataset:
Accuracy vs. complexity: A trade-off in visual question answering models

Visual Question Answering (VQA) has emerged as a Visual Turing Test to validate the reasoning ability of AI agents. The pivot to existing VQA models is the joint embedding that is learned by combining the visual features from an image and the semantic features from a given question. Consequently, a large body of literature has focused on developing complex joint embedding strategies coupled with visual attention mechanisms to effectively capture the interplay between these two modalities. However, modelling the visual and semantic features in a high dimensional (joint embedding) space is computationally expensive, and more complex models often result in trivial improvements in the VQA accuracy. In this work, we systematically study the trade-off between the model complexity and the performance on the VQA task. VQA models have a diverse architecture comprising of pre-processing, feature extraction, multimodal fusion, attention and final classification stages. We specifically focus on the effect of "multi-modal fusion" in VQA models that is typically the most expensive step in a VQA pipeline. Our thorough experimental evaluation leads us to three proposals, one optimized for minimal complexity, one for balanced complexity-accuracy and the last one for state-of-the-art VQA performance.
MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts

Extracting relations in social media posts is challenging when sentences lack of contexts. However, images related to these sentences can supplement such missing contexts and help to identify relations precisely. To this end, we present a multimodal neural relation extraction dataset (MNRE), consisting of 10000+ sentences on 31 relations derived from Twitter and annotated by crowdworkers. The subject and object entities are recognized by a pretrained NER tool and then filtered by crowdworkers. All the relations are identified manually. One sentence is tagged with one related image. We develop a multimodal relation extraction baseline model and the experimental results show that introducing multimodal information improves relation extraction performance in social media texts. Still, our detailed analysis points out the difficulties of aligning relations in texts and images, which can be addressed for future research. All details and resources about the dataset and baselines are released on https://github.com/thecharm/MNRE.
Unsupervised Part Segmentation through Disentangling Appearance and Shape

We study the problem, of unsupervised discovery and segmentation of object parts, which, as an intermediate local representation, are capable of finding intrinsic object structure and providing more explainable recognition results. Recent unsupervised methods have greatly relaxed the dependency on annotated data which are costly to obtain, but still rely on additional information such as object segmentation mask or saliency map. To remove such a dependency and further improve the part segmentation performance, we develop a novel approach by disentangling the appearance and shape representations of object parts followed with reconstruction losses without using additional object mask information. To avoid degenerated solutions, a bottleneck block is designed to squeeze and expand the appearance representation, leading to a more effective disentanglement between geometry and appearance. Combined with a self-supervised part classification loss and an improved geometry concentration constraint, we can segment more consistent parts with semantic meanings. Comprehensive experiments on a wide variety of objects such as face, bird, and PASCAL VOC objects demonstrate the effectiveness of the proposed method.
Variational quantum anomaly detection: Unsupervised mapping of phase diagrams on a physical quantum computer

One of the most promising applications of quantum computing is simulating quantum many-body systems. However, there is still a need for methods to efficiently investigate these systems in a native way, capturing their full complexity. Here, we propose variational quantum anomaly detection, an unsupervised quantum machine learning algorithm to analyze quantum data from quantum simulation. The algorithm is used to extract the phase diagram of a system with no prior physical knowledge and can be performed end-to-end on the same quantum device that the system is simulated on. We showcase its capabilities by mapping out the phase diagram of the one-dimensional extended Bose Hubbard model with dimerized hoppings, which exhibits a symmetry protected topological phase. Further, we show that it can be used with readily accessible devices nowadays and perform the algorithm on a real quantum computer.
BugGraph: Differentiating Source-Binary Code Similarity with Graph Triplet-Loss Network

Binary code similarity detection, which answers whether two pieces of binary code are similar, has been used in a number of applications,such as vulnerability detection and automatic patching. Existing approaches face two hurdles in their efforts to achieve high accuracy and coverage: (1) the problem of source-binary code similarity detection, where the target code to be analyzed is in the binary format while the comparing code (with ground truth) is in source code format. Meanwhile, the source code is compiled to the comparing binary code with either a random or fixed configuration (e.g.,architecture, compiler family, compiler version, and optimization level), which significantly increases the difficulty of code similarity detection; and (2) the existence of different degrees of code similarity. Less similar code is known to be more, if not equally, important in various applications such as binary vulnerability study. To address these challenges, we design BugGraph, which performs source-binary code similarity detection in two steps. First, BugGraph identifies the compilation provenance of the target binary and compiles the comparing source code to a binary with the same provenance.Second, BugGraph utilizes a new graph triplet-loss network on the attributed control flow graph to produce a similarity ranking. The experiments on four real-world datasets show that BugGraph achieves 90% and 75% true positive rate for syntax equivalent and similar code, respectively, an improvement of 16% and 24% overstate-of-the-art methods. Moreover, BugGraph is able to identify 140 vulnerabilities in six commercial firmware.
Deterministic Multi-kernel based extreme learning machine for pattern classification

The Extreme learning machine (ELM) designed by Huang et al. is proved to be a fast and good classifier over a decade, but existing ELM is non-deterministic in nature as well as kernel dependent and needs attention to optimize the selection of kernels. In ELM feature space is obtained with the help of single kernel function. The choice of kernel depends on perceptiveness of classification problem. So a generalized framework with deterministic nature along with optimized kernel is ought to be designed that can be applied to large domain of real world heterogeneous pattern classification problems. This paper presents a deterministic extreme learning machine for neural network with feedforward architecture which is formulated with multiple kernel learning. We further enhance this approach by incorporating Gray level co-occurrence matrix (GLCM) for multi-modal feature extraction. Two formulation of kernel extreme learning machine are introduced, with target kernel function as a linear combination of different base kernels. The first one is based on deterministic multiple kernel learning while the second one uses deterministic multiple kernel learning along with GLCM for extracting the invariant feature vectors. The performance of proposed algorithms are analyzed on pattern recognition problem for face recognition by changing the number of training set, types of kernel used and coefficients used for combining base kernels. The superior recognition rate is achieved for prominent multi-class face databases, when compared with contemporary methods that prove the efficacy of proposed algorithms.
Contrastive Multi-View Multiplex Network Embedding with Applications to Robust Network Alignment

Despite its success in learning network node representations, network embedding is still relatively new for multiplex networks (MNs) with multiple types of edges. In such networks, the inter-layer anchor links are usually missing, which represent the alignment relations between nodes on different layers and are a crucial prerequisite for many cross-network applications like network alignment. For mining such anchor links between layers for MNs, multiplex network embedding (MNE) has become one of the most promising techniques. In this paper, we consider two problems for MNs: 1) edges can be missing to different extent, and data augmentation may mitigate this issue; 2) the known alignment anchor links between layers can be misleading since the behaviors of nodes on different layers are not always consistent, so the most informative ones should be emphasized compared with those misleading ones. However, most existing works neglect the two problems and simply 1) adopt one structural view for all the layers (e.g. random walk with the same window size) and 2) equally extract information from all the anchor links. We propose an end-to-end contrastive framework called cM2NE for MNE, utilizing multiple structural views for each layer and learning with several plug-in components for different scenarios. Through end-to-end optimization on three levels, the intra-view, inter-view, and inter-layer level, our framework achieves to select the fitted views for different layers and maximize the inter-layer mutual information by emphasizing those most informative anchor links. Extensive experimental results on real-world datasets for node classification and multi-network alignment show that our approach consistently outperforms peer methods.
A Comparative Study of Using Pre-trained Language Models for Toxic Comment Classification

As user-generated contents thrive, so does the spread of toxic comment. Therefore, detecting toxic comment becomes an active research area, and it is often handled as a text classification task. As recent popular methods for text classification tasks, pre-trained language model-based methods are at the forefront of natural language processing, achieving state-of-the-art performance on various NLP tasks. However, there is a paucity in studies using such methods on toxic comment classification. In this work, we study how to best make use of pre-trained language model-based methods for toxic comment classification and the performances of different pre-trained language models on these tasks. Our results show that, Out of the three most popular language models, i.e. BERT, RoBERTa, and XLM, BERT and RoBERTa generally outperform XLM on toxic comment classification. We also prove that using a basic linear downstream structure outperforms complex ones such as CNN and BiLSTM. What is more, we find that further fine-tuning a pre-trained language model with light hyper-parameter settings brings improvements to the downstream toxic comment classification task, especially when the task has a relatively small dataset.
Feature fusion using deep learning for smartphone based human activity recognition

Identification of human physical activities is an active research area since long due to its application in personalized health and fitness monitoring. The performance accuracy of human activity recognition (HAR) models mainly depend on the features which are extracted from domain knowledge. The features are the input of the classification algorithm to efficiently identify human physical activities. Manually extracted features (handcrafted) need expert domain knowledge. Thus these features have significant importance to identify different human activities. Recently deep learning methods are utilized to extract the features automatically from raw sensory data for HAR models. However, state-of-the-art HAR literature established that the importance of handcrafted features can’t be ignored as it is extracted from expert domain knowledge. Thus, in this paper we use the fusion of both the handcrafted features and automatically extracted features using deep learning (DL) for HAR model to enhance the performance of HAR. Extensive experimental results demonstrate that our proposed feature fusion based HAR model gives higher accuracy compared with state-of-the-art HAR literature for both the self collected and public dataset.
Systematic Literature Review of Realistic Simulators Applied in Educational Robotics Context

This paper presents a systematic literature review (SLR) about realistic simulators that can be applied in an educational robotics context. These simulators must include the simulation of actuators and sensors, the ability to simulate robots and their environment. During this systematic review of the literature, 559 articles were extracted from six different databases using the Population, Intervention, Comparison, Outcomes, Context (PICOC) method. After the selection process, 50 selected articles were included in this review. Several simulators were found and their features were also analyzed. As a result of this process, four realistic simulators were applied in the review’s referred context for two main reasons. The first reason is that these simulators have high fidelity in the robots’ visual modeling due to the 3D rendering engines and the second reason is because they apply physics engines, allowing the robot’s interaction with the environment.
FleetRec: Large-Scale Recommendation Inference on Hybrid GPU-FPGA Clusters

We present FleetRec, a high-performance and scalable recommendation inference system within tight latency constraints. FleetRec takes advantage of heterogeneous hardware including GPUs and the latest FPGAs equipped with high-bandwidth memory. By disaggregating computation and memory to different types of hardware and bridging their connections by high-speed network, FleetRec gains the best of both worlds, and can naturally scale out by adding nodes to the cluster. Experiments on three production models up to 114 GB show that FleetRec outperforms optimized CPU baseline by more than one order of magnitude in terms of throughput while achieving significantly lower latency.
3D robotic navigation using a vision-based deep reinforcement learning model

In this paper, we address a problem of vision-based 3D robotic navigation using deep reinforcement learning for an Autonomous Underwater Vehicle (AUV). Our research offers conclusions from the experimental study based on one of the RoboSub 2018 competition tasks. However, it can be generalized to any navigation task consisting of movement from a starting point to the front of the next station. The presented reinforcement learning-based model predicts the robot's steering settings using the data acquired from the robot's sensors. Its Vision Module may be based on a built-in convolutional network or a pre-trained TinyYOLO network so that a comparison of various levels of features' complexity is possible. To enable evaluation of the proposed solution, we prepared a test environment imitating the real conditions. It provides the ability to steer the agent simulating the AUV and calculate values of rewards, used for training the model by evaluating its decisions. We study the solution in terms of the reward function form, the model's hyperparameters and the exploited camera images processing method, and provide an analysis of the correctness and speed of the model's functioning. As a result, we obtain a valid model able to steer the robot from the starting point to the destination based on visual cues and inputs from other sensors.
STBC: A Novel Blockchain-Based Spectrum Trading Solution

The ever-developing 5G and Internet of Things (IoT) technology has recently drawn increased attention to the use and management of underutilized spectrum. Blockchain is a good potential solution, however, the Nakamoto consensus-based scheme has a low spectrum utilization rate, high transaction delay, and a huge waste of energy. In this paper, we present a novel decentralized distributed spectrum trading protocol STBC (Spectrum Trading Blockchain) base on blockchain, which explicitly aims for efficiency, simplicity, safety, and energy-saving. STBC uses a new consensus mechanism to quickly confirm transactions while tolerating up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${n}$ </tex-math></inline-formula> /3 malicious nodes ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${n}$ </tex-math></inline-formula> is the total number of nodes), and its implementation code is very concise. We adopt the similar idea of sharding to improve the efficiency and scalability of the system. Due to the specificality of spectrum trading, it is necessary to protect the privacy of trading nodes. Therefore we propose a temporarily anonymous transaction that can effectively prevent DDoS (Distributed Denial of Service) attack. Moreover, theoretical analysis shows that STBC is also secure against mainstream attacks such as double-spending attacks and the power consumption of STBC is significantly less than the Nakamoto consensus-based scheme. The prototype evaluation of our protocol demonstrates that it improves the spectrum utilization by about 30% compared with the state-of-the-art blockchain spectrum trading schemes, and meanwhile reducing the transaction confirmation delay by about 12.5x.
An optimized Generative Adversarial Network based continuous sign language classification

Classifying manual and non-manual gestures in sign language recognition is a complex and challenging task. Sign language gestures are the combination of hand, face, and body postures, which often have self-occlusions and inter-object occlusions of both the hands, hands with face, or hands with upper body postures. This paper addresses the characterization of manual and non-manual gestures in recognizing the sign language gestures from continuous video sequences. This paper introduces a novel hyperparameter based optimized Generative Adversarial Networks (H-GANs) to classify the sign gestures, and it works in three phases. In phase-I, it adapts the stacked variational auto-encoders (SVAE) and Principal Component Analysis (PCA) to get the pre-tuned data with reduced feature dimensions. In Phase-II, the H-GANs employed Deep Long Short Term Memory (LSTM) as generator and LSTM with 3D Convolutional Neural Network (3D-CNN) as a discriminator. The generator generates random sequences with noise from the real sequence of frames, and the discriminator detects and classifies the real frames of sign gestures. In Phase-III, the proposed approach employs Deep Reinforcement Learning (DRL) for hyperparameter optimization and regularization. By getting the reward points, Proximal Policy Optimization (PPO) optimizes the hyperparameters, and Bayesian Optimization (BO) regularizes the hyperparameters. The proposed H-GANs used two different large vocabulary sign corpus of continuous sign videos to evaluate the performance and efficiency of the system. The experimental results on different dimensions reveal that the H-GANs improved the accuracy and recognition rate when compared with the state-of-the-art classification methods with reduced complexity.
Facial Chirality: Using Self-Face Reflection to Learn Discriminative Features for Facial Expression Recognition

As a fundamental vision task, facial expression recognition has made substantial progress recently. However, the recognition performance often degrades largely in real-world scenarios due to the lack of robust facial features. In this paper, we propose a simple but effective facial feature learning method that takes the advantage of facial chirality to discover the discriminative features for facial expression recognition. Most previous studies implicitly assume that human faces are symmetric. However, our work reveals that the facial asymmetric effect can be a crucial clue. Given a face image and its reflection without additional labels, we decouple the reflection-invariant facial features from the input image pair and then demonstrate that the new features with a standard and lightweight learning model (e.g. ResNet-18) are sufficiently robust to outperform the state-of-the-art methods (e.g. SCN in CVPR 2020 and ESRs in AAAI 2020). Our experiments also show the potential of the new features for other facial vision tasks such as expression image retrieval.
Developing a Virtual Museum: Experience from the Design and Creation Process

Virtual reality technology has grown significantly in recent years. The arrival of Head Mounted Displays (HDM) on the market for end-users has positioned these technologies as a new channel to promote new simulated or contextualized experiences. We have used the design and creation strategy to develop a virtual reality experience for the Oculus GO and Quest HDM. We digitized 30 pieces from nine local museums to provide an experience guided by a character that represents the main artisan work of the local region. A usability test was performed, showing that participants felt a high degree of immersion and realism. They were able to complete the assigned tasks, and results suggest that the software meets the main objective. Furthermore, the creation of this virtual reality (VR) experience has shown how important it is to make users a part of the creation process, as well as to develop a process to make the software useful to them and other users. Some recommendations are made based on the experience of the development, and comments are given on each step of the design and creation strategy.
Reinforced KGs reasoning for explainable sequential recommendation

We explore the semantic-rich structured information derived from the knowledge graphs (KGs) associated with the user-item interactions and aim to reason out the motivations behind each successful purchase behavior. Existing works on KGs-based explainable recommendations focus purely on path reasoning based on current user-item interactions, which generally result in the incapability of conjecturing users' subsequence preferences. Considering this, we attempt to model the KGs-based explainable recommendation in sequential settings. Specifically, we propose a novel architecture called Reinforced Sequential Learning with Gated Recurrent Unit (RSL-GRU), which is composed of a Reinforced Path Reasoning Network (RPRN) component and a GRU component. RSL-GRU takes users' sequential behaviors and their associated KGs in chronological order as input and outputs potential top-N items for each user with appropriate reasoning paths from a global perspective. Our RPRN features a remarkable path reasoning capacity, which is regulated by a user-conditioned derivatively action pruning strategy, a soft reward strategy based on an improved multi-hop scoring function, and a policy-guided sequential path reasoning algorithm. Experimental results on four of Amazon's large-scale datasets show that our method achieves excellent results compared with several state-of-the-art alternatives.