Reviewerly/exHarmony huggingface dataset:
The human source memory system struggles to distinguish virtual reality and reality

Virtual Reality (VR) is used in a variety of fields with the goal to increase ecological validity compared to traditional monitor-based setups. Here we report additional evidence for the adequacy of this research strategy. In a memory confusion paradigm spanning over stimuli presentation in reality, VR and on a computer monitor, participants were more prone to confusing reality with VR than with a traditional monitor-based setup, indicating a relative proximity of experiences in VR and reality. We speculate that the human source memory's difficulty in distinguishing VR from reality may provide a basis for the good generalizability of treatment effects in VR to daily life. At the same time, the effect may demonstrate a potential danger of a mindless use of VR technology.
Semantic Diversity Learning for Zero-Shot Multi-label Classification

Training a neural network model for recognizing multiple labels associated with an image, including identifying unseen labels, is challenging, especially for images that portray numerous semantically diverse labels. As challenging as this task is, it is an essential task to tackle since it represents many real-world cases, such as image retrieval of natural images. We argue that using a single embedding vector to represent an image, as commonly practiced, is not sufficient to rank both relevant seen and unseen labels accurately. This study introduces an end-to-end model training for multi-label zero-shot learning that supports the semantic diversity of the images and labels. We propose to use an embedding matrix having principal embedding vectors trained using a tailored loss function. In addition, during training, we suggest up-weighting in the loss function image samples presenting higher semantic diversity to encourage the diversity of the embedding matrix. Extensive experiments show that our proposed method improves the zero-shot model’s quality in tag-based image retrieval achieving SoTA results on several common datasets (NUS-Wide, COCO, Open Images).
DEAAN: Disentangled Embedding and Adversarial Adaptation Network for Robust Speaker Representation Learning

Despite speaker verification has achieved significant performance improvement with the development of deep neural networks, do-main mismatch is still a challenging problem in this field. In this study, we propose a novel framework to disentangle speaker-related and domain-specific features and apply domain adaptation on the speaker-related feature space solely. Instead of performing domain adaptation directly on the feature space where domain information is not removed, using disentanglement can efficiently boost adaptation performance. To be specific, our model’s input speech from the source and target domains is first encoded into different latent feature spaces. The adversarial domain adaptation is conducted on the shared speaker-related feature space to encourage the property of domain-invariance. Further, we minimize the mutual information between speaker-related and domain-specific features for both do-mains to enforce the disentanglement. Experimental results on the VOiCES dataset demonstrate that our proposed framework can effectively generate more speaker-discriminative and domain-invariant speaker representations with a relative 20.3% reduction of EER com-pared to the original ResNet-based system.
Design of parallel reservoir computing by mutually-coupled semiconductor lasers with optoelectronic feedback

Via implementing parallel tasks including Santa-Fe time series prediction and nonlinear channel equalization, we studied the characteristics of reservoir computing (RC) using two mutually-coupled semiconductor lasers (MC-SLs) with optoelectronic feedback. The influence of system parameters (injection current, coupling strength, scaling factor and coupling time-delays) on RC performance have been investigated in detail. Results show that, the optimized performance is always achieved around the edge of bifurcation points; positive feedback is more preferred when the scaling factor exceeds a certain level; under optimized parameters, both of lower normalized mean squared error (NMSE) and symbol error rate (SER) can be achieved to perform parallel tasks, due to the richer dynamics and states of virtual nodes. The effect of system parameters on RC performance was also discussed. Our study would be helpful for the design and optimization of proposed parallel RC with optoelectronic feedback.
QuickLook: Movie summarization using scene-based leading characters with psychological cues fusion

Due to recent advances in the film industry, the production of movies has grown exponentially, which has led to challenges in what is referred to as discoverability: given the overwhelming number of choices, choosing which film to watch has become a tedious task for audiences. Movie summarization (MS) could help, as it presents the central theme of the movie in a compact format and makes browsing more efficient for the audience. In this paper, we present an automatic MS framework coined as 'QuickLook', which identifies the leading characters and fuses multiple cues extracted from a movie. Firstly, the movie data is preprocessed for its division into scenes, followed by shot segmentation. Secondly, the leading characters in each segmented scene are determined. Next, four visual cues that capture the film's scenic beauty, memorability, informativeness and emotional resonance are extracted from shots containing the leading characters. These extracted features are then intelligently fused based on the assignment of different weights; shots with a fusion score above a certain threshold are selected for the final summary. The proposed MS framework is assessed by comparison with official trailers from ten Hollywood movies, providing a novel baseline for future fair comparison in the MS literature. The proposed framework is shown to outperform other state-of-the-art MS methods in terms of enjoyability and informativeness.
FantastIC4: A Hardware-Software Co-Design Approach for Efficiently Running 4Bit-Compact Multilayer Perceptrons

With the growing demand for deploying Deep Learning models to the “edge”, it is paramount to develop techniques that allow to execute state-of-the-art models within very tight and limited resource constraints. In this work we propose a software-hardware optimization paradigm for obtaining a highly efficient execution engine of deep neural networks (DNNs) that are based on fully-connected layers. The work's approach is centred around compression as a means for reducing the area as well as power requirements of, concretely, multilayer perceptrons (MLPs) with high predictive performances. Firstly, we design a novel hardware architecture named FantastIC4, which (1) supports the efficient on-chip execution of multiple compact representations of fully-connected layers and (2) minimizes the required number of multipliers for inference down to only 4 (thus the name). Moreover, in order to make the models amenable for efficient execution on FantastIC4, we introduce a novel entropy-constrained training method that renders them to be robust to 4bit quantization and highly compressible in size simultaneously. The experimental results show that we can achieve throughputs of 2.45 TOPS with a total power consumption of 3.6W on a Virtual Ultrascale FPGA XCVU440 device implementation, and achieve a total power efficiency of 20.17 TOPS/W on a 22nm process ASIC version. When compared to other state-of-the-art accelerators designed for the Google Speech Command (GSC) dataset, FantastIC4 is better by 51× in terms of throughput and 145× in terms of area efficiency (GOPS/mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ).
Congested Crowd Counting via Adaptive Multi-Scale Context Learning

In this paper, we propose a novel congested crowd counting network for crowd density estimation, i.e., the Adaptive Multi-scale Context Aggregation Network (MSCANet). MSCANet efficiently leverages the spatial context information to accomplish crowd density estimation in a complicated crowd scene. To achieve this, a multi-scale context learning block, called the Multi-scale Context Aggregation module (MSCA), is proposed to first extract different scale information and then adaptively aggregate it to capture the full scale of the crowd. Employing multiple MSCAs in a cascaded manner, the MSCANet can deeply utilize the spatial context information and modulate preliminary features into more distinguishing and scale-sensitive features, which are finally applied to a 1 × 1 convolution operation to obtain the crowd density results. Extensive experiments on three challenging crowd counting benchmarks showed that our model yielded compelling performance against the other state-of-the-art methods. To thoroughly prove the generality of MSCANet, we extend our method to two relevant tasks: crowd localization and remote sensing object counting. The extension experiment results also confirmed the effectiveness of MSCANet.
Gendering algorithms in social media

Social media platforms employ inferential analytics methods to guess user preferences and may include sensitive attributes such as race, gender, sexual orientation, and political opinions. These methods are often opaque, but they can have significant effects such as predicting behaviors for marketing purposes, influencing behavior for profit, serving attention economics, and reinforcing existing biases such as gender stereotyping. Although two international human rights treaties include express obligations relating to harmful and wrongful stereotyping, these stereotypes persist both online and offline, and platforms often appear to fail to understand that gender is not merely a binary of being a 'man' or a 'woman,' but is socially constructed. Our study investigates the impact of algorithmic bias on inadvertent privacy violations and the reinforcement of social prejudices of gender and sexuality through a multidisciplinary perspective including legal, computer science, and queer media viewpoints. We conducted an online survey to understand whether and how Twitter inferred the gender of users. Beyond Twitter's binary understanding of gender and the inevitability of the gender inference as part of Twitter's personalization trade-off, the results show that Twitter misgendered users in nearly 20% of the cases (N=109). Although not apparently correlated, only 8% of the straight male respondents were misgendered, compared to 25% of gay men and 16% of straight women. Our contribution shows how the lack of attention to gender in gender classifiers exacerbates existing biases and affects marginalized communities. With our paper, we hope to promote the online account for privacy, diversity, and inclusion and advocate for the freedom of identity that everyone should have online and offline.
Back to the Basics: Seamless Integration of Side-Channel Pre-Processing in Deep Neural Networks

Deep learning approaches have become popular for Side-Channel Analysis (SCA) in the recent years. Especially Convolutional Neural Networks (CNN) due to their natural ability to overcome jitter-based as well as masking countermeasures. Most of the recent works have been focusing on optimising the performance on given dataset, for example finding optimal architecture and using ensemble, and bypass the need for trace pre-processing. However, trace pre-processing is a long studied topic and several proven techniques exist in the literature. There is no straightforward manner to integrate those techniques into deep learning based SCA. In this paper, we propose a generic framework which allows seamless integration of multiple, user defined pre-processing techniques into the neural network architecture. The framework is based on Multi-scale Convolutional Neural Networks ( MCNN) that were originally proposed for time series analysis. MCNN are composed of multiple branches that can apply independent transformation to input data in each branch to extract the relevant features and allowing a better generalization of the model. In terms of SCA, these transformations can be used for integration of pre-processing techniques, such as phase-only correlation, principal component analysis, alignment methods, etc. We present successful results on generic network which generalizes to different publicly available datasets. Our findings show that it is possible to design a network that can be used in a more general way to analyze side-channel leakage traces and perform well across datasets.
Relation Extraction in Dialogues: A Deep Learning Model Based on the Generality and Specialty of Dialogue Text

Relation extraction from dialogue text is an innovative task in natural language processing. In addition to the general characteristics of general relation extraction from news or scientific publication text, the task is of certain special features. For example, the context in dialogues frequently switches between speakers, and there exist rich pronoun anaphora in the dialogue text. Thus, it is important for the model to be aware of such features to improve the performance. Taking these factors together, we propose an end to-end neural model for dialogue-based relation extraction, which includes four modules to handle the problems existing in the task from different aspects: (1) the word-relation attention to model a natural intuition that different words contribute differently for the identification of different relations; (2) the graph reasoning to consider the global context information in the dialogue that contains many inter-sentence relations; (3) the speaker embeddings to incorporate speaker information into our model; (4) the speaker coreference to associate pronouns with speakers and enrich the information of graph reasoning. Our model was evaluated on a recently-proposed dataset for dialogue-based relation extraction, and achieved the state of the-art performance. We show that our proposed modules are effective through ablation studies. Our work can be a competitive benchmark for the study of dialogue based relation extraction.
Four-Dimensional Hurwitz Signal Constellations, Set Partitioning, Detection, and Multilevel Coding

The Hurwitz lattice provides the densest four-dimensional packing. This fact has motivated research on four-dimensional Hurwitz signal constellations for optical and wireless communications. This work presents a new algebraic construction of finite sets of Hurwitz integers that is inherently accompanied by a respective modulo operation. These signal constellations are investigated for transmission over the additive white Gaussian noise (AWGN) channel. It is shown that these signal constellations have a better constellation figure of merit and hence a better asymptotic performance over an AWGN channel when compared with conventional signal constellations with algebraic structure, e.g., two-dimensional Gaussian-integer constellations or four-dimensional Lipschitz-integer constellations. We introduce two concepts for set partitioning of the Hurwitz integers. The first method is useful to reduce the computational complexity of the symbol detection. This suboptimum detection approach achieves near-maximum-likelihood performance. In the second case, the partitioning exploits the algebraic structure of the Hurwitz signal constellations. We partition the Hurwitz integers into additive subgroups in a manner that the minimum Euclidean distance of each subgroup is larger than in the original set. This enables multilevel code constructions for the new signal constellations.
A fast constrained state transition algorithm

When solving constrained optimization problems in real industrial processes, both optimality and computational efficiency need to be considered. However, most existing meta-heuristic algorithms are slow to find the global optimum. The first reason is that the way to generate and select candidate solutions is time-consuming. The low probability to generate and select potential solutions in assisting the computational efficiency is another reason. In this paper, a simplified state transition algorithm (STA) and a novel constraint-handling technique are proposed to address the above issues for small size constrained optimization problems. Firstly, three out of four operators in basic STA to produce candidate solutions are selected and two operators are modified with adaptive parameter tuning, which have a large probability to generate potential solutions, but consumes less time. Secondly, the constraint-handling technique considers not only the objective function value and the constraint violation but also the difference among candidate solutions. Thirdly, the sequential quadratic programming embedded into the simplified STA can further speed up the convergence. Experiments are conducted on 22 well-known test functions from IEEE CEC2006 and 4 engineering constrained optimization problems, in comparison with state-of-the-art algorithms. The experimental results show that the proposed method is competitive in finding the optimum faster.
Multi-View Mouse Social Behaviour Recognition With Deep Graphic Model

Home-cage social behaviour analysis of mice is an invaluable tool to assess therapeutic efficacy of neurodegenerative diseases. Despite tremendous efforts made within the research community, single-camera video recordings are mainly used for such analysis. Because of the potential to create rich descriptions of mouse social behaviors, the use of multi-view video recordings for rodent observations is increasingly receiving much attention. However, identifying social behaviours from various views is still challenging due to the lack of correspondence across data sources. To address this problem, we here propose a novel multiview latent-attention and dynamic discriminative model that jointly learns view-specific and view-shared sub-structures, where the former captures unique dynamics of each view whilst the latter encodes the interaction between the views. Furthermore, a novel multi-view latent-attention variational autoencoder model is introduced in learning the acquired features, enabling us to learn discriminative features in each view. Experimental results on the standard CRMI13 and our multi-view Parkinson's Disease Mouse Behaviour (PDMB) datasets demonstrate that our model outperforms the other state of the arts technologies and effectively deals with the imbalanced data problem.
Attention-Based Neural Architecture Search for Person Re-Identification

Recent years have witnessed significant progress of person reidentification (reID) driven by expert-designed deep neural network architectures. Despite the remarkable success, such architectures often suffer from high model complexity and time-consuming pretraining process, as well as the mismatches between the image classification-driven backbones and the reID task. To address these issues, we introduce neural architecture search (NAS) into automatically designing person reID backbones, i.e., reID-NAS, which is achieved via automatically searching attention-based network architectures from scratch. Different from traditional NAS approaches that originated for image classification, we design a reID-based search space as well as a search objective to fit NAS for the reID tasks. In terms of the search space, reID-NAS includes a lightweight attention module to precisely locate arbitrary pedestrian bounding boxes, which is automatically added as attention to the reID architectures. In terms of the search objective, reID-NAS introduces a new retrieval objective to search and train reID architectures from scratch. Finally, we propose a hybrid optimization strategy to improve the search stability in reID-NAS. In our experiments, we validate the effectiveness of different parts in reID-NAS, and show that the architecture searched by reID-NAS achieves a new state of the art, with one order of magnitude fewer parameters on three-person reID datasets. As a concomitant benefit, the reliance on the pretraining process is vastly reduced by reID-NAS, which facilitates one to directly search and train a lightweight reID model from scratch.
Self-Taught Semi-Supervised Anomaly Detection On Upper Limb X-Rays

Detecting anomalies in musculoskeletal radiographs is of paramount importance for large-scale screening in the radiology workflow. Supervised deep networks take for granted a large number of annotations by radiologists, which is often prohibitively very time-consuming to acquire. Moreover, supervised systems are tailored to closed set scenarios, e.g., trained models suffer from overfitting to previously seen rare anomalies at training. Instead, our approach's rationale is to use task agnostic pretext tasks to leverage unlabeled data based on a cross-sample similarity measure. Besides, we formulate a complex distribution of data from normal class within our framework to avoid a potential bias on the side of anomalies. Through extensive experiments, we show that our method outperforms baselines across unsupervised and self-supervised anomaly detection settings on a real-world medical dataset, the MURA dataset. We also provide rich ablation studies to analyze each training stage's effect and loss terms on the final performance.
An Information Theoretic Study of a Duffing Oscillator Array Reservoir Computer

Abstract Typically, nonlinearity is considered to be problematic and sometimes can lead to dire consequences. However, the nonlinearity in a Duffing oscillator array can enhance its ability to be used as a reservoir computer. Machine learning and artificial neural networks, inspired by the biological computing framework, have shown their immense potential, especially in the real-time temporal data processing. Here, the efficacy of a Duffing oscillator array is explored as a reservoir computer by using information theory. To do this, a reservoir computer model is studied numerically, which exploits the dynamics of the array. In this system, the complex dynamics stem from the Duffing term in each of the identical oscillators. The effects of various system parameters of the array on the information processing ability are discussed from the perspective of information theory. By varying these parameters, the information metric was found to be topologically mixed. Additionally, the importance of asynchrony in the oscillator array is also discussed in terms of the information metric. Since such nonlinear oscillators are used to model many different physical systems, this research provides insight into how physical nonlinear oscillatory systems can be used for dynamic computation, without significantly modifying or controlling the underlying dynamical system. To the authors' knowledge, this is the first use of Shannon's information rate for quantifying a reservoir computer of this kind, as well as the first comparison between synchronization phenomena and the computing ability of a reservoir.
Multi-EmoNet: A Novel Multi-Task Neural Network for Driver Emotion Recognition

Driver’s emotion affects driving safety Hu et al. (2013), therefore monitoring driver’s emotion could benefit road safety. However, the complex illumination conditions in a vehicle cockpit significantly challenge the effectiveness of camera-based facial expression recognition (FER) systems. To solve this problem, we proposed Multi-EmoNet, a novel multi-task neural network, to classify human facial expression under illumination variations and to restore noisy images. Our experiments demonstrate these two tasks are complementary and together facilitate better network representation learning. Our approach obtains significantly better classification accuracy on images with illumination variation compared to the baseline networks. More importantly, the proposed multi-task network is a general architecture that can be applied to any noise involved image classification problem.