Reviewerly/exHarmony huggingface dataset:
Multimodal Cross-Attention Bayesian Network for Social News Emotion Recognition

Multimodal emotion recognition comprehensively identifies the emotion contained in multimodal data by bridging the gaps between heterogeneous dataset. In recent years, multimodal emotion recognition methods have gained significant attention and been shown to surpass single-modal approaches. Most of the existing multimodal emotion analysis methods simply combine different modalities to improve the recognition capability of consistent emotion expressions across multimodal data. However, it remains challenging to recognize the right emotion when multiple modalities' contents carry inconsistent or even contradictory emotions. To solve this problem, we propose a novel image-text emotion recognition model named Multimodal Cross-Attention Bayesian Network(MCABN). The entire network exploits Bayesian theory to learn the distribution of its weight parameters, making optimization directions and results of parameters interpretable. What's more, the model leverages the consistency and complementarity between visual content and textual description to arrive at accurate decisions. Specifically, for each modality, multiple explainable features (color, texture, and shape feature in the image, while adjective, adverb, verb, noun, and negative feature in the text) and one unexplainable feature are fused as its feature representation to reinforce emotion-related features. Then two single-modal attention modules(Visual Attention Module and Textual Attention Module) capture the most discriminative features in a single image and text; Two cross-modal attention modules(Image-guided Text Attention Module and Text-guided Image Attention Module) extract the complementary and dominant features between two modalities by interactive learning. Finally, the outputs of four attention modules are integrated through intermediate fusion to predict the final emotion. The experimental results on the NVTD and MVSA-Multiple dataset indicate that the proposed MCABN outperforms state-of-the-art baselines by substantial margins.
Going Deeper with Recursive Convolutional Layers

The development of Convolutional Neural Networks (CNNs) trends towards models with an ever growing number of Convolutional Layers (CLs) and increases the number of trainable parameters significantly. Such models are sensitive to these structural parameters, which implies that large models have to be carefully tuned using hyperparameter optimisation, a process that can be very time consuming. In this paper, we study the usage of Recursive Convolutional Layers (RCLs), a module relying on an algebraic feedback loop wrapped around a CL, which can replace any CL in CNNs. Using three publicly available datasets, CIFAR10, CIFAR100 and SVHN, and a simple model comprised of 4 RCLs, we compare its performances with those obtained by its feedforward counterpart, and exhibit some core properties and use-cases of RCLs. In particular, we show that RCLs can lead to models of better performances, and that reducing the number of modules from four to one lead to a decrease in accuracy of 3.5% on average for models using RCLs, compared to 23% using CLs. Hence, the resulting architecture is much more robust to the addition or the removal of layers. We conclude by relating the effects obtained using additional CLs with those obtained using additional recursion on RCLs, which provides incentives that the latter can simulate an increase of depth but with no extra cost of parameters. Such results point to the potential benefits of either selectively or replacing all CLs by RCLs, in most recently introduced CNNs.
Invariant and Sufficient Supervised Representation Learning

Improving the generalization of neural networks under domain shift is an important and challenging task in computer vision. Obtaining an invariant representation across domains is a benchmark method in the literature. In this paper, we propose an invariant and sufficient supervised representation learning (ISSRL) approach to learn a domain invariant representation which is also preserving information used for downstream tasks. To this end, we formulate ISSRL by finding a nonlinear map <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\boldsymbol{g}$</tex> such that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$Y\perp X\vert \boldsymbol{g}(X)$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(Y,\boldsymbol{g}(X))\perp D$</tex> at the population level, where D is the label of the domains and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(X, Y)$</tex> is the paired data sampled from domains with label. We use distance correlation to characterize the (conditional) independence. At the sample level, we construct a novel loss function through an unbiased empirical version of distance correlation. We train the representation map by parameterizing it with deep neural networks. Both simulation study and real data evaluation show that ISSRL outperforms the state-of-the-art on out-of-distribution generalization. The PyTorch code for ISSRL is available at https://github.com/CaC033/ISSRL.
Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits

The use of Neural Architecture Search (NAS) techniques to automate the design of neural networks has become increasingly popular in recent years. The proliferation of devices with different hardware characteristics using such neural networks, as well as the need to reduce the power consumption for their search, has led to the realisation of Once-For-All (OFA), an algorithm engineered for low CO2 consumption and characterised by the ability to generate easily adaptable models through a single learning process. In order to improve this paradigm and develop high-performance yet eco-friendly NAS techniques, this paper presents OFAv2, the extension of OFA aimed at improving its performance while maintaining the same ecological advantage. The algorithm is improved from an architectural point of view by including early exits, parallel blocks and dense skip connections. The training process is extended by two new steps called Elastic Level and Elastic Exit. A new Knowledge Distillation technique is presented to handle multi-output networks, and finally a new strategy for dynamic teacher network selection is proposed. These modifications allow OFAv2 to improve its accuracy performance on the Tiny ImageNet dataset by up to 12.07% compared to the original version of OFA, while maintaining the algorithm flexibility and advantages.
Conditional Human Activity Signal Generation and Generative Classification with a GPT-2 Model

In this work, the results of an exploratory study into the use of the publicly available GPT-2 model for simultaneous conditional human activity signal synthesis and classification are presented. To accomplish this, the small variant of a pre-trained GPT-2 model is fine-tuned on quantized and windowed human activity signal sequences. The conditional generation is achieved by appending and prepending class labels to each window to introduce class information. During the generation phase, the model is prompted either with a class label for synthesizing signals, or a signal sequence for classification by synthesizing a class label. The study is conducted on the publicly available WISDM and UCI-HAR datasets, and the generated signals are evaluated using an LSTM-CNN model. As a classifier, the fine-tuned GPT-2 model achieved an overall accuracy of 90.47% on the WISDM, and 82.29% on the UCI-HAR dataset, which is 4% and 6% lower than the LSTM-CNN model evaluated on the same test subsets. When used as a multi-class generator, on average, 86.33% of the generated data are classified as valid samples by the LSTM-CNN model. While there is room for improvement, these results demonstrate that GPT family architectures can be used for simultaneous conditional signal synthesis and classification, opening up new opportunities for human activity recognition.
Explicitly Learning Policy Under Partial Observability in Multiagent Reinforcement Learning

We explore explicit solutions for multiagent reinforcement learning (MARL) under the constraint of partial observability. With a general framework of centralized training with decentralized execution (CTDE), existing methods implicitly alleviate partial observability by introducing global information during centralized training. However, such implicit solution cannot well address partial observability and shows low sample efficiency in many MARL problems. In this paper, we focus on the influence of partial observability on the policy of agents, and formally derive an ideal form of policy that maximizes MARL objective under partial observability. Furthermore, we develop a new method named Explicitly Learning Policy (ELP), which adopts a novel teacher-student structure and utilizes knowledge distillation to explicitly learn individual policy under partial observability for each agent. Compared to prior methods, ELP presents a more general and interpretable training process, and the procedure of ELP can be easily extended to existing methods for performance boost. Our empirical experiments on StarCraft II micromanagement benchmark show that ELP significantly outperforms prevailing state-of-the-art baselines, which demonstrates the advantage of ELP in addressing partial observability and improving sample efficiency.
LpAdvGAN: Noise Optimization Based Adversarial Network Generation Adversarial Example

Deep neural networks (DNNs) are susceptible to adversarial examples that arise due to adding small-magnitude perturbations to the input. Such adversarial examples can mislead DNNs to produce adversarially selected results. Different attack strategies have been proposed to generate adversarial examples, but more research is needed on effectively generating adversarial examples with high perceptual quality. This paper proposes an improved generative adversarial network, LpAdvGAN, to generate adversarial examples using generative adversarial networks (GANs). This algorithm learns and approximates the distribution of the original examples. For LpAdvGAN, the noise generator uses a differential evolution algorithm to generate high-quality noise. After training with relatively average GANs (RaGANs), it reduces the probability that real samples are recognized as real. Adversarial samples generated by LpAdvGAN perform well in terms of perceived similarity and have a higher attack success rate than the current mainstream attack strategies. Specifically, the success rate of LpAdvGAN is 9.88% and 8.7% higher than that of AdvGAN under the semi-white-box settings of MNIST and CIFAR 10, respectively.
Connection-Based Knowledge Transfer for Class Incremental Learning

We consider the problem of class incremental learning (CIL), where an agent aims to learn new classes continually without forgetting previous ones. As one of the mainstream paradigms of incremental learning, parameter isolation methods prevent forgetting by allocating different model parameters to each task, but knowledge transfer across tasks is difficult and usually overlooked. As a consequence, the discriminability between old and new classes is limited, especially when training data of old classes is not accessible. In this paper, we propose a new data-free approach named Twin Contrastive Networks (TCN) for CIL by utilizing the connections among tasks and network parameters. Specifically, we treat CIL as a sequence of one-class classification tasks and train separate classifiers to identify each class. To facilitate knowledge transfer and make full use of accumulated knowledge, a twin network structure is adopted to learn different feature representations for future use. While encountering new classes, previous twin networks are utilized directly by a contrastive loss to improve the model's discriminability. TCN avoids catastrophic forgetting by fixing all learnt parameters and leverages prior knowledge contained in networks. Experiments on three widely used incremental learning benchmarks verify the effectiveness of TCN.
Few-shot learning for trajectory outlier detection with only normal trajectories

Trajectory outlier detection has a profound impact on many real-world applications. Most existing methods, whether supervised or unsupervised, require adequate historical data as a prerequisite. However, due to the uneven distribution of the trajectories in space, the detection in some regions(e.g. rural areas) may suffer from data scarcity. Furthermore, because the occurrence of outliers is a small probability event, there are frequently no outlier trajectories in such limited data which makes it even worse. To handle such issues, we in this paper study a new problem, i.e. few-shot trajectory outlier detection with only normal trajectories. And we propose a novel model, named MetaTAD, which consists of a Multi-scale Encoder and an Ab-detection Meta Learner. The Multi-scale Encoder aims to learn the diverse features of trajectories for outlier detection, while the Ab-detection Meta Learner guides the model to detect with few labeled normal trajectories. Extensive experiments on two real taxi trajectory datasets show that MetaTAD achieves state-of-the-art performance compared with the baselines.
FedGrad: Mitigating Backdoor Attacks in Federated Learning Through Local Ultimate Gradients Inspection

Federated learning (FL) enables multiple clients to train a model without compromising sensitive data. The decentralized nature of FL makes it susceptible to adversarial attacks, especially backdoor insertion during training. Recently, the edge-case backdoor attack employing the tail of the data distribution has been proposed as a powerful one, raising questions about the shortfall in current defenses' robustness guarantees. Specifically, most existing defenses cannot eliminate edge-case backdoor attacks or suffer from a trade-off between backdoor-defending effectiveness and overall performance on the primary task. To tackle this challenge, we propose FedGrad, a novel backdoor-resistant defense for FL that is resistant to cutting-edge backdoor attacks, including the edge-case attack, and performs effectively under heterogeneous client data and a large number of compromised clients. FedGrad is designed as a two-layer filtering mechanism that thoroughly analyzes the ultimate layer's gradient to identify suspicious local updates and remove them from the aggregation process. We evaluate FedGrad under different attack scenarios and show that it significantly outperforms state-of-the-art defense mechanisms. Notably, FedGrad can almost 100% correctly detect the malicious participants, thus providing a significant reduction in the backdoor effect (e.g., backdoor accuracy is less than 8%) while not reducing main accuracy on the primary task.
Using Connectome Features to Constrain Echo State Networks

We report an improvement to the conventional Echo State Network (ESN) across three benchmark chaotic time-series prediction tasks using fruit fly connectome data alone. We also investigate the impact of key connectome-derived structural features on prediction performance – uniquely bridging neurobiological structure and machine learning function; and find that both increasing the global average clustering coefficient and modifying the position of weights – by permuting their synapse-synapse partners – can lead to increased model variance and (in some cases) degraded performance. In all we consider four topological point modifications to a connectome-derived ESN reservoir (null model): namely, we alter the network sparsity, re-draw nonzero weights from a uniform distribution, permute nonzero weight positions, and increase the network global average clustering coefficient. We compare the four resulting ESN model classes – and the null model – with a conventional ESN by conducting time-series prediction experiments on size-variants of the Mackey-Glass 17 (MG-17), Lorenz, and Rossler chaotic time series; denoting each model's performance and variance across train-validate trials.
Non-exemplar Class-incremental Learning via Dual Augmentation and Dual Distillation

Non-exemplar class incremental learning is the challenge of recognizing new and old classes without storing samples of old classes. The incremental learner is usually faced with how to strike a balance between transferability and stability. On the one hand, if the model overfits the current task data, the performance will collapse when encountering and solving unseen tasks. On the other hand, if the model fails to maintain the knowledge already learned, the feature space will be covered by the new features during the incremental learning process, which leads to catastrophic forgetting. In this paper, we propose a new DADD method to balance transferability and stability better. Firstly, we propose a new method named Rotation class augmentation (Rot-classAug) to provide additional classes to avoid overfitting at the current stage and to improve transferability. Secondly, we propose a new Noise-semantic augmentation (N-semanAug) that contains rich old-class information, which significantly alleviates catastrophic forgetting in continuous learning. Experiments on benchmark datasets show that our method has achieved superior performance, outperforming the state-of-the-art methods by a margin of 5%, 3% and 4%, respectively. Code is available at https://github.com/Ke1Song/DADD
Online Semi-supervised Pairwise Learning

Online learning is a machine learning method that sequentially updates the predictive model. It is a significant learning technique for massive and streaming data, where it is impractical to store the data for training. For such large-scale real-world data, it is also infeasible to label the whole training samples. Online semi-supervised learning concerns learning a model on both labeled and unlabeled examples of streaming data. The online semi-supervised pairwise learning optimizes an objective function in which its loss function is based on pairs of examples. The recent online semi-supervised pairwise learning method builds a first-order pairwise classifier that lacks the generalization ability of batch semi-supervised methods. To improve the generalization capacity of the pairwise model, we propose a second-order online semi-supervised pairwise learning algorithm that exploits second-order information of the features. More specifically, we adopt a confidence-weighted model and reformulate its objective function for pairwise semi-supervised learning. The model treats unlabeled data as positive and negative while pairing the current example with the previous opposite one in building the model. Experiments on different benchmark and real-world datasets show that the proposed model achieves AUC results that surpass the existing state-of-the-art online semi-supervised method. Also, the proposed method shows a comparable AUC results to the batch semi-supervised method.
Remote Sensing Image Object Detection by Fusing Multi-Scale Contextual Features and Channel Enhancement

Remote sensing technology is becoming more sophisticated and is extensively used for object tracking, urban planning, military reconnaissance and other fields. Complex backgrounds and diverse object scales are two important factors that affect the object detection effect of remote sensing images. To address this problem, this paper proposes a remote sensing object detection model that incorporates channel enhancement and multi-scale contextual features. Firstly, the multi-scale contextual feature enhancement module is constructed, which performs multi-order spatial interaction by cascading recursive convolution to obtain contextual information of feature maps at different scales, and introduces attention to reinforce unique features of objects and suppress background interference. Then, the spatial pyramid channel enhancement module combining sub-pixel convolution and adaptive sampling factor is designed to mitigate the semantic weakening of the depth feature maps caused by channel downscaling, thus enhancing the sampling effect between feature maps of different scales and reducing information loss. Finally, the effectiveness of the model is verified on the large-scale remote sensing image object detection dataset DIOR.
Dynamic Analysis and an Eigen Initializer for Recurrent Neural Networks

In recurrent neural networks, learning long-term dependency is the main difficulty due to the vanishing and exploding gradient problem. Many researchers are dedicated to solving this issue and they proposed many algorithms. Although these algorithms have achieved great success, understanding how the information decays remains an open problem. In this paper, we study the dynamics of the hidden state in recurrent neural networks. We propose a new perspective to analyze the hidden state space based on an eigen decomposition of the weight matrix. We start the analysis by linear state space model and explain the function of preserving information in activation functions. We provide an explanation for long-term dependency based on the eigen analysis. We also point out the different behavior of eigenvalues for regression tasks and classification tasks. From the observations on well-trained recurrent neural networks, we proposed a new initialization method for recurrent neural networks, which improves consistently the performance. It can be applied to vanilla-RNN, LSTM, and GRU. We test is on many datasets, such as Tomita Grammars, pixel-by-pixel MNIST dataset, and machine translation dataset (Multi30k). It outperforms Xavier initializer and kaiming initializer as well as other RNN-only initializers like IRNN and sp-RNN in several tasks.
UGTransformer: Unsupervised Graph Transformer Representation Learning

This paper mainly studies graph representation learning in unsupervised scenarios combined with Transformer models. Transformer network models have been widely used in many fields of machine learning and deep learning, and the application of transformer architectures to graph data has been very popular recently. For graph data, the field of graph representation learning has recently attracted a lot of attention. Graph-level representation is widely used in the real world, such as drug molecule design and disease classification in biochemistry. Traditional graph kernel methods, which design different graph kernels for different substructures, are simple but have poor generalization performance. Recently methods based on language models, such as graph2vec, use a particular substructure as the graph representation, which is also similar to the hand-crafted approach and also leads to poor generalization ability. In this paper, we propose the UGTransformer model, which builds on the standard Transformer architecture. We introduce several simple and effective structural encoding methods in order to encode the structural information of the graph into the model efficiently. The unsupervised representation of graphs is learned through a multi-headed attention mechanism and by using powerful aggregation functions. We conducted experiments on a benchmark date set for graph classification, and the experimental results validate the effectiveness of our proposed model.