Reviewerly/exHarmony huggingface dataset:
Mitigating Negative Transfer with Task Awareness for Sexism, Hate Speech, and Toxic Language Detection

This paper proposes a novelty approach to mitigate the negative transfer problem. In the field of machine learning, the common strategy is to apply the Single-Task Learning approach in order to train a supervised model to solve a specific task. Training a robust model requires a lot of data and a significant amount of computational resources, making this solution unfeasible in cases where data are unavailable or expensive to gather. Therefore another solution, based on the sharing of information between tasks, has been developed: Multi-task Learning (MTL). Despite the recent developments regarding MTL, the problem of negative transfer has still to be solved. Negative transfer is a phenomenon that occurs when noisy information is shared between tasks, resulting in a drop in performance. This paper proposes a new approach to mitigate the negative transfer problem based on the task awareness concept. The proposed approach results in diminishing the negative transfer together with an improvement of performance over classic MTL solution. Moreover, the proposed approach has been implemented in two unified architectures to detect Sexism, Hate Speech, and Toxic Language in text comments. The proposed architectures set a new state-of-the-art both in EXIST-2021 and HatEval-2019 benchmarks.
Schema Item Matters in Knowledge Base Question Answering

Knowledge base question answering is a challenging task that aims to answer questions by querying knowledge bases. Recently, state-of-the-art methods tend to include an enumerator module and a ranker module. They first enumerate by searching the knowledge base and then rank the candidates to select the target logical form. However, these methods sometimes fail to cover the candidates which involve more complex combinations. A recent solution to this issue is to add a generator module after the ranker to generate the uncovered target logical form. However, the enumerator and ranker always discard partial ground truth schema items. Consequently, the lack of them in the generator input results in the failure to generate the target logical form. To address this problem, we present a novel framework, SIMQA, to reuse the neglected schema items, i.e., classes and relations. Specifically, we adopt a matcher module to select the most related schema items for the given question, and feed them to the generator. On this basis, we propose a novel generation model based on contrastive learning to force the model to focus on the supplemental schema items. Experiment results on GRAILQA and WEBQSP datasets demonstrate the highly competitive performance of the proposed method, and verify that schema item matters in KBQA.
An Efficient Frequency Domain Separation Network for Paired and Unpaired Image Super-Resolution

Although existing super-resolution (SR) techniques have made great progress, they are often tailored for either paired or unpaired scenery, thus may result in poor migration ability. In this work, we propose a generalized Frequency Domain Separation Network (FDSNet) for both paired and unpaired SR settings. Firstly, through statistical analysis, we found that real-world low-resolution (LR) images and high-resolution (HR) images differ greatly in high frequencies but less in low frequencies. Inspired by this, we perform high and low-frequency separation of LR images and guide our model to reconstruct the HR contents in the different frequency domains. Then, according to the varying attention on frequencies of traditional CNN and Transformer models, we design a parallel pipeline: LFNet based on Transformer for low-frequency feature extraction, and HFNet based on CNN for high frequencies. In LFNet, to further alleviate the high complexity and data dependency of Transformer, Simplified Multi-head Self Attention (SMSA) is proposed at a low computational cost. And original MLP is replaced by our Spatial Enhancement MLP (SEMLP) to take full advantage of local spatial contexts. Finally, to further facilitate frequency separation and learning, a Frequency attention block is designed to impose guidance on high frequencies. Experiments indicate that our FDSNet achieves promising performance in terms of quantitative and qualitative evaluations while enjoying a faster speed and much fewer parameters.
TRMER: Transformer-Based End to End Printed Mathematical Expression Recognition

As a fundamental task of transcribing formula images into structural mathematical expressions, Printed Mathematical Expression Recognition (PMER) is wildly used in many fields. However, there is still a lack of an end-to-end approach toward fully exploring the spatial structure and semantic information in the formula to achieve high recognition accuracy. In this work, a Transformer-based Mathematical Expression Recognition (TRMER) model, is proposed to enhance the recognition accuracy. A Dual-Branch Encoder (DBE) is developed to extract multi-scaled feature maps from a formula image so that the spatial and semantic information can be obtained synchronously, and the different feature maps are fused with a Fusion Enhancement Module (FEM) by merging and reinforcing the spatial-semantic information. A standard transformer-based decoder is developed to decode the rich spatial-semantic information of the image and output a recognized mathematical expression in LaTex sequence. The experimental results have illustrated that the TRMER has achieved state-of-the-art recognition performance.
Multi-level Feature Joint Learning Methods for Emotional Speaker Recognition

In the real scene, changes in speaker features caused by different emotional states have a great impact on the performance of speaker recognition. To improve the robustness of the speaker recognition system, the existing emotional speaker recognition technologies tend to cascade different models, ignoring the frame-level acoustic features and the segment-level discourse habits feature. To this end, we combine frame- and segment-level features in different ways to build a robust recognition system for emotional speakers. The frame-level features and segment-level features are jointly learned to retain emotional information and speaker information. Four joint learning methods, namely, Joint in series, Joint in Parallel, Joint under the guidance, and Joint with Original Feature, are discussed to explore the correlations between fragment-level features and frame-level features. The experimental results illustrate that the speaker feature will change greatly in different emotional states. Compared with the accuracy of 90.95% by x-vector, the proposed methods of Joint in parallel and Joint with Original Features can achieve the accuracy of 95.06% and 94.67% respectively for emotional speaker recognition in the experiment on Mandarin Affective Speech Corpus (MASC). Our findings provide a novel aspect to improve speaker recognition robustness.
Towards Understanding the Link Between Modularity and Performance in Neural Networks for Reinforcement Learning

Modularity has been widely studied as a mechanism to improve the capabilities of neural networks through various techniques such as hand-crafted modular architectures and automatic approaches. While these methods have sometimes shown improvements towards generalisation ability, robustness, and efficiency, the mechanisms that enable modularity to give performance advantages are unclear. In this paper, we investigate this issue and find that the amount of network modularity for optimal performance is likely entangled in complex relationships between many other features of the network and problem environment. Therefore, direct optimisation or arbitrary designation of a suitable amount of modularity in neural networks may not be beneficial. We used a classic neuroevolutionary algorithm which enables rich, automatic optimisation and exploration of neural network architectures and weights with varying levels of modularity. The structural modularity and performance of networks generated by the NeuroEvolution of Augmenting Topologies algorithm was assessed on three reinforcement learning tasks, with and without an additional modularity objective. The results of the quality-diversity optimisation algorithm, MAP-Elites, suggest intricate conditional relationships between modularity, performance, and other predefined network features.
A two-layer BiLSTM model with linear gating for Chinese named entity recognition

Chinese named entity recognition (CNER) is one of the most fundamental tasks in natural language processing (NLP), and is key to extracting information from unstructured texts. In recent years, advances in neural network models and pretrained word-level information embedding techniques have played a driving role in the development of NLP. In this context, how to make full use of word vectors to extract information has become one of the research emphases. The diversity of Chinese expressions and the irregular expressions of texts lead to poor recognition results. This paper proposes a two-layer BiLSTM network model with linear gating logic to enhance the model's learning effect of word vectors within sentences and word memory. The aim is to solve the problem of gradient disappearance and improve the model's generalization ability and entity recognition. Through experiments, our model proved effective on three Chinese benchmark datasets: MSRA, the People's Daily Corpus (PRF), and Boson. The precision of NER performs best among similar models. In addition, using the lab-constructed medical dataset of Chinese Drugs for the Heart for testing, our model outperforms the existing BiLSTM model. Finally, statistical analysis of the changes in F1 during training demonstrated faster convergence of our model.
The Backpropagation algorithm for a math student

A Deep Neural Network (DNN) is a composite function of vector-valued functions, and in order to train a DNN, it is necessary to calculate the gradient of the loss function with respect to all parameters. This calculation can be a non-trivial task because the loss function of a DNN is a composition of several nonlinear functions, each with numerous parameters. The Backpropagation (BP) algorithm leverages the composite structure of the DNN to efficiently compute the gradient. As a result, the number of layers in the network does not significantly impact the complexity of the calculation. The objective of this paper is to express the gradient of the loss function in terms of a matrix multiplication using the Jacobian operator. This can be achieved by considering the total derivative of each layer with respect to its parameters and expressing it as a Jacobian matrix. The gradient can then be represented as the matrix product of these Jacobian matrices. This approach is valid because the chain rule can be applied to a composition of vector-valued functions, and the use of Jacobian matrices allows for the incorporation of multiple inputs and outputs. By providing concise mathematical justifications, the results can be made understandable and useful to a broad audience from various disciplines.
HTP: Exploiting Holistic Temporal Patterns for Sequential Recommendation

Sequential recommender systems have demonstrated a huge success for next-item recommendation by explicitly exploiting the temporal order of users' historical interactions. In practice, user interactions contain more useful temporal information beyond order, as shown by some pioneering studies. In this paper, we systematically investigate various temporal information for sequential recommendation and identify three types of advantageous temporal patterns beyond order, including absolute time information, relative item time intervals and relative recommendation time intervals. We are the first to explore item-oriented absolute time patterns. While existing models consider only one or two of these three patterns, we propose a novel holistic temporal pattern based neural network, named HTP, to fully leverage all these three patterns. In particular, we introduce novel components to address the subtle correlations between relative item time intervals and relative recommendation time intervals, which render a major technical challenge. Extensive experiments on three real-world benchmark datasets show that our HTP model consistently and substantially outperforms many state-of-the-art models. Our code is publically available at https://github.com/623851394/HTP/tree/main/HTP-main.
An Improved Branch Entropy Based Method for Chinese New Words Detection

To improve the performance of Chinese new words detection, this paper enhances the traditional method through two aspects. Firstly, a new and more effective metric for branch entropy is chosen through an extensive empirical study of different metrics; besides, punctuation are classified to two categories and assigned with different weights according to their contribution to differentiating new words. After the first step, a set of candidate new words are generated. Secondly, a multi-criteria Transformer-CRF segmentation probability model is introduced to filter the noisy candidates. The new method is tested on SIGHAN 2005 and 2008 Bakeoff data sets, and the performance demonstrates that for the resulted top 1000 ~100 words, our new metric contributes to 2.8% ~9.05% improvement compared with the commonly used minimum entropy metric; while punctuation processing leads to 3.26% ~3.77% improvement. Experiments on the legal corpus show that the Transformer-CRF model's top 1000 MAP results are further improved by 2.34% comparing with the unsupervised method.
Referenceless User Controllable Semantic Image Synthesis

Despite recent progress in semantic image synthesis, complete control over image style remains a challenging problem. Existing methods require reference images to feed style information into semantic layouts, which indicates that the style is constrained by the given image. In this paper, we propose a model named RUCGAN for user controllable semantic image synthesis, which utilizes a singular color to represent the style of a specific semantic region. The proposed network achieves reference-free semantic image synthesis by injecting color as user-desired styles into each semantic layout, and is able to synthesize semantic images with unusual colors. Extensive experimental results on various challenging datasets show that the proposed method outperforms existing methods, and we further provide an interactive UI to demonstrate the advantage of our approach for style controllability. The codes and UI are available at: https://github.com/BenjaminJonghyun/RUCGAN
FAU-Gaze: Fast and Accurate User-specific Gaze Estimation Framework

Gaze estimation has a wide range of applications such as neuroscience and clinical research. In this paper, we propose and implement a fast and accurate user-specific gaze estimation system, called FAU-Gaze. FAU-Gaze supports online real-time training with an inference speed of up to 7-11.5 ms in 100 FPS. Compared with existing models, the kernel model FPGC (Feature-based Personalized Gaze Calibrator) of FAU-Gaze increases the accuracy by 36.4% and 33.7% on MPIIFaceGaze and TabletGaze respectively. By mining each user's potential characteristics, FAU-Gaze can more accurately locate each user's real gaze position. In order to test FAU-Gaze, we also introduce a low-resolution and low-definition laptop gaze estimation dataset TobiiGaze containing 41,000 images. Through our experiments on both TobiiGaze, MPIIFaceGaze, and TabletGaze, the prediction error of FAU-Gaze is reduced to 1.61 cm and the robustness outperforms the state-of-the-art.
SFGAN: Style-Former GAN for Disentanglement in Image Generation

With rich semantic knowledge, the latent space in StyleGAN can be mapped to the high-dimensional space of the generated images. To produce a new latent space with distinct semantics properties, researchers have tried integrating the Transformer module into the structure of StyleGAN. While achieving better performance on semantic control of vectors (so called the disentanglement of latent space), this approach however faces steep challenges such as more training space (parameters) required as the size of output images increases and more time required to disentangle the semantic direction for the complex latent space. To address the challenges, in this paper we propose a new generator Style-Former GAN (SFGAN) based on channel and spatial attention. SFGAN has a style convolution attention block to compute spatial attention features while maintaining adaptive instance normalization as the channel attention block. A parallel equal-linear block is developed in the spatial dimension that significantly simplifies parameter calculation. Moreover, we propose a new unsupervised disentanglement algorithm, Open Semantic Factorization (OSF), that completes the entire task from generation to disentanglement of the face images. The algorithm enables the decoupling of single attribute from a face image while maintaining the face's identity. Our experiments have demonstrated that compared to existing approaches, SFGAN has a stronger ability to untangle the latent space while generating images of high quality.
Imperceptible Adversarial Attack on S Channel of HSV Colorspace

Deep neural network models are vulnerable to subtle but adversarial perturbations that alter the model. Adversarial perturbations are typically computed for RGB images and, therefore, are evenly distributed among RGB channels. Compared with RGB images, HSV images can express the Hue, saturation, and brightness more intuitively. We find that the adversarial perturbation in the S-channel ensures a high attack success rate, while the perturbation is small, and the visual quality of the adversarial examples is good. Using this finding, we propose an attack method, SPGD, to improve the visual quality of adversarial examples by generating perturbations on the S-channel. Based on the attack principle of the PGD method, the RGB image was converted into an HSV image. The gradient calculated by the model on the S channel was superimposed on the S channel and then combined with the non-interference H and V channels to convert back to the RGB image. The iteration stops until the attack succeed. We compare the SPGD method with the existing state-of-the-art attack methods. The results show that SPGD minimizes pixel perturbation while maintaining a high attack success rate and achieves the best results in terms of structural similarity, imperceptibility, the minimum number of iterations, and the shortest run time.
A Multimodal Hierarchical Variational Autoencoder for Saliency Detection

Existing multimodal Salient Object Detection (SOD) methods do not generalize well for more complex and scalable multimodal learning scenarios. In this paper, we propose a multimodal hierarchical variational auto-encoder for generalized multimodal SOD. By introducing joint inference factorization methods in multimodal VAEs, our model is scalable to partially missing modality data. A latent hierarchy is proposed which enhances expressiveness in latent space and allows multi-level interaction between features across modalities. By further exploring the latent hierarchy, we provide intuitive uncertainty visualizations and observe that the main source of uncertainty in SOD derives from the lower-level features. Based on this, we propose a simple yet effective sampling-based confidence estimation method, that brings robustness when encountering untrustworthy modality data for inference. Extensive experimental analysis illustrate that our model can satisfy crucial properties that make a desirable multimodal SOD framework.
Hierarchical Knowledge Transfer Network for Distantly Supervised Relation Extraction

Distantly supervised relation extraction (DSRE) aims to identify the relation between the two entities (e.g. name and location). Most existing methods extract semantic features from each level separately, without taking into account the transfer of hierarchical knowledge obtained at various levels. As a result, a large amount of knowledge that can improve the quality of the feature representations is lost, resulting in decreased performance for predicting entity relations. In this paper, we propose a novel framework termed the Hierarchical Knowledge Transfer Network (HKTN) that is capable of transferring hierarchical knowledge learned from different levels to improve the performance of predicting entity relations. Specifically, the two representation refinement blocks with re-calibrators at the bag and group levels construct robust bag features and comprehensive group features, respectively. During the construction process, the high-level features are capable of guiding the learning of the bottom-level features using the two re-calibrators. As the construction of the high-level feature representations is based on the bottom-level feature representations, prediction-based contrastive learning fully excavates bottom-level features, which can improve the quality of the feature representation at each level. The experimental results demonstrate that our proposed HKTN achieves an obvious improvement on the two benchmark datasets, including NYT-10 and GDS.
From Implicit to Explicit: A Simple Generative Method for Aspect-Category-Opinion-Sentiment Quadruple Extraction

Aspect-Category-Opinion-Sentiment Quadruple Extraction (ACOS) is a critical subtask of Aspect-Based Sentiment Analysis (ABSA), aiming to extract all quadruples <aspect, category, opinion, sentiment> in a review sentence. Existing ACOS methods are categorized into pipeline methods and unified methods. Pipeline methods face error propagation and ignore the interdependency among four sentiment elements. Unified methods generate a long target sequence when the number of quadruples increases, which degrades the model performance. In this paper, we pay attention to the implicit aspects and opinions to obtain comprehensive aspect-level sentiment information. To this end, we propose a novel sequence generation model for ACOS named SG-ACOS. We design a linearization method to express ACOS quadruples as a sequence. The proposed linearization method incorporates natural language tokens and two special tokens into the target sequence. Natural language tokens are employed to represent the four sentiment elements, thus facilitating the model to learn the semantics of sentiment elements. Special tokens reduce the length of the target sequence, thereby improving the model efficiency and performance. Our proposed model obtains an absolute <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$F_{1}$</tex> improvement of 3.52% and 2.74% against previous state-of-the-art methods on two ACOS datasets, respectively. Further experimental results show the model effectiveness in implicit sentiment detection and the robustness of our model.