Reviewerly/exHarmony huggingface dataset:
Facial Image Manipulation via Discriminative Decomposition of Semantic Space

Generative Adversarial Networks (GANs) can learn a highly informative latent space. By manipulating the latent representation in the latent space, we can control attributes in the generated images without modifying the model of GAN itself. Most existing studies find the direction vector of each semantic in the latent space and steer the latent vectors using linear computation to control semantic changes in the images. However, moving against the semantic direction still makes the latent representation subject to the entanglement of attributes in the latent space. Therefore, to solve the problem of attribute entanglement in latent space, we propose a decomposition method that decomposes the identity information subspace and the attribute information subspace from the pre-trained latent space and realize the semantic change of the latent representation by moving in the attribute information subspace. In our experiments, we evaluated its effectiveness and compared the images after the attribute change. The experiments demonstrated that our method can further solve the attribute entanglement problem in the latent space and effectively reduce the impact on other contents in the latent representation after changing the attributes.
Learning Unified Video-Language Representations via Joint Modeling and Contrastive Learning for Natural Language Video Localization

Natural language video localization (NLVL) aims to locate the matching span relevant to a given query sentence from an untrimmed video. This task requires not only understanding video and text but also aligning the semantics between video and language. Existing methods obtain vision-language representations via separate encoders, cross-modal interactions are not fine-grained enough, and the semantics are not fully aligned. In this paper, we address the vision-language alignment via joint modeling and contrastive learning. We propose a unified Video-Language Representation Network (UniNet), employing a transformer encoder to learn vision-language representations aligned. Simultaneously taking video and text as input, the encoder jointly learns the representations of both and captures the inter-relations between video and text. Then the representations are used by the predictor to locate the grounding video span. Besides, we train our model with contrastive learning to enhance vision-language representations in the training stage. Experiments on three benchmark datasets show that UniNet outperforms the baseline methods and adopting unified representation and contrastive learning can improve vision-language semantic alignment.
Interactive Capsule Networks with a Novel Dynamic Routing Mechanism for Implicit Discourse Relation Recognition

Implicit discourse relation recognition aims to understand and infer the relations between discourse arguments, which is a classification task. Existing models mostly model the interaction of discourse arguments from the perspective of a single relation, which may fail to learn the complicated interaction patterns between two arguments under different discourse relations. In this paper, we propose a novel neural approach based on capsule networks with dynamic interactive routing mechanism, named Interactive Capsule Networks (ICN) model to address the issues. It dynamically maps argument feature capsules to discourse relation capsules in an iterative refinement manner, which can mine the discourse relation-indicative semantic clues of argument features. Then, the designed dynamic interactive routing mechanism captures the more comprehensive argument interaction from the perspective of multiple relations. The experimental results on the Penn Discourse TreeBank (PDTB) demonstrate the effectiveness of our proposed ICN model.
TransD-based Multi-hop Meta Learning for Few-shot Knowledge Graph Completion

Few-shot knowledge graph completion (FKGC), which aims to infer missing facts about a relation from only a few reference triples, has recently attracted great attention. The core of solving the FKGC task is to learn a vector representation for each few-shot relation using the corresponding entity represen-tations. To this end, existing models generally enhance entity representations with their direct neighbors. However, a large number of entities have few direct neighbors. Hence, encoding only direct neighborhood is insufficient to obtain satisfactory en-tity representations. In addition, current models typically utilize static embeddings to represent entities, ignoring their diverse semantics, i.e., an entity may show distinct semantics within different few-shot relations. To address these issues, we propose a new FKGC framework, namely TransD-based Multi-hop Meta Learning (TDML). TDML consists of three main components: a multi-hop neighbor encoder to enhance entity representations by aggregating heterogeneous multi-hop neighbors, a transformer encoder to generate the relation meta representations, and a TransD-based relation representation updater that allows each entity to exhibit relation-specific semantics and tune the relation meta representations. Extensive experiments on two public datasets demonstrate that our model outperforms state-of-the-art FKGC methods.
3DSEAVNet: 3D-Squeeze-and-Excitation Networks for Audio-Visual Saliency Prediction

Video saliency prediction is an important task in the field of computer vision. Most of the existing video saliency prediction methods only focus on image information, and the audio information is often ignored. This leads to an incomplete perception mode, which makes it difficult to achieve optimal performance. SENet is an excellent attention mechanism-based network. It significantly enhances the performance of 2D convolutional networks. However, whether the 3D convolutional network can be applied to this attention mechanism network remains to be studied. In order to solve the above problems, we propose a saliency prediction network for audio-visual fusion to extract and predict various information in videos. At the same time, we improve the traditional SENet to make it applicable in 3D convolutional neural networks and discuss its role. Compared with the state-of-the-art methods, our model has strong competitiveness in multiple data sets.
Neural Radiance Fields with Regularizer Based on Differences of Neighboring Pixels

Neural Radiance Fields (NeRF) have significantly impacted the novel view synthesis task. While it can render realistic images from unseen viewpoints, the boundaries of the object or scene in the areas rapidly changing the color and density are blurred. We present a novel NeRF regularization based on neighboring pixels’ differences to resolve this issue. This proposed regularized NeRF is named Neighboring Pixels Differences Regularized Neural Radiance Fields. Our key insight is that the pixel neighborhood relation in the image has essential information about the structure of the object and scene. Therefore, we propose to incorporate the neighborhood information into the original NeRF. We can easily implement the proposed method by adding a regularization term based on pixel neighborhood relation into the loss function of the vanilla NeRF. In the experiments, to evaluate the effectiveness of our method, we compared the proposed method with the original NeRF using synthetic and real-world multiview data. The experimental results showed that our method allowed for rendering high-quality novel views for both settings than the original NeRF.
RAN: A Relation-aware Network for Relation Extraction

Relation extraction is extracting relational facts between target entities from plain text. It has attracted much interest and achieved improvement from research communities recently. However, effectively capturing relational information while ignoring intra-sentence noise from the text remains a challenge. To this end, we propose a novel RAN (Relation-aware Network) model for relation extraction. The basic idea of RAN is to model relation features, providing more accurate guidance for directly capturing relational information in a sentence, and improving the performance of relation extraction. Specifically, we first propose a multi-channel relation representation learning module. This module studies dependencies between the target entity pair in multiple sub-channels to model relation vectors. Then a dual attention mechanism-based relational words selection module is designed to highlight the relational information in the sentence of different dimensions, including semantic- and syntactic-level features. We evaluate our model on two public datasets: SemEval-2010 Task 8 and TACRED. Experiments on two benchmark datasets demonstrate the inspiring performance improvement over state-of-the-art models.
Proxy-Aware Cross-Domain Sequential Recommendation

Cross-domain sequential recommendation (CDSR) aims to predict the next item that a user is most likely to interact with based on past sequential behavior from multiple domains. Existing works on CDSR usually transfer knowledge across different domains by linking items between domains via common users, which suffers from the following limitations: (1) due to the inherent differences between the domains, transferring information across domains can be affected by different representations of related items in different domains. (2) None of existing studies consider the time interval information among items, which is essential in sequential recommendation to capture user intents over time. In this work, we propose a novel cross-domain sequential recommendation model to address the above challenges. Specifically, we first design a shared proxy item encoder to generate a universal representation for each item in all domains by using its textual descriptions. Then, we develop a time-interval-aware attention encoder to represent sequences by considering the time interval information. Moreover, we present a contrastive learning auxiliary task to enhance a cross-domain sequence by weighing the importance of the items in the auxiliary domain with respect to the objective domain. Experiments demonstrate the superiority of our proposed method from various aspects.
Improving Chinese Spelling Correction by Ranking

Chinese Spelling Check (CSC) aims to detect and correct Chinese spelling errors. Most Chinese spelling errors are the misuse of semantically, phonetically or graphically similar characters. Previous state-of-the-art works on the CSC task pursue transitions from misspelled sentences to correct sentences directly. However, it is difficult to force the current CSC methods to find the correct answer at one run. Thus, we propose a simple and effective method for CSC task by making fully use of the trained model to generate multiple candidate sentences and simply ranking to select the best, in which no additional training and parameters are required. The experimental results show that our approach outperforms previous methods and achieves the state-of-the-art performances.
Modeling Zero-Shot Relation Classification as a Multiple-Choice Problem

Zero-shot relation classification (ZeroRC) aims to infer the semantic relations between entity pairs in sentences, while the relation sets at the training and testing stages are disjoint. It is a crucial task in information extraction and is much more challenging than traditional relation classification. We propose a novel method named MC-BERT to model ZeroRC as a Multiple-Choice problem employing BERT as the backbone model. A semantic template is designed to infuse the information of entities and context. It serves as the question stem of the multiple-choice, followed by a relation label which serves as a choice. Moreover, we propose a grouping strategy to improve training efficiency. We perform comprehensive experiments on two datasets, Wiki-ZSL and FewRel. The results show that our proposed method significantly outperforms previous works. Specifically, it achieves performance gains ranging from 1.17% to 6.87% in the F1 score with only 40% of the model parameters against to state-of-the-art method, demonstrating the simplicity and effectiveness of our proposed method.
What Causes a Driver's Attention Shift? A Driver's Attention-Guided Driving Event Recognition Model

Despite much effort to try to research driver's spatial attention allocation in driving situations, the computer vision community rarely focuses on what causes driver's attention shifts. In this paper, we built an attention-based driving event dataset (ADED) constructed from the attention distributed on the traffic participants or elements and proposed a model using driver's attention as the guidance to better recognize the events that lead driver's attention shifts. We relabeled and redivided BDD-A, a driver attention dataset in critical traffic situations, into six different semantic categories of driving events. The new dataset is introduced for driving event recognition. In addition, we proposed a special driving event recognition model (called DER-Net) with driver's attention guidance to recognize the event causes a driver's attention shift. In DER-Net, a driver's attention-guided (DAG) branch is constructed to consider the driver's spatiotemporal attention information. The proposed model achieves a superior performance compared to other state-of-the-art models in action recognition. In ablation study, many experiments are conducted to discuss proper length of the sequence to input the model, the optimal criterion to find out the frame when driver's attention shifts and the appropriate function to quantify different attention maps in neural network.
Flexible Contribution Estimation Methods for Horizontal Federated Learning

Federated Learning (FL), as an emerging distributed learning paradigm to protect data privacy, facilitates model training collaborations among entities. However, performance of FL model in such a promising paradigm would drop dramatically without adequate participating entities' data. To maintain sustainability and efficiency of FL system, it is inevitable to design a reasonable and fair contribution estimation method. Despite of success, existing studies of contribution estimation do not adequately balance efficiency and fairness. To overcome such a limitation, we propose two plug-and-play methods, named as vector projection method (VPM) and income exclusion method (IEM), to estimate the contribution of each entity in FL. As standalone contribution measurement modules, they can be deployed to almost any FL server flexibly. We conduct extensive experiments on widely-used Fashion-MNIST and CIFAR-10 datasets. The experimental results demonstrate the promising effectiveness of contribution estimation, and show the convergence acceleration under contribution-weighted aggregation mechanism.
Boosting External-Reference Image Quality Assessment by Content-Constrain Loss and Attention-based Adaptive Feature Fusion

With the development of deep learning, image quality assessment (IQA) methods have made significant progress, but full-reference (FR) methods are limited by the reference image, and the performance of no-reference (NR) methods is relatively poor. Therefore, some researchers have introduced a new scheme, external-reference (ER) IQA, which uses an arbitrary high-quality image as the reference image (external reference image). The key to ER-IQA is how to extract useful reference information from external reference images and use it effectively. Since the content of the external reference image is independent of the distorted image, we think that the content information of the external reference image is harmful to the algorithm. Therefore, a content-constrain loss is designed for training the network to suppress the content information of external reference images. To utilize the external reference information more effectively, we design an attention-based adaptive feature fusion (AAFF) module. Experimental results demonstrate the effectiveness of the designed loss and feature fusion module.
Worldwide COVID-19 Topic Knowledge Graph Analysis From Social Media

Current research on online public opinion regarding the coronavirus disease in 2019 (COVID-19) leverages keyword extraction, sentiment analysis, and topic modeling to analyze online public opinion. The multi-granularity features of online public opinions and semantic relations between the features, how-ever, remain less explored. Reliance on only topics or keywords for measuring public opinion is insufficient as topics are too broad and keywords too narrow. Analysis at an intermediate level, which most studies overlook, is crucial in gaining a clearer insight into public opinion. Additionally, exploring the semantic relationships between components of online public opinion can shed light on the logical connections between them and help understand how they interact in the dissemination of online public opinion, leading to a better understanding of its evolution mechanism. We conducted a public opinion analysis on Worldwide COVID-19 outbreaks via Topic Knowledge Graph. Specifically, we first use the Combined Topic Model to extract public opinion topics. Then multi-dimensions attributes of the topic such as [subject, predicate, object] triples, topic popularity, and topic emotion intensity are extracted. Subsequently, the semantic relations of different public opinion topics are calculated from the two levels of predicates co-occurrence and subject-object sharing. Finally, we applied the constructed framework to the public opinion information related to COVID-19 and analyzed the characteristics in the evolution of online public opinion.
Enhanced Loss Function based on Laplacian Eigenmaps for Graph Classification

Recently, many works on Graph Neural Networks (GNNs) have been well developed for graph-level representation learning tasks and continuously improve graph classification accuracy. These works, however, mainly focus on the principle of different blocks of GNNs to strengthen their representation learning capability and pay less attention to the choice of a more appropriate loss function as learning objectives. In this paper, we aim to facilitate the representation learning process of any existing GNN frameworks by incorporating more task-oriented objectives. To this end, we propose approaches based on Laplacian eigenmaps to enhance the common-used cross-entropy loss, called LEELoss. Our study shows, with the property of Laplacian eigenmaps for classification problems, that utilizing the Laplacian eigenmaps as a regularizer in the original loss function can further enhance the performance of the graph-level learning tasks. Finally, via extensive experiments on popular benchmark datasets, we demonstrate that the proposed approach actually facilitates the classification performance on various GNN frameworks.
GamMa: Efficient Fine-Tuning of Pre-Trained Language Models Using Gradient Activation Mapping Masking

Recently pre-trained language models have shown great success in natural language processing. To fully leverage these models on downstream tasks, full fine-tuning has become a ubiquitous training paradigm. However, it is inefficient and resource-intensive to fine-tune the entire parameters of the model since it always employs a huge scale of parameters (e.g., GPT-3 has 175B parameters). To address these issues, we propose a simple yet effective method to fine-tune the pre-trained models, named Gradient activation mapping Masking (GamMa). Specifically, GamMa first detects the class-specific parameters from the perspective of gradient activation mapping, and then aggregates them to work out the particular parameters that can greatly influence the model predictive boundaries. During the fine-tuning process, only these significant parameters are updated necessarily. Besides, GamMa takes several advantages over the previous approaches: (i) it does not require modifying the original model structure; (ii) neither extra parameters nor training budgets are introduced. Extensive experiments on the GLUE benchmark demonstrate that our method outperforms the current state-of-the-art counterparts.
A Generative Data Augmentation Trained by Low-quality Annotations for Cholangiocarcinoma Hyperspectral Image Segmentation

Microscopic hyperspectral imaging technology combined with deep learning method emerges medical field recently as a multiplexed imaging technology. With the semantic segmentation of hyperspectral histopathological image of pathological tissue, doctors can quickly locate suspicious areas, diagnose and arrange treatment accurately and rapidly, reducing the workload of them. Cholangiocarcinoma is a rare and devastating disease with few hyperspectral histopathological data. Moreover, achieving high-quality annotations of hyperspectral histopathological image is challenging and costs time for pathologists, so generally, rough labels are annotated, but directly using the low-quality labels will reduce the performance of segmentation networks. So how to fully utilize few high-quality annotations and dozens of low-quality labels to enhance the segmentation performance of cholangiocarcinoma hyperspectral image remains to be resolved. In this paper, we proposed a two-stage hyperspectral segmentation deep learning framework based on Labels-to-Photo translation and Swin-Spec Transformer(L2P-SST). In stage-I, the OASIS generative network and the Swin-Spec Transformer discriminative network are used for adversarial training, and a spectral perceptual loss function is proposed to generate highquality hyperspectral images; in stage-II, parameters of the generative network is fixed and the generated hyperspectral images are used as data augmentation in the training of Swin-Spec Transformer segmentation network. The proposed framework achieved 76.16% mIoU(mean Intersection over Union), 85.80% mDice(mean Dice), 90.96% Accuracy and 71.65% Kappa coefficient in the semantic segmentation task of the Multidimensional Choledoch Database. Compared with other methods, the results demonstrate our framework provides a competitive segmentation performance.
Visual Explanation for Cooperative Behavior in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) can acquire cooperative behavior among agents by training multiple agents in the same environment. Therefore, it is expected to be applied to complex tasks in real environments, such as traffic signal control in a traffic environment and cooperative behavior of robots. In this study, using the multi-actor-attention-critic (MAAC) with the actor-critic method as a basis, we introduce an attention head for the actor that calculates the agent's action. In contrast to the critic in MAAC, which shares the attention head among all the agents, the attention head of the actor in our method is constructed independently for each agent. This allows the attention head of the actor to calculate actor-attention (indicating which other agents are gazed at by each agent) and to acquire cooperative behavior. We visualize actor-attention to analyze the basis of agents’ decisions for cooperative behavior. Using single_spread, which is a multi-agent environment for cooperative problems, we show that the basis of decisions for cooperative behavior can be easily analyzed. We also demonstrate that our method efficiently obtains cooperative behavior considering other agents through quantitative evaluation of the cooperative behavior.