Reviewerly/exHarmony huggingface dataset:
Reputation-Aware Opportunistic Budget Optimization for Auction-based Federation Learning

As an emerging privacy-preserving collaborative machine learning paradigm, the success of federated learning (FL) relies heavily on effectively motivating data owners (a.k.a. FL clients) to contribute high-quality local data and computational resources towards FL model training. Rewarding clients with incentives has been identified as a useful approach towards this goal. It is often implemented in conjunction with auction-based federated learning (AFL). Nevertheless, this approach introduces new challenges for FL servers (a.k.a. federations) as they must work within a limited budget and make trade-off decisions between hiring FL clients, saving costs and achieving target model performance. The dynamic nature of AFL further complicates this problem as federations need to consider potential competition from other federations. To address this challenge, we propose the Reputation-aware Opportunistic Budget Optimization approach for Auction-based Federated Learning (ROBO-AFL). Based on Lyapunov optimization, it helps federations maximize their utility by determining the time-averaged optimal allocation of budget for the hiring FL clients. Extensive experimental evaluation based on real-world data demonstrates that compared to four state-of-the-art approaches, ROBO-AFL achieves the most advantageous trade-off between fairness, cost-effectiveness and utility with 1.80% higher utility with 62.85% lower cost than the best-performing baseline, while maintaining a comparably high level of fairness. To the best of our knowledge, ROBO-AFL is the first decision support approach designed to help federations optimize their budget usage in competitive and open AFL markets.
Few-Shot Action Recognition with A Transductive Maximum Margin Classifier

Few-shot action recognition aims to train a classifier that can generalize well when just a small number of labeled videos per class are given. We introduce a transductive maximum margin classifier for few-shot action recognition, which leverages the unlabeled query videos to improve the recognition performance in the test task. The basic idea of the classical maximum margin classifier is to search for a classifier with the largest geometric margin so that training data can be correctly classified. Due to the insufficient number of labeled videos in the support set, it is challenging to find such a classifier with good generalization ability. We observe that exploring the geometric relationship between the separating hyperplane of the classifier and the feature vectors of the query videos can bring improvements to the classifier. In order to improve data utilization efficiency in the few-shot setting, the class prototypes are also treated as examples, which participate in the iterative training process of the model. Experimental results on two action recognition datasets including Kinetics and Something-Something V2 show that our method achieves state-of-the-art performance.
Snippet-level Supervised Contrastive Learning-based Transformer for Temporal Action Detection

Anchor-free temporal action detection methods have recently achieved many good results in solving the problem of flexible boundaries and different duration of actions. But the anchor-free methods use local features to predict the action boundaries so that it is sensitive to noises and prone to generate incomplete action proposals. Moreover, there exist long-term temporal dependencies between actions and temporal semantic consistency between action primitives in the same classes of actions. Therefore, we propose a snippet-level supervised contrastive learning-based transformer (SSCL-T) model for temporal action detection, which can learn semantically local and global temporal relationships in actions. This model learns the local temporal dynamic features of actions through local temporal coding and uses the transformer to model the global semantic dependencies between long-term actions. In addition, we utilize the action class information to learn the high-level semantic features of actions by designing a snippet-level supervised contrastive learning, forcing the temporal dynamic features of the same class of actions to be as close as possible and the features of different classes of actions to be as far away as possible, thus effectively realizing accurate prediction of action boundaries. Our model has been verified on two benchmark datasets ActivityNet-v1.3 and THUMOS14. The experimental results demonstrate that the proposed model has significantly improved on both datasets. Compared with the benchmark method BMN, the average mAP value has increased by 2.91% and 8.4% on ActivityNet-v1.3 and THUMOS14, respectively.
Document-level Relation Extraction with Entity Interaction and Commonsense Knowledge

Document-Level Relation Extraction(DLRE) is a more challenging task than sentence-level relation extraction because of the characteristics such as more extended context, more interactions between entities, and the need for common sense to help the relation inference. In this paper, we propose an effective model to address the problems of complex entity interactions and the lack of commonsense knowledge. Specifically, we propose a Transformer-based entity interaction module instead of graph neural networks to model the correlation across entities, thus avoiding the information loss problem triggered by predefined edge-building rules. In addition, the initial word vector from the word embedding layer of a pre-trained language model is injected into entity representation to boost the performance in the extraction of relational facts that need commonsense knowledge. Experiments show that our model obtains competitive performance, especially compared with graph-based methods, which is faster and more effective. The source code, trained checkpoint files, and the commit results will be released to the public.
KETM:A Knowledge-Enhanced Text Matching method

Text matching is the task of matching two texts and determining the relationship between them, which has extensive applications in natural language processing tasks such as reading comprehension, and Question-Answering systems. The mainstream approach is to compute text representations or to interact with the text through attention mechanism, which is effective in text matching tasks. However, the performance of these models is insufficient for texts that require commonsense knowledge-based reasoning. To this end, in this paper, We introduce a new model for text matching called the Knowledge Enhanced Text Matching model (KETM), to enrich contextual representations with real-world common-sense knowledge from external knowledge sources to enhance our model understanding and reasoning. First, we use Wiktionary to retrieve the text word definitions as our external knowledge. Secondly, we feed text and knowledge to the text matching module to extract their feature vectors. The text matching module is used as an interaction module by integrating the encoder layer, the co-attention layer, and the aggregation layer. Specifically, the interaction process is iterated several times to obtain in-depth interaction information and extract the feature vectors of text and knowledge by multi-angle pooling. Then, we fuse text and knowledge using a gating mechanism to learn the ratio of text and knowledge fusion by a neural network that prevents noise generated by knowledge. After that, experimental validation on four datasets are carried out, and the experimental results show that our proposed model performs well on all four datasets, and the performance of our method is improved compared to the base model without adding external knowledge, which validates the effectiveness of our proposed method.
Antonymy-Synonymy Discrimination through Repelling Parasiamese Neural Networks

Discriminating antonyms and synonyms is a challenging NLP task for which a number of methods have been proposed. A main challenge of this task is that both relations involve semantically similar words, despite the contrariness of antonyms, often occurring nearby in word embedding spaces. In this work, we introduce the repelling parasiamese neural network, a model that considers a parasiamese network for antonyms and a siamese network for synonyms relying on the same encoder and repels their outputs. The resulting model is a fully differentiable neural network for supervised antonymy-synonymy discrimination on pretrained word vectors. We show that this model achieves state-of-the-art results on this task.
Semi-supervised Multi-Label Learning with Missing Labels via Correlation Information

In multi-label learning, each instance is associated with a set of labels simultaneously. Most existing studies assume that the set of labels for each instance is complete. However, it is generally difficult to obtain all the relevant labels of each instance, and only a partial or even empty set of relevant labels is available, which is called semi-supervised multi-label learning with missing labels. To tackle this problem, we propose a novel framework that considers label correlations and instance correlations to recover the missing labels and utilizes a large amount of unlabeled data simultaneously to improve the classification performance. Specifically, a new supplementary label matrix is firstly obtained by learning the label correlation. Secondly, considering each class label may be decided by some specific characteristics of its own, a label-specific data representation is hence learned for each class label. Thirdly, instance correlations are utilized not only to recover the missing labels, but also to propagate the supervision information from labeled instances to unlabeled ones. In addition, a united objective function is designed to facilitate the above processing and an accelerated proximal gradient method is adopted to solve the optimization problem. Finally, extensive experimental results conducted on several benchmark datasets demonstrate the effectiveness of the proposed method compared to competing ones.
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability

Sparsely-gated Mixture of Expert (MoE) layers have been recently successfully applied for scaling large transformers, especially for language modeling tasks. An intriguing side effect of sparse MoE layers is that they convey inherent interpretability to a model via natural expert specialization. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability. To stabilize MoE training, we present both soft and hard constraint-based approaches. With hard constraints, the weights of certain experts are allowed to become zero, while soft constraints balance the contribution of experts with an additional auxiliary loss. As a result, soft constraints handle expert utilization better and support the expert specialization process, while hard constraints maintain more generalized experts and increase overall model performance. Our findings demonstrate that experts can implicitly focus on individual sub-domains of the input space. For example, experts trained for CIFAR-100 image classification specialize in recognizing different domains such as flowers or animals without previous data clustering. Experiments with RetinaNet and the COCO dataset further indicate that object detection experts can also specialize in detecting objects of distinct sizes.
How Image Corruption and Perturbation Affect Out-of-Distribution Generalization and Calibration

In scenarios such as people-flow analysis and autonomous driving using camera footage, the performance of deep neural networks can be impacted by environmental changes such as the degradation of the camera lenses over time and changes in weather conditions. Understanding the impact of environmental changes on model performance is critical to the safe use of the model but has not been systematically investigated. In this study, we cast environmental changes as distribution shifts in the data and tackle the out-of-distribution generalization problem. In particular, we investigate the impact of performance degradation and uncertainty estimation under more than 30 distribution shifts considered in real-world applications. Our experimental results show that there is a correlation between classification accuracy and uncertainty in out-of-distribution environments. Furthermore, it is suggested that uncertainty calibration is not necessarily effective in environments where severe corruption occurs. We believe these findings will contribute to the safe use of the model in the real world.
SCIRNet: Skeleton-based cattle interaction recognition network with inter-body graph and semantic priority

Cattle are social and sensitive animals that are held together by an intricate social web. Following the inter-action between cattle is of great importance to the producer for effective herd management. To recognize the interaction, computer vision technology has been widely employed and current solutions in the fields are mainly dominated by frame-based approaches using CNNs and skeleton-based approaches using GCNs. Recently, skeleton-based interaction recognition has been gaining increasing attention due to its robustness to learn the representation of behavioral features. One crucial cue in recognizing an interaction is the interactive body parts. In this work, we propose a novel interaction recognition network applicable to cattle called SCIRNet with an inter-body graph generated from the respective skeleton graph of each cattle. More specifically, the inter-body graph enables the network to focus on interactive body parts by connecting some inter-body joints which are considered to be able to represent features of cattle interaction. In addition, we introduce a multi-stream architecture that accounts for relative information between interactive cattle to improve accuracy. In practice, we combine the graph feature with the image feature extracted from the interaction area to extract a visual representation of the interaction area, as well as the semantic priority obtained from our dataset to capture our prior knowledge of the relationship between the action and interaction of cattle. Qualitative and quantitative evaluation evidences the performance of our framework as an effective method to recognize cattle interaction.
Heterogeneous-graph Attention Reinforcement Learning for Football Matches

Football player's decision-making problem is quite challenging because of the essential feature of football matches: many players with different and complex cooperative or competitive relationships. To better leverage these relationships, this paper proposes a player-policy learning method with heterogeneous-graph attention reinforcement learning (PPL-HGARL) to enable an active player closest to the ball to learn effective policies for playing football matches. Specifically, a multi-head feature representation module is designed to reconstruct raw observations using prior expert knowledge. Furthermore, a heterogeneous-graph player-relation attention network is designed by use of graph among different roles, in order to model cooperative and competitive relations among players. The network learns proper state representation for the active player, making the player pay attention to other important players. Besides, an actor-critic algorithm is adopted to train the policies efficiently. Competing with rule-based opponents of different difficulty levels in the Google Research Football environment, the active player achieves excellent results, which validates the effectiveness and superiority of the proposed method.
A Multi-correspondence Object Detection Algorithm Based on Keypoints

Keypoints-based object detection algorithms such as CenterNet and CornerNet have been proved to be very efficient. They use inherent pixels on the feature map to predict whether a pixel is a keypoint of an object, thereby eliminating the need for anchor calculation. However, in cases where the keypoints of multiple objects overlap with each other, Keypoints-based object detection algorithms cannot effectively distinguish them. The reason for the problem is that there is a one-to-one relationship in the network, that is, one location only corresponds to one prediction box. In this paper, we mainly address the problem caused by overlapping center. We achieve our purpose by a kind of multi-correspondence relationship in the network, one location corresponding to multiple prediction boxes. In order to achieve this goal, our work mainly consists of the following two parts. First, we improved the keypoint pooling method. The existing keypoint pooling methods focus on emphasizing the keypoints, but we believe that keypoints detection should suppress the surrounding information while emphasizing the true keypoints. Second, in order to make the corner better match to the correct center, we adopt Pointing Offset to match them. Specifically, the corner will match center by its offset, in order to predict Pointing Offset better, we design a module named corner-to-center (CTC) to improve the matching effect.
Unsupervised Product Title Optimization Based on Search Behavior Knowledge in E-commerce

Nowadays, E-commercial platforms such as Taobao and Amazon, provide massive products which greatly improve the convenience of online shopping. However, there have been many redundant and lengthy titles, which are misleading and hurt the experience of consumers. Under this background, the product title compression task has been proposed to generate readable short titles. The existing supervised models rely on paired corpora, which is often expensive to annotate. Whereas unsupervised methods are not adopted in the E-commercial scenario due to lack of domain knowledge. To overcome the above limitations, we propose an unsupervised title compression method based on search behavior graph. Specifically, we build the knowledge graph extracted from search queries to detect redundancy. Moreover, click behaviors are utilized to give a clue to recognize inaccurate information. Then we employ a novel graph-to-text model consisting of an edge-importance encoder and a length-countdown decoder. The encoder combines the title with both query knowledge and click behaviors, and the decoder generates the compressed title with the desired length. Experiments on datasets show that our model outperforms the state-of-the-art models on both automatic and human evaluations. Moreover, further analysis demonstrates that our method has the ability to detect redundancy and keep the appropriate information.
Reinforcement Learning for Reducing the Interruptions and Increasing Fault Tolerance in the Cloud Environment

Cloud computing delivers robust computational services by processing tasks on its virtual machines (VMs) using resource-scheduling algorithms. The cloud’s existing algorithms provide limited results due to inappropriate resource scheduling. Additionally, these algorithms cannot process tasks generating faults while being computed. The primary reason for this is that these existing algorithms need an intelligence mechanism to enhance their abilities. To provide an intelligence mechanism to improve the resource-scheduling process and provision the fault-tolerance mechanism, an algorithm named reinforcement learning-shortest job first (RL-SJF) has been implemented by integrating the RL technique with the existing SJF algorithm. An experiment was conducted in a simulation platform to compare the working of RL-SJF with SJF, and challenging tasks were computed in multiple scenarios. The experimental results convey that the RL-SJF algorithm enhances the resource-scheduling process by improving the aggregate cost by 14.88% compared to the SJF algorithm. Additionally, the RL-SJF algorithm provided a fault-tolerance mechanism by computing 55.52% of the total tasks compared to 11.11% of the SJF algorithm. Thus, the RL-SJF algorithm improves the overall cloud performance and provides the ideal quality of service (QoS).
Enhancing Chinese Calligraphy Generation with Contour and Region-aware Attention

Chinese calligraphy generation is an important problem involved in many applications. Existing methods generally regard it as an image-to-image translation problem. However, existing models meet the challenge of poor generation performance for the Chinese calligraphy generation due to the lack of effective guided information. In this paper, we propose a novel model called CRA-GAN for the Chinese calligraphy generation through utilizing the contours of calligraphy characters as certain guided information and introducing a region-aware attention module to capture their content regions, motivated by the observation that the contour and content region provide certain delicate characteristics on the calligraphy style and content, respectively. Noticing that there is usually a large glyph difference between the source and target fonts, resulting in the degradation of model performance, we borrow the idea of adaptive pre-deformation from the literature to address this issue. A series of experiments are conducted to show the effectiveness of the suggested contour and region-aware attention, as well as the used adaptive pre-deformation operation. The outperformance of the proposed model over the state-of-the-art models is also demonstrated through extensive quantitative and qualitative comparisons over nine Chinese calligraphic font datasets.
CEntRE: A paragraph-level Chinese dataset for Relation Extraction among Enterprises

Enterprise relation extraction aims to detect pairs of enterprise entities and identify the business relations between them from unstructured or semi-structured text data, and it is crucial for several real-world applications such as risk analysis, rating research and supply chain security. However, previous work mainly focuses on getting attribute information about enterprises like personnel and corporate business, and pays little attention to enterprise relation extraction. To encourage further progress in the research, we introduce the CEntRE, a new dataset constructed from publicly available business news data with careful human annotation and intelligent data processing. Moreover, we propose a joint entity and relation extraction network, which is capable of discovering enterprise entities and extracting business relations between them accurately. The network firstly encodes input sequences with strong semantic augmentation to learn contextual representation for each token, then a conditional random field (CRF) module is used for entity extraction. Subsequently, entity pairs are built and a new encoder based on the entity pairs is applied to get global information for relation extraction. Finally, a biaffine classifier is deployed to classify the relations. Extensive experiments on CEntRE demonstrate the effectiveness of our proposed method compared with other six excellent models, and thus our model can be considered as one strong baseline. The data and code are available at: https://github.com/LiuPeiP-CStMining_Entity_Relations_Among_Enterprises