Reviewerly/exHarmony huggingface dataset:
Automated Keyphrase Generation for Brazilian Legal Information Retrieval

Due to the high volume of cases in transit in the Brazilian judicial system, it is of great interest to automate processes that help legal professionals. With the recent advances in text generation methods, there is potential to apply these techniques in tasks that require automation. In this article, we address the problem of automating the writing of keyphrases, a sequence of key terms present in documents used in courts throughout Brazil. For this, we are proposing the use of a text-to-text framework based on generative Transformers. We evaluated several generative Transformer models (such as PTT5, mT5, OPT, and BLOOM) for the proposal, and a comparison of their performance is presented. The best-performing model, PTT5, was adopted as the text generator since it achieved a BLEU score of 37.54% in the test set, outperforming other evaluated methods by up to 24.6%. Finally, to assess the influence of keyphrases and the quality of the generated ones, we performed several experiments based on a real case of legal information retrieval. By using traditional information retrieval methods, such as TF-IDF and BM25, in combination with the original, generated keyphrases, or both, we observed gains statistically significant (p-value <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${&lt;}$</tex> 0.05) in all experiments.
A Determinantal Point Process Based Novel Sampling Method of Abstractive Text Summarization

In recent years abstractive text summarization (ATS) research has made considerable progress attributed to two key improvements, deep neural modeling and likelihood estimation based sampling, in the end-to-end optimization training. While modeling has grounded on a few de facto highly capable base models within encoder-decoder architecture, novel sampling ideas, such as random masking classification and generative prediction by unsupervised learning, have also been explored. They aim at improving prior knowledge, particularly of language modeling for downstream tasks. It has led to the notable performance gain of ATS. But several challenges remain, for example, undesirable word repeats. In this paper, we propose a determinantal point process (DPP) based novel sampling method to address the issue. It can be easily integrated with the existing ATS models. Our experiments and subsequent analysis have revealed that the adopted models trained by our sampling method reduce undesirable word repeats and improve word coverage while achieving competitive ROUGE scores.
Self-calibrated U-Net for Document Segmentation

Based on the need to digitalize identification documents per several institutions and companies, the segmentation task of textual information acquired importance. Commonly, Convolutional Neural Networks are applied to solve such problems. Vanilla convolution in deep learning models can provide sub-optimal performance in some learning tasks, such as semantic segmentation. Due to this, more specific proposals may be applied to better fit this context, such as self-calibrated convolutions, which consider unique characteristics of different feature maps. Considering this context, we propose a new fully convolutional network architecture based on U-Net for segmentation associated with self-calibrated convolution. We consider changing all vanilla convolution layers in this new neural network by self-calibrating convolutions. In this work, we evaluate our proposal in three different tasks of document segmentation. All experiments demonstrate an increase in the proposed neural network's segmentation performance compared with traditional U-Net in the analyzed context, but the inference time performance decreases.
Clustering joint Locality Preserving Projections

When clustering high-dimensional data, dimensionality reduction techniques are usually employed to project the data into a low-dimensional subspace. However, the existing graph-based dimensionality reduction algorithms only consider the weights among samples and don't consider the cluster structure of the raw data, which leads to the weakening or even disappearance of the cluster structure after dimensionality reduction. On the one hand, we hope to obtain well-formed clusters in the projected data, on the other hand, the local structure of data is also maintained. Therefore, we integrate the dimensionality reduction model and clustering model into one model, especially, the proposed model also considers the relationship between the cluster information in the original space and the cluster information in the subspace. The proposed model learns the optimal projection matrix while preserving the local information. Because the clustering model is embedded, the projected data can obtain well-formed clusters. The proposed model is solved by an iterative algorithm and the convergence is proved. We conducted a lot of experiments on six high-dimensional image data sets and the experimental results show the effectiveness of our algorithm.
A Clustering Algorithm for Tunnel Boring Machine Data Based on Ridge Regression and Fuzzy C-Means

Abstract A fuzzy clustering data partitioning method based on ridge regression is proposed to address the high correlation between the data attributes of the tunnel boring machine. The method utilizes the fuzzy partition matrix as the weight for ridge regression analysis and then employs the resulting regression equation as the clustering model to iteratively approach the target minimum value. This process enables data clustering based on the relevant characteristics of the data attributes. Experimental results using functional data sets demonstrate that this method achieves higher accuracy in clustering functional correlation data compared to traditional methods. Additionally, the proposed algorithm is evaluated using measured data from a bid section of a city subway. Results indicate that the algorithm effectively addresses the problem of data classification and key performance index prediction, with a misclassification rate and prediction error of 23.8% and 2.11%, respectively. These findings meet the engineering requirements and provide support for the subsequent data analysis of the tunnel boring machine.
Image Alone Are Not Enough: A General Semantic-Augmented Transformer-Based Framework for Image Captioning

Image captioning has long been widely regarded as a modal transformation task from visual to linguistic modality. Most current research focuses on the information transformation between single modalities dominated by visual features, while less attention is paid to the interaction between visual features and linguistic features. This rigid single-modal conversion method is prone to information confusion and loss during the conversion process, making it difficult for the model to generate accurate and detailed captions. In this paper, we propose a general Semantic-Augmented Transformer-Based (SAT) framework to facilitate smoother transformation between the two modalities. In the encoding stage, we use the fine-grained description of each region to fuse with the corresponding image features to make the image feature representation closer to the text feature representation. In the decoding stage, the caption's part-of-speech information is used as prior knowledge to constrain the model to pay more attention to the details in the image rather than only to the prominent entities for fine-grained captions. We extensively evaluate our framework on various state-of-the-art transformer-based models. Experiments demonstrate that these models have superior performance on the MS-COCO dataset under our framework.
Time-series Anonymization of Tabular Health Data using Generative Adversarial Network

Data anonymization has been used as a fundamental tool in various domains, e.g. healthcare, to alter personal data such that individuals can no longer be identified directly or indirectly in a way to enable broader sharing of data. For example, data perturbation techniques add noise to original data allowing individual record confidentiality while maintaining high-quality data for analytical purposes. In this paper, we propose a perturbation technique for anonymizing longitudinal tabular data such as electronic health records (EHRs). Our model starts by learning a latent space of original data to better capture temporal trends, then employs a generative adversarial network together to train a perturbation generator. During model training, a time-supervised loss function for handling sequence-dependent noise, together with the adversarial unsupervised, anonymization, and reconstruction loss functions are utilized. To evaluate our model quantitatively, we use multiple evaluation metrics for the fidelity, utility, and identifiability of generated data, in addition, the model is evaluated qualitatively by visualizing generated and original data. The results confirm that our model preserves the privacy of the original data and generates a perturbed version with high fidelity and utility compared to some state-of-the-art techniques.
Retrieval-Based Natural 3D Human Motion Generation

It is challenging to generate 3D human motions automatically from the text. In an ideal scenario, the generated motions should explore the text-grounded motion space while accurately depicting the content in the prescribed text descriptions. Text2length and text2motion training have been used to address this problem in previous research. There is, however, a lack of knowledge about the relationship between motion length and text. In this work, context-aware retrieval-based approaches are proposed for predicting motion lengths and generating proper 3D motions (C-MO). Specifically, we train a context-aware encoder-decoder model that leverages the previous output of the decoder or the embedding vector of a ground truth motion as the context so that the model becomes increasingly aware of previous alignments. Retrieving the most similar motions from the training set is based on the trained model given a text. Finally, we use the retrieval motion to guide the probability distribution for the final generated motions. Our method combines the advantages of both information retrieval and neural machine translation. C-MO is evaluated on a large-scale dataset, KIT, and its experimental results show that it achieves great improvements over the state-of-the-art.
FA-Net: More Accurate Encrypted Network Traffic Classification Based on Burst with Self-Attention

Encrypted network traffic classification (ENTC) is crucial in fields including network cyberspace security, network administration and service quality. Combining the machine learning algorithms with manual-designed burst features has been studied extensively in the ENTC community. However, these features depend on professional experience heavily, which needs lots of human effort. These hand-crafted features are task-oriented and incomplete in various complex tasks. What's more, they are also affected by the potential network jitters. In this paper, we propose a novel encrypted traffic classification method FA-Net to mine burst features. We adopt two hierarchical multi-head self-attention encoders to enumerate all potential intra-burst features and inter-burst dependencies completely, and select the optimal associations automatically. For more robust against network jitter, we design an additional burst positional encoding to loose the model's sensitivity about out-of-order packets within bursts. We evaluate the FA-Net on multiple datasets, including website and mobile application classification tasks. The results show the FA-Net model outperforms other state-of-the-art methods in all the datasets, even gains more than 5% absolute improvement in accuracy. Additionally, the quantitative measurements about burst feature similarity show that the burst features learned by FA-Net exhibits more intraclass similarity and more inter-class separation.
Discriminating Information of Modality Contributions Network by Gating Mechanism and Multi-Task Learning

Multimodal sentiment analysis is a hot topic in the field of multimodal research, which uses the multiple modal forms contained in a video (including speech, video, audio, etc.) to further obtain information such as textual information, facial expressions, and vocal intonation for sentiment understanding. The output obtained is analyzed for multimodal emotion, mainly through the complex fusion of different modal information. However, the inconsistent contribution of different modal information in multimodal networks and the undifferentiated fusion between modalities causes the fused multimodal information to contain a lot of noisy information. How to reduce the noise information between different modalities so that the multimodal contains rich and accurate messages is a challenging research task. In this paper, we propose gating mechanisms applied to within-modal and multitasking processes respectively to reduce the noise information. The intra-modal gating mechanism is a fusion control by discriminating the similarity between modalities after projecting features into the subspace. The multi-task gating mechanism is a loss calculation by discriminating the degree of difference between unimodal and multimodal labels. The performance of the downstream MSA task is improved by reducing noise information in many ways. Our gating mechanism can also be used as a framework to help improve the performance of other baseline models. Our experiments on the commonly used multimodal datasets CMU-MOSI, and CMU-MOSEI show that we achieve better performance than other good baseline models. We also consider the arrangement of the gating mechanism on the baseline model and demonstrate its performance improvement. In addition, we verify the effectiveness of the different modules through ablation experiments.
Biased or Debiased: Polarization-aware Embedding Learning from Social Media Knowledge Graph

Nowadays social media has become a major information source for understanding various aspects of social activities, such as political polarization and shopping preferences. To use social media data with machine learning algorithms, the data can be transformed into graphs, from which graph embeddings can be learned. However, it has been shown that embeddings usually inherit the social bias prevalent in the data. Sometimes such bias is necessary, and sometimes they are undesirable. Previous work that deals with embedding bias usually tries to neutralize embeddings with regard to a bias measure. In this paper, we propose an embedding bias model and a control method, which can reduce or enlarge the bias by changing the value of one parameter. In addition to explicit bias scores, we also test our method in two downstream tasks, namely, party support prediction and tweet recommendation. Through experimental evaluations, we show that when we reduce the embedding bias, the task irrelevant to the sensitive attribute can be improved. On the other hand, when we enlarge the bias, the task relevant to the sensitive attribute can be improved.
AdaptiveTune: Adaptive Value With Bernoulli Noise Can Help Us Finetune Better

Finetuning allows pretrained models (PLMs) to be adapted to various downstream tasks, thus saving significant computational resources and time. However, on the one hand, PLMs may overfit the pretrained tasks and be less applicable to downstream tasks. On the other hand, they maybe overly dependent on limited labeled data and need better generalization ability. An important research direction is how to effectively finetune the PLMs to overcome these problems. In this paper, we propose an efficient finetuning method called “Adaptive Tune” to prevent the model from overfitting the pretraining and downstream tasks by adding some adaptive noise according to the noise addition strategy in different finetuning stages. Specifically, we extract representative adaptive values as noise based on the data distribution characteristics of different parameter matrices. We inject different noises into different parameter matrices by matrix perturbation methods and specific noise addition strategies. Extensive experiments with multiple models on GLUE general-purpose language test benchmarks and emotion sentiment datasets demonstrate that AdaptiveTune can effectively finetune PLMs and improve performance on different tasks.
PGLR: Pseudo Graph and Label Reuse for Entity Relation Extraction

Named entity recognition and relation extraction are essential for structured information construction. However, existing methods, particularly in the field of relation prediction, struggled with information redundancy. In addition, conventional classification models only focus on predicting labels and comparing them to the ground truth, without considering the values and semantics of the labels. In this paper, we propose a novel end-to-end joint entity recognition and relation extraction model named PGLR (Pseudo Graph and Label Reuse for Entity Relation Extraction), which mainly decomposes the task into three hierarchical components to cope with the above issues. Firstly, the model generates a fully connected pseudo-graph that connects all possible entities and relations, followed by a confidence-based module to remove nodes and edges associated with non-relational facts. Secondly, a label reuse approach is employed to aggregate predicted labels and external knowledge, and the representations are reconstructed using an optimal re-parameterization technique. Thirdly, a gating mechanism is utilized to gauge the confidence of the reconstructed representations. The experimental results from three benchmark datasets (ACE04, ACE05, and SciERC) indicate that the proposed model achieves greater accuracy, faster inference speed, and requires fewer parameters.
Semi-Supervised Generalized Source-Free Domain Adaptation (SSG-SFDA)

Continual learning aims to learn on a sequence of new tasks while maintaining the performance on previous tasks. Source-free domain adaptation (SFDA), which adapts a pretrained source model to a target domain, is useful in protecting the source domain data privacy. Generalized SFDA (G-SFDA) combines continual learning and SFDA to achieve outstanding performance on both the source and the target domains. This paper proposes semi-supervised G-SFDA (SSG-SFDA) for domain incremental learning, where a pre-trained source model (instead of the source data), few labeled target data, and plenty of unlabeled target data, are available. The goal is to achieve good performance on all domains. To cope with domain-ID agnostic, SSG-SFDA trains a conditional variational auto-encoder (CVAE) for each domain to learn its feature distribution, and a domain discriminator using virtual shallow features generated by CVAE to estimate the domain ID. To cope with catastrophic forgetting, SSG-SFDA uses soft domain attention to improve the sparse domain attention in G-SFDA. To cope with insufficient labeled target data, SSG-SFDA uses MixMatch to augment the unlabeled target data and better exploit the few labeled target data. Experiments on three datasets demonstrated the effectiveness of SSG-SFDA.
CA-SSD: Channel-Independent Rare Class Attention for 3D Object Detection

The current 3D object detection datasets suffer from a serve problem of uneven data distribution. In autonomous driving scenarios, rare classes are also critical. The existing methods perform poorly in these classes. Generic methods of solving long-tailed distributions such as re-sampling and re-weighting would over-fit or destroy the original distribution. The popular solution is to design detectors for different classes, which improves detection accuracy but reduces generalizability. Motived by this, we propose a simple and effective single-stage detector named CA-SSD, which can detect multiple categories simultaneously. The key to our approach is the channel transformation of the features, which corrects for the bias in task distribution from the training phase to the testing phase. In addition, to reduce the information loss due to multiple downsampling, a self-attention module is introduced to retain richer contextual information. Experimental results on the KITTI benchmark demonstrate the superiority of CA-SSD, especially on small samples. Besides, we designed ablation experiments to verify the effectiveness of both modules.
Offline Reinforcement Learning with Uncertainty Critic Regularization Based on Density Estimation

By utilizing previously offline data, offline reinforcement learning (offline RL) can develop effective policies for the environment with complex online interaction. However, due to the incomplete coverage of offline datasets, the estimation errors of Q-functions caused by out-of-distribution (OOD) actions may cause current off-policy methods to fail. Offline RL uses policy constraints, value function regularization, or uncertainty estimation to drive the learned policy from resembling behavioral policy. Unfortunately, the policy constraints approach restricts the learned policy to a region near the sub-optimal behavioral policy. In addition, the value function regularization approach does not accurately assess OOD actions, which can cause it to be too conservative in estimating the Q-value of actions within the proximity distribution. Finally, the uncertainty estimation is biased due to the complex environment or inaccurate valuation early in training. We suggest Density-UCR as a solution to the aforementioned issues. Density-UCR makes the Q-function estimate have a lower-confidence bound (LCB) and penalizes the OOD actions by using the estimation error of the ensemble Q-functions as a penalty value. Additionally, Density-UCR models the offline data's distribution using a density estimator to derive more accurate uncertainty weights for the penalty value. Density-UCR employs uncertainty estimates as the weight of the priority replay buffer to increase the stability of online fine-tuning and prevent performance degradation caused by the distribution shift of offline samples over online samples. Our experiments on the D4RL benchmark show that Density-UCR significantly outperforms the policy constraints approach with the value function regularization approach. Furthermore, Density-UCR also offers excellent fine-tuning performance.
GIT: Detecting Uncertainty, Out-Of-Distribution and Adversarial Samples using Gradients and Invariance Transformations

Deep neural networks tend to make overconfident predictions and often require additional detectors for misclassifi-cations, particularly for safety-critical applications. Existing detection methods usually only focus on adversarial attacks or out-of-distribution samples as reasons for false predictions. However, generalization errors occur due to diverse reasons often related to poorly learning relevant invariances. We therefore propose GIT, a holistic approach for the detection of generalization errors that combines the usage of gradient information and invariance transformations. The invariance transformations are designed to shift misclassified samples back into the generalization area of the neural network, while the gradient information measures the contradiction between the initial prediction and the corre-sponding inherent computations of the neural network using the transformed sample. Our experiments demonstrate the superior performance of GIT compared to the state-of-the-art on a variety of network architectures, problem setups and perturbation types.