Mao-Zedong At SemEval-2023 Task 4: Label Represention Multi-Head Attention Model With Contrastive Learning-Enhanced Nearest Neighbor Mechanism For Multi-Label Text Classification
Year: 2023

Abstract:
The study of human values is essential in both practical and theoretical domains. With the development of computational linguistics, the creation of large-scale datasets has made it possible to automatically recognize human values accurately. SemEval 2023 Task 4(Kiesel et al., 2023) provides a set of arguments and 20 types of human values that are implicitly expressed in each argument. In this paper, we present our team’s solution. We use the Roberta(Liu et al.) model to obtain the word vector encoding of the document and propose a multi-head attention mechanism to establish connections between specific labels and semantic components. Furthermore, we use a contrastive learning-enhanced K-nearest neighbor mechanism(Su et al.) to leverage existing instance information for prediction. Our approach achieved an F1 score of 0.533 on the test set and ranked fourth on the leaderboard. we make our code publicly available at https://github.com/peterlau0626/semeval2023task4-HumanValue.

heading: 1 Introduction
text: The identification and analysis of human values in texts has been an important area of research. With the development of computational linguistics, this research has gained widespread attention because of its potential impact on areas such as sentiment analysis, social science.
One of the challenges in this area is to accurately categorize all the human value. Several notable research achievements have been made in the categorization of human values. One of the approach is that classifies human values into 54 categories across four different levels(Kiesel et al., 2022). SemEval2023 task4 uses the classification method in this paper, where an argument is given to identify whether a value is included in the instrument, and the F1 scores of the results at the level2 level are used for the total ranking. There are 20 cate-
gories of human values in level 2, and a argument could belong to multiple value categories or not to any one value category. this is a typical multilabel text classification (MLTC) problem which has been applied in many scenarios such as news emotion analysis(Bhowmick et al.) and web page tagging(Jain et al.).
In this paper, we propose a model that combines the label-specific attention network with the contrastive learning-enhanced nearest neighbor mechanism(Su et al.). The multi-headed attention mechanism allows our model to overcome the shortcomings of traditional attention mechanism models and to be able to focus on different parts of a document, resulting in more accurate labeled attention results. And the nearest neighbor mechanism enables our model to not waste the rich knowledge that can be directly obtained from the existing training instances and helps enhance the interpretability and robustness of the model.

heading: 2 Background

heading: 2.1 Datasets
text: The dataset comprises of arguments from six different domains such as news releases, online platforms, etc. originating from four different countries/regions, which are composed of 80% data from IBM argument quality dataset (95% from the original dataset), 15% from the European Future Conference (New), and 5% from group discussion ideas (2% from the original dataset). The training dataset comprises of more than 6500 arguments, whereas the validation and test datasets consist of around 1500 arguments each. In addition, the organizers of the competition provided three additional datasets to evaluate the robustness of methods:validation set Zhihu (labels available), test set Nahj al-Balagha (labels confidential), test set The New York Times (labels confidential). All datasets have been manually annotated.
426
Each sample in the dataset contains an argument ID, conclusion, stance towards the premise, and the premise itself. The labels consist of the argument ID and a column for each of the 20 value categories, indicating whether the sample belongs to each category (0 or 1).

heading: 2.2 Related Work
text: Before the widespread adoption of deep learning, models such as SVM were widely used to minimize an upper bound of the generalization error(Qin and Wang). Simple neural network (NN) models were later used for MLTC and achieved good performance(Nam et al.). Additionally, convolutional neural networks (CNNs) and recurrent networks with gated recurrent units (GRUs) have been successfully used with pre-trained word2vec embeddings(Berger). Feature selection has been shown to be effective in speeding up learning and improving performance by identifying representative words and removing unimportant ones(Spolaor and Tsoumakas).
In recent years, with the development of pretrained models, the ability to extract semantic information has become increasingly powerful. there have been several representative works that have focused on improving the MLTC models. For example, (Pal et al.) utilized graph neural networks based on label graphs to explicitly extract labelspecific semantic components from documents. seq2seq model can capture the correlation between tags(Yang et al.). LSAN(Xiao et al.) can focus on different tokens when predicting each label.

heading: 3 System Overview
text: In this section, we will present our model, which consists of two main parts. The first part is a multiheaded attention mechanism based on a specific label representation, while the second part is a nearest neighbor mechanism enhanced using contrast learning.
The MLTC problem can be described as follows: assuming a set of data D = {(xi, yi)}Ni=1, N labeled documents, where xi represents the text and yi ∈ {0, 1}l represents the label of xi, and l represents the total number of labels. Each document xi consists of a series of words. Our goal is to learn a classifier to establish a mapping from xi to yi, so that when a new document x is presented, its label vector y can be correctly predicted.As pretrained language models (PLMs) show remarkable performance in extracting natural language representations, we use PLMs as base encoder to get document and label feature. A input sample can be expressed as xi = {w1, w2, w3 . . . wn−1, wn}, wp ∈ Rd denotes the pth word vector of a document. After calculated by PLMs , the input matrix of the whole sentence is obtained as H ∈ Rn×d, where d is the hidden dimension of PLMs.

heading: 3.1 Label-specific multi-head attention network
text: In order to explicitly capture the corresponding label-related semantic components from each document, the approach of using label-guided attention mechanisms to learn label-specific text representations has been widely used in previous studies, and such a method is used in LSAN(Xiao et al.). In addition, the success of the Transformer model(Vaswani et al.) illustrates the ability of multi-headed attention mechanisms to extend the model’s ability to focus on different locations more effectively than single-headed attention mechanisms. The usefulness of this method for text classification is very intuitive.For example in the following sentence, "Social media is good for us. Although it may make some people rude, social media makes our lives easier." Focusing on the words “although”, “but” and “makes life easier” at the same time is a more accurate way of getting at the value of comfort in life, while ignoring the the disadvantages of social media.As mentioned above, we next show our model.
Firstly, to make use of the semantic information of labels, we initialize the trainable label representation matrix C ∈ Rl×d with the mean-pooling of the label features vector which is obtained by the pretrained encoder. Then, the multi-headed attention mechanism is used to compute the label-aware attention score. With the input document representation matrix as H ∈ Rn×d and the label representation matrix C, the query Q, key K, and value
V of the attention mechanism can be expressed as follows:
Q = WqC
K = WkH
V = WvH
(1)
Where WQ,WK ,WV ∈ Rd×d is the weight matrix to be learned.We use the h-head attention mechanism, then the three matrices Q,K,V can be expressed in the following form.
(Q1, Q2, . . . Qh) = Q
(K1,K2, . . .Kh) = K
(V1, V2, . . . Vh) = V
(2)
Where Qi ∈ Rl×da , Ki, Vi ∈ Rn×da correspond to the query, key and value of each attention header, and da = d/h denotes the dimensionality of a single attention mechanism representation space. Attention scores are then computed for each attention head similar to the method used in the Transformer model. Since the length of the document is different in a data batch, we perform a mask operation on the result of the QK matrix multiplication, set the value corresponding to the padding part to 1e−12, and then use the softmax activation function to activate it.
scorei = softmax
( mask ( QiKi da )) (3)
scorei ∈ R1×n denotes the attention score of the label for each word vector in the document. Then we obtain the attention results for each label with respect to the document content.
Attention =
Concat (attention1, . . . , attentionh)W O
where attentioni = scoreiVi (4)
Where Attention ∈ Rl×d can be considered as the representation vector of the document under the view of L labels. To obtain the representation vector of the document Z, the row vectors of the Attention matrix for each labeled view are averaged:
Z = mean(row(Attention)) (5)
After obtaining a comprehensive document representation with label-specific correlation, we can construct a multi-label text classifier by means of a perceptron consisting of fully connected layers. Mathematically, the predicted probability of each label of the next document can be determined by: ŷ = sigmod ( W 1ZT ) . Where W 1 ∈ Rl×dis trainable parameters for the fully connected and output layers, which can transfer the output value into a probability. Since multi-label classification has the problem of unbalanced positive and negative samples, in order to balance the coefficients of positive and negative samples in the loss function and obtain a better trained model, we use the cross-entropy loss function with weights as the loss function of the model:
LBCE = b∑
i=1
l∑
j=1
−(w · yij log (pij)
+ (1− yij) log (1− pij)) (6)
Where b is the size of a data batch, w is the weighting factor, yij is the true value of the jth label of the ith sample, pij is the probability that the model predicts that label to be yij, and l is the total number of labels. The positive sample size/negative sample size in the training set is taken as the value of w.

heading: 3.2 Contrastive Learning-Enhanced Nearest Neighbor Mechanism
text: We use the k nearest neighbor (KNN) mechanism enhanced by contrast learning(Su et al.). This approach innovatively proposes a KNN mechanism for multi-label text classification that can make good use of the information of existing instances. And a contrastive learning approach is designed to enhance this KNN mechanism mechanism effectively. Specifically, this approach designs a loss function for contrastive learning based on dynamic coefficients of label similarity, which compares the documents representation vectors at training time to let the vector with more same labels be more similar as possible, while the vectors of documents with fewer identical labels are as far away as possible. Assuming a data batch of size b, we define a function to output all other instances of a particular instance in this batch g(i) = {k | k ∈ {1, 2, . . . ,b}, k ̸= i}.The contrastive loss(CL loss) of each instance pair (i, j) can be calculated as:
Lijcon = −βij log e−d(zi,zj)/τ
′
∑ k∈g(i) e −d(zi,zk)/τ ′
Cij = y T i · yj , βij = Cij∑ k∈g(i)Cik
(7)
where d(·, ·) is the euclidean distance, τ ′ is the contrastive learning temperature and Z denotes the document representation. Cijdenotes the label similarity between i, j, and normalized to obtain βij. The CL loss of the whole batch can be expressed as:Lcon = ∑ i ∑ j∈g(i) L ij con. The crossentropy loss function is expressed as LBCE , then the whole Loss function is L = LBCE + γLcon , where a controls the ratio of the coefficients of the contrast learning loss function and the crossentropy loss function. Then, we construct a data store of training instances so that we can later use the existing instance information as a comparison. Based on the training set (xi, yi) ∈ D, the storage of a training set of document representation vectorsD′ = {( hi, yi )}Ni=1 is obtained by a trained model. where hidenotes the document representation vector of the training set, which is calculated by the model.
In the inference stage, give an input X , after the model calculation we obtain its document representation vector Z, and the prediction of the model ŷM ∈ {p | p ∈ [0, 1]}1. Next, we compare with the data repository to find the nearest k nearest
neighbors N = {(hi, yi)}ki=1, then the KNN prediction can be calculated as:
ŷKNN =
k∑
i=1
αiyi
αi = e−d(hi,Z)/τ∑ j e −d(hj,Z)/τ
(8)
where d(·, ·) is the euclidean distance, τ is the temperature of KNN, αi is the weight coefficient of the ith neighbor, when the closer the test instance vector representation is to this neighbor, the larger the weight will be. The final prediction form is expressed as follows:
ŷ = λŷKNN + (1− λ)ŷM (9) where λ is the weight coefficient that regulates the KNN prediction and the model prediction.

heading: 4 Experimental Setup
text: In the dataset, many pronouns in the premise were used such as "this research", "it", "this way" etc,
and these pronouns refer to objects contained in the conclusion. Whereas our model tries to establish the attention scores of different semantic components of a document for a specific label, it is clear that the presence of these words with unclear denotations affects the attention results. In addition, stance toward conclusions also influence value judgments. Therefore, we use a simple strategy: combining the three parts of conclusion, preference, and stance into a sentence that conforms to natural language conventions. Specifically, the data were preprocessed uniformly and simply, and the input structure was: "I agree (disagree) that" + conclusion content + ", because" + premise content.
We use the Roberta model(Liu et al.) as the base pre-trained model to obtain word representation. Experimenting with our architecture on the base model. And we use the K-fold cross-validation method. We merge the training and validation sets and then randomly divide them into six copies. We perform the training process six times, each time using 5/6 of the data as the training set and 1/6 of the data as the validation. During the training process, the best-trained model for each fold is saved, and the average output probability of all models is taken as the final prediction score.

heading: 5 Results
text: On the leaderboard of the TIRA, our method achieved an macro-F1 score of 0.53 and ranks fourth, while the baseline which use a bert model(Devlin et al.) achieved 0.42, with the best result for the whole competition being 0.56. In addition, our model also achieved an F1-score of 0.32 on each of the two test sets, Nahj al-Balagha and New York Times. This effect is relatively high among all participating teams, which fully demonstrates the robustness and stability of our approach.1
To illustrate the effectiveness of our architecture, we conducted ablation experiments. The ablation experiments evaluate the performance effect of the model directly on the validation set merged with the dataset from Zhihu. In the ablation experiments, we did not use the strategy of k-fold cross-validation. The results of the ablation experiment are shown in Table 1, which shows all strategy results with Precision, Recall, and marco-F1. We show the results for each method using the average results from three runs. At first, we use the word vector correspond-
1Find it from Appendix
ing to the output of the Roberta model [CLS] as the representation vector of the document to connect the classifier as the baseline. We then compared the effect of baseline with LSAN(Xiao et al.), just use multi-attention mechanism, and the effect of removing the multi-headed attention mechanism part. As can be seen in the table, after using the multiheaded attention mechanism, the marco-F1 value improves by about 0.3% compared to the baseline model, while the LSAN mechanism get 0.2% improvement on F1-score. And after adding the KNN mechanism augmented with contrastive learning alone, the marco-F1-score is improved by about 0.4%. In the case of the full model, the marcoF1-score improves by about 0.7% compared to the baseline. This result is within our expectation and illustrates the effectiveness of our method. Then in order to increase the stability and robustness, and to avoid overfitting generation, we use the K-fold cross-validation method, so that our experimental results can be shown relatively stable, which leads to an improvement of about 0.4 percentage points in the F1-scores.

heading: 6 Conclusion
text: We propose a multi-label text classification model using a label-specific multi-headed attention mechanism. Compared to previous models of attention mechanisms, the use of multi-headed attention enables specific labels to focus on different semantic components of the document more effectively. Besides, we use the KNN mechanism to exploit the instance information in the training set. We then perform ablation experiments on our architecture to analyze the role of each part and demonstrate the superiority of using a multi-headed attention mechanism.

heading: A Appendix

heading: Future Work