# Augmenting Feature Representation with Gradient Penalty for Robust Text Categorization

Authors: Depei Wang; Lianglun Cheng; Zhuowei Wang
Journal: International Journal of Intelligent Systems
 (2023)

---

## Abstract

The capabilities of deep models are constantly mined for extraction and representation of features among text classification tasks. However, these models are sensitive to changes in input data, resulting in poor robustness. Meanwhile, the model lacks information interaction and weak representation ability. In this work, for feature extraction, a joint model that consists of a convolutional neural network, a bidirectional gated recurrent unit, and an attention mechanism is proposed. This new model can improve versatility and fully discover category information in text. For feature representation, a projector under the supervised contrastive learning method is introduced. The method can improve the representation of an encoder and realize aggregation of the same category. Considering the robustness of the PCRA, the gradient penalty is added to a contrastive loss function. Experiments are performed on four datasets to assess the proposed model (PCRA and PCRA-GP) using an accuracy metric. The experimental results show that our model is suitable for variable-length and bilingual texts. Compared with the baseline model, it remains competitive, and it reaches SOTA on the 20 Newsgroups dataset. Moreover, the performance of the model is evaluated under different hyperparameters to clarify its working mechanism.


---

## Body

## 1. Introduction

Deep learning is constantly expanding its application areas, such as convolutional and recurrent neural networks and their increasing use in image processing [1], natural language processing (NLP) [2], and many other scenarios [3], accompanied by the steady increase in computing power and data volume which are able to be processed. In text category tasks, deep learning models can automatically obtain the features of input samples and perform self-learning feature representation. However, the extracted features from training data are solidified, causing the fragility of representation. Due to the weak generalization ability of representation, models are vulnerable to the attack of the input side, resulting in wrong decisions. Therefore, there are great needs for studying model robustness in text classification tasks at present.Initially, adversarial examples appeared in image-processing tasks. Szegedy et al. [4] added small disturbances to samples to propose the concept of adversarial samples for the first time. Inspired by this, many data augmentation methods are used to improve the performance of deep learning models, and common methods entail elastic distortion, scaling, translation, and rotation in image classification. In the NLP domain, which is analogous to pixel-level disturbances in images, perturbation is added to the input by manipulating words in the sentence, such as adding or subtracting characters and swapping positions. Specifically for classification tasks, it not only requires enhanced features but also high-quality representation capabilities of the input.Representation learning (RL) encodes a high-dimensional input into a much lower dimensional space, which captures high-level and useful concepts, and the mapping process involves the conversion of raw input data into a feature vector or tensor. Fortunately, contrastive learning (CL) is an easy way to meet these constraints [5]. It uses a self-supervised paradigm to compare and learn the distribution of positive samples and negative samples so that similar species are closer and heterogeneous species are further apart in the mapping space. Besides, the selection of the projection head can be linear or nonlinear after encoding. Inspired by triplet, max-margin, and N-pairs loss, supervised contrastive learning (SCL) introduces CL to the fully supervised method. Researchers have never stopped in their quest to model expressiveness. Therefore, the feature representation ability is still worthy of attention.In this paper, a multichannel joint architecture, PCRA-GP, is established based on SCL with the gradient penalty. It aims to address the model’s robustness through adversarial training of feature representation. First, the multichannel convolutional neural network (CNN, C) that enhances the adaptability to sentence length, the bidirectional gated recurrent unit (BiGRU, R) that extracts contextual information, and the attention mechanism (A) that calculates the contribution of different features are integrated. Second, the projector (P) of contrastive learning is pretrained to improve the representation ability of features with a contrastive loss function. Finally, the gradient penalty (GP) is used to generate perturbation at the feature level, and the robust representation of text features produced by using the PCRA encoder is realized. The main contributions of our work are summarized as follows:(i)
To realize feature extraction, a joint deep learning model that consists of the CNN, BiGRU and attention mechanism is established, which are used to adapt to sentence length, capture contextual information, and filter features. The joint model can efficiently improve versatility and fully detect category information in text.(ii)
For feature representation, contrastive learning and a projector are introduced under the supervised method. The method can improve the representation of an encoder and realize aggregation of data that belong to the same category.(iii)
For model robustness, the gradient penalty is added to the contrastive loss function, which achieves robust training.The remainder of this paper is organized as follows: Section2 reviews contrastive learning, generation of adversarial samples, and deep learning models for text classification. Section 3 describes the proposed PCRA-GP architecture. Section 4 introduces the experimental setup in detail, conducts comparison experiments, and analyzes the experimental results. Section 5 summarizes the research.


## 2. Related Work

### 2.1. Contrastive Learning

Recently, CL has been widely applied to self-supervised RL for CV [6, 7], NLP audio [8], graph [9], multimodal methods, and other domains. CL is a self-supervised learning method inspired by noise contrastive estimation or N-pair losses. It learns representations through comparison between different input samples, closing similar inputs and repelling dissimilar inputs. Le-Kha et al. [5] declared that CL methods provide a simple yet powerful approach to learning representations in a discriminative manner in both supervised and self-supervised setups. PIRL [7] verified this conclusion by learning invariant representations based on pretext task solving jigsaw puzzles and found that the resulting invariant representations perform better than their covariant counterparts across a range of vision tasks in self-supervised and supervised manners.There remains much research exploring the potential of CL. To maintain the current negative candidate pool and adopt momentum update strategy passing parameters between positive and negative samples, He et al. proposed the MoCo [6] method, a mechanism for building dynamic dictionaries for CL. The SimCLR [10] framework introduced learnable nonlinear transformations and two data augmentation methods to train an encoder and compared the effects of data augmentation, batch sizes, and training steps on the training results. Common cropping and rotation transformations are no longer applicable to serialized data such as texts and audio. A universal unsupervised learning approach CPC [8] architecture that extracts features from high-dimensional data and uses autoregressive models to predict the future in latent space was proposed.Beyond self-supervised classification, SCL [1] is an extension of traditional CL. It pulls the same class closer using SupCon loss by leveraging label information. SsCL [11] combined the CL loss branch with the cross-entropy loss branch in semisupervised learning and introduced a cocalibration mechanism to interchange predictions between the two branches. RoCL [12] trained a robust neural network without label information by using a contrastive self-supervised learning framework. Also, the CosG [9] model is a graph-based CL method for fact verification including label-supervised and unsupervised graph-contract. The former helps the model learn discriminative representation for items of various classes, and the latter trains a graph convolutional encoder for reducing the loss of sole node features in graph propagation.


### 2.2. Adversarial Example Generation

In NLP systems, studies have shown that the model is highly dependent on the input data and have used this discovery to conduct robust training of the model through adversarial samples. Belinkov and Bisk [13] found that even with spell checking, existing models are still sensitive to the spelling order of words in machine translation tasks. For character-level attacks, perturbation can be approximated by the number of character edits. For word-level attacks, perturbation can be achieved by substitution. EDA [14], an easy data augmentation method, consists of four simple operations: synonym replacement, random insertion, random swap, and random deletion. Similarly, there are some methods based on word semantic similarity. Kobayashi proposed a contextual augmentation [15] strategy, which uses a bidirectional language model to generate a diverse alternative vocabulary according to the context. Exploring the broadening of the scope of application, round-trip translation constructs data through translation, which is suitable for words, phrases, sentences, and texts.Recent adversarial attack example generation algorithms can be generally categorized into two groups: white-box or black-box methods. In the white-box setting, an adversary has access to the model, model parameters, and feature set of inputs. For perturbation at the embedding level, TextTricker [16] calculated the gradient magnitude of each input unit and added the scores of each dimension in the embedding space as the word-level importance score when identifying key words from input. Perturbating at the grade level, HotFlip [17] relies on an atomic flip operation to generate white-box adversarial examples that trick character-level and word-level neural models. It swaps one token for another based on the gradients of one-hot input vectors. Papernot et al. proposed FGSM [18], a fast gradient sign method widely applied in the image domain [19], which could be solved by linearizing the cost function of the model around its input and selecting perturbation using the gradient of the cost function with respect to the input itself. GBDA [20], a gradient-based distributional text attack against a transformer framework, defines a parameterized distribution of adversarial examples and uses Gumbel-softmax approximation to derive a smooth estimate of the gradient. In the black-box setting, an adversary is only allowed to query the target classifier and does not know the details of learned models or the feature representations of inputs. In [21], AEG was proposed, a reinforcement-learning-based approach, to generate adversarial examples in black-box settings.Intuitively, the generative adversarial network (GAN) generator captured the data distribution and made the discriminant equal to ½ everywhere in the space of its arbitrary function. After continuous research on GAN, WGAN [22] was developed by using the directed Wasserstein distance to produce a value function to solve the problem regarding original GAN training instability. Facing weight flipping caused by poor samples in WGAN, Gulrajani proposed a gradient penalty (WGAN-GP) algorithm [23] which can penalize the norm of the gradient of the critic with respect to its input. Moreover, in adversarial examples producing high quality and efficiency, Xiao produced AdvGAN [24] to generate adversarial examples with GANs in both semiwhite-box and black-box attack settings.


### 2.3. Deep Learning Models for Text Classification

The main basic components of the deep learning model in the text classification task are the recurrent neural network (RNN) [25], CNN [26, 27], attention mechanism [28, 29], and graph neural network (GNN) [30]. These basic components have tended to develop towards a fusion model. In the joint model, the advantages of local information, context information, and feature screening are combined. Huang and Liu proposed GCNN [31] and used CNN to compensate for the shortcoming of gated recurrent units (GRUs) in the extraction of long sentence information. Yang and Tang [32] tested the effect of the attention mechanism on different positions of the CNN-based model and performed experimental verification in ATCNN-1, ATCNN-2, and ATCNN-3 which superimposes formers combining feature weighting and feature selecting. To strengthen the feature extraction ability, Wenzhen et al. proposed the C-BiGRU-ATT [33] method to reduce the impact of text representation on the classification results and increase the features of the text at the vocabulary level and character level to improve classification accuracy. Also, in the face of semantic ambiguity, Liu and Guo proposed AC-BiLSTM [34], which is oriented to high-dimensional, discrete, and complex semantic phenomena for use on text classification tasks. This entailed the use of bidirectional long short-term memory (BiLSTM), an attention mechanism, and a convolutional layer.The success of TextGCN [30] turned the serialized text model to the graph-structured text model. Wu et al. proposed SGC [35] to accelerate GCN by removing nonlinearities and collapsing weight matrices between consecutive layers. Zhu and Koniusz proposed SSGC to improve the performance of GCN and capture the global and local context of each node using simple spectral graph convolution. With the enrichment of the corpus and the development of increased computing power, a series of transformer models represented by Bert emerged. Lin et al. proposed that BertGCN [36] adopt transductive learning, learning representations for both training data and unlabeled test data, by propagating label influence through graph convolution. Zhu and Koniusz proposed SSGC [37] to solve transition smoothing and reduce computing and storage costs.Inspired by large-scale pretraining models [38], researchers have begun to use external information [39, 40] to improve the classification accuracy of the model. Pan et al. [41] proposed SWEMs to transfer source domain knowledge to unseen text sequences. Abid et al. proposed BiGRU-CNN [28] based on pretrained GloVe embedding which retains domain knowledge. There are also models that do not rely on external knowledge. In the sparse classification of short text data, Liu et al. [42] proposed a bilevel attention model that does not rely on external knowledge to capture word-level information which is presented to explore the topic-word association and sequence-level information which is used to extract the relationship between local and global sentiment expression.So far, we have discussed the development of CL, adversarial sample generation strategies, and deep models used in text classification tasks. Obviously, there is still potential for improvement in CL’s robust representation. The adversarial sample generation technique based on the entire feature embedding is not addressed. The majority of deep models are focused on enhancing feature acquisition while ignoring model stability. To make the best of the text feature and robust representation content, a new neural network significantly improving the model classification robustness is proposed in this paper. The proposed model can efficiently represent enhanced features and learn the stable representation by input attacks.


## 2.1. Contrastive Learning

Recently, CL has been widely applied to self-supervised RL for CV [6, 7], NLP audio [8], graph [9], multimodal methods, and other domains. CL is a self-supervised learning method inspired by noise contrastive estimation or N-pair losses. It learns representations through comparison between different input samples, closing similar inputs and repelling dissimilar inputs. Le-Kha et al. [5] declared that CL methods provide a simple yet powerful approach to learning representations in a discriminative manner in both supervised and self-supervised setups. PIRL [7] verified this conclusion by learning invariant representations based on pretext task solving jigsaw puzzles and found that the resulting invariant representations perform better than their covariant counterparts across a range of vision tasks in self-supervised and supervised manners.There remains much research exploring the potential of CL. To maintain the current negative candidate pool and adopt momentum update strategy passing parameters between positive and negative samples, He et al. proposed the MoCo [6] method, a mechanism for building dynamic dictionaries for CL. The SimCLR [10] framework introduced learnable nonlinear transformations and two data augmentation methods to train an encoder and compared the effects of data augmentation, batch sizes, and training steps on the training results. Common cropping and rotation transformations are no longer applicable to serialized data such as texts and audio. A universal unsupervised learning approach CPC [8] architecture that extracts features from high-dimensional data and uses autoregressive models to predict the future in latent space was proposed.Beyond self-supervised classification, SCL [1] is an extension of traditional CL. It pulls the same class closer using SupCon loss by leveraging label information. SsCL [11] combined the CL loss branch with the cross-entropy loss branch in semisupervised learning and introduced a cocalibration mechanism to interchange predictions between the two branches. RoCL [12] trained a robust neural network without label information by using a contrastive self-supervised learning framework. Also, the CosG [9] model is a graph-based CL method for fact verification including label-supervised and unsupervised graph-contract. The former helps the model learn discriminative representation for items of various classes, and the latter trains a graph convolutional encoder for reducing the loss of sole node features in graph propagation.


## 2.2. Adversarial Example Generation

In NLP systems, studies have shown that the model is highly dependent on the input data and have used this discovery to conduct robust training of the model through adversarial samples. Belinkov and Bisk [13] found that even with spell checking, existing models are still sensitive to the spelling order of words in machine translation tasks. For character-level attacks, perturbation can be approximated by the number of character edits. For word-level attacks, perturbation can be achieved by substitution. EDA [14], an easy data augmentation method, consists of four simple operations: synonym replacement, random insertion, random swap, and random deletion. Similarly, there are some methods based on word semantic similarity. Kobayashi proposed a contextual augmentation [15] strategy, which uses a bidirectional language model to generate a diverse alternative vocabulary according to the context. Exploring the broadening of the scope of application, round-trip translation constructs data through translation, which is suitable for words, phrases, sentences, and texts.Recent adversarial attack example generation algorithms can be generally categorized into two groups: white-box or black-box methods. In the white-box setting, an adversary has access to the model, model parameters, and feature set of inputs. For perturbation at the embedding level, TextTricker [16] calculated the gradient magnitude of each input unit and added the scores of each dimension in the embedding space as the word-level importance score when identifying key words from input. Perturbating at the grade level, HotFlip [17] relies on an atomic flip operation to generate white-box adversarial examples that trick character-level and word-level neural models. It swaps one token for another based on the gradients of one-hot input vectors. Papernot et al. proposed FGSM [18], a fast gradient sign method widely applied in the image domain [19], which could be solved by linearizing the cost function of the model around its input and selecting perturbation using the gradient of the cost function with respect to the input itself. GBDA [20], a gradient-based distributional text attack against a transformer framework, defines a parameterized distribution of adversarial examples and uses Gumbel-softmax approximation to derive a smooth estimate of the gradient. In the black-box setting, an adversary is only allowed to query the target classifier and does not know the details of learned models or the feature representations of inputs. In [21], AEG was proposed, a reinforcement-learning-based approach, to generate adversarial examples in black-box settings.Intuitively, the generative adversarial network (GAN) generator captured the data distribution and made the discriminant equal to ½ everywhere in the space of its arbitrary function. After continuous research on GAN, WGAN [22] was developed by using the directed Wasserstein distance to produce a value function to solve the problem regarding original GAN training instability. Facing weight flipping caused by poor samples in WGAN, Gulrajani proposed a gradient penalty (WGAN-GP) algorithm [23] which can penalize the norm of the gradient of the critic with respect to its input. Moreover, in adversarial examples producing high quality and efficiency, Xiao produced AdvGAN [24] to generate adversarial examples with GANs in both semiwhite-box and black-box attack settings.


## 2.3. Deep Learning Models for Text Classification

The main basic components of the deep learning model in the text classification task are the recurrent neural network (RNN) [25], CNN [26, 27], attention mechanism [28, 29], and graph neural network (GNN) [30]. These basic components have tended to develop towards a fusion model. In the joint model, the advantages of local information, context information, and feature screening are combined. Huang and Liu proposed GCNN [31] and used CNN to compensate for the shortcoming of gated recurrent units (GRUs) in the extraction of long sentence information. Yang and Tang [32] tested the effect of the attention mechanism on different positions of the CNN-based model and performed experimental verification in ATCNN-1, ATCNN-2, and ATCNN-3 which superimposes formers combining feature weighting and feature selecting. To strengthen the feature extraction ability, Wenzhen et al. proposed the C-BiGRU-ATT [33] method to reduce the impact of text representation on the classification results and increase the features of the text at the vocabulary level and character level to improve classification accuracy. Also, in the face of semantic ambiguity, Liu and Guo proposed AC-BiLSTM [34], which is oriented to high-dimensional, discrete, and complex semantic phenomena for use on text classification tasks. This entailed the use of bidirectional long short-term memory (BiLSTM), an attention mechanism, and a convolutional layer.The success of TextGCN [30] turned the serialized text model to the graph-structured text model. Wu et al. proposed SGC [35] to accelerate GCN by removing nonlinearities and collapsing weight matrices between consecutive layers. Zhu and Koniusz proposed SSGC to improve the performance of GCN and capture the global and local context of each node using simple spectral graph convolution. With the enrichment of the corpus and the development of increased computing power, a series of transformer models represented by Bert emerged. Lin et al. proposed that BertGCN [36] adopt transductive learning, learning representations for both training data and unlabeled test data, by propagating label influence through graph convolution. Zhu and Koniusz proposed SSGC [37] to solve transition smoothing and reduce computing and storage costs.Inspired by large-scale pretraining models [38], researchers have begun to use external information [39, 40] to improve the classification accuracy of the model. Pan et al. [41] proposed SWEMs to transfer source domain knowledge to unseen text sequences. Abid et al. proposed BiGRU-CNN [28] based on pretrained GloVe embedding which retains domain knowledge. There are also models that do not rely on external knowledge. In the sparse classification of short text data, Liu et al. [42] proposed a bilevel attention model that does not rely on external knowledge to capture word-level information which is presented to explore the topic-word association and sequence-level information which is used to extract the relationship between local and global sentiment expression.So far, we have discussed the development of CL, adversarial sample generation strategies, and deep models used in text classification tasks. Obviously, there is still potential for improvement in CL’s robust representation. The adversarial sample generation technique based on the entire feature embedding is not addressed. The majority of deep models are focused on enhancing feature acquisition while ignoring model stability. To make the best of the text feature and robust representation content, a new neural network significantly improving the model classification robustness is proposed in this paper. The proposed model can efficiently represent enhanced features and learn the stable representation by input attacks.


## 3. Proposed Model

Here, we introduce the structure of the proposed model, main components, and realization of data-processing flow in detail. The model combines the powerful instance representation ability of contrastive learning and robustness brought about by GP. In the feature vector extraction part, the joint model of CNN, RNN, and the attention mechanism is used. The SCL projection method is adopted to obtain better representation capabilities. Finally, we add the gradient penalty to the SCL loss to obtain the robustness of the model to perturbation of the feature vector. The model processes the input data in a two-step process. In the first step, the embedding layer is used to vectorize the input. Next, CNNs, BiGRU, an attention mechanism, and a projector are integrated to construct an encoder. Then, the dataset is utilized to train the encoder to achieve a better representation of the input text. In the second step, the frozen encoder with the classifier layer realizes text classification. The overall structure of PCRA-GP is shown in Figure1. The pseudocode of PCRA-GP is presented in Algorithm 1. The key points of our approach are described as follows.Figure 1
The framework of the PCRA-GP model.Algorithm 1: Pseudocode for PCRA-GP.
Input: a set of text TOutput: Accuracy#Step 1: training an encoder()Ttrain∈T;fort∈Ttrain doF⟵concatCNNt,3,CNNt,4,CNNt,5;O⟵BiGRUF;c⟵AttO;z⟵projectorc;losstrain⟵ equation (6);updateθtrain_e to minimize losstrain_e;#Step 2: training a model PCRA-GP() with a frozen encoder encoder()fort∈Ttrain doz⟵encodert;y^⟵classifierzlosstrain_c⟵crossentropyy,y^;updateθtrain_c to minimize losstrain_c;#Step 3: testing the PCRA-GP() modelTtest∈T;fort∈Ttest doreturnaccuracy⟵ PCRA-GP (t). evaluate();The following is a summary of the implementation process of the PCRA-GP model for the text classification task:Step 1: Embedding is implemented on training dataTtrain, and a joint feature extraction model is established. Next, equations (2), (3), and (4) are employed to capture the feature representation of training items. Then, GP perturbation is added to the feature representation. Finally, a robustness projector is trained under the loss function, as shown in equation (6).Step 2: The trained encoder in Step 1 is used to extract the training sample feature vectors. Besides, the cross-entropy function with the Adam optimizer is adopted to train the classifier, and the parameters are updated according to the results.Step 3: The trained encoder and the classifier of the PCRA-GP model are applied to predict the label of testing dataTtest. The comparison between the predicted value and the true value reveals the model classification accuracy.


### 3.1. CNN Layer

CNNs are widely used for image tasks and are gradually being applied to NLP. They can extract local correlation of spatial or temporal structures and can also be used to capture sequence information and reduce the number of input dimensions. TextCNN [43] added a convolutional layer after the word vector to build a simple model and achieved good sentence classification results. The CNN layer used is the same as TextCNN. The structure of TextCNN is shown in Figure 2.Figure 2
Structural diagram of the TextCNN process.The input text of the CNN is denoted asT=x1,x2,…,xn, T∈Rn×d. The text length is n, and each word is represented as a vector xn. After the embedding layer, the word is represented as a vector of the dimension d. Because the input text matrix must be of the same length, we cut off lengths greater than n and pad those lengths less than n. For the selection of n-gram features, the height of convolution kernels is set to 3, 4, and 5. The step size is set to 1. After one-dimensional convolutional operation, the mth local features encoding fm are calculated using equation (1). Regarding unilateral suppression operation, the ReLU function is used to change negative values to zero, all positive values and zero remain unchanged, and a pooling layer is adopted to retain the main features while reducing parameters and computation. Finally, the feature sequence is derived as Lf=f1,f2,…,fm. The output of the three-layer CNN is concatenated as the downstream input (as given by equation (2)):(1)fm=rWt∙xt+b,(2)F=F1,F2,…,Fm=Lf3⊕Lf4⊕Lf5=f13⊕f14⊕f15,…,fm3⊕fm4⊕fm5,where r∙ is the ReLU function, Wt and b represent the filter weight and bias term in a window of the word xt, m denotes the total number of filters, F represents the output through the CNN layer, ∙∙ represents the feature under different kernel sizes, and ⊕ represents the concatenate operator. The length of F is equal to m.


### 3.2. Bi-GRU Layer

RNNs have a recurrent hidden state throughout the computational flow which can maintain continuous sequence information and has a good effect on processing sequence data. Therefore, in addition to text local features, we also pay attention to sequence global information. A GRU is a type of RNN architecture and has become the popular structure. It performs in a manner similar to long short-term memory and is less computationally onerous. In our architecture, BiGRU is used to encode the forward and backward information of the sentence. The structure of the BiGRU is shown in Figure3.Figure 3
Structural diagram of the BiGRU.The forward GRU (GRU⟶) calculates each timestep hidden state from 1 to m. In the backward GRU GRU⃖ layer, the reverse calculation is performed from m to 1. Finally, the bidirectional output is merged as the result of BiGRU. The whole calculation process can be described using the following formulae:(3)Ym=GRU⟶F=fw1Fm+w2Ym−1,Ym′=GRU⟵F=fw3Fm+w4Ym−1′,om=BiGRUw5Ym+w6Ym′,where Ym and Ym′ are the forward and backward output, om represents the output results of BiGRU at time m, the final output of the BiGRU network is O=o1,o2,…,om, and w∙ denotes the weights corresponding to the forward and reverse hidden states.


### 3.3. Attention Layer

An attention mechanism is a feature selection method, which focuses on the strength and weakness of local information and obtains dependent information for correct judgment. In practice, the attention function is commonly divided into two categories: dot-product attention and additive attention [44]. Dot-product attention is much faster and more space-efficient and uses a highly optimized matrix multiplication code, while additive attention is committed to dealing with long-term memory problems and applied to longer and wide varying sequences. The schematic of the attention mechanism is illustrated in Figure 4.Figure 4
Schematic of the attention mechanism.The attention mechanism can be simplified as follows:(4)et=aO,αt=expet∑k=1mexpek,c=∑t=1mαtot,where a∙ is a learnable function dependent only on O, c denotes the attention weight embedding sequence that matches the product of the input hidden state o and the weight α, and the parameters t and k indicate different positions.


### 3.4. SCL Method

SCL adopts a fully supervised method to further reveal label information. The SCL method [1] comprises three parts: data augmentation, an encoder network, and a projection network. In our work, the feature extraction network consists of a CNN layer (Section 3.1), a RNN layer (Section 3.2), and attention (Section 3.3) components. Besides, we do not directly use the data augmentation part in our model but indirectly adopt the embedding penalty gradient (Section 3.5). In this section, the project network and supervised contrastive loss function are introduced.The projection network is instantiated as a dense layer. First, this layer maps the encoder network outputc to a feature vector z=projectc∈RDp, where Dp=128 and z is normalized to place the output on the unit hypersphere; then, an inner product is used to measure the output in the projection space.Khosla et al. [1] argued that the SCL loss is equivalent to the N-pair loss when more than one negative item is present. The N-pair loss [45] is a metric-learning objective, which accelerates convergence through strengthening the interaction between positive classes and negative classes in each update. The contrastive loss is shown as follows:(5)Lscl=1N∑i=1Nlog1+∑j≠iexpziTzj+−ziTzi+τ,where N is the total number of sample data categories and i,j∈N, ∙T denotes the transpose operation, zi+ represents the positive example representation while zj+, i≠j denotes the negative example representation, and τ is the temperature parameter.


### 3.5. Gradient Penalty

Commonly, text classification mainly relies on text representation and feature extraction to obtain better classification results. However, the model trained by this method is vulnerable to attacks, which leads to the model having poor robustness. In the field of NLP, data augmentation is a way to improve the robustness of the model, but it has its own limitations. Text enhancement methods are usually suitable for the character level, word level, and sentence level and are rarely applied to document-level materials. The GP method adds perturbation to the loss function, realizing a universal adversarial method. As a regularization method, the gradient penalty reduces overfitting and improves model generalizability.In our method, the disturbance part is calculated by the differential of the contrastive lossLscl to the feature vector z, defined as ∇zLsclz. A hyperparameter ε is also introduced to control the extent of the penalty applied. Then, loss function equation (5) is updated to equation (6):(6)Ltotal=Lscl+ε∇zLsclz2.In PCRA-GP, the pooling layer, dropout layer, and normalizing layer are used to avoid overfitting. The softmax layer is used to generate the probability distribution of the text label class, and the Adam optimizer is selected to optimize the loss function of the network.


## 3.1. CNN Layer

CNNs are widely used for image tasks and are gradually being applied to NLP. They can extract local correlation of spatial or temporal structures and can also be used to capture sequence information and reduce the number of input dimensions. TextCNN [43] added a convolutional layer after the word vector to build a simple model and achieved good sentence classification results. The CNN layer used is the same as TextCNN. The structure of TextCNN is shown in Figure 2.Figure 2
Structural diagram of the TextCNN process.The input text of the CNN is denoted asT=x1,x2,…,xn, T∈Rn×d. The text length is n, and each word is represented as a vector xn. After the embedding layer, the word is represented as a vector of the dimension d. Because the input text matrix must be of the same length, we cut off lengths greater than n and pad those lengths less than n. For the selection of n-gram features, the height of convolution kernels is set to 3, 4, and 5. The step size is set to 1. After one-dimensional convolutional operation, the mth local features encoding fm are calculated using equation (1). Regarding unilateral suppression operation, the ReLU function is used to change negative values to zero, all positive values and zero remain unchanged, and a pooling layer is adopted to retain the main features while reducing parameters and computation. Finally, the feature sequence is derived as Lf=f1,f2,…,fm. The output of the three-layer CNN is concatenated as the downstream input (as given by equation (2)):(1)fm=rWt∙xt+b,(2)F=F1,F2,…,Fm=Lf3⊕Lf4⊕Lf5=f13⊕f14⊕f15,…,fm3⊕fm4⊕fm5,where r∙ is the ReLU function, Wt and b represent the filter weight and bias term in a window of the word xt, m denotes the total number of filters, F represents the output through the CNN layer, ∙∙ represents the feature under different kernel sizes, and ⊕ represents the concatenate operator. The length of F is equal to m.


## 3.2. Bi-GRU Layer

RNNs have a recurrent hidden state throughout the computational flow which can maintain continuous sequence information and has a good effect on processing sequence data. Therefore, in addition to text local features, we also pay attention to sequence global information. A GRU is a type of RNN architecture and has become the popular structure. It performs in a manner similar to long short-term memory and is less computationally onerous. In our architecture, BiGRU is used to encode the forward and backward information of the sentence. The structure of the BiGRU is shown in Figure3.Figure 3
Structural diagram of the BiGRU.The forward GRU (GRU⟶) calculates each timestep hidden state from 1 to m. In the backward GRU GRU⃖ layer, the reverse calculation is performed from m to 1. Finally, the bidirectional output is merged as the result of BiGRU. The whole calculation process can be described using the following formulae:(3)Ym=GRU⟶F=fw1Fm+w2Ym−1,Ym′=GRU⟵F=fw3Fm+w4Ym−1′,om=BiGRUw5Ym+w6Ym′,where Ym and Ym′ are the forward and backward output, om represents the output results of BiGRU at time m, the final output of the BiGRU network is O=o1,o2,…,om, and w∙ denotes the weights corresponding to the forward and reverse hidden states.


## 3.3. Attention Layer

An attention mechanism is a feature selection method, which focuses on the strength and weakness of local information and obtains dependent information for correct judgment. In practice, the attention function is commonly divided into two categories: dot-product attention and additive attention [44]. Dot-product attention is much faster and more space-efficient and uses a highly optimized matrix multiplication code, while additive attention is committed to dealing with long-term memory problems and applied to longer and wide varying sequences. The schematic of the attention mechanism is illustrated in Figure 4.Figure 4
Schematic of the attention mechanism.The attention mechanism can be simplified as follows:(4)et=aO,αt=expet∑k=1mexpek,c=∑t=1mαtot,where a∙ is a learnable function dependent only on O, c denotes the attention weight embedding sequence that matches the product of the input hidden state o and the weight α, and the parameters t and k indicate different positions.


## 3.4. SCL Method

SCL adopts a fully supervised method to further reveal label information. The SCL method [1] comprises three parts: data augmentation, an encoder network, and a projection network. In our work, the feature extraction network consists of a CNN layer (Section 3.1), a RNN layer (Section 3.2), and attention (Section 3.3) components. Besides, we do not directly use the data augmentation part in our model but indirectly adopt the embedding penalty gradient (Section 3.5). In this section, the project network and supervised contrastive loss function are introduced.The projection network is instantiated as a dense layer. First, this layer maps the encoder network outputc to a feature vector z=projectc∈RDp, where Dp=128 and z is normalized to place the output on the unit hypersphere; then, an inner product is used to measure the output in the projection space.Khosla et al. [1] argued that the SCL loss is equivalent to the N-pair loss when more than one negative item is present. The N-pair loss [45] is a metric-learning objective, which accelerates convergence through strengthening the interaction between positive classes and negative classes in each update. The contrastive loss is shown as follows:(5)Lscl=1N∑i=1Nlog1+∑j≠iexpziTzj+−ziTzi+τ,where N is the total number of sample data categories and i,j∈N, ∙T denotes the transpose operation, zi+ represents the positive example representation while zj+, i≠j denotes the negative example representation, and τ is the temperature parameter.


## 3.5. Gradient Penalty

Commonly, text classification mainly relies on text representation and feature extraction to obtain better classification results. However, the model trained by this method is vulnerable to attacks, which leads to the model having poor robustness. In the field of NLP, data augmentation is a way to improve the robustness of the model, but it has its own limitations. Text enhancement methods are usually suitable for the character level, word level, and sentence level and are rarely applied to document-level materials. The GP method adds perturbation to the loss function, realizing a universal adversarial method. As a regularization method, the gradient penalty reduces overfitting and improves model generalizability.In our method, the disturbance part is calculated by the differential of the contrastive lossLscl to the feature vector z, defined as ∇zLsclz. A hyperparameter ε is also introduced to control the extent of the penalty applied. Then, loss function equation (5) is updated to equation (6):(6)Ltotal=Lscl+ε∇zLsclz2.In PCRA-GP, the pooling layer, dropout layer, and normalizing layer are used to avoid overfitting. The softmax layer is used to generate the probability distribution of the text label class, and the Adam optimizer is selected to optimize the loss function of the network.


## 4. Experiments

In this section, the experimental set-up and comparative experimental design are demonstrated. The experimental setup is about the presence of datasets and the initial settings of model parameters used in the experiment. In the comparative test part, the results are compared with the deep learning baselines and the state-of-the-art text classification methods. Finally, the model classification accuracy changes under the two main components of the model, and hyperparameter settings are evaluated. The detailed experimental data are recorded in Appendix A (Tables1–8). In addition, as a verification of the versatility of our idea, the GP method is also used in image classification (Appendix B) (Figure 5).Table 1
IMDB with various epsilon (temperature=0.05) on RTX2080.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores.Table 2
IMDB with various epsilon (temperature=0.05) on K80.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores.Table 3
ChnsentiCorp-Htl with various epsilon (temperature=0.05) on RTX2080.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Table 4
ChnsentiCorp-Htl with various epsilon (temperature=0.05) on K80.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Table 5
20news with various epsilon (temperature=0.05) on RTX2080.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Table 6
20news with various epsilon (temperature=0.05) on K80.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Table 7
Cnews with various epsilon (temperature=0.05) on RTX2080.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Table 8
Cnews with various epsilon (temperature=0.05) on K80.
Bold values are the better performances with different epsilon. Values with color red are the best accuracy scores among PCRA-GP and PCRA.Figure 5
Result of SCL with GP on the cifar10 dataset.


### 4.1. Experimental Setup

#### 4.1.1. Datasets

Our model is used to solve bilingual text classification tasks (including sentiment and news category) at the chapter level and paragraph level in English and Chinese (Table9). The datasets used in this article are as follows:(i)
IMDB: it is a dataset of 50,000 movie reviews in English, labeled by sentiment (positive/negative).(ii)
ChnSentiCorp-Htl (https://github.com/SophonPlus/ChineseNlpCorpus): it is a corpus including more than 7,000 hotel reviews in Chinese.(iii)
20news: The 20 Newsgroups (https://qwone.com/~jason/20Newsgroups/) dataset is a collection of approximately 20,000 English newsgroup documents, partitioned evenly across 20 different newsgroups. It has become a popular dataset for experiments in text applications of machine learning techniques, such as text classification and text clustering.(iv)
Cnews: It is a subset of the THUCnews (https://github.com/thunlp/THUCTC) dataset produced by NLP and Computational Social Science Lab, Tsinghua University. The THUCnews dataset mainly collects Sohu news, and Cnews contains 10 categories and 65,000 Chinese texts.Table 9
Dataset statistics.
CorpusClassesInstanceAvg lengthTrain/testLanguageIMDB250,00023125,000/25,000EnglishChnSentiCorp-Htl277671283000/3000Chinese20news2018,846221.2611,314/7532EnglishCnews1065,00053050,000/15,000Chinese


#### 4.1.2. Baseline Methods

BALB: it is a backdoor attack against the LSTM-based text classification system [46].GCNN: it is a model that introduces contextual information obtained by the RNN and local information captured by the CNN and was proposed by Huang and Liu [31].AEG: it is a reinforcement-learning-based approach to generating adversarial examples in black-box settings [21].AC-BiLSTM: it is a novel and unified architecture that contains the bidirectional LSTM (BiLSTM), attention mechanism, and convolutional layer [34].BiGRU-CNN: A model composed of GloVe, CNN, RNN and max-pooling layer was proposed by Abid et al. [28].BAM: it is a bilevel attention model that does not rely on external knowledge and was proposed by Liu et al. [42].Text GCN: in this model, the first task is to build the whole corpus into a heterogeneous graph and use GNN to jointly learn word and document embeddings, as performed by Yao et al. [30].SGC: it is a method aimed to simplify single linear transformation and was proposed by Wu et al. [35].SSGC: this model is a variant model derived from Markov diffusion kernels, deals with degradation caused by the increased depth, and was proposed by Zhu and Koniusz [37].C-CNN: this model can realize centrality convolution based on a separate encoding function and was proposed by Dong et al. [26].BertGCN: it is a model that combines Bert’s large-scale pretraining knowledge and the transferability of the GCN and was proposed by Lin et al. [36].RoBERTaGCN: this method entails joint RoBERTa and GCN as a comparative experiment and was proposed by Lin et al. [36].SWEM: this method uses a modified hierarchical pooling strategy in few-shot transfer learning and was proposed by Pan et al. [41].C-BiGRU-ATT: this model uses the CNN, attention mechanism, and BiGRU, extracting contextual and local information at the character and vocabulary levels, and was proposed by Wenzhen et al. [33].ATCNN-3: this method extracts and filters features by adding attention mechanisms in different positions and was proposed by Tang and Yang. [32].


#### 4.1.3. Parameter Settings

The PCRA-GP model uses accuracy as the evaluation metric to estimate the overall classification performance. The first layer of PCRA-GP is an embedding layer which converts words into 300-dimensional vectors. The kernels of multichannel CNNs are set to 3, 4, and 5. The number of hidden units of BiGRUs and CNNs is set to 256 which is suitable for document classification and 128 which is suitable for paragraph classification. The training batch sizes are set to 256, 128, and 64. The dropout rate is 0.5. To better avoid model overfitting, we add dropout behind the embedding layer and before the classification dense layer and set it to 0.2 when solving IMDB and ChnSentiCorp-Htl models. The two hyperparameters (epsilon and temperature) are set to 0.01–0.06. The backpropagation algorithm based on the Adam stochastic optimization method is used to train the network through time, and the learning rates are 0.001 and 0.0001, which are used for different datasets. The normalization layer involves batch normalization and layer normalization. Each corpus is split into training and testing sets in the proportion of 8 : 2. All experiments are conducted with Python 3.7.9 and Keras 2.4.3 on GeForce RTX 2080 Ti and NVIDIA Tesla K80 systems. The specific parameters of the model are listed in Table10.Table 10
The specific parameters of PCRA-GP.
ComponentParameterWord embedding300CNN[3, 4, 5], [256, 128]BiGRU[256, 128]Batch size[256, 128, 64, 32]Dropout[0.2, 0.5]Dense128Epsilon[0.01, 0.02, 0.031, 0.04, 0.05, 0.06]Temperature[0.01, 0.02, 0.03, 0.04, 0.05, 0.06]Learning rate[0.001, 0.0001]NormalizationBatch normalization, layer normalizationPython3.7.9Keras2.4.3GPUGeForce RTX 2080 Ti, NVIDIA Tesla K80


### 4.2. Experiments

#### 4.2.1. Overall Comparison

In this section, all the experimental data are summarized and the best classification accuracy is selected for comparisons with the baseline model accuracy. The results recorded in Table11 involve the classification of sentiment and news texts, and the text languages are Chinese and English. The baseline results are all reported in the cited papers.Table 11
Accuracy of the models.
DatasetModelsAccuracy (%)IMDBBALB [46]84.92GCNN [31]90.0AEG [21]89.95PCRA (ours)90.26PCRA-GP (ours)90.34ChnSentiCorp-HtlAC-BiLSTM [34]90.60BiGRU-CNN [28]91.60BAM [42]90.0PCRA (ours)91.75PCRA-GP (ours)91.7520newsTextGCN [30]86.34SGC [35]88.50SSGC [37]88.60C-CNN [26]82.59BertGCN [36]89.3RoBERTaGCN [36]89.5PCRA (ours)88.42PCRA-GP (ours)89.86CnewsSWEMs [41]90.07C-BiGRU-ATT [33]95.55ATCNN-3 [32]96.58PCRA (ours)96.92PCRA-GP (ours)97.45Bold values are the best accuracy scores.The results for the IMDB dataset show that the PCRA is higher than GCNN by 0.26%, while the PCRA-GP is 0.34% higher. In comparison with backdoor attack models, our model classification accuracy improvement is significantly ahead of theirs by 5.42% and 5.34%. The black-box setting AEG model is better than backdoor attack models, but it is weaker than our model by 0.39% and 0.31%. The results indicate that perturbation on the model loss function is an effective way to improve accuracy.The models applied to the ChnsentiCorp-Htl dataset show that BiGRU-CNN lacks an attention layer compared to AC-BiLSTM, and AC-BiLSTM lacks projection compared to the PCRA model. The model using only bilevel attention returns the lowest result. Due to the small amount of data, the simple model BiGRU-CNN is better than the complex point model AC-BiLSTM, and the addition of the GP method results in the highest accuracy of classification. The results for the Cnews dataset indicate that PCRA and PCRA-GP are, respectively, 1.37% and 1.9% better than C-BiGRU-ATT that does not use a projector and GP components. Compared with ATCNN-3, PCRA is 0.34% higher after increasing feature extraction ability. Compared with SWEM, which is a few-shot model, the overall training method can maintain a high level of expressiveness. The results show that a projector under the SCL training process brings improved results with the GP constraint.Since TextGCN was proposed, GNN has demonstrated high-quality performance and gradually replaced the traditional deep model based on CNNs and RNNs. The results for the 20 Newsgroups dataset show that the classification accuracies of GNN-based models (TextGCN, SGC, and SSGC) are gradually improving. After fusing the pretrained Bert model, the performance of the GNN model is further improved. Our proposed model is 0.36% better than RoBERTaGCN and achieved SOTA results. The results indicate that our model improves the performance of traditional deep models based on the CNN, RNN, and attention mechanism.Combined with the performance of our proposed model for four public datasets, our results are found to be consistently better than the baseline results recorded in most published papers. In view of the above discussion, the PCRA-GP model that we proposed is highly competitive.


#### 4.2.2. Effect of Components of PCRA-GP

In Figure6, the classification accuracies are plotted for all datasets, in which the temperature hyperparameter is set to 0.05 and comes from the experiment using the RTX 2080Ti system. Since the experimental results are sensitive to parameter settings, our experimental process mainly focuses on the type of normalization, batch size, epoch, and epsilon.Figure 6
The accuracy of datasets with the type of normalizations, epoch, and batch size on RTX2080 Ti. The symbolL/B_XXX_YYY means layer (L) and batch (B) normalizations, epsilon, epoch, and batch size. The PCRA column represents contrastive learning without the gradient penalty. (a) The PCRA-GP’s accuracy on IMDB, trained with L. (b) The PCRA-GP’s accuracy on IMDB, trained with B. (c) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with L. (d) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with B. (e) The PCRA-GP’s accuracy on 20news, trained with L. (f) The PCRA-GP’s accuracy on 20news, trained with B. (g) The PCRA-GP’s accuracy on Cnews, trained with L. (h) The PCRA-GP’s accuracy on Cnews, trained with B.
(a)(b)(c)(d)(e)(f)(g)(h)Overall, as shown in Figure6, the performance of the PCRA model is relatively stable and less affected. Batch normalization could withstand larger epsilon disturbances than layer normalization. When the batch size is set to 256, the model classification accuracy can be maintained in response to variations in other factors. The various epoch settings have little effect on the performance. When epsilon is greater than 0.04, the classification accuracy decreases significantly.In Table12, the maximum value is counted under different parameter settings, which ignores epsilon, and the data are selected from Figure 6. Different from the trend analysis of the data, we pay more attention to specific accuracy value information. By summarizing the classification results of all datasets, the layer normalization classification results are found to be 33 times higher than those of batch normalization among 48 groups, and the best accuracy occurrence ratio is 5 : 3. Moreover, eight locations, where the minimum value is obtained, are counted, five of them appear at a batch size of 64, and the epoch is 200. The maximum value appears 5 times when the batch size is 256.Table 12
Best results obtained from different parameters without epsilon. The minimum and maximum values are in bold in the table, and the maximum values are marked in red.In summary, batch normalization methods are stable, while layer normalization has high accuracy and often leads to the optimal classification. A large batch size can acquire good accuracy and offset the impact of different epochs. In terms of overall results, the result is best balanced when epsilon is 0.01, and the second best choice is 0.02.


#### 4.2.3. Ablation Experiment

We tested the performance of different combinations, and the results are shown in Table13. The experimental hyperparameters are fixed and executed on RTX 2080Ti. We set the epoch size to 100, batch size to 256, and temperature to 0.05. Besides, epsilon is set to 0.01 during adversarial training. Considering PC as the baseline, the introduction of R reduces the classification effect and the integration of A brings a huge improvement in the results. The adversarial training with GP also improves the classification accuracy of the model. 20news is not well trained in methods PC and PCR. Additionally, we visualize the loss value and accuracy during the training process to understand the results. The training process data are shown in Figure 7. During the training stage of the encoder and classifier, the loss values of PC and PCR do not drop significantly, resulting in low accuracy. However, metrics for PCRA and PCRA-GP perform well. In addition, during the training of the encoder, the introduction of GP gives a delayed effect to the loss compared to PCRA. The results indicate that joint features enhance the performance of the model and that adversarial training is an effective way for the classification task.Table 13
Experimental results of different component combinations.
ModelIMDBChnSentiCorp-Htl20newsCnewsPC64.3567.338.9861.69PCR59.5757.587.0454.85PCRA89.4790.3388.3496.91PCRA-GP89.7389.3389.3396.83Bold values are the best accuracy scores.Figure 7
The training process data of different datasets. e_l means the encoder loss, c_l denotes the classifier loss, and acc means accuracy. (a) The training process data of IMDB. (b) The training process data of ChnSentiCorp-Htl. (c) The training process data of 20news. (d) The training process data of Cnews.
(a)(b)(c)(d)


#### 4.2.4. Tuning of the Temperature Parameter

In addition to the changes brought about by the above hyperparameter changes, temperature is also a hyperparameter introduced by SimCLR [10], which is an adjustable value to help learn hard negatives well. Table 14 lists the accuracy under a change in temperature. Obviously, PCRA-GP is easier to be affected than PCRA, and only three results are improved by adjusting temperature. In the IMDB dataset, the PCRA method is improved by 0.12% at a temperature of 0.04. Besides, the PCRA method improved by 0.21% at a temperature of 0.02 in the Cnews dataset. In addition, only PCRA is higher than PCRA-GP in the Cnews dataset. The results indicate that the best value of temperature is 0.05.Table 14
Classification results under different temperature parameters. The temperature is adjusted for the maximum value in Table12, which is trained at a fixed temperature of 0.05.
TemperatureAccuracyIMDBChnSentiCorp-Htl20newsCnewsPCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)0.0151.4389.2359.4290.5012.9884.5566.3497.350.0249.5087.7188.8389.5812.3782.1068.9297.390.0350.1081.2789.1790.0813.0486.8656.4697.180.0488.8089.8488.9289.2588.1086.9196.5096.920.0589.9189.7291.4291.0889.1788.4897.2197.180.0689.5689.6589.8389.2587.7687.9797.2397.08Bold values are the best accuracy scores.


#### 4.2.5. Model Training Efficiency

The training time is also recorded in Table15 to observe the effect of different components on the training time. The experimental tests were performed on RTX 2080Ti. The results show that R increases the training time per step. The attention mechanism reduces the training time and verifies the effect of automatic feature selection. However, the role of the attention mechanism fails for ChnSentiCorp-Htl. This is mainly due to the small size of the dataset and the short text length. Furthermore, the lower time reduction during the classification stage is due to the use of the frozen encoder. The results indicate that the training time can be reduced by the attention mechanism in the multilevel feature fusion strategy.Table 15
Model training time comparison (ms/step).
ModelIMDBChnSentiCorp-Htl20newsCnewsEncoderClassifierEncoderClassifierEncoderClassifierEncoderClassifierPC95.7727.9851.3811.94138.7135.78211.8758.18PCR99.1035.3957.3116.52138.2251.47215.8180.77PCRA94.3135.257.7016.8130.2447.01199.3477PCRA-GP94.7834.8149.2715131.1247213.3779.67Bold values are the most time-consuming results.


## 4.1. Experimental Setup

### 4.1.1. Datasets

Our model is used to solve bilingual text classification tasks (including sentiment and news category) at the chapter level and paragraph level in English and Chinese (Table9). The datasets used in this article are as follows:(i)
IMDB: it is a dataset of 50,000 movie reviews in English, labeled by sentiment (positive/negative).(ii)
ChnSentiCorp-Htl (https://github.com/SophonPlus/ChineseNlpCorpus): it is a corpus including more than 7,000 hotel reviews in Chinese.(iii)
20news: The 20 Newsgroups (https://qwone.com/~jason/20Newsgroups/) dataset is a collection of approximately 20,000 English newsgroup documents, partitioned evenly across 20 different newsgroups. It has become a popular dataset for experiments in text applications of machine learning techniques, such as text classification and text clustering.(iv)
Cnews: It is a subset of the THUCnews (https://github.com/thunlp/THUCTC) dataset produced by NLP and Computational Social Science Lab, Tsinghua University. The THUCnews dataset mainly collects Sohu news, and Cnews contains 10 categories and 65,000 Chinese texts.Table 9
Dataset statistics.
CorpusClassesInstanceAvg lengthTrain/testLanguageIMDB250,00023125,000/25,000EnglishChnSentiCorp-Htl277671283000/3000Chinese20news2018,846221.2611,314/7532EnglishCnews1065,00053050,000/15,000Chinese


### 4.1.2. Baseline Methods

BALB: it is a backdoor attack against the LSTM-based text classification system [46].GCNN: it is a model that introduces contextual information obtained by the RNN and local information captured by the CNN and was proposed by Huang and Liu [31].AEG: it is a reinforcement-learning-based approach to generating adversarial examples in black-box settings [21].AC-BiLSTM: it is a novel and unified architecture that contains the bidirectional LSTM (BiLSTM), attention mechanism, and convolutional layer [34].BiGRU-CNN: A model composed of GloVe, CNN, RNN and max-pooling layer was proposed by Abid et al. [28].BAM: it is a bilevel attention model that does not rely on external knowledge and was proposed by Liu et al. [42].Text GCN: in this model, the first task is to build the whole corpus into a heterogeneous graph and use GNN to jointly learn word and document embeddings, as performed by Yao et al. [30].SGC: it is a method aimed to simplify single linear transformation and was proposed by Wu et al. [35].SSGC: this model is a variant model derived from Markov diffusion kernels, deals with degradation caused by the increased depth, and was proposed by Zhu and Koniusz [37].C-CNN: this model can realize centrality convolution based on a separate encoding function and was proposed by Dong et al. [26].BertGCN: it is a model that combines Bert’s large-scale pretraining knowledge and the transferability of the GCN and was proposed by Lin et al. [36].RoBERTaGCN: this method entails joint RoBERTa and GCN as a comparative experiment and was proposed by Lin et al. [36].SWEM: this method uses a modified hierarchical pooling strategy in few-shot transfer learning and was proposed by Pan et al. [41].C-BiGRU-ATT: this model uses the CNN, attention mechanism, and BiGRU, extracting contextual and local information at the character and vocabulary levels, and was proposed by Wenzhen et al. [33].ATCNN-3: this method extracts and filters features by adding attention mechanisms in different positions and was proposed by Tang and Yang. [32].


### 4.1.3. Parameter Settings

The PCRA-GP model uses accuracy as the evaluation metric to estimate the overall classification performance. The first layer of PCRA-GP is an embedding layer which converts words into 300-dimensional vectors. The kernels of multichannel CNNs are set to 3, 4, and 5. The number of hidden units of BiGRUs and CNNs is set to 256 which is suitable for document classification and 128 which is suitable for paragraph classification. The training batch sizes are set to 256, 128, and 64. The dropout rate is 0.5. To better avoid model overfitting, we add dropout behind the embedding layer and before the classification dense layer and set it to 0.2 when solving IMDB and ChnSentiCorp-Htl models. The two hyperparameters (epsilon and temperature) are set to 0.01–0.06. The backpropagation algorithm based on the Adam stochastic optimization method is used to train the network through time, and the learning rates are 0.001 and 0.0001, which are used for different datasets. The normalization layer involves batch normalization and layer normalization. Each corpus is split into training and testing sets in the proportion of 8 : 2. All experiments are conducted with Python 3.7.9 and Keras 2.4.3 on GeForce RTX 2080 Ti and NVIDIA Tesla K80 systems. The specific parameters of the model are listed in Table10.Table 10
The specific parameters of PCRA-GP.
ComponentParameterWord embedding300CNN[3, 4, 5], [256, 128]BiGRU[256, 128]Batch size[256, 128, 64, 32]Dropout[0.2, 0.5]Dense128Epsilon[0.01, 0.02, 0.031, 0.04, 0.05, 0.06]Temperature[0.01, 0.02, 0.03, 0.04, 0.05, 0.06]Learning rate[0.001, 0.0001]NormalizationBatch normalization, layer normalizationPython3.7.9Keras2.4.3GPUGeForce RTX 2080 Ti, NVIDIA Tesla K80


## 4.1.1. Datasets

Our model is used to solve bilingual text classification tasks (including sentiment and news category) at the chapter level and paragraph level in English and Chinese (Table9). The datasets used in this article are as follows:(i)
IMDB: it is a dataset of 50,000 movie reviews in English, labeled by sentiment (positive/negative).(ii)
ChnSentiCorp-Htl (https://github.com/SophonPlus/ChineseNlpCorpus): it is a corpus including more than 7,000 hotel reviews in Chinese.(iii)
20news: The 20 Newsgroups (https://qwone.com/~jason/20Newsgroups/) dataset is a collection of approximately 20,000 English newsgroup documents, partitioned evenly across 20 different newsgroups. It has become a popular dataset for experiments in text applications of machine learning techniques, such as text classification and text clustering.(iv)
Cnews: It is a subset of the THUCnews (https://github.com/thunlp/THUCTC) dataset produced by NLP and Computational Social Science Lab, Tsinghua University. The THUCnews dataset mainly collects Sohu news, and Cnews contains 10 categories and 65,000 Chinese texts.Table 9
Dataset statistics.
CorpusClassesInstanceAvg lengthTrain/testLanguageIMDB250,00023125,000/25,000EnglishChnSentiCorp-Htl277671283000/3000Chinese20news2018,846221.2611,314/7532EnglishCnews1065,00053050,000/15,000Chinese


## 4.1.2. Baseline Methods

BALB: it is a backdoor attack against the LSTM-based text classification system [46].GCNN: it is a model that introduces contextual information obtained by the RNN and local information captured by the CNN and was proposed by Huang and Liu [31].AEG: it is a reinforcement-learning-based approach to generating adversarial examples in black-box settings [21].AC-BiLSTM: it is a novel and unified architecture that contains the bidirectional LSTM (BiLSTM), attention mechanism, and convolutional layer [34].BiGRU-CNN: A model composed of GloVe, CNN, RNN and max-pooling layer was proposed by Abid et al. [28].BAM: it is a bilevel attention model that does not rely on external knowledge and was proposed by Liu et al. [42].Text GCN: in this model, the first task is to build the whole corpus into a heterogeneous graph and use GNN to jointly learn word and document embeddings, as performed by Yao et al. [30].SGC: it is a method aimed to simplify single linear transformation and was proposed by Wu et al. [35].SSGC: this model is a variant model derived from Markov diffusion kernels, deals with degradation caused by the increased depth, and was proposed by Zhu and Koniusz [37].C-CNN: this model can realize centrality convolution based on a separate encoding function and was proposed by Dong et al. [26].BertGCN: it is a model that combines Bert’s large-scale pretraining knowledge and the transferability of the GCN and was proposed by Lin et al. [36].RoBERTaGCN: this method entails joint RoBERTa and GCN as a comparative experiment and was proposed by Lin et al. [36].SWEM: this method uses a modified hierarchical pooling strategy in few-shot transfer learning and was proposed by Pan et al. [41].C-BiGRU-ATT: this model uses the CNN, attention mechanism, and BiGRU, extracting contextual and local information at the character and vocabulary levels, and was proposed by Wenzhen et al. [33].ATCNN-3: this method extracts and filters features by adding attention mechanisms in different positions and was proposed by Tang and Yang. [32].


## 4.1.3. Parameter Settings

The PCRA-GP model uses accuracy as the evaluation metric to estimate the overall classification performance. The first layer of PCRA-GP is an embedding layer which converts words into 300-dimensional vectors. The kernels of multichannel CNNs are set to 3, 4, and 5. The number of hidden units of BiGRUs and CNNs is set to 256 which is suitable for document classification and 128 which is suitable for paragraph classification. The training batch sizes are set to 256, 128, and 64. The dropout rate is 0.5. To better avoid model overfitting, we add dropout behind the embedding layer and before the classification dense layer and set it to 0.2 when solving IMDB and ChnSentiCorp-Htl models. The two hyperparameters (epsilon and temperature) are set to 0.01–0.06. The backpropagation algorithm based on the Adam stochastic optimization method is used to train the network through time, and the learning rates are 0.001 and 0.0001, which are used for different datasets. The normalization layer involves batch normalization and layer normalization. Each corpus is split into training and testing sets in the proportion of 8 : 2. All experiments are conducted with Python 3.7.9 and Keras 2.4.3 on GeForce RTX 2080 Ti and NVIDIA Tesla K80 systems. The specific parameters of the model are listed in Table10.Table 10
The specific parameters of PCRA-GP.
ComponentParameterWord embedding300CNN[3, 4, 5], [256, 128]BiGRU[256, 128]Batch size[256, 128, 64, 32]Dropout[0.2, 0.5]Dense128Epsilon[0.01, 0.02, 0.031, 0.04, 0.05, 0.06]Temperature[0.01, 0.02, 0.03, 0.04, 0.05, 0.06]Learning rate[0.001, 0.0001]NormalizationBatch normalization, layer normalizationPython3.7.9Keras2.4.3GPUGeForce RTX 2080 Ti, NVIDIA Tesla K80


## 4.2. Experiments

### 4.2.1. Overall Comparison

In this section, all the experimental data are summarized and the best classification accuracy is selected for comparisons with the baseline model accuracy. The results recorded in Table11 involve the classification of sentiment and news texts, and the text languages are Chinese and English. The baseline results are all reported in the cited papers.Table 11
Accuracy of the models.
DatasetModelsAccuracy (%)IMDBBALB [46]84.92GCNN [31]90.0AEG [21]89.95PCRA (ours)90.26PCRA-GP (ours)90.34ChnSentiCorp-HtlAC-BiLSTM [34]90.60BiGRU-CNN [28]91.60BAM [42]90.0PCRA (ours)91.75PCRA-GP (ours)91.7520newsTextGCN [30]86.34SGC [35]88.50SSGC [37]88.60C-CNN [26]82.59BertGCN [36]89.3RoBERTaGCN [36]89.5PCRA (ours)88.42PCRA-GP (ours)89.86CnewsSWEMs [41]90.07C-BiGRU-ATT [33]95.55ATCNN-3 [32]96.58PCRA (ours)96.92PCRA-GP (ours)97.45Bold values are the best accuracy scores.The results for the IMDB dataset show that the PCRA is higher than GCNN by 0.26%, while the PCRA-GP is 0.34% higher. In comparison with backdoor attack models, our model classification accuracy improvement is significantly ahead of theirs by 5.42% and 5.34%. The black-box setting AEG model is better than backdoor attack models, but it is weaker than our model by 0.39% and 0.31%. The results indicate that perturbation on the model loss function is an effective way to improve accuracy.The models applied to the ChnsentiCorp-Htl dataset show that BiGRU-CNN lacks an attention layer compared to AC-BiLSTM, and AC-BiLSTM lacks projection compared to the PCRA model. The model using only bilevel attention returns the lowest result. Due to the small amount of data, the simple model BiGRU-CNN is better than the complex point model AC-BiLSTM, and the addition of the GP method results in the highest accuracy of classification. The results for the Cnews dataset indicate that PCRA and PCRA-GP are, respectively, 1.37% and 1.9% better than C-BiGRU-ATT that does not use a projector and GP components. Compared with ATCNN-3, PCRA is 0.34% higher after increasing feature extraction ability. Compared with SWEM, which is a few-shot model, the overall training method can maintain a high level of expressiveness. The results show that a projector under the SCL training process brings improved results with the GP constraint.Since TextGCN was proposed, GNN has demonstrated high-quality performance and gradually replaced the traditional deep model based on CNNs and RNNs. The results for the 20 Newsgroups dataset show that the classification accuracies of GNN-based models (TextGCN, SGC, and SSGC) are gradually improving. After fusing the pretrained Bert model, the performance of the GNN model is further improved. Our proposed model is 0.36% better than RoBERTaGCN and achieved SOTA results. The results indicate that our model improves the performance of traditional deep models based on the CNN, RNN, and attention mechanism.Combined with the performance of our proposed model for four public datasets, our results are found to be consistently better than the baseline results recorded in most published papers. In view of the above discussion, the PCRA-GP model that we proposed is highly competitive.


### 4.2.2. Effect of Components of PCRA-GP

In Figure6, the classification accuracies are plotted for all datasets, in which the temperature hyperparameter is set to 0.05 and comes from the experiment using the RTX 2080Ti system. Since the experimental results are sensitive to parameter settings, our experimental process mainly focuses on the type of normalization, batch size, epoch, and epsilon.Figure 6
The accuracy of datasets with the type of normalizations, epoch, and batch size on RTX2080 Ti. The symbolL/B_XXX_YYY means layer (L) and batch (B) normalizations, epsilon, epoch, and batch size. The PCRA column represents contrastive learning without the gradient penalty. (a) The PCRA-GP’s accuracy on IMDB, trained with L. (b) The PCRA-GP’s accuracy on IMDB, trained with B. (c) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with L. (d) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with B. (e) The PCRA-GP’s accuracy on 20news, trained with L. (f) The PCRA-GP’s accuracy on 20news, trained with B. (g) The PCRA-GP’s accuracy on Cnews, trained with L. (h) The PCRA-GP’s accuracy on Cnews, trained with B.
(a)(b)(c)(d)(e)(f)(g)(h)Overall, as shown in Figure6, the performance of the PCRA model is relatively stable and less affected. Batch normalization could withstand larger epsilon disturbances than layer normalization. When the batch size is set to 256, the model classification accuracy can be maintained in response to variations in other factors. The various epoch settings have little effect on the performance. When epsilon is greater than 0.04, the classification accuracy decreases significantly.In Table12, the maximum value is counted under different parameter settings, which ignores epsilon, and the data are selected from Figure 6. Different from the trend analysis of the data, we pay more attention to specific accuracy value information. By summarizing the classification results of all datasets, the layer normalization classification results are found to be 33 times higher than those of batch normalization among 48 groups, and the best accuracy occurrence ratio is 5 : 3. Moreover, eight locations, where the minimum value is obtained, are counted, five of them appear at a batch size of 64, and the epoch is 200. The maximum value appears 5 times when the batch size is 256.Table 12
Best results obtained from different parameters without epsilon. The minimum and maximum values are in bold in the table, and the maximum values are marked in red.In summary, batch normalization methods are stable, while layer normalization has high accuracy and often leads to the optimal classification. A large batch size can acquire good accuracy and offset the impact of different epochs. In terms of overall results, the result is best balanced when epsilon is 0.01, and the second best choice is 0.02.


### 4.2.3. Ablation Experiment

We tested the performance of different combinations, and the results are shown in Table13. The experimental hyperparameters are fixed and executed on RTX 2080Ti. We set the epoch size to 100, batch size to 256, and temperature to 0.05. Besides, epsilon is set to 0.01 during adversarial training. Considering PC as the baseline, the introduction of R reduces the classification effect and the integration of A brings a huge improvement in the results. The adversarial training with GP also improves the classification accuracy of the model. 20news is not well trained in methods PC and PCR. Additionally, we visualize the loss value and accuracy during the training process to understand the results. The training process data are shown in Figure 7. During the training stage of the encoder and classifier, the loss values of PC and PCR do not drop significantly, resulting in low accuracy. However, metrics for PCRA and PCRA-GP perform well. In addition, during the training of the encoder, the introduction of GP gives a delayed effect to the loss compared to PCRA. The results indicate that joint features enhance the performance of the model and that adversarial training is an effective way for the classification task.Table 13
Experimental results of different component combinations.
ModelIMDBChnSentiCorp-Htl20newsCnewsPC64.3567.338.9861.69PCR59.5757.587.0454.85PCRA89.4790.3388.3496.91PCRA-GP89.7389.3389.3396.83Bold values are the best accuracy scores.Figure 7
The training process data of different datasets. e_l means the encoder loss, c_l denotes the classifier loss, and acc means accuracy. (a) The training process data of IMDB. (b) The training process data of ChnSentiCorp-Htl. (c) The training process data of 20news. (d) The training process data of Cnews.
(a)(b)(c)(d)


### 4.2.4. Tuning of the Temperature Parameter

In addition to the changes brought about by the above hyperparameter changes, temperature is also a hyperparameter introduced by SimCLR [10], which is an adjustable value to help learn hard negatives well. Table 14 lists the accuracy under a change in temperature. Obviously, PCRA-GP is easier to be affected than PCRA, and only three results are improved by adjusting temperature. In the IMDB dataset, the PCRA method is improved by 0.12% at a temperature of 0.04. Besides, the PCRA method improved by 0.21% at a temperature of 0.02 in the Cnews dataset. In addition, only PCRA is higher than PCRA-GP in the Cnews dataset. The results indicate that the best value of temperature is 0.05.Table 14
Classification results under different temperature parameters. The temperature is adjusted for the maximum value in Table12, which is trained at a fixed temperature of 0.05.
TemperatureAccuracyIMDBChnSentiCorp-Htl20newsCnewsPCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)0.0151.4389.2359.4290.5012.9884.5566.3497.350.0249.5087.7188.8389.5812.3782.1068.9297.390.0350.1081.2789.1790.0813.0486.8656.4697.180.0488.8089.8488.9289.2588.1086.9196.5096.920.0589.9189.7291.4291.0889.1788.4897.2197.180.0689.5689.6589.8389.2587.7687.9797.2397.08Bold values are the best accuracy scores.


### 4.2.5. Model Training Efficiency

The training time is also recorded in Table15 to observe the effect of different components on the training time. The experimental tests were performed on RTX 2080Ti. The results show that R increases the training time per step. The attention mechanism reduces the training time and verifies the effect of automatic feature selection. However, the role of the attention mechanism fails for ChnSentiCorp-Htl. This is mainly due to the small size of the dataset and the short text length. Furthermore, the lower time reduction during the classification stage is due to the use of the frozen encoder. The results indicate that the training time can be reduced by the attention mechanism in the multilevel feature fusion strategy.Table 15
Model training time comparison (ms/step).
ModelIMDBChnSentiCorp-Htl20newsCnewsEncoderClassifierEncoderClassifierEncoderClassifierEncoderClassifierPC95.7727.9851.3811.94138.7135.78211.8758.18PCR99.1035.3957.3116.52138.2251.47215.8180.77PCRA94.3135.257.7016.8130.2447.01199.3477PCRA-GP94.7834.8149.2715131.1247213.3779.67Bold values are the most time-consuming results.


## 4.2.1. Overall Comparison

In this section, all the experimental data are summarized and the best classification accuracy is selected for comparisons with the baseline model accuracy. The results recorded in Table11 involve the classification of sentiment and news texts, and the text languages are Chinese and English. The baseline results are all reported in the cited papers.Table 11
Accuracy of the models.
DatasetModelsAccuracy (%)IMDBBALB [46]84.92GCNN [31]90.0AEG [21]89.95PCRA (ours)90.26PCRA-GP (ours)90.34ChnSentiCorp-HtlAC-BiLSTM [34]90.60BiGRU-CNN [28]91.60BAM [42]90.0PCRA (ours)91.75PCRA-GP (ours)91.7520newsTextGCN [30]86.34SGC [35]88.50SSGC [37]88.60C-CNN [26]82.59BertGCN [36]89.3RoBERTaGCN [36]89.5PCRA (ours)88.42PCRA-GP (ours)89.86CnewsSWEMs [41]90.07C-BiGRU-ATT [33]95.55ATCNN-3 [32]96.58PCRA (ours)96.92PCRA-GP (ours)97.45Bold values are the best accuracy scores.The results for the IMDB dataset show that the PCRA is higher than GCNN by 0.26%, while the PCRA-GP is 0.34% higher. In comparison with backdoor attack models, our model classification accuracy improvement is significantly ahead of theirs by 5.42% and 5.34%. The black-box setting AEG model is better than backdoor attack models, but it is weaker than our model by 0.39% and 0.31%. The results indicate that perturbation on the model loss function is an effective way to improve accuracy.The models applied to the ChnsentiCorp-Htl dataset show that BiGRU-CNN lacks an attention layer compared to AC-BiLSTM, and AC-BiLSTM lacks projection compared to the PCRA model. The model using only bilevel attention returns the lowest result. Due to the small amount of data, the simple model BiGRU-CNN is better than the complex point model AC-BiLSTM, and the addition of the GP method results in the highest accuracy of classification. The results for the Cnews dataset indicate that PCRA and PCRA-GP are, respectively, 1.37% and 1.9% better than C-BiGRU-ATT that does not use a projector and GP components. Compared with ATCNN-3, PCRA is 0.34% higher after increasing feature extraction ability. Compared with SWEM, which is a few-shot model, the overall training method can maintain a high level of expressiveness. The results show that a projector under the SCL training process brings improved results with the GP constraint.Since TextGCN was proposed, GNN has demonstrated high-quality performance and gradually replaced the traditional deep model based on CNNs and RNNs. The results for the 20 Newsgroups dataset show that the classification accuracies of GNN-based models (TextGCN, SGC, and SSGC) are gradually improving. After fusing the pretrained Bert model, the performance of the GNN model is further improved. Our proposed model is 0.36% better than RoBERTaGCN and achieved SOTA results. The results indicate that our model improves the performance of traditional deep models based on the CNN, RNN, and attention mechanism.Combined with the performance of our proposed model for four public datasets, our results are found to be consistently better than the baseline results recorded in most published papers. In view of the above discussion, the PCRA-GP model that we proposed is highly competitive.


## 4.2.2. Effect of Components of PCRA-GP

In Figure6, the classification accuracies are plotted for all datasets, in which the temperature hyperparameter is set to 0.05 and comes from the experiment using the RTX 2080Ti system. Since the experimental results are sensitive to parameter settings, our experimental process mainly focuses on the type of normalization, batch size, epoch, and epsilon.Figure 6
The accuracy of datasets with the type of normalizations, epoch, and batch size on RTX2080 Ti. The symbolL/B_XXX_YYY means layer (L) and batch (B) normalizations, epsilon, epoch, and batch size. The PCRA column represents contrastive learning without the gradient penalty. (a) The PCRA-GP’s accuracy on IMDB, trained with L. (b) The PCRA-GP’s accuracy on IMDB, trained with B. (c) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with L. (d) The PCRA-GP’s accuracy on ChnSentiCorp-Htl, trained with B. (e) The PCRA-GP’s accuracy on 20news, trained with L. (f) The PCRA-GP’s accuracy on 20news, trained with B. (g) The PCRA-GP’s accuracy on Cnews, trained with L. (h) The PCRA-GP’s accuracy on Cnews, trained with B.
(a)(b)(c)(d)(e)(f)(g)(h)Overall, as shown in Figure6, the performance of the PCRA model is relatively stable and less affected. Batch normalization could withstand larger epsilon disturbances than layer normalization. When the batch size is set to 256, the model classification accuracy can be maintained in response to variations in other factors. The various epoch settings have little effect on the performance. When epsilon is greater than 0.04, the classification accuracy decreases significantly.In Table12, the maximum value is counted under different parameter settings, which ignores epsilon, and the data are selected from Figure 6. Different from the trend analysis of the data, we pay more attention to specific accuracy value information. By summarizing the classification results of all datasets, the layer normalization classification results are found to be 33 times higher than those of batch normalization among 48 groups, and the best accuracy occurrence ratio is 5 : 3. Moreover, eight locations, where the minimum value is obtained, are counted, five of them appear at a batch size of 64, and the epoch is 200. The maximum value appears 5 times when the batch size is 256.Table 12
Best results obtained from different parameters without epsilon. The minimum and maximum values are in bold in the table, and the maximum values are marked in red.In summary, batch normalization methods are stable, while layer normalization has high accuracy and often leads to the optimal classification. A large batch size can acquire good accuracy and offset the impact of different epochs. In terms of overall results, the result is best balanced when epsilon is 0.01, and the second best choice is 0.02.


## 4.2.3. Ablation Experiment

We tested the performance of different combinations, and the results are shown in Table13. The experimental hyperparameters are fixed and executed on RTX 2080Ti. We set the epoch size to 100, batch size to 256, and temperature to 0.05. Besides, epsilon is set to 0.01 during adversarial training. Considering PC as the baseline, the introduction of R reduces the classification effect and the integration of A brings a huge improvement in the results. The adversarial training with GP also improves the classification accuracy of the model. 20news is not well trained in methods PC and PCR. Additionally, we visualize the loss value and accuracy during the training process to understand the results. The training process data are shown in Figure 7. During the training stage of the encoder and classifier, the loss values of PC and PCR do not drop significantly, resulting in low accuracy. However, metrics for PCRA and PCRA-GP perform well. In addition, during the training of the encoder, the introduction of GP gives a delayed effect to the loss compared to PCRA. The results indicate that joint features enhance the performance of the model and that adversarial training is an effective way for the classification task.Table 13
Experimental results of different component combinations.
ModelIMDBChnSentiCorp-Htl20newsCnewsPC64.3567.338.9861.69PCR59.5757.587.0454.85PCRA89.4790.3388.3496.91PCRA-GP89.7389.3389.3396.83Bold values are the best accuracy scores.Figure 7
The training process data of different datasets. e_l means the encoder loss, c_l denotes the classifier loss, and acc means accuracy. (a) The training process data of IMDB. (b) The training process data of ChnSentiCorp-Htl. (c) The training process data of 20news. (d) The training process data of Cnews.
(a)(b)(c)(d)


## 4.2.4. Tuning of the Temperature Parameter

In addition to the changes brought about by the above hyperparameter changes, temperature is also a hyperparameter introduced by SimCLR [10], which is an adjustable value to help learn hard negatives well. Table 14 lists the accuracy under a change in temperature. Obviously, PCRA-GP is easier to be affected than PCRA, and only three results are improved by adjusting temperature. In the IMDB dataset, the PCRA method is improved by 0.12% at a temperature of 0.04. Besides, the PCRA method improved by 0.21% at a temperature of 0.02 in the Cnews dataset. In addition, only PCRA is higher than PCRA-GP in the Cnews dataset. The results indicate that the best value of temperature is 0.05.Table 14
Classification results under different temperature parameters. The temperature is adjusted for the maximum value in Table12, which is trained at a fixed temperature of 0.05.
TemperatureAccuracyIMDBChnSentiCorp-Htl20newsCnewsPCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)PCRA-GP (%)PCRA (%)0.0151.4389.2359.4290.5012.9884.5566.3497.350.0249.5087.7188.8389.5812.3782.1068.9297.390.0350.1081.2789.1790.0813.0486.8656.4697.180.0488.8089.8488.9289.2588.1086.9196.5096.920.0589.9189.7291.4291.0889.1788.4897.2197.180.0689.5689.6589.8389.2587.7687.9797.2397.08Bold values are the best accuracy scores.


## 4.2.5. Model Training Efficiency

The training time is also recorded in Table15 to observe the effect of different components on the training time. The experimental tests were performed on RTX 2080Ti. The results show that R increases the training time per step. The attention mechanism reduces the training time and verifies the effect of automatic feature selection. However, the role of the attention mechanism fails for ChnSentiCorp-Htl. This is mainly due to the small size of the dataset and the short text length. Furthermore, the lower time reduction during the classification stage is due to the use of the frozen encoder. The results indicate that the training time can be reduced by the attention mechanism in the multilevel feature fusion strategy.Table 15
Model training time comparison (ms/step).
ModelIMDBChnSentiCorp-Htl20newsCnewsEncoderClassifierEncoderClassifierEncoderClassifierEncoderClassifierPC95.7727.9851.3811.94138.7135.78211.8758.18PCR99.1035.3957.3116.52138.2251.47215.8180.77PCRA94.3135.257.7016.8130.2447.01199.3477PCRA-GP94.7834.8149.2715131.1247213.3779.67Bold values are the most time-consuming results.


## 5. Conclusion

In this research, PCRA-GP that takes the best advantages from both feature extraction and representation is proposed for text classification. First, the CNN, BiGRU, and attention mechanism are combined to obtain text features. Second, contrastive learning is used to train a projector which enhances feature vector embedding. Finally, a gradient penalty is used to generate feature vector disturbance by improving the robustness of PCRA. Experiments are performed on Chinese and English texts, involving sentiment classification and news text classification. The comparisons with some state-of-the-art baseline models demonstrate that the PCRA-GP model is more effective, efficient, and adaptable in terms of the classification quality and domain in most cases. In subsequent research, we will optimize the details of the model. The feature extractor will be adjusted to reduce the dimensionality appropriately to accelerate training speed. In addition, this work will be useful in other domain classification tasks.



---

*Source: 7386888-2023-02-20.xml*