Enhancing Transformer-based Semantic Matching for Few-shot Learning through Weakly Contrastive Pre-training

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 13:51:51

Abstract: The task of semantic text matching focuses on measuring the semantic similarity between two distinct texts and is widely applied in search and ranking scenarios. In recent years, pre-trained models based on the Transformer architecture have demonstrated powerful semantic representation capabilities and have become the mainstream method for text representation. The pipeline of fine-tuning pre-trained language models on downstream semantic matching tasks has achieved promising results and widespread adoption. However, practical downstream scenarios often face severe challenges in terms of data quality and quantity. Ensuring high-quality and large quantities of samples is often difficult. Current research on enhancing pre-trained models for few-shot semantic text matching tasks is still not advanced enough. Therefore, this paper focuses on providing a general enhancement scheme for few-shot semantic text matching tasks. Specifically, we propose an Enhanced Transformer-based Semantic Matching method for few-shot learning through weakly contrastive pre-training, which is named as EBSIM. Firstly, considering the characteristics of semantic text matching tasks, we design a simple and cost-effective data augmentation method for constructing weakly supervised samples. Then, we design a contrastive learning objective based on alignment-aspect to achieve effective semantic matching by optimizing the bidirectional semantic perception between constructed texts. We conduct comprehensive experiments on five Chinese and English semantic text matching datasets using various Transformer-based pre-trained models. The experimental results confirm that our proposed method significantly improves the model's performance on semantic text matching tasks. Further ablation experiments and case studies validate the effectiveness of our approach. Our code and data will be made publicly available at a later stage.

Main paper: 
 introduction: 
 The semantic text similarity (STS) [20, 31, 40] task measures the semantic similarity between texts. It is a fundamental task in natural language processing (NLP). And it plays an essential role in various fields, such as information retrieval and machine translation evaluation. Plenty of methods have been proposed for the task, including traditional feature engineering techniques, hybrid approaches, and purely neural architectures [6, 33, 46, 54, 61] .Recently, pre-trained language models (PLMs) have led to impressive performance gains across various NLP tasks, such as sentiment classification [18, 52] , natural language inference [10, 25] , named entity recognition [27, 43] and so on. In detail, people generally load the PLM such as BERT [16] , RoBERTa [38] , then fine-tune it on the specific downstream task to obtain better performances. In this way, PLMs can be fine-tuned with just one additional output layer to create state-of-the-art models.Despite the numerous research efforts that have introduced various pre-trained language models (PLMs) to enhance the performance of semantic matching tasks, there remain several critical issues that have not been effectively addressed. Firstly, there exists a natural gap between the learning objectives during the pre-training stage and those during the fine-tuning stage. The pre-training phase primarily focuses on language modeling aspects, such as the masked language model (MLM) in BERT, dynamic masking in RoBERTa, or sentence ordering relations modeling, exemplified by tasks like next sentence prediction (NSP) in BERT and sentence order prediction (SOP) in ALBERT [32] . However, semantic matching tasks during the fine-tuning stage are primarily concerned with learning semantic similarity between texts. The significant differences in learning objectives can easily constrain the performance of PLMs on downstream tasks. Secondly, obtaining high-quality and high-quantity downstream task datasets for fine-tuning presents a significant challenge. The most critical factor affecting fine-tuning model effectiveness is the quality of training samples. Limited, noisy, and low-quality training samples can hinder model convergence and potentially disrupt the general language knowledge acquired during pre-training. The question of how to design effective pre-trained language models to endow them with strong text representation capabilities in few-shot learning scenarios is a pivotal issue.In addition, semantic matching tasks place a greater emphasis on a model's ability to judge the semantic similarity between texts, as opposed to textual similarity judgments. Currently, there is limited research focused on designing pre-training models that are more tailored to downstream semantic matching tasks. To enhance a model's ability to recognize semantic consistency in textual content, a recent study introduced an approach involving the inclusion of sentiment recognition auxiliary tasks during the pre-training phase. Given the ubiquity and importance of semantic matching tasks, the development of pre-training methods specifically tailored to such tasks is a significant and valuable endeavor. Furthermore, when confronted with situations where two texts share the same synonyms, models can easily make inaccurate semantic judgments due to the presence of identical tokens. Therefore, devising effective pre-training strategies to mitigate a model's overreliance on synonyms and enhance its capacity for learning semantic structures is also a crucial consideration.To address the aforementioned issues, this paper focuses on providing a general enhancement pre-training approach tailored to semantic text matching tasks with limited samples. Specifically, we propose an Enhanced Transformer-based Semantic Matching method for Few-shot Learning through Weakly Contrastive Pretraining, which is named as EBSIM. Firstly, to improve the alignment between the learning objectives in the pre-training and finetuning stages, we design a weakly supervised dataset for semantic text matching tasks based on general corpora. Considering the need to align the enhanced samples and training objectives with the characteristics of semantic text matching, we devise a triplet data augmentation method based on both text token-level and sentence structure-level. Particularly, we leverage high-quality article headlines from general corpora, such as Wikipedia, as anchors. We select relevant samples from the articles based on the similarity at the token level to construct pairs of samples. Given that models can be easily perturbed by similar sentence structures in matching tasks, we adopt a sentence structure-level approach by masking synonymous entity words to preserve the sentence structures in the text. This process helps construct pairs of samples.Next, to effectively leverage the information from the weakly supervised dataset for consistent learning between pre-training and fine-tuning, we introduce a multi-aspect contrastive pre-training enhancement approach. We approach this from a relation-aspect perspective, utilizing the global semantic space obtained through Transformer encoding of text. We construct contrastive learning objectives based on the semantic representations of text pairs. Subsequently, we design contrastive learning objectives based on Alignment-Aspect, facilitating bidirectional semantic alignment between texts to achieve effective semantic matching. By enhancing Transformer-based pre-training models within this adaptable framework, we achieve improved performance in semantic text matching tasks. During the fine-tuning stage, there is no need to introduce additional network structures or modify pre-trained models; training and predictions can be directly performed based on downstream datasets. The whole pipeline of EBSIM is given in Figure 1 .The major contributions can be summarized as follows:• We have designed a low cost and efficient Weakly-Supervised Data Augmentation method. From both text token-level and sentence structure-level perspectives, we have constructed triplets composed of anchor text, positive text, and negative text as augmented dataset. This dataset has been validated to significantly enhance the performance of pre-trained models.We intend to make this dataset publicly available. and English semantic matching datasets using multiple pretrained language models. The experimental results confirm that our enhanced pre-training method significantly improves the model's performance on semantic matching tasks.Our source code and data will be publicly released. 
related work 2.1 semantic text similarity methods: 
 Generally, deep semantic text similarity (STS) modeling includes two kinds of methods: interaction-based method and representationbased method. The former usually constructs feature interactions between two sentences, and further obtains the matching result [13, 19, 64] , which obtain better performances with interaction mechanisms. The latter usually learns a representation and calculates the similarity score through a deep learning model [1, 9, 34] .In recent years, STS has rapidly developed with the help of pretraining models. Based on pre-trained BERT models, two kinds of fine-tuned architecture are typically used: bi-encoders and crossencoders. A lot of work aims to balance these two methods in terms of efficiency and performance [24, 28, 41, 47] . Different from the above works, we focus on enhancing pre-training for STS tasks. We utilize the cross-encoders fine-tuning method, and this stage is identical to the fine-tuning of the conventional PLMs.PLMs can capture rich language information from the text and benefit many applications by fine-tuning [45] . Previous PLMs focus on the general language model, most of which are following BERT [16] . BERT-wwm [15] adapts whole word masking in Chinese text, masking the whole word instead of masking Chinese characters. RoBERTa [38] is an improved version of BERT, including the use of dynamic masking and more training data, while removing the next sentence prediction.More related to our work is the enhancement of pre-training for specific tasks. MWA [35] enhances pre-trained Chinese character representation with the word-aligned attention. SKEP [53] introduces sentiment knowledge enhanced pre-training to learn a unified sentiment representation. To better model dialogue behaviors, TOD-BERT [58] proposes a contrastive objective function to simulate the response selection task. TaskPT [21] adds a taskguided pre-training stage with selective masking to capture the domain-specific and task-specific patterns. Similar but different to these works, we are the first to enhance pre-training for the STS tasks. There has not been the same job before.In recent years, contrastive learning plays an important role in self-supervised learning for computer vision, natural language processing, and other domains [26, 56, 65] . Contrast visual representation learning focuses on generating effective visual representations [23, 42] . Recent methods have produced results comparable to the state-of-the-art supervised method on some tasks [4, 11, 22] . Moreover, some previous works solve specific tasks through contrastive learning. LS-Score [59] evaluates the quality of abstracts through unsupervised contrastive learning. GCL [3] solves the neural dialogue generation with group-wise contrastive learning.More related to our work is the language representation learning. Contrastive learning assumes that observed pairs of text are more semantically similar than randomly sampled text [2] . Some methods try to model the language representation learning with the idea of contrastive learning [12, 30, 60] . Unlike these previous works, we construct sentence pairs and design the contrast objectives to enhance the sentence similarity relation modeling. 
enhancing pre-training for semantic text similarity: 
 This section will describe the proposed method in detail. Section 3.1 formalizes STS task and the pipeline of our method, followed by weakly-supervised text similarity corpus construction in Section 3.2. Finally, contrastive learning objectives are described in Section 3.3.In this section, we formalize the semantic text similarity (STS) task and describe the pipeline of EBSIM. Given a data set D = {(𝑠 𝑎1 , 𝑠 𝑏1 ), (𝑠 𝑎2 , 𝑠 𝑏2 ), ..., (𝑠 𝑎𝑛 , 𝑠 𝑏𝑛 )}, (𝑛 ∈ 𝑁 ), we aim to learn a similarity model M from D. Thus, M (𝑠 𝑎 , 𝑠 𝑏 ) measures the similarity degree between 𝑠 𝑎 and 𝑠 𝑏 for any new sentence pair (𝑠 𝑎 , 𝑠 𝑏 ), where the similarity degree is a regression score or classification label. In BERT-based semantic text similarity measurement, we use the embedding of '[CLS]' tokens to predict similarity labels. The pipeline we propose for enhanced pre-training in semantic matching mainly consists of three stages, as outlined in Figure 1 .As previously analyzed, there exists a natural gap between pretraining learning objectives like the masked language model, sentence order prediction, and semantic matching objectives. To enhance the consistency between the pre-training and fine-tuning stages, optimization is required from both the pre-training data and pre-training methods. A critical challenge is to design effective data augmentation and training methods that align with the characteristics of semantic matching tasks. This allows models to capture semantic relationships between texts from the augmented data and accurately recognize semantics through enhanced training methods.In particular, semantic similarity primarily focuses on the semantic connections between texts or concepts rather than surface similarity. Models in semantic similarity tasks can easily be misled by two types of similar schemes, namely token-level similarity and sentence structure-level similarity. Token-level similarity refers to situations where two texts share similar or overlapping words in their textual forms. In such cases, models may mistakenly assume that tokens with similar forms also share similar semantics. Sentence structure-level similarity pertains to situations where two texts have similar sentence structures or expressions. In these cases, models may incorrectly assume that texts with similar structures also convey similar semantics. Therefore, we design two data enhancement methods to construct weakly supervised data sets from the perspective of enhancing the model's ability to recognize these two schemes. Subsequently, we guide the model's learning through finely designed enhancement pre-training objectives that correspond to these two schemes.Token-level Similarity Scheme. In order to construct samples with token-level similarity for enhancing pre-training, a costeffective data augmentation method needs to be designed. Given that titles and sentence contents in chapter-level corpora simultaneously possess rich semantic and token similarity relationships, we build samples based on general pre-training corpora such as Wikipedia.For each article, including the title 𝑡 and the content set of sentences 𝑆, we consider retrieving related sentences through its title. We encode the title 𝑡 and all sentences in content {𝑠 1 , 𝑠 2 , ..., 𝑠 𝐾 } ∈ 𝑆 through a sentence encoder, where 𝐾 is the number of sentences in the current article content. Then we calculate the similarity and sort these sentence vectors {𝑣 1 , 𝑣 2 , ..., 𝑣 𝐾 } ∈ 𝑉 using cosine similarity:EQUATIONwhere 𝑣 𝑎 and 𝑣 𝑏 are sentence vectors, and 𝜖 is a small value to avoid division by zero. In our work, we obtain the sentence vector by computing the mean of all output vectors of PLMs.After sorting, we can get relevant candidates. These relevant candidates can be combined with the title to form sentence pairs. The order of candidates represents the similarity order of sentencepairs:           < title, relevant 1 > < title, relevant 2 > ... < title, relevant 𝐾 > (2)where < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 > is more similar than < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑥 > (1 < 𝑥 ≤ 𝐾). As a result, we can construct the weakly-supervised sentence triples < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 , 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑥 >. to mask synonym entities in the text while preserving other tokens, thereby retaining similar sentence patterns and structures. Since the < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 > pairs obtained earlier are of high quality, we directly perform data augmentation on such sample pairs. Specifically, we calculate the word similarity between the title and relevant1 based on the text representation vectors. Then, we mask words in 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 with a similarity greater than a certain threshold, resulting in a new text 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 (𝑑𝑟𝑜𝑝 ) 1, with similar sentence structures. The function for constructing similar sentence structures can be represented as follows:EQUATION)where 𝑠 𝑖,𝑗 represents the 𝑗-th word in the sentence 𝑠 𝑖 , which is expressed as 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 above. 𝑠 ′ 𝑖,𝑗 denotes the generated word by structure similar function, which is expressed as 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 (𝑑𝑟𝑜𝑝 ) 1 above.[𝑀𝐴𝑆𝐾] token represents a meaningless placeholder token, just like in other general pre-trained language models. 𝑠𝑖𝑚(𝑠 𝑖,𝑗 , * ) denotes the similarity between the 𝑗-th word in 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 and the word in 𝑡𝑖𝑡𝑙𝑒 calculated by cosine similarity. 𝜂 indicates the similarity threshold used as a hyperparameter to control the level of masking.Based on sentence pair < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 >, we can use the similar sentence structure component method above to get the second kind of weakly supervised augmentation sample as follows:EQUATIONThis kind of sentence triples makes the model face the challenge of similar sentence structure during training, so as to help the model recognize semantically similar information more accurately. It is worth mentioning that for < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 >, if there are no synonyms that exceed the threshold, no triples will be constructed. Therefore, for each document, 1-2 triples are constructed.Compared to the labeled high-quality similar sentences, the similarity labels of weakly-supervised sentence pairs above are relatively weak in reliability. As a result, it can't be used directly for optimizing the classification objective. Inspired by contrastive learning's success, we find the order of similarity is valuable and trustworthy. Therefore, we introduce the comparative prediction task, which is helpful for similarity relation learning.Given the input sentence triple < 𝑡, 𝑠 𝑝 , 𝑠 𝑛 >, where 𝑠 𝑝 is more similar to 𝑡 than 𝑠 𝑛 . We want to learn a classifying function with the ability to identify 𝑠 𝑝 is more similar than 𝑠 𝑛 . A more common practice is Triplet Loss [49] , which encourages a large margin between the similarity of the anchor 𝑡 and its positive 𝑠 𝑝 and the similarity of the anchor and negative 𝑠 𝑛 . Mathematically, we minimize the following loss function:EQUATIONwhere 𝜉 is the similarity measurement function. Margin 𝛽 ensures that 𝑠 𝑝 is at least closer to 𝑡 with gap 𝛽 than 𝑠 𝑛 , and we set 𝛽 = 1 in our experiments.Consequently, our goal is to define the appropriate function 𝜉, which can better model the semantic similarity relation between sentences. Our proposed contrastive objectives are committed to multiple mapping functions.In the process of BERT-based semantic text similarity, we first concatenate two sentences to the required format. '[SEP]' is adopted to concatenate two sentences, and '[CLS]' is added at beginning. In detail, we concatenate a sentence pair < 𝑠 𝑎 , 𝑠 𝑏 > and fed them into the PLM. With the help of semantic knowledge of PLM, the hidden states are calculated as follows:EQUATIONwhere ℎ 𝑐𝑙𝑠 denotes the representation of the first token '[CLS]'. 𝐻 𝑠 𝑎 :𝑠 𝑏 represents the sequence representation of the whole input.In this paper, we aim to make full use of ℎ 𝑐𝑙𝑠 and 𝐻 𝑠 𝑎 :𝑠 𝑏 to model the semantic similarity relation between 𝑠 𝑎 and 𝑠 𝑏 via multi-aspect contrast.' is regarded as the global representation of semantic relations, which contains the current sentence relation information and serves as the input for the classification layer in the fine-tuning stage. As a result, our first object is to model the similarity of sentences in the relation aspect. Through a distance transformation matrix, we simulate to calculate the distance of sentence pairs:EQUATIONwhere𝑊 𝑑 , 𝑏 𝑑 are learnable parameters and 𝜎 is the sigmoid function.Finally, we redefine the mapping function as the similarity function of the triple loss function:EQUATIONThe Relation-Aspect (RA) contrastive objective is:EQUATION3.3.2 Alignment-Aspect Contrast. The relation-aspect contrast focuses on the relationship between sentence pairs based on BERTs with multi-head self-attention. However, for each concatenated sentence pair, the global relation-aspect encoding relies mainly on the whole tokens' self-attention mechanism. There lacks of attention to local sentence alignment, which leads to insufficient interaction between the two sentences. A simple form of alignment based on the attention mechanism is used following [61] with minor modification to help the BERT-based STS. In this subsection, we leverage the alignment mechanism and propose an alignment-aspect contrast strategy to enhance interaction between sentences. The alignment component takes features from 𝐻 𝑠 𝑎 :𝑠 𝑏 as input and computes the aligned representations as output. Input from the first sentence 𝑠 𝑎 of length 𝑙 𝑎 is denoted as 𝑠 𝑎 = (𝑎 1 , 𝑎 2 , ..., 𝑎 𝑙 𝑎 ) and input from the second sentence 𝑠 𝑏 of length 𝑙 𝑏 is denoted as 𝑠 𝑏 = (𝑏 1 , 𝑏 2 , ..., 𝑏 𝑙 𝑏 ). The output representation 𝑎 EQUATIONThe fusion layer compares raw token and aligned representations from two perspectives (• and -). The fusion result of sentence 𝑠 𝑎 is computed by:EQUATIONwhere 𝑊 𝑓 , 𝑏 𝑓 are learnable parameters, and • denotes elementwise multiplication. The subtraction operator highlights the difference between the two vectors while the multiplication highlights similarity. Then, we obtain 𝑠 𝑓 𝑎 = MaxPooling(𝐴 𝑓 ) and 𝑠 𝑓 𝑏 omitted here. We simulate to calculate the similarity confidence of sentence pairs based on the alignment mechanism:EQUATIONThe Alignment-Aspect (AA) contrastive objective is:EQUATIONThe overall objective function is:EQUATIONIn the fine-tuning phase, our primary focus is on training and testing the model for semantic text matching tasks. To simplify, we consider the semantic text matching task as determining whether two texts have similar semantic information, which can be defined as a classification problem. If two texts are semantically similar, the corresponding label is set to 1. If two texts do not share semantic similarity, the corresponding label is set to 0. The model's task is to predict the probability of semantic similarity between the two input texts.Based on the encoded [CLS] representation ℎ 𝑐𝑙𝑠 , we can obtain the semantic similarity probability as:EQUATION)where 𝑔 represents the similarity function, which is replaced by the inner product. Let 𝑦 𝑖 ∈ {0, 1} represent the real label of the sample.The cross entropy loss function can be expressed as:EQUATIONwhere 𝑦 𝑖 represents the label of the sample. 𝑁 denotes the number of training samples. 
experiments: 
 In this section, we conduct extensive experiments to answer the following questions:• RQ1 What kind of effect does our proposed EBSIM enhancement pretraining framework have on pretrained language models in semantic matching tasks? • RQ2 Can our proposed EBSIM improve the effectiveness of few shot learning on downstream semantic matching tasks? 1 . BQ is the largest manually annotated Chinese public corpus in the bank domain [7] . LCQMC is a large-scale Chinese question matching corpus [36] . Followed with Chen et al. [7] , we mainly report Accuracy scores for BQ and LCQMC. MRPC is a corpus of sentence pairs automatically extracted from online news sources [17] . QQP is a collection of question pairs from the community question-answering website Quora 1 . STS-B consists of English datasets used in the semantic textual similarity tasks, which are organized in the context of SemEval between 2012 and 2017 [5] . We follow common practice and report both Pearson and Spearman scores. Following Wang et al. [57] , we use the training set, validation set and test set corresponding to the dataset. Because the classes in MRPC(68% positive) and QQP(37% negative) are imbalanced, we follow common practice and report both Accuracy and F1 scores.All the experiments are executed on the base model (12 layers, hidden size 768), including BERT-base, BERT-wwm-ext-base, RoBERTa-base, and so on. All the experiments are tested on five random seeds. We report the average score to evaluate the performance of these models. Corpus Construction. During constructing the weakly-supervised corpus, we use BERT 𝑏𝑎𝑠𝑒 as a encoder to obtain the sentence embedding. The document-level corpus include English corpus [55] and Chinese corpus 2 . We construct weakly-supervised sentence triples < 𝑡𝑖𝑡𝑙𝑒, 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 1 , 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 2 > as described in Section 3.2. During the synonyms dropping, the embedding size is set to 200. For Chinese corpus, we initialize the Chinese word embedding from Tencent Embedding Corpus 3 . For English corpus, word2vec [39] is initialized randomly and fine-tuned on our English corpus. The similarity threshold for word pairs to be masked is 0.8. The number of masked words does not exceed 50% of the entire sentence. For all experiments, the number of documents is 100w, except Table 4 uses different scales (200,000 and 500,000). After the weakly supervised dataset construction as described in 3.2, the numbers of final 2 : The Accuracy(%) and F1(%) for semantic text similarity on the MRPC, QQP and STS-B datasets.triples: 1,750,000(English), 1,750,000(Chinese). These final triples include 1,000,000 triples constructed through retrieval and others constructed by synonyms deletion.Pre-Training and Fine-Tuning Configuration. The pre-training is based on the initialization parameters of existing PLMs such as BERT, RoBERTa. During further pre-training, We use Adam optimizer [29] . The learning rate is set to 5e-6, and the batch size is 32. We choose the best pre-trained model for the fine-tuning according to the dev set, including 20,000 samples. This fine-tuning stage remains unchanged. For the hyper-parameters of fine-tuning, we refer to the followed existing works [15, 16, 37, 51] and explore the most suitable ones. The details of fine-tuning for benchmarks are as Table 1 .To evaluate the effectiveness of our approach, we compare the model with the following baselines: ERNIE2.0 [51] , R 2 -Net [63] , tBERT [44] , DistilBERT [48] , XLNET [62] , ALBERT [32] , BiMPM [57] , BERT-wwm [15] , BERT-wwm-ext [15] , RoBERTawwm-ext [15] , MacBERT [14] , ERNIE [50] , ERNIE2.0 [51] , GMN-BERT [8] , BERT [16] , BERT-wwm-ext [15] , RoBERTa [38] .In order to achieve fair and objective performance evaluation, we conducted enhanced pretraining experiments on three representative Transformer-based models, namely BERT, BERT-wwm-ext [15] , and RoBERTa [38] , on the aforementioned five Chinese and English datasets. The experimental results for the Chinese datasets are presented in Table 2 , while the results for the English datasets can be found in Table 3 . Based on the experimental comparisons, we made the following observations:• For the Chinese semantic matching tasks, BQ and LCQMC, our proposed enhanced pretraining method, EBSIM, resulted in significant performance improvements. Specifically, for the BQ dataset, the enhanced BERT-base model achieved an accuracy improvement of 0.5%, the BERT-wwm-ext model improved by 0.69%, and the RoBERTa model improved by 0.93%. Due to the relatively large sample size of the LCQMC dataset, the improvement range for the three pretrained language models ranged from 0.36% to 0.63%. The experimental results effectively confirm the effectiveness of our weakly supervised contrastive enhancement pretraining method. It's worth noting that our weakly supervised dataset was constructed based on the original training data of these pretrained language models. • For the English semantic matching tasks, MRPC, QQP, and STS-B, our method also demonstrated superior performance. Specifically, for the high-quality STS-B dataset, the enhanced BERT-base model achieved a 1.68% improvement in Pearson correlation, while the enhanced RoBERTa model improved by 0.58%. For the MRPC and QQP datasets, the accuracy improvement of the enhanced pretrained language models ranged from 0.22% to 0.69%. Whether in tasks with larger sample sizes or smaller semantic matching tasks, our enhanced pretraining strategy provided noticeable benefits to the models. Model BQ LCQMC Previous Systems BiMPM [57] 81.9 83.4 BERT-wwm [15] 85.2 87.0 BERT-wwm-ext [15] 85.3 87.1 RoBERTa-wwm-ext [15] 85.0 86.4 MacBERT [14] 85.2 87.0 ERNIE [50] 84.8 87.4 ERNIE2.0 [51] 85.0 87.9 GMN-BERT [8] 85.6 87.3 Our Implementation BERT 𝑏𝑎𝑠𝑒 [16] 84.87 87.05 + EBSIM 85.37 87.47 BERT-wwm-ext [15] 84.95 87.02 + EBSIM 85.64 87.38 RoBERTa 𝑏𝑎𝑠𝑒 [38] 84.42 86.59 + EBSIM 85.35 87.22 Table 3 : The Accuracy(%) for semantic text similarity on the BQ and LCQMC datasets.One of the research objectives of this study is to enhance pretrained language models for semantic matching, so that they can maintain strong representational capabilities in the downstream few-shot learning scenario, where the training samples are limited. For the QQP dataset with 390K training samples, we conducted experiments with subsets of the original training set containing 20%, 40%, 60%, 80%, and 100% of the samples for fine-tuning, followed by testing the model's performance on the original test set. The model's performance under different proportions of training samples is presented in Figure 2 . Additionally, to investigate whether our proposed method also helps with the convergence of models on semantic matching tasks, we calculated the model's performance on the test set based on different training steps, as shown in Figure 3 .• Visually, it can be observed that our method not only outperforms the original BERT model when using 100% of the training samples but also maintains a stable advantage even when using only 60% or 20% of the training samples. This indicates that our proposed enhanced pretraining method indeed effectively aids the model in accurately recognizing semantic similarity, regardless of the scale of the training sample. • It is evident that our proposed method achieves significantly higher performance when using a limited number of training samples. Particularly, the model enhanced by EBSIM achieves better performance with only 60% of the training samples compared to the original pretrained model using 100% of the training samples. In other words, our proposed method can directly improve the few-shot learning performance on consistent downstream tasks solely through effective enhancement in the pretraining phase. • Based on the test results corresponding to different training steps shown in Figure 3 , EBSIM accelerates the model's convergence during the fine-tuning phase. In the initial 5000 steps, the model enhanced by EBSIM already exhibits significantly better performance than the original pretrained language model.In order to validate the effectiveness of our contrastive pretraining optimization objectives based on relation-aspect (RA) and Alignment-Aspect (AA), we conducted ablation experiments. The experimental results are presented in Table 5 , where we removed each of the two contrastive learning objectives separately. The results show a significant drop in performance on the BQ and MRPC datasets when we removed both contrastive learning objectives. This indicates that our adopted weakly supervised training method can indeed help the model improve its judgment of semantic similarity. Further observation reveals that RA has a greater impact on the model's performance compared to AA. This is because RA primarily utilizes global multi-head attention to perceive semantic tokens, which inherently contains global semantic information between the two texts. On the other hand, the introduction of AA mainly helps the model align more fine-grained token information between the two texts, compensating for the insufficient perception of some local tokens caused by global attention. In fact, since our weakly supervised samples were constructed while considering the characteristics of semantic matching tasks, our weakly supervised training objectives capture the modeling approach for semantic similarity. Therefore, the combination of these two objectives has a significant effect.The goal of this paper is to explore a low-cost training strategy for enhancing existing pretrained language models. In order to gain a more detailed understanding of the impact of enhanced pretrained samples on model performance, we conducted an in-depth analysis of the effect of pretrained sample size. Specifically, we performed model-enhanced pretraining using weakly supervised samples of 200,000, 500,000, and 1,000,000. We then tested the performance of models with different pretrained sample sizes on downstream semantic matching tasks. For convenience, we used BERT-base as the pretrained language model for validation. The experimental results are shown in weakly supervised samples used for enhanced pretraining increases, the model's performance shows an upward trend. When training with the complete 1,000,000 samples, the model achieves optimal performance on both datasets. The experimental results validate the effectiveness of the weakly supervised samples we constructed and demonstrate that this type of sample can consistently improve the model's performance as the quantity increases.The main purpose of introducing the Alignment-Aspect in the contrastive pretraining optimization objective is to achieve finegrained semantic alignment between tokens in the text, thereby assisting the model in accurate semantic matching recognition. We aim to optimize the model's semantic perception of certain keywords more effectively. Therefore, we conducted a case study to further explore the enhanced model's capabilities. By visualizing the distribution of attention weights among different models in Figure 4 , we can observe that by introducing the alignment-aspect contrastive mechanism, our model can focus more on key words in sentences, which may be the reason why the alignment-aspect contrastive mechanism improves the performance. In this case, in order to judge the semantic similarity of two sentences: How do I quit smoking and How do I give up on cigarette smoking, EBSIM pays more attention to quit smoking and give up on cigarette smoking than BERT. They are keywords to identify the similarity.  
: 
 Figure 1 : Illustration of the EBSIM's pipeline. Firstly, the sentence-triplet corpus is constructed from a large-scale document-level corpus via token level and structure level data augmentation. Then, the BERTs are further pre-trained through contrastive objectives on the weakly-supervised corpus. Finally, the enhanced BERTs adapt to STS tasks in the fine-tuning stage. The subgraph of the BERT's structure is derived from Devlin et al. [16] . 
 
 Appendix:  


Decision: The reviewers did not reach a consensus regarding this paper. The authors' rebuttal did not address all the issues raised. In particular, there are still concerns related to generalization (mentioned by two reviewers) and the overall relevance of the paper for the conference (the approach is unimodal).
In the end, the paper is borderline even after the reviewer went carefully through both the paper and the rebuttal. The small tendencies in the final rating toward acceptance make me suggest this decision.

Reviews: review: paper_summary: paper_summary: The authors introduce a pre-training method (called EBSIM) for enhancing performance on downstream semantic text matching when only a few fine-tuning examples are available. In particular, at pre-training time, they enforce token-similarity and structure similarity through two triplet loss objectives, creating ad-hoc triplets through a corpus of English and Chinese sentences composed of titles and relevant document sentences. The authors show the effectiveness of their pre-training method by fine-tuning the pre-trained model on downstream semantic matching datasets, showing some improvements over methods pre-trained using standard BERT objectives (MLM). The authors also perform some ablation studies to show that the novel model is more sample efficient when fine-tuned.
strength_weakness: strength_weakness: - The paper is well-written and quite easy to follow. 
- The motivation is clear and the introduction helps to clearly present the problem.
- The experiments have been conducted on sentence corpora spanning two very different languages (English, Chinese).
limitations: limitations: - Despite the reasonable motivation and design, the proposed method seems not achieving a remarkable margin improvement (below 1%) over the standard MLM pre-training method. It is true that, from the ablation study, the model seems easier to fine-tune, but, even from Figure 3, it seems that there is not a significant statistical difference between BERT and BERT + EBSIM.
- The idea of pre-training large transformer models is that a single pre-training schema may benefit many downstream tasks. However, in this work, pre-training has been designed to benefit only the downstream semantic matching task. To be more solid, I would have expected a different experimental design, in which the EBSIM pre-training method is then fine-tuned on other downstream tasks. Otherwise, it may be that EBSIM helps semantic matching tasks while worsening performance, for example, on text summarization or next-sentence prediction.
- Given that pre-training and fine-tuning are very much correlated in the proposed approach, I would have expected in the ablation studies also an experiment where the pre-training data was mixed with the downstream semantic matching data, instead of being used for pre-training. This would have evaluated how much the overall pre-training schema - which includes specific token-level and structure-level objectives - helped over the data themselves. It could be that, in the end, the whole method works a few decimal percentage points better only because more data is being used.
- Given that the authors employ BERT pre-trained on MLM as a starting base, I would have expected the same pre-training EBSIM schema to be used on other pre-trained models (e.g., RoBERTa), to evaluate how much the model-specific pre-training strategies influence the whole pipeline.

Minor comments:
- In the related works, the authors can consider citing some pre-trained vision-language models like VinVL [1] or VL-BERT [2], in which semantic image-text matching is the key pre-training objective.
- A more in-detail figure showing the architecture of the pre-training schema would have better summarized the proposed architecture.
- In Section 4.2, line 636-637, Chinese datasets should be in Table 3 and English in Table 2. Currently, they are inverted.

Considering that this paper only concerns unimodal scenarios (text), the weaknesses of the experimental results, and the potential design flaws, I lean towards the rejection of this paper at ACM MM.

[1] Zhang, Pengchuan, et al. "Vinvl: Revisiting visual representations in vision-language models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Su, Weijie, et al. "VL-BERT: Pre-training of Generic Visual-Linguistic Representations." International Conference on Learning Representations. 2019.
paper_summary: paper_summary: This paper proposes addressing the quality and quantity issues of downstream data when fine-tuning pre-trained models. Focusing on the task of semantic matching, the authors suggest using few-shot learning through weakly contrastive pre-training. Comprehensive experiments on five Chinese and English semantic text-matching datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper proposes weakly-supervised data augmentation and contrastive pre-training for Transformer-based semantic matching.

2. The proposed method achieves state-of-the-art performance on five Chinese and English semantic text matching datasets.
limitations: limitations: In Figure 3, why does the EBSIM method accelerate the model's convergence compared to BERT base (blue line)? A detailed explanation and analysis are needed.
paper_summary: paper_summary: The paper presents an innovative approach to enhancing Transformer-based semantic text matching for few-shot learning scenarios through a method named EBSIM (Enhanced Transformer-based Semantic Matching for Few-shot Learning through Weakly Contrastive Pre-training). The authors focus on addressing the challenges of data quality and quantity in practical downstream applications and propose a general enhancement scheme for few-shot semantic text matching tasks. EBSIM introduces a weakly supervised data augmentation method and a contrastive learning objective to improve the semantic similarity measurement between texts. The method was evaluated on five Chinese and English semantic text matching datasets using various Transformer-based pre-trained models, demonstrating significant improvements in performance.
strength_weakness: strength_weakness: (1) Innovative Approach: The paper proposes a novel pre-training strategy that bridges the gap between pre-training and fine-tuning stages for semantic text matching tasks.

(2) Weakly Supervised Data Augmentation: A cost-effective method for creating weakly supervised samples that enhances the model's ability to understand semantic similarity.

(3) Contrastive Learning Objective: The introduction of a contrastive learning objective based on alignment aspects leads to effective semantic matching by optimizing bidirectional semantic perception.

(4) Comprehensive Experiments: The method was tested on multiple datasets in both Chinese and English, providing a robust evaluation of its effectiveness.

(5) Performance Improvement: The results show significant performance improvements over existing models on semantic text matching tasks.

(6) Ablation Studies: The paper includes ablation experiments that validate the contribution of different components of the proposed method.
limitations: limitations: (1) Limited Generalizability: The enhancements are specifically tailored for semantic text matching tasks and may not be directly applicable to other types of NLP tasks.

(2) Resource Intensive: Although the data augmentation method is cost-effective, the pre-training process with contrastive learning objectives might still require considerable computational resources.

(3) Overreliance on Synonyms: The method might still be sensitive to the presence of synonyms, which could affect the accuracy of semantic judgments.

(4) Dataset Bias: The effectiveness of EBSIM was evaluated on specific datasets, and its performance on other types of datasets or domains might vary.

(5) Complexity of Implementation: The introduction of new objectives and mechanisms may increase the complexity of the model, potentially making it harder to implement and maintain.

(6) Potential for Overfitting: With the focus on enhancing performance on specific tasks, there may be a risk of overfitting to the particularities of the training datasets used.