## 5 Experiments and Analysis of Results
### Exploring the Number of Inserted Punctuation Marks  
First insert 1 to 3 punctuation marks, using the English punctuation marks \".\",  
As can be observed from Table 9, the best results are achieved when 1 to 3 punctuation marks are inserted. While the insertion of only one punctuation mark also has a positive effect on the model, the insertion of more punctuation marks (>3) does not lead to the best performance of the model but does not have a significant impact on the original semantics of the utterance, and the model still outperforms the original SimCSE method.  
As can be seen from Table 10, when only one symbol is inserted, the effect on the \"roberta-base\" of the model is negligible; when more symbols are inserted, the model is susceptible to poor robustness; when 1-4 symbols are inserted, the effect is weaker than the original model; when more symbols are inserted (6 or 7), the model tends to perform the same as the original model; when 1-2 symbols are inserted, the model has the best performance.  
As can be seen from Table 11, for the 'roberta-large' model, the performance of the model increases as the maximum number of symbols inserted increases from 1 to 5, with the best performance achieved when 1 to 5 symbols are inserted. There is no negative impact on the model when more symbols are inserted, and the model still outperforms the baseline (SimCSE).  
In summary, the bert-base model achieves the best performance when 1-3 punctuation symbols are inserted; the roberta-large model achieves the best performance when  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Number of Symbols** & **ATEC** & **BQ** & **LCQMC** & **PAWS-X** & **STS-B** & **Mean** \\ \hline
0 & 34.11 & 50.18 & 65.39 & 10.27 & 29.54 & 37.89 \\
1–1 & 32.78 & 50.83 & 65.83 & 10.04 & 29.78 & 37.78 \\
1–2 & 33.90 & 51.52 & 67.76 & 10.92 & 29.56 & 38.73 \\
1–3 & 32.89 & 50.41 & 66.36 & 10.45 & 30.00 & 38.02 \\
1–4 & 31.20 & 50.79 & 66.55 & 9.83 & 29.91 & 37.65 \\
1–5 & 31.74 & 50.54 & 67.92 & 10.42 & 30.11 & 38.14 \\
1–6 & 32.40 & 50.58 & 66.26 & 10.61 & 29.73 & 37.91 \\
1–7 & 33.68 & 50.85 & 66.02 & 9.20 & 30.19 & 37.98 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Effect of the number of insertion symbols on the roberta-base model.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Number of Symbols** & **ATEC** & **BQ** & **LCQMC** & **PAWS-X** & **STS-B** & **Mean** \\ \hline
1–1 & 33.38 & 50.51 & 62.00 & 9.46 & 27.29 & 36.52 \\
1–2 & 32.41 & 51.11 & 64.46 & 11.14 & 26.72 & 37.16 \\
1–3 & 34.23 & 50.67 & 64.9 & 10.15 & 27.76 & 37.54 \\
1–4 & 32.68 & 50.17 & 61.98 & 9.27 & 25.98 & 36.01 \\
1–5 & 33.43 & 50.75 & 63.12 & 10.4 & 24.92 & 36.52 \\
1–6 & 32.99 & 49.54 & 63.7 & 10.12 & 26.94 & 36.65 \\
1–7 & 32.95 & 50.1 & 65.44 & 9.78 & 25.8 & 36.8 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Effect of the number of insertion symbols on the bert-base model.  
1-5 symbols are inserted, and the insertion of more symbols does not negatively affect both models, and the model performance is still better than the baseline (SimCSE). The roberta-base model has the best performance when 1-2 symbols are inserted, and the model performance when more symbols are inserted is comparable to the baseline (SimCSE) effect.