# Grouped Contrastive Learning of Self-Supervised Sentence Representation
## 3 Methodology
### Partial Word Vector Augmentation  
As discussed above, performing data augmentation in contrastive learning is done in order to obtain positive samples. However, most existing methods are discrete and performed on full words of a sentence, which could deteriorate original semantic information for discrete and sparse text data. Therefore, we design a continuous and partial-word vector augmentation (PWVA) for contrastive learning. Furthermore, a word vector is a vector with fixed dimensionality, and every element in a word vector is a real value. Therefore, a word vector can be treated as a 1D discrete signal. In this way, word vectors can be processed by strategies of digital signal processing. Our proposed PWVA is based on this insight in order to implement data augmentation. To be exact, PWVA is conducted by two probability choices. Let \(W=\{w_{i}\in\mathbb{R}^{d}\}_{i=1}^{N}\) be the \(N\) word vectors with \(d\) dimensionality. The first probability choice of PWVA is represented by:  
\[w_{aug}=\rho(A_{gwn}(w_{i}),A_{rzs}(w_{i}),A_{lift}(w_{i}),A_{rbn}(w_{i});p1,p2,p3,p4), \tag{1}\]where \(\rho(\cdot)\) is a function aiming to choose data augmentation strategies from Random Zero Setting (RZS), Inverse Fast Fourier Transformation (IFFT), Gaussian White Noise (GWN), and Random Background Noise (RBN) by the probabilities \(p1,p2,p3\), and \(p4\), respectively. The \(w_{aug}\) denotes the augmented word vectors. The second choice of PWVA can be expressed below: \[w_{pww}=\varrho(w_{aug},w_{i};p),\] (2) where \(\varrho\) is a function to select the final PWVA output \(w_{pww}\) from \(w_{aug}\) and \(w_{i}\), with probability \(p\). The visualization of PWVA is shown in the lower right plot of Figure 1. In addition, four data augmentation strategies employed in Equation (1) are explained below. * **Gaussian White Noise (GWN)** In order to improve the robustness of our model, we introduce Gaussian white noise (as illustrated in Figure 2a) into the word vectors. This approach is inspired by the work of [PERSON] et al. [30]. Gaussian white noise can be mathematically represented as follows: \[A_{gun}(w_{i})=w_{i}+\lambda\cdot\mathcal{N}(0,1),\] (3) where \(\lambda\) represents the trade-off parameter, while \(\mathcal{N}(0,1)\) refers to the standard normal distribution. * **Random Zero Setting (RZS)** To mitigate data dependence and enhance generalization ability, we employ a technique called random zero setting \(A_{rzs}(w_{i})=Dropout(w_{i})\) (as illustrated in Figure 2b). This technique enables us to randomly assign zero values to certain word vector components. * **Inverse Fast Fourier Transformation (IFFT)** To extract features in the frequency domain, we utilize word vectors and subsequently apply the inverse fast Fourier transform (IFFT) as illustrated in Figure 2c to convert them into the time domain. The word vectors undergo slight modifications after undergoing the IFFT process, thereby enhancing the resilience of the data boundary. The mathematical representation of the IFFT can be expressed as follows: \[A_{ifft}(w_{i})=Real(IFFT(FFT(w_{i}))),\] (4) where \(Real(\cdot)\) denotes the real part. * **Random Background Noise (RBN)** Random background noise cannot be learned by a model, as stated in the research conducted by [31]. Therefore, to enhance training stability, we introduce random  
Figure 1: The GCLSR architecture. background noise into the word vectors, as depicted in Figure 2d. The formulation for random background noise (RBN) is given below:  
\[A_{rhn}(w_{i})=w_{i}+uniform(0,0.1), \tag{5}\]  
where \(uniform(\cdot)\) is the uniform distribution. In summary, continuous and PWVA can enhance and facilitate contrastive learning. The characteristic of \"continuous\" PWVA can ensure that there is no semantic gap in word vectors, while the \"partial\" can retain more original semantics of word vectors. In this way, a model can readily acquire a richer set of discriminative features by assimilating the disparities between the initial word vectors and their respective augmented counterparts.