## 4 The Proposed Sentence Representation Model  
As Figure 1 shows, an entry consists of a word and the corresponding explanatory note. Usually, an explanatory note is a complete sentence, so we extract the semantics of the explanatory note and treat it as the representation of the corresponding word.  
We use \(s_{1:m}=T_{1}\dots T_{i}\dots T_{m}\) to represent a sentence with length \(m\). The sentence representation of \(s_{(1:m)}\) is denoted by \(SEMTS(s_{(1:m)})\). We assume that the more tokens a  
Figure 2: An infeasible semantics learning model composed of WE and ENE. The model aims to extract the semantics of words from the explanatory notes but fails.  
sentence contains, the more semantics it conveys. Let us consider two token sequences, \(s_{(1:k)}\) and \(s_{(1:k+1)}\). \(s_{(1:k+1)}\) has one more token \(T_{k+1}\) than \(s_{(1:k)}\). According to the hypothesis, \(s_{(1:k+1)}\) contains more semantics than \(s_{(1:k)}\). The added semantics of \(s_{(1:k+1)}\) than \(s_{(1:k)}\) is mainly caused by \(T_{k+1}\), so we can use the added semantics to predict \(T_{k+1}\), that is,  
\[SEMTS(s_{(1:k+1)})-SEMTS(s_{(1:k)})\overset{\text{predict}}{\longrightarrow} T_{k+1}. \tag{2}\]  
In Equation (2), the sentence representation \(SEMTS(s_{(1:k+1)})\) is calculated by the self-attention mechanism. The self-attention mechanism in our sentence representation model shown in Figure 3 is slightly different from the traditional self-attention mechanism [1]. Suppose that \(Z=Z_{1}Z_{2}\cdots Z_{m}\) is the output of the encoder when the input token sequence is \(s_{(1:m)}\). \(Z_{i}\) is the encoding of \(T_{i}\). The calculation process of the sentence representation of \(s_{(1:m)}\) is shown in Figure 3.  
In Figure 3, \(\bar{q}\), \(W_{s}\), and \(b\) represent the query vector, the shape transformation matrix, and the bias, respectively. The \(e\), \(tan\), and \(\odot\) operations are all element-wise.  
To make full use of the strong encoding ability of BERT, we use BERT as the backbone of our sentence representation model. Another benefit of building and fine-tuning the model based on BERT is that it can save a lot of computation power. Thus, we add our semantics computing module to BERT as a sub-task, and we call our semantics representation model SEMTS-BERT. The architecture of SEMTS-BERT is shown in Figure 4. NSP and MWP are two sub-tasks of the original BERT [1]. Final Word Prediction (FWP) sub-task corresponds to our semantics computing model, and its structure is shown in Figure 5. The loss of FWP sub-task is defined as  
\[\text{Loss}_{\text{fwp}}=-\frac{1}{N}\sum_{1}^{N}p(T_{k+1}), \tag{3}\]  
where \(p(T_{k+1})\) is the probability of \(T_{k+1}\) defined in Equation (2) when using the added semantics to predict \(T_{k+1}\). \(N\) is the number of predicted words in a batch. \(p(T_{k+1})\) is calculated by FWP, shown in Figure 5. When we minimize the objective, the loss of FWP defined in this way makes the probability of predicted words as large as possible.  
Figure 3: The calculation process of the sentence representation denoted by \(SEMTS(s_{(1:m)})\).  
The total loss of SEMTS-BERT is the sum of \(\text{Loss}_{fwp}\), \(\text{Loss}_{nsp}\), and \(\text{Loss}_{mwp}\), that is,  
\[\text{Loss}=\text{Loss}_{fwp}+\text{Loss}_{nsp}+\text{Loss}_{mwp}\,. \tag{4}\]  
The detail of \(\text{Loss}_{nsp}\) and \(\text{Loss}_{mwp}\) can be seen in [1].