# Improving Sentence Representations via Component Focusing
## 3 Methods  
### Model Architecture  
Our model CF-BERT aims to focus on crucial components and syntactic relations of a sentence to get a more powerful sentence representation that yields better performance in downstream NLP tasks. As mentioned before, the sentence representation consists of two parts, and to make it easy to describe a sentence representation that consists of two parts, we give the raw sentence text of the two parts names: \(S_{\textit{basic}}\) and \(S_{\textit{cf}}\). The basic sentence part \(S_{\textit{basic}}\), contains the complete sentence information, and the component-enhanced sentence part \(S_{\textit{cf}}\), contains the crucial (primarily from the subject, predicate, and object of a sentence) sentence information. \(S_{\textit{basic}}\) essentially equals to the raw text of a sentence in NLP datasets which cover the global information of a sentence, while \(S_{\textit{cf}}\) enhances semantic information by keeping only crucial information and removing meaningless or noisy words. The example sentence results of basic component \(S_{\textit{basic}}\) and component-enhanced \(S_{\textit{cf}}\) are listed in Table 1.  
Based on this idea, the overall architecture of the CF-BERT is shown in Figure 2. A similar network structure as that in SBERT [32] is used to design the CF-BERT model.  
According to Figure 2, the two sentences A and B are passed to the CF-BERT model to generate fixed-sized sentence embeddings \(\textit{emb\_SA}_{\textit{basic}}\) and \(\textit{emb\_SB}_{\textit{basic}}\) as the basic components of A and B, respectively (described in Section 3.2). Subsequently, the CF-BERT performs the dependency parsing of sentences A and B to obtain the component-enhanced part with their relations (described in Section 3.3).  
\begin{table}
\begin{tabular}{l l} \hline \hline \(S_{\textit{basic}}\) & \(S_{\textit{sf}}\) \\ \hline An Asian woman in a crowd is not carrying a black bag & woman not carrying bag \\ \hline A man attacks a woman & man attacks woman \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of basic part \(S_{\textit{basic}}\) and component-enhanced part \(S_{\textit{sf}}\).  
By the same way as the basic components, sentence embeddings \(\text{emb\_SA}_{cf}\) and \(\text{emb\_SB}_{cf}\) for the component-enhanced parts of A and B are also generated. Next, the weight factor \(W_{cf}\) is introduced to adjust the ratio of the component-enhanced part embeddings \(\text{emb\_Scf}\) to the basic part embeddings \(\text{emb\_S}_{basic}\) to generate the complete sentence representation. A complete sentence representation \(\text{emb\_S}\) (fixed-sized vector) is finally expressed as follows:  
\[\text{emb\_S}=\text{emb\_S}_{cf}*W_{cf}+\text{emb\_S}_{basic} \tag{1}\]  
when \(W_{cf}=0\), CF-BERT equals SBERT.  
The best weight factor \(W_{cf}\) is obtained through a grid search method in the experiment (described in Section 5). Finally, we use the complete sentence representations from the previous step as the input of the last output layer. The output layer of the model can be changed according to the specific NLP tasks. This study investigates semantic textual similarity. We focus on textual similarity tasks only in this section and then evaluate the broader utility of our sentence representations on entailment classification in the next section.