# BRFP: An Efficient and Universal Sentence Embedding Learning Model Method Supporting Fused Syntax Combined with Graph Embedding Representation Algorithm

Authors: Zhifeng Li; Wenwei Wu; Chunlei Shen
Journal: Journal of Sensors
 (2022)

---

## Abstract

Due to the rapidly growing volume of data on the Internet, the methods of efficiently and accurately processing massive text information have been the focus of research. In natural language processing theory, sentence embedding representation is an important method. This paper proposes a new sentence embedding learning model called BRFP (Factorization Process with Bidirectional Restraints) that fuses syntactic information, uses matrix decomposition to learn syntactic information, and fuses and calculates with word vectors to obtain the embedded representation of sentences. In the experimental chapter, text similarity experiments are conducted to verify the rationality and effectiveness of the model and analyzed experimental results on Chinese and English texts with the current mainstream learning methods, and potential improvement directions are summarized. The experimental results on Chinese and English datasets, including STS, AFQMC, and LCQMC, show that the model proposed in this paper outperforms the CNN method in terms of accuracy and F1 value by 7.6% and 4.8. The comparison experiment with the word vector weighted model shows that when the sentence length is longer, or the corresponding syntactic structure is complex, the model’s advantages in this paper are more prominent than TF-IDF and SIF methods. Compared with the TF-IDF method, the effect improved by 14.4%. Compared with the SIF method, it has a maximum advantage of 7.9%, and the overall improvement in each comparative experimental task is between 4 and 6 percentage points. In the neural network model comparison experiment, the model in this paper compared the CNN, RNN, LSTM, ST, QT, and InferSent models, and the effect significantly improved on the 14’OnWN, 14’Tweet-news, and 15’Ans.-forum datasets. For example, in the 14’OnWN dataset, the BRFP method has a 10.9% improvement over the ST method. The 14’Tweet-news dataset has a 22.9% advantage over the LSTM method, and the 15’Ans.-forum dataset has a 24.07% improvement over the RNN method. The article also demonstrates the generality of the model, proving that the model proposed in this paper is also a universal learning framework.


---

## Body

## 1. Introduction

In the Internet era, the amount of information shows a geometric progression, and the massive amount of text data hides inestimable social and economic value. Efficient and reasonable text information processing has become a hot and challenging point in current research [1]. The emergence of the word embedding method [2] provides an excellent idea, and researchers have focused their attention on the vector representation of natural language, usually based on semantic similarity to model the words and sentences into continuous vectors. With the maturity and widespread application of word embedding methods, using vectors to represent [3–5] those higher language levels is also feasible.Natural language is a product of the evolution and development of human society. As a carrier of information and a tool for communication and thinking, it has extensively promoted the development of human history. The difficulty of natural language processing lies in the flexibility of natural language semantic connotation and the ambiguity of syntactic rules. Because of these characteristics, natural language modeling is complicated. In addition, the accuracy of the evaluation criteria for many downstream tasks in natural language processing also needs to be studied. Unlike the field of computer vision, for example, in the image classification task, even if the image labels are manually labeled, the labels’ correctness can be significantly guaranteed due to the image’s intuition. However, the high abstraction of natural language determines. Uncertainty in labeling training samples in supervised learning adds random and chance factors to the model’s training process. In the final analysis, natural language, as a symbolic system, needs to be converted into a numerical model to realize computer processing and analysis.The emergence of word embedding methods provides an excellent idea, and researchers have focused their attention on the vectorized representation of natural language, usually based on semantic similarity for modeling to map language into continuous vectors. In natural language, the words, sentences, paragraphs, and chapters constitute the four levels of division of natural language, among which words are the most basic language units. In addition, a sentence’s general semantics must contain its constituent words’ semantics. However, it is not a simple superposition of semantics but a fusion of lexical semantics under the constraints of specific syntactic rules.The current main sentence embedding learning methods include bag-of-words model-based and neural network methods. Most of these methods start from the perspective of semantic similarity, directly regard the sentence as a whole without considering its internal composition, and learn the embedded representation of sentences by simply predicting various semantic connections between sentences. Therefore, these methods generally have the problem of a low degree of syntactic information fusion. The lack of syntactic information will lead to the overall semantic analysis and understanding deviation, ultimately affecting the semantic accuracy of sentence embedding.This paper proposes a sentence embedding learning method based on the above background that fuses syntactic information. With the help of the syntactic analysis [6–8] method, the sentence is modeled as an attribute network containing node information [9], and matrix decomposition is used to learn the vector representation of syntactic information and fuse it with word vectors [10]. Embedding representation of sentences is dedicated to solving the deficiency of insufficient syntactic information fusion in the current research on sentence embedding [11]. The sentence embedding representation studied in this paper belongs to the fundamentally public nature problem in natural language processing. Sentence embedding can be understood as an extension of word embedding [12], which maps long text fragments into digital vectors, which points out potential fields for natural language processing research. At the same time, it is also the core link for solving many practical application problems such as machine translation, automatic question answering, sentiment analysis, and personalized recommendation. Therefore, this research topic has far-reaching theoretical significance and application value.


## 2. Related Research

Embedding representation of sentences is a fundamental theory in natural language processing and has become an indispensable part of natural language processing theory. In the early days, unsupervised representation learning has always occupied the mainstream position but supervised learning, and multitask learning is the current mainstream trends [13]. In addition, the emergence of pretrained language models has profoundly changed the landscape of natural language processing. So far, scholars from various countries have proposed many embedding representation learning methods. These methods mainly fall into the bag-of-words model-based and neural network methods.


### 2.1. The Methods Based on Bag-of-Words Model

The bag-of-words model was used initially in text classification tasks to represent documents as feature vectors by position index [14]. Methods based on the bag-of-words model usually ignore word order, grammar, and syntax in the text and only regard the text as a collection of words or a collection of word vectors.Deerwester et al. believed that text generally contains several topics [15], and the similarity of text semantics can be approximated as the similarity of topics. A dimensionality-reduced latent semantic space was constructed and applied to text classification tasks by performing singular value decomposition on the vocabulary-text matrix. In 2017, Arora et al. proposed a smooth inverse frequency weighting model [16]. First, they presented the concept of public discourse vector, arguing that the generation of the corpus has obtained from a random walk of discourse vector, and introduced two smoothing factors to suppress those useless highs in the contribution of frequent words and ensure that information outside the context of words can also appear in sentences.


### 2.2. The Methods Based on Neural Network Model

Kiros et al. proposed the Skip-Thought model [17], which adopts the most commonly used encoder-decoder architecture in machine translation and uses the midsentence to predict the sentence of the context to learn the sentence. Experiments show that the sentence vectors generated by Skip-Thought are not all optimal but generally perform well in multiple tasks. This result proves that the model has good generality. However, Skip-Thought predicts the context without any syntactic, semantic, and contextual information after training is finished and discards the encoder, resulting in a lack of model training efficiency. In 2018, Logeswaran et al. proposed Quick-Thought [18], which improved the decoder structure of Skip-Thought and replaced the original prediction behavior with a classification task, which improved the efficiency of the sentence encoding process and downstream jobs.Collobert and Weston proposed the C&W model [19] to design models and objective functions directly from the distributed hypothesis, with the ultimate goal of learning word vectors. Mikolov et al. proposed word2vec in 2013. The method includes the CBOW model and the Skip-gram model [20], which are calculated based on local sliding windows and entirely use local contextual features. Pennington et al. proposed the Glove model [21], which fuses global statistical elements and local contextual features. In 2016, Facebook released the FastText model, similar to the CBOW model of word2vec, including the input, hidden, and output layers. FastText can be used for embedding learning in different languages and is an excellent general term embedding model. In 2018, Paul G. Allen published the ELMo [22] model. Before it appeared, almost all word embedding learning models could not solve the problem of polysemy because each word has only a unique vector. The ELMo model is no longer a direct correspondence between words and vectors but infers the word vector corresponding to each word according to an input sentence or paragraph. Even in the context of polysemy, the model can combine contexts for polysemy understanding.


### 2.3. Syntax Analysis Method

Different from artificial language, there are many ambiguities in natural language, such as part-of-speech ambiguity, structural ambiguity, and referential ambiguity, and the process of syntactic analysis can eliminate the ambiguity problem [23] existing in natural language processing, which is an important method to solve natural language understanding [24]. Syntactic analysis is a key challenge [25] in natural language processing. Its purpose is to analyze the grammatical functions of words in sentences. The syntactic analysis includes syntactic structure analysis and dependency analysis.Syntactic structure analysis [25] is aimed at obtaining the syntactic structure or temporary sentence structure of the whole sentence and organizing the structural relationship into a tree structure, called the syntactic structure tree, in which each node is composed of part-of-speech tags, phrase tags, or clause structure tags to refer to syntactic elements at different levels.


### 2.4. Graph Embedding Representation Algorithm

The core idea of graph embedding [26] is to find a mapping function to convert each node or the entire graph in the graph structure into a low-dimensional vector representation. This concept was originally an extension of the word embedding technology in natural language processing [27]. In the research field of graph embedding, many excellent algorithms, such as DeepWalk [28], node2vec [29], LINE [30], and GarphGAN [31], have been proposed, but the most influential is DeepWalk, proposed by Perozzi et al. [28] in 2014. The main idea is to perform random walks on the graph structure to generate node sequences, treat the node sequences as text sequences in natural language, input them into the word2vec model, and train them to obtain the vector representation of graph nodes. In 2015, Yang et al. proved that the DeepWalk algorithm based on the random walk, which in Figure 1 shows its algorithm matrix, is equivalent to the matrix decomposition of matrix M [32], in the form of M=WTH.Figure 1
DeepWalk algorithm equivalent matrix.


## 2.1. The Methods Based on Bag-of-Words Model

The bag-of-words model was used initially in text classification tasks to represent documents as feature vectors by position index [14]. Methods based on the bag-of-words model usually ignore word order, grammar, and syntax in the text and only regard the text as a collection of words or a collection of word vectors.Deerwester et al. believed that text generally contains several topics [15], and the similarity of text semantics can be approximated as the similarity of topics. A dimensionality-reduced latent semantic space was constructed and applied to text classification tasks by performing singular value decomposition on the vocabulary-text matrix. In 2017, Arora et al. proposed a smooth inverse frequency weighting model [16]. First, they presented the concept of public discourse vector, arguing that the generation of the corpus has obtained from a random walk of discourse vector, and introduced two smoothing factors to suppress those useless highs in the contribution of frequent words and ensure that information outside the context of words can also appear in sentences.


## 2.2. The Methods Based on Neural Network Model

Kiros et al. proposed the Skip-Thought model [17], which adopts the most commonly used encoder-decoder architecture in machine translation and uses the midsentence to predict the sentence of the context to learn the sentence. Experiments show that the sentence vectors generated by Skip-Thought are not all optimal but generally perform well in multiple tasks. This result proves that the model has good generality. However, Skip-Thought predicts the context without any syntactic, semantic, and contextual information after training is finished and discards the encoder, resulting in a lack of model training efficiency. In 2018, Logeswaran et al. proposed Quick-Thought [18], which improved the decoder structure of Skip-Thought and replaced the original prediction behavior with a classification task, which improved the efficiency of the sentence encoding process and downstream jobs.Collobert and Weston proposed the C&W model [19] to design models and objective functions directly from the distributed hypothesis, with the ultimate goal of learning word vectors. Mikolov et al. proposed word2vec in 2013. The method includes the CBOW model and the Skip-gram model [20], which are calculated based on local sliding windows and entirely use local contextual features. Pennington et al. proposed the Glove model [21], which fuses global statistical elements and local contextual features. In 2016, Facebook released the FastText model, similar to the CBOW model of word2vec, including the input, hidden, and output layers. FastText can be used for embedding learning in different languages and is an excellent general term embedding model. In 2018, Paul G. Allen published the ELMo [22] model. Before it appeared, almost all word embedding learning models could not solve the problem of polysemy because each word has only a unique vector. The ELMo model is no longer a direct correspondence between words and vectors but infers the word vector corresponding to each word according to an input sentence or paragraph. Even in the context of polysemy, the model can combine contexts for polysemy understanding.


## 2.3. Syntax Analysis Method

Different from artificial language, there are many ambiguities in natural language, such as part-of-speech ambiguity, structural ambiguity, and referential ambiguity, and the process of syntactic analysis can eliminate the ambiguity problem [23] existing in natural language processing, which is an important method to solve natural language understanding [24]. Syntactic analysis is a key challenge [25] in natural language processing. Its purpose is to analyze the grammatical functions of words in sentences. The syntactic analysis includes syntactic structure analysis and dependency analysis.Syntactic structure analysis [25] is aimed at obtaining the syntactic structure or temporary sentence structure of the whole sentence and organizing the structural relationship into a tree structure, called the syntactic structure tree, in which each node is composed of part-of-speech tags, phrase tags, or clause structure tags to refer to syntactic elements at different levels.


## 2.4. Graph Embedding Representation Algorithm

The core idea of graph embedding [26] is to find a mapping function to convert each node or the entire graph in the graph structure into a low-dimensional vector representation. This concept was originally an extension of the word embedding technology in natural language processing [27]. In the research field of graph embedding, many excellent algorithms, such as DeepWalk [28], node2vec [29], LINE [30], and GarphGAN [31], have been proposed, but the most influential is DeepWalk, proposed by Perozzi et al. [28] in 2014. The main idea is to perform random walks on the graph structure to generate node sequences, treat the node sequences as text sequences in natural language, input them into the word2vec model, and train them to obtain the vector representation of graph nodes. In 2015, Yang et al. proved that the DeepWalk algorithm based on the random walk, which in Figure 1 shows its algorithm matrix, is equivalent to the matrix decomposition of matrix M [32], in the form of M=WTH.Figure 1
DeepWalk algorithm equivalent matrix.


## 3. Methodology

This article will innovate and propose new algorithm models based on syntax analysis methods [33] and graph embedding algorithms [34]. We model syntactic and semantic information as attribute networks containing vertex information and apply algorithmic ideas in the field of graph embedding [35] to sentence embedding [26] representation learning.


### 3.1. The Principle of the BRFP Model

In the study of sentence embedding, syntactic information is an important aspect [36] because, from a grammatical point of view, the overall semantics of a sentence not only depends on the independent semantics of each word in the sentence [37] but also generates new semantics, even opposite semantics under the action of syntax.In addition, the high abstraction of natural language determines the abstraction of syntactic structure, so a quantifiable modeling method is needed to describe syntactic information.In this paper, the BRFP model introduced the Stanford parser [38] to parse the sentence to generate a syntactic structure tree. Because the tree can be regarded as a graph without rings in the data structure, BRFP constructs an undirected graph structure containing node information, shown in Figure 2, based on the syntactic structure tree and defines each component label in the tree as a graph vertex.Figure 2
Undirected graph with node information.In the graph, triple represents the node information of the vertex and the semantic vector. There are two types of vertices in the graph. The first type is terminal vertex, such as vertex 4, 5, 8, 10, 12, 13, and 14. Each vertex corresponds to an entity word in the sentence, and the semantic vector of such vertex is the trained word vector. The rest of the vertices are nonterminal vertex and only correspond to syntactic components rather than an entity word. The BRFP model uses the mean of the semantic vectors of all subvertex vertices as the semantic vector of the vertices. The calculation rule is as Equation (1), which represents the syntax vector, which is also the part that requires model training. Furthermore, it represents various syntactic labels in the syntactic parse tree:
(1)Vi=1C∑vi∈Cveci,otherwise,veci,viisterminal,Cischildsetofvi.The abovementioned undirected graph model contains the semantic information represented by word vectors and the syntactic information represented by the graph topology and vertex syntax labels. The embedding representation of the sentence is transformed into the embedding representation of the learning graph, which is precisely the embedding of the attribute network containing the vertex information.BRFP decomposes a matrixM into the product form of four matrices E, W, H, and R, where W and H are parameter matrices. R represents the semantic matrix, which is formed by splicing the semantic vectors of each vertex in the graph structure horizontally. E represents the syntactic mutual information matrix, and the element Eij is defined as follows:
(2)Eijn=Ωin∩ΩjnΩin+Ωjn.Ω⋅n is a multilabel set of order n corresponding to a vertex, representing the set of syntactic labels of all neighbor vertex within n steps away from a vertex. In the following example, the parameter n is set to 2, and E56 is calculated: the vertex within 2 steps from vertex 5 is 1, 3, and 6, and the vertex within 2 steps from vertex 6 is 1, 3, 5, 7, 8, and 9; the specific calculation process is as follows:
(3)Ω5k=S,VP,SΩ6k=S,VP,VBD,VP,TO,VP⇒E56=S,VPΩ5k+Ω6k=23+6=29.The matrixE reflects the syntactic similarity between vertices. The BRFP model uses matrix E and matrix R to add syntactic and semantic information in the decomposition process. E and R being fixed can also be regarded as two constraints on the matrix decomposition process, corresponding to the bidirectional restraints in the BRFP model. For matrix decomposition, this paper adopts the following loss function:
(4)minW,HM−ETWTHRF2+λ2WF2+HF2.Further, the second term is the regularization part, andλ is the introduced harmonic factor. During the optimization process, E and R are fixed, and it is only necessary to alternately fix and update W and H until the model converges:
(5)∂∂WM−STWTHTF2+λ2WF2+HF2=λW−2SαMT−SαTWEET,Sα=HR,(6)∂∂HM−STWTHTF2+λ2WF2+HF2=λH−2SβM−SβTHRRT,Sβ=WE.Finally, by splicing the matrixE∗W with the matrix H∗R, the k-dimensional network representation vector with attribute information can be obtained. It is assigned to the syntactic vectors Ei of V vertices in turn. These representation vectors not only contain the topological structure information of the syntactic parse tree but also contain the syntactic label information of the nodes and the semantic information of the word vector.Many natural language processing models process sentences as time series, such as recurrent neural networks [29, 39]. From the perspective of syntactic parsing, sentences are organized into a tree-like structure with hierarchical relationships. The structure is expanded spatially, which is more in line with the process in which words combine through syntactic rules to form complex structures in grammar.For example, the computational linguist Schubert believes that a phrase is a language unit that has an aggregate relationship with other words and phrases. A syntactic relationship exists between the words within a phrase, forming a language combination [30, 40]. Therefore, according to the parse tree’s recursive structure and the subsequent tree traversal, the BRFP model fuses the semantic vectors of each node in the parse tree. The basis for the fusion is the syntactic vector trained by the model, and finally, the embedded representation of the entire sentence is obtained from the root node. The node recursive fusion process can also be regarded as a semantic fusion process based on syntactic information. The fusion calculation rules refer to
(7)Vi=∑Vj∈CEi,EjZiVj,whereZi=∑jEi,Ej,where C is the set of child nodes of node Vi. Zi represents the normalization factor. The larger the inner product value of Ej and Ei is, the closer the syntactic component represented by node Vj is to the syntactic component represented by node Vi, which means that the semantic component corresponding to Vj occupies a greater proportion in the upper-level syntactic structure.


### 3.2. The Calculation Process

The author improved the DeepWalk method proposed by Perozzi et al. [28] and proposed a BRFP model, which constructs an undirected graph network [35] with node information based on the syntactic structure tree and word vectors, uses matrix decomposition to learn the syntactic information in the graph structure, and fuses the word vectors to generate the embedded representation of the sentence.Figure3 shows the decomposition process of the model, and the calculation steps are shown in Algorithm 1.Figure 3
BRFP model decomposition process.Algorithm 1: The model calculation steps.
BRFP AlgorithmInputWord embeddingsViw:w∈CSyntactic Structure Tree of sentenceT with labels LiOrders of multi-label setnMaximum transition stepstDimensions of syntactic embeddingkRegularization factorλ1. Build graphGfromTwhere nodeihas informationVi,Ei,Li2. Get decomposed matrix MComputeM=A+A2+⋯+At/t3. Construct syntactic matrixEand semantic matrixREijn=Ωin∩Ωjn/Ωin+Ωjn,R=V1,V2,⋯,VG4. Work out each k-dimension representationswhile not converge do:Wij⟵Wij−ηλWij−2SαMij−SαWijEijEijwhereSα=HijRijHij⟵Hij−ηλHij−2SβMij−SβHijRijRijwhereSβ=WijEijend while.ConcatenateE∗W,RT∗HT as k-dimension representations, then every row assigns to each Ei5. Compute sentence embeddingVSfor each nodei in G do :Vi=∑Vj∈CEi,Ej/ZiVj,wherenodeihaschildnodesandZi=∑Vj∈CEi,Ejend for.OutputSentence embeddingVS


## 3.1. The Principle of the BRFP Model

In the study of sentence embedding, syntactic information is an important aspect [36] because, from a grammatical point of view, the overall semantics of a sentence not only depends on the independent semantics of each word in the sentence [37] but also generates new semantics, even opposite semantics under the action of syntax.In addition, the high abstraction of natural language determines the abstraction of syntactic structure, so a quantifiable modeling method is needed to describe syntactic information.In this paper, the BRFP model introduced the Stanford parser [38] to parse the sentence to generate a syntactic structure tree. Because the tree can be regarded as a graph without rings in the data structure, BRFP constructs an undirected graph structure containing node information, shown in Figure 2, based on the syntactic structure tree and defines each component label in the tree as a graph vertex.Figure 2
Undirected graph with node information.In the graph, triple represents the node information of the vertex and the semantic vector. There are two types of vertices in the graph. The first type is terminal vertex, such as vertex 4, 5, 8, 10, 12, 13, and 14. Each vertex corresponds to an entity word in the sentence, and the semantic vector of such vertex is the trained word vector. The rest of the vertices are nonterminal vertex and only correspond to syntactic components rather than an entity word. The BRFP model uses the mean of the semantic vectors of all subvertex vertices as the semantic vector of the vertices. The calculation rule is as Equation (1), which represents the syntax vector, which is also the part that requires model training. Furthermore, it represents various syntactic labels in the syntactic parse tree:
(1)Vi=1C∑vi∈Cveci,otherwise,veci,viisterminal,Cischildsetofvi.The abovementioned undirected graph model contains the semantic information represented by word vectors and the syntactic information represented by the graph topology and vertex syntax labels. The embedding representation of the sentence is transformed into the embedding representation of the learning graph, which is precisely the embedding of the attribute network containing the vertex information.BRFP decomposes a matrixM into the product form of four matrices E, W, H, and R, where W and H are parameter matrices. R represents the semantic matrix, which is formed by splicing the semantic vectors of each vertex in the graph structure horizontally. E represents the syntactic mutual information matrix, and the element Eij is defined as follows:
(2)Eijn=Ωin∩ΩjnΩin+Ωjn.Ω⋅n is a multilabel set of order n corresponding to a vertex, representing the set of syntactic labels of all neighbor vertex within n steps away from a vertex. In the following example, the parameter n is set to 2, and E56 is calculated: the vertex within 2 steps from vertex 5 is 1, 3, and 6, and the vertex within 2 steps from vertex 6 is 1, 3, 5, 7, 8, and 9; the specific calculation process is as follows:
(3)Ω5k=S,VP,SΩ6k=S,VP,VBD,VP,TO,VP⇒E56=S,VPΩ5k+Ω6k=23+6=29.The matrixE reflects the syntactic similarity between vertices. The BRFP model uses matrix E and matrix R to add syntactic and semantic information in the decomposition process. E and R being fixed can also be regarded as two constraints on the matrix decomposition process, corresponding to the bidirectional restraints in the BRFP model. For matrix decomposition, this paper adopts the following loss function:
(4)minW,HM−ETWTHRF2+λ2WF2+HF2.Further, the second term is the regularization part, andλ is the introduced harmonic factor. During the optimization process, E and R are fixed, and it is only necessary to alternately fix and update W and H until the model converges:
(5)∂∂WM−STWTHTF2+λ2WF2+HF2=λW−2SαMT−SαTWEET,Sα=HR,(6)∂∂HM−STWTHTF2+λ2WF2+HF2=λH−2SβM−SβTHRRT,Sβ=WE.Finally, by splicing the matrixE∗W with the matrix H∗R, the k-dimensional network representation vector with attribute information can be obtained. It is assigned to the syntactic vectors Ei of V vertices in turn. These representation vectors not only contain the topological structure information of the syntactic parse tree but also contain the syntactic label information of the nodes and the semantic information of the word vector.Many natural language processing models process sentences as time series, such as recurrent neural networks [29, 39]. From the perspective of syntactic parsing, sentences are organized into a tree-like structure with hierarchical relationships. The structure is expanded spatially, which is more in line with the process in which words combine through syntactic rules to form complex structures in grammar.For example, the computational linguist Schubert believes that a phrase is a language unit that has an aggregate relationship with other words and phrases. A syntactic relationship exists between the words within a phrase, forming a language combination [30, 40]. Therefore, according to the parse tree’s recursive structure and the subsequent tree traversal, the BRFP model fuses the semantic vectors of each node in the parse tree. The basis for the fusion is the syntactic vector trained by the model, and finally, the embedded representation of the entire sentence is obtained from the root node. The node recursive fusion process can also be regarded as a semantic fusion process based on syntactic information. The fusion calculation rules refer to
(7)Vi=∑Vj∈CEi,EjZiVj,whereZi=∑jEi,Ej,where C is the set of child nodes of node Vi. Zi represents the normalization factor. The larger the inner product value of Ej and Ei is, the closer the syntactic component represented by node Vj is to the syntactic component represented by node Vi, which means that the semantic component corresponding to Vj occupies a greater proportion in the upper-level syntactic structure.


## 3.2. The Calculation Process

The author improved the DeepWalk method proposed by Perozzi et al. [28] and proposed a BRFP model, which constructs an undirected graph network [35] with node information based on the syntactic structure tree and word vectors, uses matrix decomposition to learn the syntactic information in the graph structure, and fuses the word vectors to generate the embedded representation of the sentence.Figure3 shows the decomposition process of the model, and the calculation steps are shown in Algorithm 1.Figure 3
BRFP model decomposition process.Algorithm 1: The model calculation steps.
BRFP AlgorithmInputWord embeddingsViw:w∈CSyntactic Structure Tree of sentenceT with labels LiOrders of multi-label setnMaximum transition stepstDimensions of syntactic embeddingkRegularization factorλ1. Build graphGfromTwhere nodeihas informationVi,Ei,Li2. Get decomposed matrix MComputeM=A+A2+⋯+At/t3. Construct syntactic matrixEand semantic matrixREijn=Ωin∩Ωjn/Ωin+Ωjn,R=V1,V2,⋯,VG4. Work out each k-dimension representationswhile not converge do:Wij⟵Wij−ηλWij−2SαMij−SαWijEijEijwhereSα=HijRijHij⟵Hij−ηλHij−2SβMij−SβHijRijRijwhereSβ=WijEijend while.ConcatenateE∗W,RT∗HT as k-dimension representations, then every row assigns to each Ei5. Compute sentence embeddingVSfor each nodei in G do :Vi=∑Vj∈CEi,Ej/ZiVj,wherenodeihaschildnodesandZi=∑Vj∈CEi,Ejend for.OutputSentence embeddingVS


## 4. Classification Experiments and Results

### 4.1. Experiment Setup

In this experiment, the Stanford parser [38] is used to analyze the Chinese and English corpus syntactically. The diversified language model provided by Stanford can support the syntactic analysis of English, Chinese, Arabic, and other languages. The English model of the Stanford analyzer has a total of 67 component labels, including 36 part-of-speech labels, 22 phrase labels, and nine clause structure labels. The standard component labels in the Chinese model of the Stanford analyzer mainly include 33 part-of-speech labels and 17 phrase labels.In the English data experiment in this paper, due to the Pearson correlation coefficient being used to respect the final evaluation index in the datasets, in the calculation step, firstly, the similarity value of all sentence pairs is calculated by cosine similarity. Then, the Pearson correlation coefficient [41] is calculated with the standard similarity score in the datasets as the final evaluation index.The Pearson correlation coefficient is a statistical concept used to measure the linear correlation between two random variables with a value between -1 and 1. Here,Xi is the text similarity prediction value, Yi is the given standard similarity value, and the calculation equation is as follows:
(8)Pearson=∑i=1nXi−X¯Yi−Y¯∑i=1nXi−X¯2∑i=1nYi−Y¯2.The Chinese data experiments are typical binary classification problems [42], so this paper uses the accuracy and F1 value as the evaluation indicators. First, the cosine similarity is used to calculate the similarity value of all sentence pairs, and then, an optimal threshold is determined by the model. In addition, this experiment uses the Chinese and English word vectors trained by word2vec, and the default length is 100 dimensions.


### 4.2. Analysis of Experimental Results in Chinese and English

The English corpus uses the text semantic similarity task of the 2012-2016 International Workshop on Semantic Evaluation [43]. The competition publishes several datasets each year. The corpus is collected from the news, videos, forums, picture descriptions, Twitter, and many other fields, and the number of datasets and corpus is adjusted annually. For example, STS-12 provides five datasets, including MSRpar, MSRvid, OnWN, SMTnews, and SMTeuroparl, and STS-13 includes four datasets of HDL, FNWN, OnWN, and SMT, replacing some datasets based on STS-12. The number of samples also fluctuates to a certain extent. The STS datasets used in the experiment are shown in Table 1. The data in brackets is the number of samples in the dataset. The STS task is designed to measure the degree of semantic similarity. Each data sample consists of a sentence pair and a standard similarity score. The similarity score ranges from 0 to 5. The higher the score, the closer the semantics of the two sentences.Table 1
The STS datasets used in the experiment.
STS-12STS-13STS-14STS-15STS-16MSRpar (1500)HDL (750)HDL (750)HDL (750)HDL (249)MSRvid (1500)FNWN (189)OnWN (750)Images (750)Plagiarism (230)OnWN (750)OnWN (561)Deft-forum (450)Ans.-student (750)Post-editing (244)SMTnews (750)SMT (750)Deft-news (300)Ans.-forum (375)Ans.-Ans. (254)SMTeuroparl (750)Images (750)Belief (375)Quest.-Quest. (209)Tweet-news (750)Due to the differences in the sample sources of the datasets included in the STS tasks in different years, the average Pearson coefficient of each dataset is used here as the final presentation index. As a widely used text semantic similarity dataset worldwide, the STS task can thoroughly verify the rationality and effectiveness of the BRFP model. For intuitive expression, all Pearson correlation coefficients in the experimental data in this paper are multiplied by 100 by default.In Table2, STS-12 to STS-16 represent the datasets of different years from 2012 to 2016 in the International Workshop on Semantic Evaluation.Table 2
English text experimental results.
ModelSTS−12STS−13STS−14STS−15STS−16TF−IDF55.0752.1457.3960.8562.39SIF57.5261.8368.5073.7371.50CNN56.1054.2259.5552.7560.35RNN48.2159.3658.6057.2960.31LSTM53.5658.4962.5666.5367.30ST50.8442.8761.4962.0355.73QT58.5757.7066.5268.4569.05InferSent59.2860.6767.7372.5072.41BRFP59.0865.3271.8073.4372.35The bold numbers in Table2 represent the best performance on the specific dataset. The experimental results show that InferSent [44] achieves the best performance on STS-12 and STS-16 datasets and has good performance on other datasets, proving that InferSent is an excellent baseline model. Our method achieves the highest scores on the STS-13 and STS-14 tasks, which are 3.5 and 2.3 higher than the other state-of-the-art performances, respectively, even though our method fails to achieve the best results on all datasets. Nevertheless, they are also close compared to the best methods, with differences of 0.2, 0.3, and 0.06, respectively. It is difficult for any single model to apply to language representations in different fields, considering the complexity of natural language and the diversity of application scenarios, but the BRFP model has shown relatively good performance.The Chinese dataset uses the Ant Financial Question Matching Corpus dataset (AFQMC) [45] and a Large-scale Chinese Question Matching Corpus dataset (LCQMC) [46], both of which are binary classification tasks to determine whether two sentences are semantically similar. For the AFQMC dataset, 100,000 pairs of labeled data are used in this experiment. For the LCQMC dataset, 238,766 training samples, 8,802 validation samples, and 12,500 test samples are used in the experiment.According to the above experimental results in Tables3 and 4 and Figure 4, the all-around performance of BRFP and LSTM methods [47] on the AFQMC dataset is significantly better than the other three models. Compared with LSTM, BFRP exceeds 1.1 and 0.3 in accuracy and F1 value, respectively, and the performance improvement is relatively apparent. Although BRFP failed to achieve the highest accuracy rate on the LCQMC dataset, it significantly surpassed neural network models such as RNN [48] and Quick-Thought (QT) [18], only 0.4 lower than the best-performing InferSent and the same as InferSent in F1 value.Table 3
The Chinese text experimental results on the AFQMC dataset.
ModelAccuracy (%)F1 (%)TF-IDF63.562.4CNN69.370.7LSTM75.775.2QT68.967.4BRFP76.875.5Table 4
The Chinese text experimental results on the LCQMC dataset.
ModelAccuracy (%)F1 (%)SIF77.478.2RNN76.775.7ST68.368.9InferSent79.478.5BRFP79.078.4Figure 4
(a) Description of the result on AFQMC dataset; (b) description of the result on LCQMC dataset.
(a)(b)The AFQMC and LCQMC datasets have a wide range of sample sources. The AFQMC dataset is derived from the interactive Q&A between customer service and users in the production environment and involves some proper nouns in finance and e-commerce, which can thoroughly test the model’s ability to deal with unknown and low-frequency words.In addition, the LCQMC dataset contains colloquial description text, which involves many different fields, which is a massive challenge to the model’s generalization ability. Therefore, the stable performance of different Chinese data further verifies the generality of the BRFP model.


### 4.3. Compared with the Word Vector Weighted Model

This subsection compares the BRFP model with the weighted model based on word vectors, including TF-IDF and SIF methods. This subsection uses the text semantic similarity of the 2012 and 2013 International Semantic Evaluation Competition, in which STS-12 includes five datasets and STS-13 includes four datasets.Since different years may contain datasets with the same name, the following experiments are distinguished by the year plus the dataset name. For example, the OnWN datasets of the STS-12 task and STS-13 task are denoted as 12’OnWN and 13’OnWN. It is not used as the experimental dataset in this section, considering that the total number of samples in the 13’FNW dataset is small. After excluding 13’FNWN, the experiment in this section actually contains 8 datasets, namely, 12’MSRpar, 12’MSRvid, 12’OnWN, 12’SMTnews, 12’SMTeuroparl, 13’HDL, 13’OnWN, and 13’SMT.The experimental results in Table5 show that the BRFP model outperforms the comparative baselines in five of the eight selected datasets and achieves the best performance, which is reasonably competitive. In addition, the BRFP model has achieved significant performance improvements on some datasets such as 12’MSRpar, 12’SMTeuroparl, 13’OnWN, and 13’SMT, and the overall improvement is between 4 and 6 percentage points.Table 5
The result of comparison with the word vector weighted model.
DatasetTF-IDFSIFBRFP12’MSRpar43.5341.8747.5012’MSRvid69.3479.0778.9312’OnWN63.5866.1567.0912’SMTnews68.0558.6366.5812’SMTeuroparl43.0753.2557.4713’HDL47.5651.3450.0913’OnWN50.0957.8063.9413’SMT51.7749.5556.30Both TF-IDF and SIF measure the contribution of each word in sentence semantics based on word frequency but do not reflect the impact of the interaction between syntactic components on the overall semantics of the sentence. As the experimental result shows, the corresponding syntactic structure is more complex, and the effect of the BRFP model is improved more obviously.


### 4.4. Compared with Neural Network Model

This section compares the BRFP model with some neural network models and baseline models including CNN [49], RNN [50], LSTM [51], Skip-Thought (ST) [17], Quick-Thought QT [18], and InferSent [44]. Given the small number of samples in each dataset of the STS-16 task, this section uses 11 datasets provided by STS-14 and STS-15. The content of these datasets involves news, forums, images, and many other fields. The experiment uses different experiments on datasets in the field to fully verify the model’s generality.From the data in Table6, it can be seen that the BRFP model’s performance exceeds the baseline model’s performance on more than half of the datasets, and the effect is significantly improved on the 14’OnWN, 14’Tweet-news, and 15’Ans.-forum datasets, which are definite improvements of 2.87, 3.52, and 2.4. However, A particular gap has been found in the BRFP model compared to the best comparison method in the two datasets of 14’Images and 15’Images. The possible reason is that these two datasets contain a large number of unique characters, such as “‰” and “3/4.” The existence of special characters affects the accuracy of syntactic analysis and reduces the accuracy of syntactic information learned by the model.Table 6
The result of comparison with neural network model.
DatasetCNNRNNLSTMSTQTInferSentBRFP14’HDL60.1855.4062.6661.0570.4667.5169.5314’OnWN63.5061.3558.3655.4761.2559.3566.3714’Deft−forum58.3759.6765.7458.3665.5870.9771.0214’Deft−news51.0342.1667.0565.0067.6066.3266.4714’Images67.2567.3961.5061.9271.0977.6074.6914’Tweet−news71.9370.0556.0467.5074.3575.4278.9415’HDL73.5465.3766.5068.5766.8371.6073.9515’Images64.3053.0260.3758.9361.0767.5264.1815’Ans.−student59.7551.6071.5353.6670.4374.3875.6715’Ans.−forum62.9560.5070.4861.1072.4570.8074.8515’Belief65.8361.7267.2570.3568.5072.9572.50According to statistics, the number of samples containing special characters in 14’Images and 15’Images accounts for 9% and 11% of the total samples, respectively. The existence of a large number of special characters weakens the ability of the BRFP model to capture effective syntax.


### 4.5. Validity Experiment of Syntactic Information

This subsection will illustrate the effectiveness of syntactic information in the model from another perspective to further illustrate competitiveness and interpretability. The BRFP model transforms sentence embedding representation learning into graph embedding representation learning with vertex information and adopts a triple to represent vertex information. Whether the syntactic vector learned by the model can reflect syntactic information is also an aspect worthy of further discussion.In this paper, 3000 sentences are randomly selected from the STS task historical dataset and input to the BRFP model, and 3000 trained undirected graphs with complete vertex information can be obtained. Then, for all undirected graphs, the average similarity between the syntactic vectors corresponding to the vertices of different component labels is calculated. Since the English syntactic analysis of the Stanford analyzer has a total of 67 component labels, there are many combinations between labels, and only the similarity relationship data of some component labels are displayed in Table7.Table 7
The English ingredient label similarity results.
ADJPNPPPQPVPCD0.170.730.650.920.20DT0.400.850.670.870.37IN0.390.140.980.130.51JJ0.960.800.480.170.46JJR0.740.690.300.140.35JJS0.830.730.350.070.38NN0.910.890.770.810.73NNS0.870.830.710.850.82NNP0.650.710.870.620.69VB0.620.340.450.340.93VBD0.570.380.330.230.84VBN0.510.270.200.300.73VBG0.570.100.160.170.90VBP0.440.140.240.360.81VBZ0.400.090.320.290.85According to the data in Table7, it can be observed the similarity distribution between the syntax vectors.For example, an adjective phrase’s most common syntactic composition (ADVP) is “adjective+noun.” From the data in Table7, it can be seen that ADVP is related to the adjective (JJ), adjective comparative (JJR), adjective superlative (JJS), and noun singular (NN). The average similarity of the syntactic vectors corresponding to the component labels such as noun plural (NNS) is significantly higher than other labels. Verb phrases (VP) and noun labels (NN, NNS) and verb labels (VB, VBD, VBN, VBG, VBP, and VBZ) are much more similar than the rest of the component labels.In addition, the syntactic similarity between prepositional phrases (PP), position conjunctions (IN), and proper nouns (NNP) is high, which also fits the syntactic structure of “preposition+place noun.”It can be seen that the syntactic vectors learned by BRFP reflect the regularity of collocation between different syntactic components, and the process of using syntactic vectors to weight and fuse word vectors is also a process of semantic fusion based on syntactic information.


### 4.6. Hyperparameter Analysis

The BRFP model has four essential parametersk, t, n, and λ, of which the maximum number of transition steps t and the multilabel set order n both reflect the utilization range of the information around the vertex, so in the practical application of the model, t and n usually take the same value to simplify model parameters.Therefore, this subsection will focus on the impact of changes in parametersk, n, and λ on model performance. First, the STS-15 and STS-16 datasets were selected to study the influence of the parameter k, and five groups of different n and λ values were randomly selected for each dataset. The experimental results are shown in Figure 5.Figure 5
(a) Description of the result on STS-15; (b) description of the result on STS-16.
(a)(b)Under different combinations ofn and λ values, most of the curves show the peak value of the Pearson coefficient in the interval of 80-100, and the model effect reaches the optimal state. Considering the overall data, 70-100 is still the most ideal value range for the syntactic vector dimension k, although a small number of curves show that k reaches a peak when 60 or 100 is taken, followed by a slight downward trend. In order to control the variables, the default k is 90 in the subsequent experiments in this subsection.It can be seen from Figure6(a) that the increase of n significantly improves the effect of the experiment. When n exceeds 8, most of the curves begin to show a downward trend; especially when n is greater than 10, the Pearson coefficient generally drops sharply.Figure 6
(a) Description of the result on parametern; (b) description of the result on parameter λ.
(a)(b)Further, it can be inferred that the ideal value ofn is between 4 and 7. The main reason is that the principle of multiple label sets is to use the component labels of the neighboring vertices of the vertices to approximate the syntactic similarity between vertices, which is similar to the K nearest neighbor algorithm.Whenn is relatively small, the increase of n means that the multilabel set can cover more label information of neighbor vertices, and it will be more accurate in the judgment of syntactic similarity between two vertices.However, when the value ofn is large enough, because the syntactic structure tree of short sentences has relatively few nodes, high-order multilabel sets will produce convergence. In extreme cases, if n exceeds the total number of nodes in the syntactic structure tree, the multiple label sets corresponding to any two nodes are approximately the same, which will weaken the constraining effect of syntactic information in the subsequent matrix decomposition process.In addition, as can be seen from Figure6(b), after the harmonic factor λ reaches 0.6, the performance is basically in a relatively ideal state. Although the subsequent curve trend fluctuates, the overall stability remains stable.


## 4.1. Experiment Setup

In this experiment, the Stanford parser [38] is used to analyze the Chinese and English corpus syntactically. The diversified language model provided by Stanford can support the syntactic analysis of English, Chinese, Arabic, and other languages. The English model of the Stanford analyzer has a total of 67 component labels, including 36 part-of-speech labels, 22 phrase labels, and nine clause structure labels. The standard component labels in the Chinese model of the Stanford analyzer mainly include 33 part-of-speech labels and 17 phrase labels.In the English data experiment in this paper, due to the Pearson correlation coefficient being used to respect the final evaluation index in the datasets, in the calculation step, firstly, the similarity value of all sentence pairs is calculated by cosine similarity. Then, the Pearson correlation coefficient [41] is calculated with the standard similarity score in the datasets as the final evaluation index.The Pearson correlation coefficient is a statistical concept used to measure the linear correlation between two random variables with a value between -1 and 1. Here,Xi is the text similarity prediction value, Yi is the given standard similarity value, and the calculation equation is as follows:
(8)Pearson=∑i=1nXi−X¯Yi−Y¯∑i=1nXi−X¯2∑i=1nYi−Y¯2.The Chinese data experiments are typical binary classification problems [42], so this paper uses the accuracy and F1 value as the evaluation indicators. First, the cosine similarity is used to calculate the similarity value of all sentence pairs, and then, an optimal threshold is determined by the model. In addition, this experiment uses the Chinese and English word vectors trained by word2vec, and the default length is 100 dimensions.


## 4.2. Analysis of Experimental Results in Chinese and English

The English corpus uses the text semantic similarity task of the 2012-2016 International Workshop on Semantic Evaluation [43]. The competition publishes several datasets each year. The corpus is collected from the news, videos, forums, picture descriptions, Twitter, and many other fields, and the number of datasets and corpus is adjusted annually. For example, STS-12 provides five datasets, including MSRpar, MSRvid, OnWN, SMTnews, and SMTeuroparl, and STS-13 includes four datasets of HDL, FNWN, OnWN, and SMT, replacing some datasets based on STS-12. The number of samples also fluctuates to a certain extent. The STS datasets used in the experiment are shown in Table 1. The data in brackets is the number of samples in the dataset. The STS task is designed to measure the degree of semantic similarity. Each data sample consists of a sentence pair and a standard similarity score. The similarity score ranges from 0 to 5. The higher the score, the closer the semantics of the two sentences.Table 1
The STS datasets used in the experiment.
STS-12STS-13STS-14STS-15STS-16MSRpar (1500)HDL (750)HDL (750)HDL (750)HDL (249)MSRvid (1500)FNWN (189)OnWN (750)Images (750)Plagiarism (230)OnWN (750)OnWN (561)Deft-forum (450)Ans.-student (750)Post-editing (244)SMTnews (750)SMT (750)Deft-news (300)Ans.-forum (375)Ans.-Ans. (254)SMTeuroparl (750)Images (750)Belief (375)Quest.-Quest. (209)Tweet-news (750)Due to the differences in the sample sources of the datasets included in the STS tasks in different years, the average Pearson coefficient of each dataset is used here as the final presentation index. As a widely used text semantic similarity dataset worldwide, the STS task can thoroughly verify the rationality and effectiveness of the BRFP model. For intuitive expression, all Pearson correlation coefficients in the experimental data in this paper are multiplied by 100 by default.In Table2, STS-12 to STS-16 represent the datasets of different years from 2012 to 2016 in the International Workshop on Semantic Evaluation.Table 2
English text experimental results.
ModelSTS−12STS−13STS−14STS−15STS−16TF−IDF55.0752.1457.3960.8562.39SIF57.5261.8368.5073.7371.50CNN56.1054.2259.5552.7560.35RNN48.2159.3658.6057.2960.31LSTM53.5658.4962.5666.5367.30ST50.8442.8761.4962.0355.73QT58.5757.7066.5268.4569.05InferSent59.2860.6767.7372.5072.41BRFP59.0865.3271.8073.4372.35The bold numbers in Table2 represent the best performance on the specific dataset. The experimental results show that InferSent [44] achieves the best performance on STS-12 and STS-16 datasets and has good performance on other datasets, proving that InferSent is an excellent baseline model. Our method achieves the highest scores on the STS-13 and STS-14 tasks, which are 3.5 and 2.3 higher than the other state-of-the-art performances, respectively, even though our method fails to achieve the best results on all datasets. Nevertheless, they are also close compared to the best methods, with differences of 0.2, 0.3, and 0.06, respectively. It is difficult for any single model to apply to language representations in different fields, considering the complexity of natural language and the diversity of application scenarios, but the BRFP model has shown relatively good performance.The Chinese dataset uses the Ant Financial Question Matching Corpus dataset (AFQMC) [45] and a Large-scale Chinese Question Matching Corpus dataset (LCQMC) [46], both of which are binary classification tasks to determine whether two sentences are semantically similar. For the AFQMC dataset, 100,000 pairs of labeled data are used in this experiment. For the LCQMC dataset, 238,766 training samples, 8,802 validation samples, and 12,500 test samples are used in the experiment.According to the above experimental results in Tables3 and 4 and Figure 4, the all-around performance of BRFP and LSTM methods [47] on the AFQMC dataset is significantly better than the other three models. Compared with LSTM, BFRP exceeds 1.1 and 0.3 in accuracy and F1 value, respectively, and the performance improvement is relatively apparent. Although BRFP failed to achieve the highest accuracy rate on the LCQMC dataset, it significantly surpassed neural network models such as RNN [48] and Quick-Thought (QT) [18], only 0.4 lower than the best-performing InferSent and the same as InferSent in F1 value.Table 3
The Chinese text experimental results on the AFQMC dataset.
ModelAccuracy (%)F1 (%)TF-IDF63.562.4CNN69.370.7LSTM75.775.2QT68.967.4BRFP76.875.5Table 4
The Chinese text experimental results on the LCQMC dataset.
ModelAccuracy (%)F1 (%)SIF77.478.2RNN76.775.7ST68.368.9InferSent79.478.5BRFP79.078.4Figure 4
(a) Description of the result on AFQMC dataset; (b) description of the result on LCQMC dataset.
(a)(b)The AFQMC and LCQMC datasets have a wide range of sample sources. The AFQMC dataset is derived from the interactive Q&A between customer service and users in the production environment and involves some proper nouns in finance and e-commerce, which can thoroughly test the model’s ability to deal with unknown and low-frequency words.In addition, the LCQMC dataset contains colloquial description text, which involves many different fields, which is a massive challenge to the model’s generalization ability. Therefore, the stable performance of different Chinese data further verifies the generality of the BRFP model.


## 4.3. Compared with the Word Vector Weighted Model

This subsection compares the BRFP model with the weighted model based on word vectors, including TF-IDF and SIF methods. This subsection uses the text semantic similarity of the 2012 and 2013 International Semantic Evaluation Competition, in which STS-12 includes five datasets and STS-13 includes four datasets.Since different years may contain datasets with the same name, the following experiments are distinguished by the year plus the dataset name. For example, the OnWN datasets of the STS-12 task and STS-13 task are denoted as 12’OnWN and 13’OnWN. It is not used as the experimental dataset in this section, considering that the total number of samples in the 13’FNW dataset is small. After excluding 13’FNWN, the experiment in this section actually contains 8 datasets, namely, 12’MSRpar, 12’MSRvid, 12’OnWN, 12’SMTnews, 12’SMTeuroparl, 13’HDL, 13’OnWN, and 13’SMT.The experimental results in Table5 show that the BRFP model outperforms the comparative baselines in five of the eight selected datasets and achieves the best performance, which is reasonably competitive. In addition, the BRFP model has achieved significant performance improvements on some datasets such as 12’MSRpar, 12’SMTeuroparl, 13’OnWN, and 13’SMT, and the overall improvement is between 4 and 6 percentage points.Table 5
The result of comparison with the word vector weighted model.
DatasetTF-IDFSIFBRFP12’MSRpar43.5341.8747.5012’MSRvid69.3479.0778.9312’OnWN63.5866.1567.0912’SMTnews68.0558.6366.5812’SMTeuroparl43.0753.2557.4713’HDL47.5651.3450.0913’OnWN50.0957.8063.9413’SMT51.7749.5556.30Both TF-IDF and SIF measure the contribution of each word in sentence semantics based on word frequency but do not reflect the impact of the interaction between syntactic components on the overall semantics of the sentence. As the experimental result shows, the corresponding syntactic structure is more complex, and the effect of the BRFP model is improved more obviously.


## 4.4. Compared with Neural Network Model

This section compares the BRFP model with some neural network models and baseline models including CNN [49], RNN [50], LSTM [51], Skip-Thought (ST) [17], Quick-Thought QT [18], and InferSent [44]. Given the small number of samples in each dataset of the STS-16 task, this section uses 11 datasets provided by STS-14 and STS-15. The content of these datasets involves news, forums, images, and many other fields. The experiment uses different experiments on datasets in the field to fully verify the model’s generality.From the data in Table6, it can be seen that the BRFP model’s performance exceeds the baseline model’s performance on more than half of the datasets, and the effect is significantly improved on the 14’OnWN, 14’Tweet-news, and 15’Ans.-forum datasets, which are definite improvements of 2.87, 3.52, and 2.4. However, A particular gap has been found in the BRFP model compared to the best comparison method in the two datasets of 14’Images and 15’Images. The possible reason is that these two datasets contain a large number of unique characters, such as “‰” and “3/4.” The existence of special characters affects the accuracy of syntactic analysis and reduces the accuracy of syntactic information learned by the model.Table 6
The result of comparison with neural network model.
DatasetCNNRNNLSTMSTQTInferSentBRFP14’HDL60.1855.4062.6661.0570.4667.5169.5314’OnWN63.5061.3558.3655.4761.2559.3566.3714’Deft−forum58.3759.6765.7458.3665.5870.9771.0214’Deft−news51.0342.1667.0565.0067.6066.3266.4714’Images67.2567.3961.5061.9271.0977.6074.6914’Tweet−news71.9370.0556.0467.5074.3575.4278.9415’HDL73.5465.3766.5068.5766.8371.6073.9515’Images64.3053.0260.3758.9361.0767.5264.1815’Ans.−student59.7551.6071.5353.6670.4374.3875.6715’Ans.−forum62.9560.5070.4861.1072.4570.8074.8515’Belief65.8361.7267.2570.3568.5072.9572.50According to statistics, the number of samples containing special characters in 14’Images and 15’Images accounts for 9% and 11% of the total samples, respectively. The existence of a large number of special characters weakens the ability of the BRFP model to capture effective syntax.


## 4.5. Validity Experiment of Syntactic Information

This subsection will illustrate the effectiveness of syntactic information in the model from another perspective to further illustrate competitiveness and interpretability. The BRFP model transforms sentence embedding representation learning into graph embedding representation learning with vertex information and adopts a triple to represent vertex information. Whether the syntactic vector learned by the model can reflect syntactic information is also an aspect worthy of further discussion.In this paper, 3000 sentences are randomly selected from the STS task historical dataset and input to the BRFP model, and 3000 trained undirected graphs with complete vertex information can be obtained. Then, for all undirected graphs, the average similarity between the syntactic vectors corresponding to the vertices of different component labels is calculated. Since the English syntactic analysis of the Stanford analyzer has a total of 67 component labels, there are many combinations between labels, and only the similarity relationship data of some component labels are displayed in Table7.Table 7
The English ingredient label similarity results.
ADJPNPPPQPVPCD0.170.730.650.920.20DT0.400.850.670.870.37IN0.390.140.980.130.51JJ0.960.800.480.170.46JJR0.740.690.300.140.35JJS0.830.730.350.070.38NN0.910.890.770.810.73NNS0.870.830.710.850.82NNP0.650.710.870.620.69VB0.620.340.450.340.93VBD0.570.380.330.230.84VBN0.510.270.200.300.73VBG0.570.100.160.170.90VBP0.440.140.240.360.81VBZ0.400.090.320.290.85According to the data in Table7, it can be observed the similarity distribution between the syntax vectors.For example, an adjective phrase’s most common syntactic composition (ADVP) is “adjective+noun.” From the data in Table7, it can be seen that ADVP is related to the adjective (JJ), adjective comparative (JJR), adjective superlative (JJS), and noun singular (NN). The average similarity of the syntactic vectors corresponding to the component labels such as noun plural (NNS) is significantly higher than other labels. Verb phrases (VP) and noun labels (NN, NNS) and verb labels (VB, VBD, VBN, VBG, VBP, and VBZ) are much more similar than the rest of the component labels.In addition, the syntactic similarity between prepositional phrases (PP), position conjunctions (IN), and proper nouns (NNP) is high, which also fits the syntactic structure of “preposition+place noun.”It can be seen that the syntactic vectors learned by BRFP reflect the regularity of collocation between different syntactic components, and the process of using syntactic vectors to weight and fuse word vectors is also a process of semantic fusion based on syntactic information.


## 4.6. Hyperparameter Analysis

The BRFP model has four essential parametersk, t, n, and λ, of which the maximum number of transition steps t and the multilabel set order n both reflect the utilization range of the information around the vertex, so in the practical application of the model, t and n usually take the same value to simplify model parameters.Therefore, this subsection will focus on the impact of changes in parametersk, n, and λ on model performance. First, the STS-15 and STS-16 datasets were selected to study the influence of the parameter k, and five groups of different n and λ values were randomly selected for each dataset. The experimental results are shown in Figure 5.Figure 5
(a) Description of the result on STS-15; (b) description of the result on STS-16.
(a)(b)Under different combinations ofn and λ values, most of the curves show the peak value of the Pearson coefficient in the interval of 80-100, and the model effect reaches the optimal state. Considering the overall data, 70-100 is still the most ideal value range for the syntactic vector dimension k, although a small number of curves show that k reaches a peak when 60 or 100 is taken, followed by a slight downward trend. In order to control the variables, the default k is 90 in the subsequent experiments in this subsection.It can be seen from Figure6(a) that the increase of n significantly improves the effect of the experiment. When n exceeds 8, most of the curves begin to show a downward trend; especially when n is greater than 10, the Pearson coefficient generally drops sharply.Figure 6
(a) Description of the result on parametern; (b) description of the result on parameter λ.
(a)(b)Further, it can be inferred that the ideal value ofn is between 4 and 7. The main reason is that the principle of multiple label sets is to use the component labels of the neighboring vertices of the vertices to approximate the syntactic similarity between vertices, which is similar to the K nearest neighbor algorithm.Whenn is relatively small, the increase of n means that the multilabel set can cover more label information of neighbor vertices, and it will be more accurate in the judgment of syntactic similarity between two vertices.However, when the value ofn is large enough, because the syntactic structure tree of short sentences has relatively few nodes, high-order multilabel sets will produce convergence. In extreme cases, if n exceeds the total number of nodes in the syntactic structure tree, the multiple label sets corresponding to any two nodes are approximately the same, which will weaken the constraining effect of syntactic information in the subsequent matrix decomposition process.In addition, as can be seen from Figure6(b), after the harmonic factor λ reaches 0.6, the performance is basically in a relatively ideal state. Although the subsequent curve trend fluctuates, the overall stability remains stable.


## 5. Model Feature of Efficient and Universal

The model proposed in this paper is an efficient and universal sentence embedding learning framework that can flexibly integrate different word embedding schemes and syntactic analysis techniques, considering word embedding and syntactic analysis as the framework’s building blocks.For the word embedding method, word2vec is a typical method, and C&W, Glove, FastText, and ELMo are also popular word embedding methods. In order to verify the usability of the model combination, this paper integrates five word embedding methods, word2vec, C&W, Glove, FastText, and ELMo, into the BRFP model and conducts experiments on the classification effect of the model in the STS2012-2015 dataset.For syntactic analysis, this paper constructs BRFP model variants based on two different syntax tree structures, syntax structure tree (SST) and Dependency Syntactic Tree (DST), and conducts comparative experiments.The model proposed in this paper has adopted the syntax tree and achieved good performance. Studying whether the dependency syntax tree can achieve good results as a model building block is also precious. In addition to the Stanford analyzer, the Language Technology Platform (LTP) [52] developed by the Harbin Institute of Technology can provide complete technical support for Chinese natural language processing.As a complete set of Chinese natural language processing systems, including a series of language processing modules such as lexical, syntactic, and semantic, LTP has become one of the most influential Chinese language processing platforms at home and abroad.As shown in Figure7, this paper will use the syntactic structure analysis and syntactic dependency analysis of the Stanford analyzer and LTP to construct four variant methods, namely, Stanford+SST, Stanford+DST, LTP+SST, and LTP+DST, where SST and DST are the abbreviations for syntax structure tree and Dependency Syntax Tree.Figure 7
Schematic diagram of model combination.


### 5.1. The Experimental Results of BRFP and Word Embedding Fusion

The dataset used in this experiment is the text semantic similarity task of the 2012-2015 International Workshop on Semantic Evaluation. In order to ensure the same experimental conditions, the experiments in this section only change the category of word vectors. All word vectors use the official pretraining model, and other variables such as syntactic analysis method, word vector dimension, and hyperparameter values are the same.As shown in Figure8, the five word embedding methods, word2vec, C&W, Glove, FastText, and ELMo, can all achieve good classification results after they are integrated into BRFP. This experiment proves efficient and universal to the BRFP framework. Different word embedding methods can be selected according to application scenarios to achieve the best results. It is also possible to use more powerful word embedding methods to improve the BRFP model’s performance in the future.Figure 8
Different word embedding method fusion experimental result.


### 5.2. The Experimental Results of BRFP and Syntactic Analysis Fusion

In this experiment, the AFQMC and the LCQMC datasets are selected, both of which are classification tasks. The F1 value is used as the evaluation index to judge whether the two sentences’ semantics are similar. In order to ensure the fairness of the comparison environment of variant methods, this section only adjusts the syntactic analysis technology, and other conditions remain the same.Syntactic analysis is a critical method in the field of NLP, and in addition to structural analysis, syntactic analysis methods also include dependency analysis. Dependency analysis considers that the syntactic functions of words to other words are described by dependencies, emphasizing the dependencies between local words and constraining them into a tree structure.The arrow direction in Figure9 points from the dominant word to the subordinate word, and the label on the edge is the relation type. Representing the dependencies of all words in a sentence in the form of directed edges results in a tree called a dependency syntax tree.Figure 9
Dependency syntax tree.In modern dependency grammars, linguist Robinson [53] proposes four binding axioms for dependency syntax trees:
(1)
There is only one word or virtual root node that does not depend on other words(2)
All words except the virtual root node must depend on other words(3)
Each word cannot depend on multiple words(4)
If A depends on B, then A and B can only depend on words between A and BThe above four axioms constrain the uniqueness, connectivity, acyclicity, and projectivity of the root node of the dependency syntax tree, respectively. Therefore, the dependency syntax tree and the syntax structure tree are structurally homogenous and can also be used as the input of the BRFP model.As shown in Figure10, the syntactic information expressed by the generated tree structure is different due to the different focuses of the two analyses on the two syntactic analysis platforms of Stanford analyzer and LTP, with the practical effect of using the syntactic structure tree significantly better than the Dependency Syntax Tree. Since the syntactic structure analysis is to decompose the sentence structure layer by layer starting from the global structure, the proposition of concern is the sentence generation process. On the other hand, dependency analysis emphasizes the grammatical connection between local words, believes there is a master-slave relationship between words, and constrains this master-slave relationship into a tree structure. The experimental results also show that the syntactic structure tree is more suitable for the text-similarity task.Figure 10
Different syntactic analysis method fusion experimental result.By comparing the Stanford+SST and LTP+SST methods horizontally, it is found that the F1 values of the two methods on the LCQMC dataset are the same, but the LTP performs better on the AFQMC dataset, and the F1 value increases by about 1.5 percentage points. The difference is the granularity of word segmentation, which is more evident in Chinese texts. Word segmentation is usually not required in English text preprocessing, or spaces are directly split. The Chinese word segmentation granularity of LTP is smaller than that of the Stanford analyzer. For example, LTP will treat 北京大学 (Peking University) as a whole noun and will not split it, while the word segmentation result of the Stanford analyzer is 北京(Peking) and 大学 (University). The original node in the tree will be split into two, thus changing the entire tree structure, which will split the original semantics of some words.


## 5.1. The Experimental Results of BRFP and Word Embedding Fusion

The dataset used in this experiment is the text semantic similarity task of the 2012-2015 International Workshop on Semantic Evaluation. In order to ensure the same experimental conditions, the experiments in this section only change the category of word vectors. All word vectors use the official pretraining model, and other variables such as syntactic analysis method, word vector dimension, and hyperparameter values are the same.As shown in Figure8, the five word embedding methods, word2vec, C&W, Glove, FastText, and ELMo, can all achieve good classification results after they are integrated into BRFP. This experiment proves efficient and universal to the BRFP framework. Different word embedding methods can be selected according to application scenarios to achieve the best results. It is also possible to use more powerful word embedding methods to improve the BRFP model’s performance in the future.Figure 8
Different word embedding method fusion experimental result.


## 5.2. The Experimental Results of BRFP and Syntactic Analysis Fusion

In this experiment, the AFQMC and the LCQMC datasets are selected, both of which are classification tasks. The F1 value is used as the evaluation index to judge whether the two sentences’ semantics are similar. In order to ensure the fairness of the comparison environment of variant methods, this section only adjusts the syntactic analysis technology, and other conditions remain the same.Syntactic analysis is a critical method in the field of NLP, and in addition to structural analysis, syntactic analysis methods also include dependency analysis. Dependency analysis considers that the syntactic functions of words to other words are described by dependencies, emphasizing the dependencies between local words and constraining them into a tree structure.The arrow direction in Figure9 points from the dominant word to the subordinate word, and the label on the edge is the relation type. Representing the dependencies of all words in a sentence in the form of directed edges results in a tree called a dependency syntax tree.Figure 9
Dependency syntax tree.In modern dependency grammars, linguist Robinson [53] proposes four binding axioms for dependency syntax trees:
(1)
There is only one word or virtual root node that does not depend on other words(2)
All words except the virtual root node must depend on other words(3)
Each word cannot depend on multiple words(4)
If A depends on B, then A and B can only depend on words between A and BThe above four axioms constrain the uniqueness, connectivity, acyclicity, and projectivity of the root node of the dependency syntax tree, respectively. Therefore, the dependency syntax tree and the syntax structure tree are structurally homogenous and can also be used as the input of the BRFP model.As shown in Figure10, the syntactic information expressed by the generated tree structure is different due to the different focuses of the two analyses on the two syntactic analysis platforms of Stanford analyzer and LTP, with the practical effect of using the syntactic structure tree significantly better than the Dependency Syntax Tree. Since the syntactic structure analysis is to decompose the sentence structure layer by layer starting from the global structure, the proposition of concern is the sentence generation process. On the other hand, dependency analysis emphasizes the grammatical connection between local words, believes there is a master-slave relationship between words, and constrains this master-slave relationship into a tree structure. The experimental results also show that the syntactic structure tree is more suitable for the text-similarity task.Figure 10
Different syntactic analysis method fusion experimental result.By comparing the Stanford+SST and LTP+SST methods horizontally, it is found that the F1 values of the two methods on the LCQMC dataset are the same, but the LTP performs better on the AFQMC dataset, and the F1 value increases by about 1.5 percentage points. The difference is the granularity of word segmentation, which is more evident in Chinese texts. Word segmentation is usually not required in English text preprocessing, or spaces are directly split. The Chinese word segmentation granularity of LTP is smaller than that of the Stanford analyzer. For example, LTP will treat 北京大学 (Peking University) as a whole noun and will not split it, while the word segmentation result of the Stanford analyzer is 北京(Peking) and 大学 (University). The original node in the tree will be split into two, thus changing the entire tree structure, which will split the original semantics of some words.


## 6. Discussion

Text semantic understanding has always been a critical problem in natural language processing. The current experimental results show that the method in this paper can surpass many baseline models on the text semantic similarity task, but there are still some areas for improvement.First, text semantic similarity is a fundamental problem in natural language processing. Currently, the English STS corpus is rich in training data, but the Chinese STS corpus is relatively scarce, and most belong to classification tasks, which have no accurate similarity score for two sentences like the English STS corpus. In future work, the authors will try to collect more Chinese datasets similar to English STS for experimental comparison.Secondly, the existence of special characters in the dataset will reduce the accuracy of syntactic analysis and affect the model effect. The method of dealing with nodes containing special characters effectively and reasonably in the process of matrix decomposition without destroying the original syntactic structure is also worthy of further study.


## 7. Conclusions

This paper proposes a new sentence embedding learning model that integrates syntactic information called BRFP, aimed at the low degree of syntactic information fusion in the current research on sentence embedding representation, which model uses matrix decomposition to learn syntactic information and fuses it with word vectors to obtain the embedded representation of sentences.The experiments in this paper prove that the sentence embedding representation learned by the BRFP model surpasses most of the baseline models in the semantic similarity task of Chinese and English texts, and the accuracy of each experiment has achieved significant advantages.



---

*Source: 7471408-2022-08-17.xml*