# A Multimodal Retrieval and Ranking Method for Scientific Documents Based on HFS and XLNet

Authors: Meichao Yan; Yu Wen; Qingxuan Shi; Xuedong Tian
Journal: Scientific Programming
 (2022)

---

## Abstract

Aiming at the defects of traditional full-text retrieval models in dealing with mathematical expressions, which are special objects different from ordinary texts, a multimodal retrieval and ranking method for scientific documents based on hesitant fuzzy sets (HFS) and XLNet is proposed. This method integrates multimodal information, such as mathematical expression images and context text, as keywords to realize the retrieval of scientific documents. In the image modal, the images of mathematical expressions are recognized, and the hesitancy fuzzy set theory is introduced to calculate the hesitancy fuzzy similarity between mathematical query expressions and the mathematical expressions in candidate scientific documents. Meanwhile, in the text mode, XLNet is used to generate word vectors of the mathematical expression context to obtain the similarity between the query text and the mathematical expression context of the candidate scientific documents. Finally, the multimodal evaluation is integrated, and the hesitation fuzzy set is constructed at the document level to obtain the final scores of the scientific documents and corresponding ranked output. The experimental results show that the recall and precision of this method are 0.774 and 0.663 on the NTCIR dataset, respectively, and the average normalized discounted cumulative gain (NDCG) value of the top-10 ranking results is 0.880 on the Chinese scientific document (CSD) dataset.


---

## Body

## 1. Introduction

Scientific literature retrieval and ranking is an important way for workers to obtain scientific and technological information. As an important part of scientific documents, mathematical expressions and contextual texts with mathematical semantics are the primary basis for scientific document retrieval and ranking. However, the traditional full-text retrieval model for one-dimensional is not effective when facing the special two-dimensional pattern retrieval of mathematical expressions. At present, research studies on mathematical expression retrieval and ranking have been carried out with some progress, and methods and prototype systems [1–6] with mathematical retrieval functions have been proposed.In terms of mathematical expression retrieval, WikiMirs3.0 [7] constructed a hybrid index composed of the formulas index and the context index to enable more comprehensive use of mathematical information. In addition, the importance of formulas in the document is calculated for distinguishment. Zhang and Youssef [8] proposed a multidimensional similarity index based on a vector model to determine and evaluate five factors: system distance, data type level, matching depth, query coverage, and whether it is a formula. According to these five factors, the similarity between the query expression and the matching expression parsed by MATHML can be calculated.In the research of mathematical expression retrieval and ranking that fuses mathematical expressions with textual information, MIaS [9] used the LRO (Leave Rightmost Out) method to split the original query generated by the combination of keywords and mathematical expressions into subqueries and merged the results using appropriate weighting to obtain more relevant results to the original topic. Zai and Tian [10] used FDS [11, 12] to parse the formulas and retrieved relevant documents using obtained operators. The cosine distance between the input word vectors and the keyword vectors in the documents after the word embedding model is calculated to obtain the similarity between the two, which enables a more reasonable and comprehensive retrieval and ranking. The textual information of a mathematical expression is usually contained in the context of the expressions. Kristianto [13] proposed the concept of mathematical expression dependency, using rich semantic information to obtain better accuracy and improve the retrieval results of the mathematical search system.Multimodality refers to any combination of two or more modalities. Piergiovanni and Ryoo [14] proposed a joint multimodal representation space method, using adversarial formulas for unmatched text and video data to improve the joint embedding space. Frome et al. [15] proposed a deep visual semantic embedding model based on the semantic information in the labeled image data and unlabeled text to identify visual objects. Jin et al. [16] proposed a generalized deep multimodal hashing framework for scalable image-text and video-text retrieval that explored feature representation learning, intermodality similarity preserving, intramodality semantic label preserving, and hash function learning with different types of loss functions simultaneously. Shen et al. [17] proposed a novel unsupervised hashing method (multiview discrete hashing) to learn compact hash codes from multiview data. The proposed method jointly learned the hash codes and cluster labels via factorization techniques and spectral analysis. And they developed an efficient alternating algorithm to optimize the proposed model. The generated hash codes not only could reflect the underlying semantics from multiple views but also enjoy high discrimination. Lu et al. [18] proposed an Online Multimodal Hashing with Dynamic Query-adaption (OMHDQ) method in a novel fashion that was designed to adaptively preserve the multimodal feature information into hash codes. Moreover, the online module was parameter-free. It could avoid time-consuming and inaccurate parameter adjustment in the unsupervised query hashing process.In the image recognition of mathematical expressions, the mathematical document INFTY system [19] utilized the optical character recognition techniques to analyze the structure of mathematical expressions and recognized printed mathematical expressions into LaTeX and XML markup formats. Deng et al. [20] explored an image-text generation technology, applied them to mathematical expression recognition, used a convolutional neural network (CNN) to extract image features, and employed a recurrent neural network (RNN) for encoding and decoding.The abovementioned research on the recognition and retrieval of mathematical expressions has achieved certain results. However, the single-modal retrieval model has great limitations because mathematical expressions in scientific documents often exist in multiple forms, such as embedding descriptions and images. Based on this, this study proposes a multimodal retrieval method for scientific documents based on HFS [21, 22] and XLNet [23]. This method integrates the functions of mathematical expression images and contextual text to improve the accuracy of retrieval results. In this study, the input form of mathematical expressions is no longer limited, and the information of mathematical expressions in images and text format can be input, which increases the flexibility and practicability of retrieval. In addition, the context of mathematical expression is closely related to the mathematical expression itself in scientific documents, and the combination between mathematical expression and context makes the retrieval and sorting of scientific documents more reasonable.The contributions of this study can be summarized as follows:(1)
Multimodal retrieval is introduced into the retrieval task of scientific documents, and the complementarity between image mode and text mode is utilized to retrieve scientific documents.(2)
Mathematical expressions and their context are combined to retrieval and ranking, and XLNet is used to generate word vector, so that a richer semantic representation of mathematical expression context can be obtained.(3)
The hesitancy fuzzy set is used to calculate the hesitancy fuzzy measure of scientific documents. The hesitancy fuzzy set considers the attributes of the documents. In addition, Chinese scientific documents (CSD) were added to the retrieved dataset.


## 2. Model Framework

The multimodal retrieval and ranking process of scientific documents based on HFS and XLNet is shown in Figure1.Figure 1
The flowchart of the multimodal retrieval and ranking for scientific documents.First, in the query module, mathematical expression images and text keywords are inputted.The processing module of the image model is used to calculate the similarity between mathematical expression in images and in candidate technical documents. The LaTeX forms of the input mathematical expressions are obtained by recognizing the images of the input mathematical expression, and FDS is used to analyze the recognition result. Then, the hesitant fuzzy set theory is introduced to calculate the similarity between the mathematical expressions and the results are returned to the document processing module.The processing module of text modal is used to calculate the similarity between the mathematical expression context. The text in the context of mathematical expressions in the dataset is extracted and used to pretrain XLNet. XLNet is used to calculate the similarity between the query text and the mathematical expression context of the candidate scientific documents.The document processing module is used to output documents in order. The document attributes are designed, the scores of the documents are calculated by hesitation fuzzy set, and the ranking results are output in descending order of similarity.


## 3. Similarity Measure of Multimodal Mathematical Expressions

### 3.1. Mathematical Expression Image Model’s Similarity Measure

#### 3.1.1. Mathematical Expression Image Recognition

The ViT and transformer models proposed in the literature [24–26] for processing sequence problems and image tasks are shown in Figure 2.Figure 2
Image recognition of mathematical expressions.The model consists of a ViT [24] encoder with a deep residual network (ResNet) [25] backbone and a Transformer [26] decoder. The encoder is used for feature extraction, and the decoder is used to convert the mathematical expression information in the image into the LaTeX form. The experimental results show that the accuracy of Bilingual Evaluation Understudy (BLEU) is 0.88.


#### 3.1.2. Mathematical Expression Image Similarity

The hesitant fuzzy set proposed by Torra [21, 22] is used to measure the similarity between query expressions and candidate expressions. The value of membership in the hesitant fuzzy set is a value set containing several possible membership degrees. Therefore, the results can be evaluated from multiple aspects. This approach avoids the errors due to a single phenomenon. The degree of hesitation of people in the process of transaction processing can be more objectively reflected.Definition 1.
(hesitating fuzzy set). LetX be a nonempty set, and the definition of the hesitation fuzzy set is(1)E=<x,hEx>|x∈X,where hEx represents the set of possible membership degrees for x∈X, which is a subset of the interval [0, 1] [21, 22]. Among them, hEx means evaluation attributes, which may be one or more. Each group of evaluation attributes contains multiple evaluation indicators.
The similarity of the analytical mathematical expression of FDS [11, 12] is calculated by the hesitant fuzzy set. The evaluation attribute of the mathematical expression is defined as a triple hS,hO,hN [27], where hS is the structural attribute of the expression, hO is the operator attribute of the expression, and hN is the operand attribute of the expression. The structure and operator characteristics of the expression are evaluated, respectively. Each evaluation attribute contains several evaluation indicators. By setting the membership function for each indicator, the query expression Eq and the hesitant membership degree of each result expression ED for each attribute are evaluated.
In conclusion, the set of hesitating fuzzy evaluation attributeshS,hO,hN and the set of hesitating fuzzy elements hEx=uhS,uhO,uhN are constructed based on the above attributes. uhS, uhO, uhN are the corresponding hesitant fuzzy membership functions of each evaluation attribute.
(1) Structural AttributehSDefinition 2.
The subformula weight distribution method [28] in the traditional tree index structure is referred, and the flag, length, and operator level in the subexpression are used to replace the structural complexity, length, and depth of nodes in the traditional method.(2)uhS=ul⋅ufulevel3,where(3)ulEq,ED=lEqlED,ufEq,ED=fEqfED,ulevelEq,ED=1nlevel1+level2+⋯+leveln.
Here,fEq is the lowest form of the flag bit of the current subexpression, fED is the flag of the subexpression in the expression, lEq is the length of the subexpression, lED is the length of the entire expression, and level is the level of operators in the subexpression. When the subexpression appears several times in the query results, the average is taken as its level attribute value.
(2) Operator AttributehO. Here, the BM25 algorithm is referenced as the membership function of the operator index:(4)uhO=∑i∈ΘlogNNEmk1+1fEqik+fEqik2+1fqk2+fq.
The formula can be disassembled into three components. The first componentN represents the total number of expressions in the database, NEm represents the total number of expressions, which contains Eq. The second component is the weight of the query word in the database, where fEq represents the frequency of the operator in the database, and k1 and k are empirical parameters. The third component is the weight of the query operator itself, where fq represents the word frequency of the query operators in the user’s queries, which is usually set to 1 for shorter queries. k2 is an empirical parameter.
The evaluation of operand attributehN is similar to the operator attribute hO, so the description will not be repeated.
(3) Similarity Calculation(5)simEq,Emi=1−13∑1lxi∑j=1lxi|hMσjxi−hNσjxi|λ1/λ,where lxi is the number of evaluation values and hMσjxi and hNσjxi represent the j-th element in hMx and hNx, respectively.
LetEq be a2+b2, and some of the retrieval results and the corresponding hesitant fuzzy sets are shown in Table 1.Table 1
Some of the retrieval results ofa2+b2 and the corresponding hesitant fuzzy sets.
No.ExpressionuhS,uhO,uhN1a2+b21.000,0.174,0.0892a2+b2=c20.622,0.089,0.0613a2+b20.532,0.065,0.0614ca2+b20.460,0.065,0.0615a2−b2/a2+b20.091,0.051,0.048Definition 3.
Let the set of mathematical expressions corresponding to the document setD=Dn|N∈Rn=1N be E=Em|M∈Rm=1M.
The mathematical expression similarity calculation algorithm is as follows:
Algorithm 1: Mathematical expression similarity calculation.
Input: a LaTeX form of the recognized mathematical expression EqOutput: a set of mathematical expressions similar to Eq(1)
//Initialize the feature vector databaseEDid,expstring(2)
FDSEq //parsed by FDS(3)
FDSEmi(4)
while (FDSEq) do(5)
forFDSEq in FDSEmi:(6)
uhSi=uli,ufi,uleveli //Structural attribute membership value(7)
uhOi,uhNi//Operator and operand attribute membership values(8)
hEmix=uhSi,uhOi,uhNi //Hesitant fuzzy set of Emi(9)
simEq,Emi=simHFSEq,HFSEmi //The similarity between expressions Eq and Emi is transformed into the similarity between the hesitant fuzzy set HFSEq and HFSEmi(10)
Add to table simexpid,expstring,simEq,Emi(11)
end for(12)
end while(13)
return simexp(14)
END


### 3.2. Mathematical Expression Context Similarity Measure

XLNet [23] is a generalized autoregressive pretraining model. The text in the documents is extracted, one-third of which is annotated to train XLNet, so that a richer semantic representation of the mathematical expression text can be obtained. The main structure is shown in Figure 3 (assuming the factorization order is 3 ⟶ 2 ⟶ 4 ⟶ 1).Figure 3
The schematic diagram of the XLNet model.The same keyword may have different meanings in different contexts, and textual information that explains a mathematical expression often appears around the expression. The example is in the document “Parasitic capacitance.html.” The expression in this document isi=Cdv/dt, and its contexts are “When two conductors at different potentials are close to one another, they are affected by each others’ electric field and store opposite electric charges like a capacitor” and “where C is the capacitance between the conductors.” The meanings of “potentials,” “electric charges,” and “capacitance” may have different meanings in other contexts, and the constructed vectors are also different.This study introduces the XLNet [23] language model to generate word vectors to be rich in semantics. XLNet solves the problem that BERT did not consider the relationship between the words that are shielded and the words that are not shielded during the training process; that is, the independence between words was not taken into account. The XLNet model implements a new bidirectional coding based on autoregressive (AR) language model. When calculating the text similarity, XLNet will fully consider the semantic information of word vectors, and therefore, the accuracy of calculating text similarity is improved.The TF-IDF algorithm is used to extract keywords and their weights in the context of mathematical expressions. By analyzing a large number of scientific literature studies, the context of the mathematical expressions is used to analyze mathematical expressions and explain symbols. It can be seen that the context of expressions is closely related to mathematical expressions, so it is very important to extract the context of mathematical expressions for the retrieval of mathematical expressions. The context and keywords corresponding to two mathematical expressions are selected as shown in Table2.Table 2
The context of two expressions and the corresponding keywords.
No.FilenameExpressionPreceding paragraphsFollowing paragraphsKeywords1Quadratic formulaax2+bx+c=0The general quadratic equation isOne can verify that the quadratic formula satisfies the quadratic equation by inserting the former into the latter.Quadratic formula, quadratic equation2Gaussian functionfx=ae−x−b2/2c2In mathematics, a Gaussian function, often simply referred to as a Gaussian, is a function of the form:For arbitrary real constants, and it is named after the mathematician Carl Friedrich Gauss.Gaussian function, Gaussian


## 3.1. Mathematical Expression Image Model’s Similarity Measure

### 3.1.1. Mathematical Expression Image Recognition

The ViT and transformer models proposed in the literature [24–26] for processing sequence problems and image tasks are shown in Figure 2.Figure 2
Image recognition of mathematical expressions.The model consists of a ViT [24] encoder with a deep residual network (ResNet) [25] backbone and a Transformer [26] decoder. The encoder is used for feature extraction, and the decoder is used to convert the mathematical expression information in the image into the LaTeX form. The experimental results show that the accuracy of Bilingual Evaluation Understudy (BLEU) is 0.88.


### 3.1.2. Mathematical Expression Image Similarity

The hesitant fuzzy set proposed by Torra [21, 22] is used to measure the similarity between query expressions and candidate expressions. The value of membership in the hesitant fuzzy set is a value set containing several possible membership degrees. Therefore, the results can be evaluated from multiple aspects. This approach avoids the errors due to a single phenomenon. The degree of hesitation of people in the process of transaction processing can be more objectively reflected.Definition 1.
(hesitating fuzzy set). LetX be a nonempty set, and the definition of the hesitation fuzzy set is(1)E=<x,hEx>|x∈X,where hEx represents the set of possible membership degrees for x∈X, which is a subset of the interval [0, 1] [21, 22]. Among them, hEx means evaluation attributes, which may be one or more. Each group of evaluation attributes contains multiple evaluation indicators.
The similarity of the analytical mathematical expression of FDS [11, 12] is calculated by the hesitant fuzzy set. The evaluation attribute of the mathematical expression is defined as a triple hS,hO,hN [27], where hS is the structural attribute of the expression, hO is the operator attribute of the expression, and hN is the operand attribute of the expression. The structure and operator characteristics of the expression are evaluated, respectively. Each evaluation attribute contains several evaluation indicators. By setting the membership function for each indicator, the query expression Eq and the hesitant membership degree of each result expression ED for each attribute are evaluated.
In conclusion, the set of hesitating fuzzy evaluation attributeshS,hO,hN and the set of hesitating fuzzy elements hEx=uhS,uhO,uhN are constructed based on the above attributes. uhS, uhO, uhN are the corresponding hesitant fuzzy membership functions of each evaluation attribute.
(1) Structural AttributehSDefinition 2.
The subformula weight distribution method [28] in the traditional tree index structure is referred, and the flag, length, and operator level in the subexpression are used to replace the structural complexity, length, and depth of nodes in the traditional method.(2)uhS=ul⋅ufulevel3,where(3)ulEq,ED=lEqlED,ufEq,ED=fEqfED,ulevelEq,ED=1nlevel1+level2+⋯+leveln.
Here,fEq is the lowest form of the flag bit of the current subexpression, fED is the flag of the subexpression in the expression, lEq is the length of the subexpression, lED is the length of the entire expression, and level is the level of operators in the subexpression. When the subexpression appears several times in the query results, the average is taken as its level attribute value.
(2) Operator AttributehO. Here, the BM25 algorithm is referenced as the membership function of the operator index:(4)uhO=∑i∈ΘlogNNEmk1+1fEqik+fEqik2+1fqk2+fq.
The formula can be disassembled into three components. The first componentN represents the total number of expressions in the database, NEm represents the total number of expressions, which contains Eq. The second component is the weight of the query word in the database, where fEq represents the frequency of the operator in the database, and k1 and k are empirical parameters. The third component is the weight of the query operator itself, where fq represents the word frequency of the query operators in the user’s queries, which is usually set to 1 for shorter queries. k2 is an empirical parameter.
The evaluation of operand attributehN is similar to the operator attribute hO, so the description will not be repeated.
(3) Similarity Calculation(5)simEq,Emi=1−13∑1lxi∑j=1lxi|hMσjxi−hNσjxi|λ1/λ,where lxi is the number of evaluation values and hMσjxi and hNσjxi represent the j-th element in hMx and hNx, respectively.
LetEq be a2+b2, and some of the retrieval results and the corresponding hesitant fuzzy sets are shown in Table 1.Table 1
Some of the retrieval results ofa2+b2 and the corresponding hesitant fuzzy sets.
No.ExpressionuhS,uhO,uhN1a2+b21.000,0.174,0.0892a2+b2=c20.622,0.089,0.0613a2+b20.532,0.065,0.0614ca2+b20.460,0.065,0.0615a2−b2/a2+b20.091,0.051,0.048Definition 3.
Let the set of mathematical expressions corresponding to the document setD=Dn|N∈Rn=1N be E=Em|M∈Rm=1M.
The mathematical expression similarity calculation algorithm is as follows:
Algorithm 1: Mathematical expression similarity calculation.
Input: a LaTeX form of the recognized mathematical expression EqOutput: a set of mathematical expressions similar to Eq(1)
//Initialize the feature vector databaseEDid,expstring(2)
FDSEq //parsed by FDS(3)
FDSEmi(4)
while (FDSEq) do(5)
forFDSEq in FDSEmi:(6)
uhSi=uli,ufi,uleveli //Structural attribute membership value(7)
uhOi,uhNi//Operator and operand attribute membership values(8)
hEmix=uhSi,uhOi,uhNi //Hesitant fuzzy set of Emi(9)
simEq,Emi=simHFSEq,HFSEmi //The similarity between expressions Eq and Emi is transformed into the similarity between the hesitant fuzzy set HFSEq and HFSEmi(10)
Add to table simexpid,expstring,simEq,Emi(11)
end for(12)
end while(13)
return simexp(14)
END


## 3.1.1. Mathematical Expression Image Recognition

The ViT and transformer models proposed in the literature [24–26] for processing sequence problems and image tasks are shown in Figure 2.Figure 2
Image recognition of mathematical expressions.The model consists of a ViT [24] encoder with a deep residual network (ResNet) [25] backbone and a Transformer [26] decoder. The encoder is used for feature extraction, and the decoder is used to convert the mathematical expression information in the image into the LaTeX form. The experimental results show that the accuracy of Bilingual Evaluation Understudy (BLEU) is 0.88.


## 3.1.2. Mathematical Expression Image Similarity

The hesitant fuzzy set proposed by Torra [21, 22] is used to measure the similarity between query expressions and candidate expressions. The value of membership in the hesitant fuzzy set is a value set containing several possible membership degrees. Therefore, the results can be evaluated from multiple aspects. This approach avoids the errors due to a single phenomenon. The degree of hesitation of people in the process of transaction processing can be more objectively reflected.Definition 1.
(hesitating fuzzy set). LetX be a nonempty set, and the definition of the hesitation fuzzy set is(1)E=<x,hEx>|x∈X,where hEx represents the set of possible membership degrees for x∈X, which is a subset of the interval [0, 1] [21, 22]. Among them, hEx means evaluation attributes, which may be one or more. Each group of evaluation attributes contains multiple evaluation indicators.
The similarity of the analytical mathematical expression of FDS [11, 12] is calculated by the hesitant fuzzy set. The evaluation attribute of the mathematical expression is defined as a triple hS,hO,hN [27], where hS is the structural attribute of the expression, hO is the operator attribute of the expression, and hN is the operand attribute of the expression. The structure and operator characteristics of the expression are evaluated, respectively. Each evaluation attribute contains several evaluation indicators. By setting the membership function for each indicator, the query expression Eq and the hesitant membership degree of each result expression ED for each attribute are evaluated.
In conclusion, the set of hesitating fuzzy evaluation attributeshS,hO,hN and the set of hesitating fuzzy elements hEx=uhS,uhO,uhN are constructed based on the above attributes. uhS, uhO, uhN are the corresponding hesitant fuzzy membership functions of each evaluation attribute.
(1) Structural AttributehSDefinition 2.
The subformula weight distribution method [28] in the traditional tree index structure is referred, and the flag, length, and operator level in the subexpression are used to replace the structural complexity, length, and depth of nodes in the traditional method.(2)uhS=ul⋅ufulevel3,where(3)ulEq,ED=lEqlED,ufEq,ED=fEqfED,ulevelEq,ED=1nlevel1+level2+⋯+leveln.
Here,fEq is the lowest form of the flag bit of the current subexpression, fED is the flag of the subexpression in the expression, lEq is the length of the subexpression, lED is the length of the entire expression, and level is the level of operators in the subexpression. When the subexpression appears several times in the query results, the average is taken as its level attribute value.
(2) Operator AttributehO. Here, the BM25 algorithm is referenced as the membership function of the operator index:(4)uhO=∑i∈ΘlogNNEmk1+1fEqik+fEqik2+1fqk2+fq.
The formula can be disassembled into three components. The first componentN represents the total number of expressions in the database, NEm represents the total number of expressions, which contains Eq. The second component is the weight of the query word in the database, where fEq represents the frequency of the operator in the database, and k1 and k are empirical parameters. The third component is the weight of the query operator itself, where fq represents the word frequency of the query operators in the user’s queries, which is usually set to 1 for shorter queries. k2 is an empirical parameter.
The evaluation of operand attributehN is similar to the operator attribute hO, so the description will not be repeated.
(3) Similarity Calculation(5)simEq,Emi=1−13∑1lxi∑j=1lxi|hMσjxi−hNσjxi|λ1/λ,where lxi is the number of evaluation values and hMσjxi and hNσjxi represent the j-th element in hMx and hNx, respectively.
LetEq be a2+b2, and some of the retrieval results and the corresponding hesitant fuzzy sets are shown in Table 1.Table 1
Some of the retrieval results ofa2+b2 and the corresponding hesitant fuzzy sets.
No.ExpressionuhS,uhO,uhN1a2+b21.000,0.174,0.0892a2+b2=c20.622,0.089,0.0613a2+b20.532,0.065,0.0614ca2+b20.460,0.065,0.0615a2−b2/a2+b20.091,0.051,0.048Definition 3.
Let the set of mathematical expressions corresponding to the document setD=Dn|N∈Rn=1N be E=Em|M∈Rm=1M.
The mathematical expression similarity calculation algorithm is as follows:
Algorithm 1: Mathematical expression similarity calculation.
Input: a LaTeX form of the recognized mathematical expression EqOutput: a set of mathematical expressions similar to Eq(1)
//Initialize the feature vector databaseEDid,expstring(2)
FDSEq //parsed by FDS(3)
FDSEmi(4)
while (FDSEq) do(5)
forFDSEq in FDSEmi:(6)
uhSi=uli,ufi,uleveli //Structural attribute membership value(7)
uhOi,uhNi//Operator and operand attribute membership values(8)
hEmix=uhSi,uhOi,uhNi //Hesitant fuzzy set of Emi(9)
simEq,Emi=simHFSEq,HFSEmi //The similarity between expressions Eq and Emi is transformed into the similarity between the hesitant fuzzy set HFSEq and HFSEmi(10)
Add to table simexpid,expstring,simEq,Emi(11)
end for(12)
end while(13)
return simexp(14)
END


## 3.2. Mathematical Expression Context Similarity Measure

XLNet [23] is a generalized autoregressive pretraining model. The text in the documents is extracted, one-third of which is annotated to train XLNet, so that a richer semantic representation of the mathematical expression text can be obtained. The main structure is shown in Figure 3 (assuming the factorization order is 3 ⟶ 2 ⟶ 4 ⟶ 1).Figure 3
The schematic diagram of the XLNet model.The same keyword may have different meanings in different contexts, and textual information that explains a mathematical expression often appears around the expression. The example is in the document “Parasitic capacitance.html.” The expression in this document isi=Cdv/dt, and its contexts are “When two conductors at different potentials are close to one another, they are affected by each others’ electric field and store opposite electric charges like a capacitor” and “where C is the capacitance between the conductors.” The meanings of “potentials,” “electric charges,” and “capacitance” may have different meanings in other contexts, and the constructed vectors are also different.This study introduces the XLNet [23] language model to generate word vectors to be rich in semantics. XLNet solves the problem that BERT did not consider the relationship between the words that are shielded and the words that are not shielded during the training process; that is, the independence between words was not taken into account. The XLNet model implements a new bidirectional coding based on autoregressive (AR) language model. When calculating the text similarity, XLNet will fully consider the semantic information of word vectors, and therefore, the accuracy of calculating text similarity is improved.The TF-IDF algorithm is used to extract keywords and their weights in the context of mathematical expressions. By analyzing a large number of scientific literature studies, the context of the mathematical expressions is used to analyze mathematical expressions and explain symbols. It can be seen that the context of expressions is closely related to mathematical expressions, so it is very important to extract the context of mathematical expressions for the retrieval of mathematical expressions. The context and keywords corresponding to two mathematical expressions are selected as shown in Table2.Table 2
The context of two expressions and the corresponding keywords.
No.FilenameExpressionPreceding paragraphsFollowing paragraphsKeywords1Quadratic formulaax2+bx+c=0The general quadratic equation isOne can verify that the quadratic formula satisfies the quadratic equation by inserting the former into the latter.Quadratic formula, quadratic equation2Gaussian functionfx=ae−x−b2/2c2In mathematics, a Gaussian function, often simply referred to as a Gaussian, is a function of the form:For arbitrary real constants, and it is named after the mathematician Carl Friedrich Gauss.Gaussian function, Gaussian


## 4. Calculation of the Similarity of Scientific Documents

Retrieval and ranking of scientific and technological documents is a comprehensive measurement with multiple attributes including mathematical expressions and keywords. Different scientific documents have different meanings, even if they contain the same formula. Therefore, hesitating fuzzy sets are used to evaluate scientific documents in an all-around way to achieve the final sorting in this study.Define the attribute of the scientific document as a five-tupleHexp,Hword,Hloc,Hef,Hwf, where Hexp is the similarity attribute of the mathematical expression, Hword is the keyword similarity attribute, Hloc is the relative position attribute of the expression, Hef is the frequency attribute of the expression, and Hwf is the frequency attribute of the keyword. The mathematical expressions and keywords of scientific documents are evaluated.Definition 4.
W=WEmm=1M is the keyword set corresponding to Emm=1M; VECEmm=1M, and VECqq=1Q are the word vectors corresponding to WEmm=1M and the query keyword Wq.Definition 5.
The functionUexpEq,Emi is used to calculate the similarity between the query expression and the expression in the candidate document.(6)UexpEq,Emi=simEq,Emi,where simEq,Emi represents the similarity between the mathematical expression Eq of the query and the mathematical expression Emi in the candidate scientific document.Definition 6.
The functionUwordWq,WEmi is used to express the similarity between the query keyword Wq and the keyword WEmi in the context.(7)UwordWq,WEmi=simVECq,VECEmi,where WEmi represents the keyword in the document retrieved in the candidate scientific document.Definition 7.
The functionUlocEmi,DNi is used to express the position of the expression Emi in the document DNi.(8)UlocEmi,DNi=1−locexpnum,where locexp is the position where the query expression Eq appears for the first time in the document DNi, and num represents the total number of characters contained in the document DNi.Definition 8.
The functionUefEq,DNi is used to express the frequency of the query expression Eq in the document DNi.(9)UefEq,DNi=1−e−αkexp/ksume,where α is the feature weight coefficient of the number of mathematical expressions in the document, which is obtained by counting the number of expressions in all documents in the database. kexp represents the number of expressions in the document DNi that matches the query expression Eq, and ksum represents the total number of expressions contained in the document DNi.Definition 9.
The functionUwfWq,DNi is used to express the frequency of the query keyword Wq in the document DNi.(10)UwfWq,DNi=1−e−αtWq/tsumw,where α is the feature weight coefficient of the number of keywords in the document, which is obtained by counting the number of keywords in all documents in the database. tWq represents the number of keywords in the document DNi that matches the query keyword Wq, and tsumw represents the total number of keywords contained in the document DNi.Definition 10.
The functionSDNi is used to calculate the score of scientific document retrieval results.(11)SDNi=1−15∑1lhi∑j=1lhiHEWqσjhi−HDNiσjhiλ1/λ,where SDNi is the scoring function of the result document DNi when querying the input expressions and keywords, and hii=exp,word,loc,ef,wf is the five evaluation attributes of the document. HEWqσjhi and HDNiσjhi are the j-th largest elements in HEWqhi and HDNihi, respectively. lhi is the number of evaluation values included in the evaluation attribute hi. The attributes of the document are shown in Table 3.
The sorting algorithm of retrieval result documents is as follows:
Algorithm 2: Ranking of the retrieval results of scientific documents.
Input: document collection of the retrieval results of scientific documentsOutput: the ranking sequence of the documents(1)
while (Result) do:(2)
UexpEq,Emi=simEq,Emi   //Mathematical expression similarity degree of membership value(3)
UwordWq,WEmi=simVECq,VECEmi(4)
location (Emi,DNi)   //The position of the mathematical expression in the document(5)
UlocEmi,DNi(6)
UefEq,DNi, UwfWq,DNi(7)
Hexp,Hword,Hloc,Hef,Hwf   //hesitant fuzzy set of EDi(8)
simText=simHexp,Hword,Hloc,Hef,Hwf,1,1,1,1,1   //The similarity between documents is transformed into the similarity between the hesitant fuzzy set Hexp,Hword,Hloc,Hef,Hwf(9)
returnsimText(10)
end while(11)
returnsimText DESC   //Sort in descending order, return results(12)
ENDTable 3
Document attributes.
FileID of fileFive-tupleDN11Hexp1,Hword1,Hloc1,Hef1,Hwf1DN22Hexp2,Hword2,Hloc2,Hef2,Hwf2………DNnnHexpn,Hwordn,Hlocn,Hefn,Hwfn


## 5. Experimental Process and Result Analysis

### 5.1. Experimental Data

For the image recognition part of mathematical expressions, we use the IM2LATEX-100K dataset for training and testing. The IM2LATEX-100K dataset contains 103,556 images of different mathematical expressions. The label data consist of the LaTeX format of mathematical expressions.For the scientific document retrieval and ranking part, the public dataset Ntcir-MathIR-Wikipedia-Corpus (NTCIR) is used, and 31,742 documents are extracted, which contains 518,929 mathematical expressions. In addition, Chinese scientific documents (CSD) are added to expand the dataset, which contains 10,372 documents and 121,495 mathematical expressions.


### 5.2. System Experiments

#### 5.2.1. Image Recognition of Mathematical Expressions

The image recognition algorithm model [24–26] is used to recognize mathematical expression images and conducts a lot of experiments on different types of mathematical expression images in this study. According to the BLEU evaluation standard, the model result reaches 0.88.For this recognition algorithm, five different types of mathematical expression images are selected for recognition and display in this study, and the recognition results are shown in Table4 (the content of the image here is expressed in text).Table 4
Image recognition results of five mathematical expressions.
No.Images containing the mathematical expressionRecognition result1a+ba + b2S=πr2\[S = \pi {r^2}\]3x=−b±b2−4ac/2ax = \frac{{ - b \pm \sqrt {{b^2} - 4ac} }}{{2a}4cosα+cosβ=2cos1/2α+βcos1/2α−β\cos \alpha + \cos \beta = 2\cos \frac{1}{2}\left({\alpha+\beta }\right)\cos\frac{1}{2}\left({\alpha - \beta } \right)5ab≤a+b/2\sqrt {ab} \le \frac{{a + b}}{2}


#### 5.2.2. Ablation Study

Ten groups of formulas and keywords are selected in Table5 as queries for retrieval. The proposed method includes three main parts, and the performance is continuously improved by gradually increasing the functions of each part. The baseline experiment was image expression retrieval. The final reordering of our has the best performance. The average recall rates of this study are 77.4% and 77.8%. And the average precision rates are 66.3% and 69.2%. All of them are shown in Table 6.Table 5
The list of documents.
No.ExpKeyword1fx=axExponential function (指数函数)2S=πr2Circular area (圆形面积)3x=−b±b2−4ac/2aQuadratic formula (求根公式)4limn⟶∞1+1/nnLimit theorem (极限)5ab≤a+b/2Inequality (不等式)6logaxLogarithm (对数)7a2−b2Square variance (方差)8f′xDifferential coefficient (微分)9tanθ=sinθ/cosθTrigonometric function (三角函数)10PX=k=λk/k!eλPoisson (泊松)Table 6
Results of ablation assessment on NTCIR and CSD data.
RecallPrecisionF1DatasetNTCIRCSDNTCIRCSDNTCIRCSDBaseline0.7530.7510.6290.6350.6850.688Baseline + XLNet0.7610.7720.6360.6490.6930.705Our0.7740.7780.6630.6920.7140.732


#### 5.2.3. Performance on NTCIR Dataset

In this section, the method in this article is compared with some traditional methods and current existing methods using the NTCIR dataset. FDS + Word Embedding [10] combines the FDS and Word Embedding to retrieve scientific documents: FDS is used to parse expressions, and Word Embedding is used to generate the word vectors of keywords in scientific documents, hereinafter referred to as Method 1. And SearchOnMath [29] is a mathematical formula retrieval tool that aims at accurately matching mathematical expressions, However, SearchOnMath implements pure mathematical expression retrieval and does not consider the important information of the scientific document itself, hereinafter referred to as Method 2. MIaS [4] is based on the full-text search engine Apache Lucene. MIaS processes text and math separately. The text is tokenized and stemmed to unify inflected word forms, hereinafter referred to as Method 3.In this study, NDCG is used to evaluate the ranking results, which is the search result after the normalization of DCG (discount cumulative gain). The calculation method is as follows:(12)nDCGl=DCGlIDCGl,where(13)DCGl=∑i=1lrilog2i+1,IDCGl=∑i=1|REL|rilog2i+1,where l is the number of search results, ri is the relevance score, IDCGl is the ideal DCG value, and |REL| indicates that the search results are all related to the query expression.The query formula and keywords in Table5 are taken as the query, and the method in this study and other methods top-10 experts ranking results are shown in Figure 4. Method 2 starts with a higher value than the method in this article, but as the number of expression retrievals increases, the method in this article is all higher than Method 2. The average NDCG of this method is higher than the other three methods. And the average value of NDCG (n = 10) is 0.865 on the NTCIR dataset in this study. The experimental results show that the ranking performance of the proposed method is better and the retrieval result is more reasonable.Figure 4
Comparison of the NDCG between the proposed method and other methods.


#### 5.2.4. Performance on CSD Dataset

In this section, the method in this article is compared with Method 1 using the NTCIR dataset. Chinese scientific documents (CSD) are added to expand the dataset, which contains 10,372 documents and 121,495 mathematical expressions. The experimental results are shown in Figure5.Figure 5
Comparison between the proposed method and Method 1.It can be seen that the NDCG of the method in this study is higher than the comparison method. The average value of NDCG (n = 10) is 0.88 on the CSD dataset in this study. So the results of the method in this study are more reasonable, and the retrieval and ranking performance is improved.


#### 5.2.5. Retrieval System

A large number of experiments are conducted for different expressions. The first ten search results are selected for display in this study. When the input formula image is “PX=k=λk/k!eλ” and the keyword is “Poisson,” some of the search results are shown in Table 7.Table 7
Partial retrieval and ranking results of scientific documents based on HFS and XLNet.
No.FileNameScore1Poisson distribution0.922Variance0.883Poisson games0.834Zero-truncated Poisson distribution0.795Geometric Poisson distribution0.766Displaced Poisson distribution0.737Poisson process0.728Poisson limit theorem0.709Inhomogeneous Poisson process0.6810Poisson point process0.67First of all, the method in this study identifies the LaTeX form of the formula as “P\left({X =k} \right) = \frac{{{\lambda ^k}}}{{k!}}{e^{-\lambda }},” finding out a collection of documents similar to the formula, and the XLNet model is used to obtain the word vector of “Poisson” and document expressions context keywords, and the similarity between them is calculated. Finally, according to the keywords and formula information, the similarity calculation of the documents is performed again using the hesitant fuzzy set so as to sort and output. FileName is the name of the document where the expression is located, and Score is the document score in Table 7.


## 5.1. Experimental Data

For the image recognition part of mathematical expressions, we use the IM2LATEX-100K dataset for training and testing. The IM2LATEX-100K dataset contains 103,556 images of different mathematical expressions. The label data consist of the LaTeX format of mathematical expressions.For the scientific document retrieval and ranking part, the public dataset Ntcir-MathIR-Wikipedia-Corpus (NTCIR) is used, and 31,742 documents are extracted, which contains 518,929 mathematical expressions. In addition, Chinese scientific documents (CSD) are added to expand the dataset, which contains 10,372 documents and 121,495 mathematical expressions.


## 5.2. System Experiments

### 5.2.1. Image Recognition of Mathematical Expressions

The image recognition algorithm model [24–26] is used to recognize mathematical expression images and conducts a lot of experiments on different types of mathematical expression images in this study. According to the BLEU evaluation standard, the model result reaches 0.88.For this recognition algorithm, five different types of mathematical expression images are selected for recognition and display in this study, and the recognition results are shown in Table4 (the content of the image here is expressed in text).Table 4
Image recognition results of five mathematical expressions.
No.Images containing the mathematical expressionRecognition result1a+ba + b2S=πr2\[S = \pi {r^2}\]3x=−b±b2−4ac/2ax = \frac{{ - b \pm \sqrt {{b^2} - 4ac} }}{{2a}4cosα+cosβ=2cos1/2α+βcos1/2α−β\cos \alpha + \cos \beta = 2\cos \frac{1}{2}\left({\alpha+\beta }\right)\cos\frac{1}{2}\left({\alpha - \beta } \right)5ab≤a+b/2\sqrt {ab} \le \frac{{a + b}}{2}


### 5.2.2. Ablation Study

Ten groups of formulas and keywords are selected in Table5 as queries for retrieval. The proposed method includes three main parts, and the performance is continuously improved by gradually increasing the functions of each part. The baseline experiment was image expression retrieval. The final reordering of our has the best performance. The average recall rates of this study are 77.4% and 77.8%. And the average precision rates are 66.3% and 69.2%. All of them are shown in Table 6.Table 5
The list of documents.
No.ExpKeyword1fx=axExponential function (指数函数)2S=πr2Circular area (圆形面积)3x=−b±b2−4ac/2aQuadratic formula (求根公式)4limn⟶∞1+1/nnLimit theorem (极限)5ab≤a+b/2Inequality (不等式)6logaxLogarithm (对数)7a2−b2Square variance (方差)8f′xDifferential coefficient (微分)9tanθ=sinθ/cosθTrigonometric function (三角函数)10PX=k=λk/k!eλPoisson (泊松)Table 6
Results of ablation assessment on NTCIR and CSD data.
RecallPrecisionF1DatasetNTCIRCSDNTCIRCSDNTCIRCSDBaseline0.7530.7510.6290.6350.6850.688Baseline + XLNet0.7610.7720.6360.6490.6930.705Our0.7740.7780.6630.6920.7140.732


### 5.2.3. Performance on NTCIR Dataset

In this section, the method in this article is compared with some traditional methods and current existing methods using the NTCIR dataset. FDS + Word Embedding [10] combines the FDS and Word Embedding to retrieve scientific documents: FDS is used to parse expressions, and Word Embedding is used to generate the word vectors of keywords in scientific documents, hereinafter referred to as Method 1. And SearchOnMath [29] is a mathematical formula retrieval tool that aims at accurately matching mathematical expressions, However, SearchOnMath implements pure mathematical expression retrieval and does not consider the important information of the scientific document itself, hereinafter referred to as Method 2. MIaS [4] is based on the full-text search engine Apache Lucene. MIaS processes text and math separately. The text is tokenized and stemmed to unify inflected word forms, hereinafter referred to as Method 3.In this study, NDCG is used to evaluate the ranking results, which is the search result after the normalization of DCG (discount cumulative gain). The calculation method is as follows:(12)nDCGl=DCGlIDCGl,where(13)DCGl=∑i=1lrilog2i+1,IDCGl=∑i=1|REL|rilog2i+1,where l is the number of search results, ri is the relevance score, IDCGl is the ideal DCG value, and |REL| indicates that the search results are all related to the query expression.The query formula and keywords in Table5 are taken as the query, and the method in this study and other methods top-10 experts ranking results are shown in Figure 4. Method 2 starts with a higher value than the method in this article, but as the number of expression retrievals increases, the method in this article is all higher than Method 2. The average NDCG of this method is higher than the other three methods. And the average value of NDCG (n = 10) is 0.865 on the NTCIR dataset in this study. The experimental results show that the ranking performance of the proposed method is better and the retrieval result is more reasonable.Figure 4
Comparison of the NDCG between the proposed method and other methods.


### 5.2.4. Performance on CSD Dataset

In this section, the method in this article is compared with Method 1 using the NTCIR dataset. Chinese scientific documents (CSD) are added to expand the dataset, which contains 10,372 documents and 121,495 mathematical expressions. The experimental results are shown in Figure5.Figure 5
Comparison between the proposed method and Method 1.It can be seen that the NDCG of the method in this study is higher than the comparison method. The average value of NDCG (n = 10) is 0.88 on the CSD dataset in this study. So the results of the method in this study are more reasonable, and the retrieval and ranking performance is improved.


### 5.2.5. Retrieval System

A large number of experiments are conducted for different expressions. The first ten search results are selected for display in this study. When the input formula image is “PX=k=λk/k!eλ” and the keyword is “Poisson,” some of the search results are shown in Table 7.Table 7
Partial retrieval and ranking results of scientific documents based on HFS and XLNet.
No.FileNameScore1Poisson distribution0.922Variance0.883Poisson games0.834Zero-truncated Poisson distribution0.795Geometric Poisson distribution0.766Displaced Poisson distribution0.737Poisson process0.728Poisson limit theorem0.709Inhomogeneous Poisson process0.6810Poisson point process0.67First of all, the method in this study identifies the LaTeX form of the formula as “P\left({X =k} \right) = \frac{{{\lambda ^k}}}{{k!}}{e^{-\lambda }},” finding out a collection of documents similar to the formula, and the XLNet model is used to obtain the word vector of “Poisson” and document expressions context keywords, and the similarity between them is calculated. Finally, according to the keywords and formula information, the similarity calculation of the documents is performed again using the hesitant fuzzy set so as to sort and output. FileName is the name of the document where the expression is located, and Score is the document score in Table 7.


## 5.2.1. Image Recognition of Mathematical Expressions

The image recognition algorithm model [24–26] is used to recognize mathematical expression images and conducts a lot of experiments on different types of mathematical expression images in this study. According to the BLEU evaluation standard, the model result reaches 0.88.For this recognition algorithm, five different types of mathematical expression images are selected for recognition and display in this study, and the recognition results are shown in Table4 (the content of the image here is expressed in text).Table 4
Image recognition results of five mathematical expressions.
No.Images containing the mathematical expressionRecognition result1a+ba + b2S=πr2\[S = \pi {r^2}\]3x=−b±b2−4ac/2ax = \frac{{ - b \pm \sqrt {{b^2} - 4ac} }}{{2a}4cosα+cosβ=2cos1/2α+βcos1/2α−β\cos \alpha + \cos \beta = 2\cos \frac{1}{2}\left({\alpha+\beta }\right)\cos\frac{1}{2}\left({\alpha - \beta } \right)5ab≤a+b/2\sqrt {ab} \le \frac{{a + b}}{2}


## 5.2.2. Ablation Study

Ten groups of formulas and keywords are selected in Table5 as queries for retrieval. The proposed method includes three main parts, and the performance is continuously improved by gradually increasing the functions of each part. The baseline experiment was image expression retrieval. The final reordering of our has the best performance. The average recall rates of this study are 77.4% and 77.8%. And the average precision rates are 66.3% and 69.2%. All of them are shown in Table 6.Table 5
The list of documents.
No.ExpKeyword1fx=axExponential function (指数函数)2S=πr2Circular area (圆形面积)3x=−b±b2−4ac/2aQuadratic formula (求根公式)4limn⟶∞1+1/nnLimit theorem (极限)5ab≤a+b/2Inequality (不等式)6logaxLogarithm (对数)7a2−b2Square variance (方差)8f′xDifferential coefficient (微分)9tanθ=sinθ/cosθTrigonometric function (三角函数)10PX=k=λk/k!eλPoisson (泊松)Table 6
Results of ablation assessment on NTCIR and CSD data.
RecallPrecisionF1DatasetNTCIRCSDNTCIRCSDNTCIRCSDBaseline0.7530.7510.6290.6350.6850.688Baseline + XLNet0.7610.7720.6360.6490.6930.705Our0.7740.7780.6630.6920.7140.732


## 5.2.3. Performance on NTCIR Dataset

In this section, the method in this article is compared with some traditional methods and current existing methods using the NTCIR dataset. FDS + Word Embedding [10] combines the FDS and Word Embedding to retrieve scientific documents: FDS is used to parse expressions, and Word Embedding is used to generate the word vectors of keywords in scientific documents, hereinafter referred to as Method 1. And SearchOnMath [29] is a mathematical formula retrieval tool that aims at accurately matching mathematical expressions, However, SearchOnMath implements pure mathematical expression retrieval and does not consider the important information of the scientific document itself, hereinafter referred to as Method 2. MIaS [4] is based on the full-text search engine Apache Lucene. MIaS processes text and math separately. The text is tokenized and stemmed to unify inflected word forms, hereinafter referred to as Method 3.In this study, NDCG is used to evaluate the ranking results, which is the search result after the normalization of DCG (discount cumulative gain). The calculation method is as follows:(12)nDCGl=DCGlIDCGl,where(13)DCGl=∑i=1lrilog2i+1,IDCGl=∑i=1|REL|rilog2i+1,where l is the number of search results, ri is the relevance score, IDCGl is the ideal DCG value, and |REL| indicates that the search results are all related to the query expression.The query formula and keywords in Table5 are taken as the query, and the method in this study and other methods top-10 experts ranking results are shown in Figure 4. Method 2 starts with a higher value than the method in this article, but as the number of expression retrievals increases, the method in this article is all higher than Method 2. The average NDCG of this method is higher than the other three methods. And the average value of NDCG (n = 10) is 0.865 on the NTCIR dataset in this study. The experimental results show that the ranking performance of the proposed method is better and the retrieval result is more reasonable.Figure 4
Comparison of the NDCG between the proposed method and other methods.


## 5.2.4. Performance on CSD Dataset

In this section, the method in this article is compared with Method 1 using the NTCIR dataset. Chinese scientific documents (CSD) are added to expand the dataset, which contains 10,372 documents and 121,495 mathematical expressions. The experimental results are shown in Figure5.Figure 5
Comparison between the proposed method and Method 1.It can be seen that the NDCG of the method in this study is higher than the comparison method. The average value of NDCG (n = 10) is 0.88 on the CSD dataset in this study. So the results of the method in this study are more reasonable, and the retrieval and ranking performance is improved.


## 5.2.5. Retrieval System

A large number of experiments are conducted for different expressions. The first ten search results are selected for display in this study. When the input formula image is “PX=k=λk/k!eλ” and the keyword is “Poisson,” some of the search results are shown in Table 7.Table 7
Partial retrieval and ranking results of scientific documents based on HFS and XLNet.
No.FileNameScore1Poisson distribution0.922Variance0.883Poisson games0.834Zero-truncated Poisson distribution0.795Geometric Poisson distribution0.766Displaced Poisson distribution0.737Poisson process0.728Poisson limit theorem0.709Inhomogeneous Poisson process0.6810Poisson point process0.67First of all, the method in this study identifies the LaTeX form of the formula as “P\left({X =k} \right) = \frac{{{\lambda ^k}}}{{k!}}{e^{-\lambda }},” finding out a collection of documents similar to the formula, and the XLNet model is used to obtain the word vector of “Poisson” and document expressions context keywords, and the similarity between them is calculated. Finally, according to the keywords and formula information, the similarity calculation of the documents is performed again using the hesitant fuzzy set so as to sort and output. FileName is the name of the document where the expression is located, and Score is the document score in Table 7.


## 6. Conclusion

Based on the retrieval and ranking mode of combining mathematical expression image and text, this study proposes a multimodal retrieval and ranking method for scientific documents based on HFS and XLNet. This method obtains the LaTeX structure information of mathematical expressions through image recognition algorithms and solves the single-modal problem of scientific document retrieval. The similarity between mathematical expressions is obtained by the evaluation of hesitant fuzzy sets, which solves the problem of the unity of evaluation of traditional mathematical expressions. In combination with the context of mathematical expression, the words with similar query keywords are obtained according to XLNet, which enriches the singleness problem of mathematical expression retrieval. Finally, the similarity between of attributes of mathematical expressions and the keywords in the documents is calculated through the hesitation fuzzy set, which makes the ranking of the retrieval results of scientific documents more reasonable.This experimental method also has some shortcomings. In the future, the following points will be considered for improvement:(1)
Only the mathematical expressions whose recognition results are in LaTeX form are analyzed, and different forms of mathematical expressions (such as MathML) will be analyzed(2)
The evaluation attributes of documents will be further improved, and the evaluation attributes of document similarity will be increased(3)
Only images and texts are analyzed, and an attempt will be made to expand the multimodality more widely and apply voice or video to retrieval



---

*Source: 5373531-2022-01-04.xml*