# A Weakly Supervised Academic Search Model Based on Knowledge-Enhanced Feature Representation

Authors: Mingying Xu; Junping Du; Feifei Kou; Meiyu Liang; Xin Xu; Jiaxin Yang
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Internet of Things search has great potential applications with the rapid development of Internet of Things technology. Combining Internet of Things technology and academic search to build academic search framework based on Internet of Things is an effective solution to realize massive academic resource search. Recently, the academic big data has been characterized by a large number of types and spanning many fields. The traditional web search technology is no longer suitable for the search environment of academic big data. Thus, this paper designs academic search framework based on Internet of Things Technology. In order to alleviate the pressure of the cloud server processing massive academic big data, the edge server is introduced to clean and remove the redundancy of the data to form a clean data for further analysis and processing by the cloud server. Edge computing network effectively makes up for the deficiency of cloud computing in the conditions of distributed and high concurrent access, reduces long-distance data transmission, and improves the quality of network user experience. For Academic Search, this paper proposes a novel weakly supervised academic search model based on knowledge-enhanced feature representation. The proposed model can relieve high cost of acquisition of manually labeled data by obtaining a lot of pseudolabeled data and consider word-level interactive matching and sentence-level semantic matching for more accurate matching in the process of academic search. The experimental result on academic datasets demonstrate that the performance of the proposed model is much better than that of the existing methods.


---

## Body

## 1. Introduction

Internet of Things technology has contributed to the realization of many Internet of Things applications that benefit the whole world and constantly changes people’s way of life. Internet of Things search service is one of the most important services provided by the Internet of Things. It can efficiently and accurately obtain information to meet the needs of users from massive, heterogeneous, and dynamic Internet of Things data. Currently, academic big data presents the characteristics of huge quantity, various types, and spanning multiple fields. Different from network data in the general sense, academic big data includes paper patents, scientific research equipment, experts and scholars, scientific research teams, national key laboratories, major scientific research infrastructure and large scientific research instruments, academic video scientific research reports, and other scientific and technological entities. Academic big data presents the characteristics of massive, multisource, heterogeneous, and related. Traditional web search technology is no longer suitable for academic big data search environment. Internet of Things search can solve the search problem of academic big data. Internet of Things search is that users send search query requests to the network system, the network system exchanges information with the physical world, and then returns the search object and its location, status, and other information to users. Academic big data search based on Internet of Things technology transfers the massive academic big data to remote cloud server for analysis and calculation, which promotes the interaction between users and intelligent system, improves the search efficiency and realizes automation of academic search. Academic search should not only ensure the search efficiency but also ensure the search accuracy. Therefore, efficient and accurate search is urgently needed.However, on the one hand, there are several key issues to be considered in the way that massive academic big data is analyzed and calculated by remote cloud server. First of all, multisource, heterogeneous massive big data has data redundancy, noise, data missing, and other problems. If it is transmitted to the remote cloud processing server, it will increase the burden of the cloud server. Secondly, a large number of data directly communicate with the cloud server, which occupies the bandwidth resources and seriously affects the transmission rate. At the same time, accurate academic resource search is another problem to be solved. Many current academic search studies pay little attention on specific search algorithms [1, 2]. There are only few studies on specific academic search task. Explicit Semantic Ranking [3] (ESR) defines academic retrieval as entity set retrieval, which represents the query and each document using knowledge graph embedding, and uses manually labeled training data to train a supervised academic search model. SetRank [4] models the relationships between entities through the type of entities while representing entities and proposes an unsupervised academic search framework. The retrieval performance of ESR and SetRank based on entity set retrieval has been improved to some extent, but there is still much room for improvement. On the one hand, ESR does not consider the relationship between entities. On the other hand, ESR only considers entity level matching, but does not consider deep semantic matching between query and text. What is more, ESR is a supervised academic retrieval model. It can show strong effectiveness [5] when large-scale manual-labeled data of document relevance are available. But manually labeled academic data usually requires domain expert knowledge, which is time-consuming, labor-intensive, and difficult to obtain. It is a serious bottleneck [6] for supervised academic retrieval. Moreover, although SetRank models interentity relationships using entity types, it still cannot model complex relationships between entities. In many cases, a user who submits such a query as “a new ranking method that leverages knowledge graph embedding” will expect to know how “ranking method” associated with “knowledge graph embedding.” The distinguishing characteristic of such queries is that they reflect user’s needs to find documents containing interentity relationships. As the results of the survey in [4], such queries are common in academic search scenarios.In response to the above-mentioned problems, this paper designs academic search framework in technology Internet of Things. In order to alleviate the pressure of the cloud server processing massive academic big data, the edge server is introduced to clean and remove the redundancy of the data to form a clean data for further analysis and processing by the cloud server. The edge computing network effectively makes up for the deficiency of cloud computing in the conditions of distributed and high concurrent access, reduces the long-distance data transmission process, has faster processing and response speed and lower computing and storage costs, and greatly improves the quality of network user experience. For academic search, this paper proposes a weakly supervised academic search model based on knowledge-enhanced feature representation. In this paper, we refer to both entities and words as features. Specifically, we first employ a scientific information organization tool SCIIE [7] to extract entities and relationships from scientific literatures. In order to obtain the feature representation that can express the entity and the relationship between entities, language model is trained with structured knowledge added as supervision signals. Based on the learned feature vector, we propose a weakly supervised academic retrieval model. In academic search, we should not only pay attention to the semantic matching of words in the query and document but also consider interactive matching between the features in the query and the document. On the other hand, a new weakly supervised method suitable for academic search is employed to obtain a large amount of pseudolabeled training data. Then, the academic search model is trained on labeled training data and pseudolabeled training data based on the knowledge-enhanced feature representation. We conduct extensive experiments on academic data sets. The experimental results demonstrate the effectiveness of our model in improving retrieval performance. We summarize the main novelties and contributions as follows:
(1)
An academic search framework based on Internet of Things technology is designed(2)
A novel weakly supervised academic search model based on the knowledge-enhanced feature representation is proposed to improve academic search performance(3)
In the process of academic search, not only the word-level interactive matching but also the sentence-level semantic matching between query and document are considered to realize the accurate matching between query and document(4)
Extensive experiments on the academic datasets prove that our proposed model is significantly better than the state-of-the-art search methodsThe rest of this paper is organized as follows: Section2 discusses related work, and Section 3 describes knowledge-enhanced feature representation for weakly supervised academic search. Section 4 reports experimental results and analysis, and we summarize this article in Section 5.


## 2. Related Work

### 2.1. IOT Search

Internet of Things search service is that users send search requests to the network system, the network system exchanges information with the physical world, and then returns information such as the location and status of the search object to users, which is directly driven by users. The existing Internet of Things search technology mainly includes location-based search, content-based search, and heterogeneous search. Location-based search mainly searches the content associated with location. The location information may be expressed as geographic coordinates or may be a logical location, such as a distance from another device [8]. Content-based search is based on the data content collected by a specific target sensor. First, the Internet of Things search engine analyzes the content and maps the corresponding index. Then, when querying, the search engine uses the corresponding index to pair the query with the content and returns the sensor information. Heterogeneous search technology mainly includes semantic or ontology-based search and resource retrieval. In particular, ontology represents concepts, types, and relationships in different fields [9]. The integration of semantic and ontology mechanism can help the system to build domain, task, and method combination search system. On the other hand, all data and IOT devices can abstract resources and provide different services.


### 2.2. Knowledge-Aware Representation

Representation learning [10] is a core issue in the information retrieval field. Distributed representation methods such as word2vec [11] and glove [12] have been successfully applied in the field of information retrieval [13]. In recent years, the emergence of large-scale knowledge graphs has promoted the development of knowledge-aware representation [14]. The knowledge graph contains rich knowledge [15]. Knowledge-aware representation brings in rich semantic information from the knowledge graph, which significantly improves the effectiveness of the search algorithms [16] and provides new opportunities for better understanding queries and documents [17, 18]. For instance, Xiong et al. introduce a bag-of-entities model, which successfully improves the retrieval accuracy by representing queries and documents using their entity annotations [19]. Hadas et al. design an entity-based language model which uniformly marks the individual features in the text and the term sequence recognized as entities by the entity linking tool as entities and effectively used for document retrieval [20]. A word-entity framework [21] is proposed that combines the bag-of-words and the entities linked to the knowledge graph to optimize information retrieval. Liu et al. employ Entity-Duet Neural Ranking [22], which introduces the knowledge graph into the neural search system. This model proposes a knowledge representation method combining words and entities to represent queries and documents and significantly improves search performance.


### 2.3. Weakly Supervised Neural Information Retrieval

With the great success of deep neural networks, several deep neural network-based retrieval models have been proposed, such as CDSSM [23], MatchPramid [24], DRMM [25], K-NRM [26], and CONV-KNRM [27]. These methods focus on either matching the whole document or word level interaction. There is no balance between the two aspects. What is more, when large-scale relevance training signals are available, the neural network retrieval models have shown strong effectiveness. However, collecting manually labeled data is time-consuming and labor-intensive [28]. Therefore, insufficient labeled training data has become the main obstacle that affects the performance of neural network-based retrieval models [29]. Weak supervision [30, 31] is an effective way to overcome this limitation. Levy et al. use the generated pseudolabeled data to train the neural network retrieval model, thereby solving the problem of difficulty in obtaining labeled data [32]. One can obtain clicked data of users as weakly supervised training data [33–35]. However, the randomness and subjectivity of user’s click behavior caused the inconsistency between user’s click and the real relevance label, which made the training set mixed with noise. Dehghani et al. propose a ranking-based method [36] to train a neural retrieval model with the help of an existing retrieval model such as BM25, which assumes that documents with higher scores are more relevant to the query than documents with lower scores. Recently, content-based methods [37, 38] are proposed to generate a set of weak pseudoqueries and related documents. MacAvaney et al. employ filters to eliminate training samples that cannot be converted well into relevance scores [37]. A training sample filtering technique based on heuristics and a new type of supervised filtering [38] is developed to remove those samples which are far away from the domains.


## 2.1. IOT Search

Internet of Things search service is that users send search requests to the network system, the network system exchanges information with the physical world, and then returns information such as the location and status of the search object to users, which is directly driven by users. The existing Internet of Things search technology mainly includes location-based search, content-based search, and heterogeneous search. Location-based search mainly searches the content associated with location. The location information may be expressed as geographic coordinates or may be a logical location, such as a distance from another device [8]. Content-based search is based on the data content collected by a specific target sensor. First, the Internet of Things search engine analyzes the content and maps the corresponding index. Then, when querying, the search engine uses the corresponding index to pair the query with the content and returns the sensor information. Heterogeneous search technology mainly includes semantic or ontology-based search and resource retrieval. In particular, ontology represents concepts, types, and relationships in different fields [9]. The integration of semantic and ontology mechanism can help the system to build domain, task, and method combination search system. On the other hand, all data and IOT devices can abstract resources and provide different services.


## 2.2. Knowledge-Aware Representation

Representation learning [10] is a core issue in the information retrieval field. Distributed representation methods such as word2vec [11] and glove [12] have been successfully applied in the field of information retrieval [13]. In recent years, the emergence of large-scale knowledge graphs has promoted the development of knowledge-aware representation [14]. The knowledge graph contains rich knowledge [15]. Knowledge-aware representation brings in rich semantic information from the knowledge graph, which significantly improves the effectiveness of the search algorithms [16] and provides new opportunities for better understanding queries and documents [17, 18]. For instance, Xiong et al. introduce a bag-of-entities model, which successfully improves the retrieval accuracy by representing queries and documents using their entity annotations [19]. Hadas et al. design an entity-based language model which uniformly marks the individual features in the text and the term sequence recognized as entities by the entity linking tool as entities and effectively used for document retrieval [20]. A word-entity framework [21] is proposed that combines the bag-of-words and the entities linked to the knowledge graph to optimize information retrieval. Liu et al. employ Entity-Duet Neural Ranking [22], which introduces the knowledge graph into the neural search system. This model proposes a knowledge representation method combining words and entities to represent queries and documents and significantly improves search performance.


## 2.3. Weakly Supervised Neural Information Retrieval

With the great success of deep neural networks, several deep neural network-based retrieval models have been proposed, such as CDSSM [23], MatchPramid [24], DRMM [25], K-NRM [26], and CONV-KNRM [27]. These methods focus on either matching the whole document or word level interaction. There is no balance between the two aspects. What is more, when large-scale relevance training signals are available, the neural network retrieval models have shown strong effectiveness. However, collecting manually labeled data is time-consuming and labor-intensive [28]. Therefore, insufficient labeled training data has become the main obstacle that affects the performance of neural network-based retrieval models [29]. Weak supervision [30, 31] is an effective way to overcome this limitation. Levy et al. use the generated pseudolabeled data to train the neural network retrieval model, thereby solving the problem of difficulty in obtaining labeled data [32]. One can obtain clicked data of users as weakly supervised training data [33–35]. However, the randomness and subjectivity of user’s click behavior caused the inconsistency between user’s click and the real relevance label, which made the training set mixed with noise. Dehghani et al. propose a ranking-based method [36] to train a neural retrieval model with the help of an existing retrieval model such as BM25, which assumes that documents with higher scores are more relevant to the query than documents with lower scores. Recently, content-based methods [37, 38] are proposed to generate a set of weak pseudoqueries and related documents. MacAvaney et al. employ filters to eliminate training samples that cannot be converted well into relevance scores [37]. A training sample filtering technique based on heuristics and a new type of supervised filtering [38] is developed to remove those samples which are far away from the domains.


## 3. Weakly Supervised Academic Search Based on Knowledge-Enhanced Feature Representation in Technology Internet of Things

### 3.1. System Model

In this part, we introduce the architecture of academic search system in academic Internet of Things. As shown in Figure1, the system is mainly divided into four layers, data acquisition layer, edge server layer, cloud server layer, and academic search layer.Figure 1
Academic Internet of Things framework.


#### 3.1.1. Data Acquisition

The data acquisition layer mainly obtains academic big data resources, including papers and patents, scientific research teams, scientific and technological talents, key laboratories, instruments, and equipment.


#### 3.1.2. Edge Server

Edge server adopts edge computing model, which is a distributed network model. Because of its high autonomy, each edge server is located in a specific area and connected to a specific device or data source in that area. The edge server is used to clean the academic resources, remove the redundancy, and submit them to the cloud server for further analysis and processing.


#### 3.1.3. Cloud Server

The cloud server layer is composed of a large number of dedicated servers with strong computing power, storage capacity, and stable connection, which can process a large number of data and complex computing in a very short time. Therefore, the most important advantage of the cloud server layer is the aggregation, mining, analysis [39, 40], and storage of massive data, which are beyond the processing capacity of the edge server. The cloud server layer is mainly used to deal with more complex and huge computing intensive data and tasks, such as the training of academic search model based on deep learning.


#### 3.1.4. Academic Search Layer

According to the query submitted by users, it aims to search a group of academic Internet of Things resources, including academic resources and equipment. The Internet of Things search engine responds to the query and returns academic resources (such as key laboratories and equipment), academic data (such as papers and patents).The academic search process is shown in Figure2 as follows.Figure 2
Academic search process.


### 3.2. Weakly Supervised Academic Search Based on Knowledge-Enhanced Representation

Weakly supervised academic search model is shown in Figure3, including two main processes: knowledge-enhanced feature learning and weakly supervised academic search.Figure 3
Weakly supervised academic search model.For the knowledge-enhanced feature learning, it models entities and their relationships by adding the knowledge extracted from scientific literatures to train word2vec to express rich semantics. The training process makes the learned features not only depend on the cooccurrence information of features in context but also learn the complex relationship between features.Weakly supervised academic search model can be trained on large-scale data with the help of weakly supervision strategy to improve the search performance. In the process of academic search, the representation-based matching model and the interaction-based matching model are deployed to the matching module. The model focuses on not only the accurate matching between words but also the semantic matching between query and document to improve the matching accuracy.


#### 3.2.1. Knowledge-Enhanced Feature Representation Learning

In this paper, we propose a knowledge-enhanced feature representation learning method by using word2vec based on skip-gram. The proposed representation learning method is able to express richer semantic information with the knowledge graph embedding model TransE [41]. This paper uses the scientific information extraction tool SCIIE to extract the entities and the relationships in the academic literature. SCIIE defines 6 relationships including “Used-for,” “Feature-of,” “Hyponym-of,” “Part-of,” “Compare,” and “Conjunction.” It can extract entity and interentity relationships in scientific papers and organize them into structured knowledge Tti,r,tj. Here, ti and tj are features extracted from the scientific literature, and r is relationship between ti and tj. This knowledge is added when training word2vec, so that the learning process of feature vectors is based on not only the cooccurrence information of the context but also the relationship between features, thereby improving the quality of semantic expression. Here, we refer to word and entity extracted from the training corpus as features. The objective function of the knowledge-enhanced feature representation learning model is as follows:
(1)L=∑ilogptc∣ti+∑rlogptj∣ti+r,where ti is a feature of the entire corpus and termc is the context of ti. r is the relationship between ti and tj. The left side of Equation (1) is the optimization loss of word2vec based on skip-gram, and the right side is the optimization loss of TransE. ptc∣ti is the probability that tc be predicted as the context of termi. ptj∣ti+r is the probability that ti+r equals to tj. They can be approximately calculated by negative sampling. Given ti and r, tc and tj are specifically positive samples of word2vec model and TransE model. We use negative sampling to extract negative samples of ti.The optimization loss of word2vec based on skip-gram can be calculated as follows:(2)∑ilogptc∣ti=∑ti,tc∈S+logσetc∙eti+∑tc′ϵNtilogσetc′∙eti.Here,ti,tc∈S+ implies that ti is the context of tc, S+ is positive sample sets of ti. tc′ϵNti implies that tc′ is not the context of ti. Nti is the negative sample set of ti. eti, etc,etc′ is the embedding of feature ti,tc, tc′.The optimization loss of TransE can be calculated as follows:(3)∑rlogptj∣ti+r=∑ti,r,tj∈T+logσeti+r∙etj+∑ti,r,tj∈T−log1−σeti+r∙etj.ti,r,tj∈T+implies that ti+r equals to tj. T+ is the positive sample set of ti,r. ti,r,tj∈T− implies that ti+r do not equal to tj. T− is the negative sample set of ti,r.We use stochastic gradient ascent to update parameters. Finally, feature vectorseti can be obtained which express semantic relations among entities and entity relationship.


#### 3.2.2. Weakly Supervised Academic Search

The weakly supervised academic search model is implemented as a pairwise ranking model in Figure1. Given a query-document pair q,d1,d2, it tries to learn a model that assigns a larger score to document d1 than document d2 if d1 matches to q better. By combining the ranking-based weak supervision method with the content-based weak supervision method, we propose a novel weak supervision method for generating query document pairs.On the one hand, for a given query and of manually labeled documents, a query-document pair is formed as labeled training data (query,d+, d−). On the other hand, given a query, top-ranked documents can be recalled from candidate documents using BM25 algorithm. Then, human-labeled documents related to the query are labeled as positive samples, and the documents recalled by BM25 are labeled as negative samples, which forms positive and negative sample pairs. At the same time, considering the title of the paper reflects the most critical content of the paper in the most appropriate and concise terms. And the abstract of the paper contains the most important and necessary information of the paper and is a summary of the main content of the paper. So, we choose the title and corresponding abstract of a paper in the document collection as a positive query-document sample. To sample a negative example of the query, we still use the BM25 algorithm to search the corresponding negative example of this query. Then, the pseudopositive and negative sample pairs are formed, denoted by Pse-training data (Pse-query, Pse-D+, Pse-D-). Finally, weakly supervised training data (Query, D+, D−) are formed including labeled training data (query, d+, d−) and pseudolabeled training data (pse-query, pse-d+, pse-d-). The weakly supervised training data are sent to knowledge-enhanced representation layer to map query and document to feature embedding etqt=1m and etidt=1n. Then, etqt=1m and etidt=1n are sent to matching layer to calculate matching score1 and score2. Score1 represents the interaction matching score of the word in query and document, and score2 represents the semantic matching score of query and document.Specifically, KNRM [26] is employed to compute interactive matching score score1. First, the transformation matrix Mij calculates the similarity between the word of query and document.
(4)Mij=coseiq,ejd.Then, kernels are employed to convertM to query-document matching features ∅M.
(5)KKMi=∑jexp−Mij−μk22σk2,K⟶Mi=K1Mi,⋯,KKMi,∅M=∑i=1nlogK⟶Mi.KKMi represents the kth RBF kernel. K⟶Mi employs K kernels to transform translation matrix into a K-dimensional feature vector. The matching features ∅M are sent to a fully connected layer to produce the final ranking score f1q,d.
(6)f1q,d=tanhWs1T∅M+bs1.At the same time,etqt=1m and etidt=1n are sent to a simple Bi-LSTM layer, and we can get the contextual representation. The specific calculation is as follows:
(7)ctq=BiLSTMqct−1q,eiq,ctid=BiLSTMqct−1id,eiid.Then, self-attention is employed to distinguish the importance of different words in the query and sentence of document when calculating the representation of query and document.(8)atq=expWqTtanhWq∙ctq∑i=1mexpWqTtanhWq∙ctq,preq=∑j=1matqcjq,atid=expWdTtanhWd∙ctid∑j=1nexpWdTtanhWd∙ctid,preid=∑t=1natidctid,pred=∑t=1lpretdl.Finally, the similarity scoref2q,d of query and document is calculated through a full connection layer.
(9)f2q,d=tanhWs2Tpreq⨀pred+bs2.Given a pair of weakly supervised training samples, we use the hinge loss [42] of information retrieval as the loss function, which is calculated as follows:
(10)loss=∑q,d+,d−∈Query,D+,D−max0,1−f1q,d++f2q,d++f1q,d−+f2q,d−,where f1q,d represents the interactive matching score between the query and the document. f2q,d represents the semantic matching score between the query and the document to balance human-judged relevance labels and BM25 model scores. The model is fine-tuned using human-labeled relevance judgments after the parameters of the network is pretrained using the weakly supervised data.


## 3.1. System Model

In this part, we introduce the architecture of academic search system in academic Internet of Things. As shown in Figure1, the system is mainly divided into four layers, data acquisition layer, edge server layer, cloud server layer, and academic search layer.Figure 1
Academic Internet of Things framework.


### 3.1.1. Data Acquisition

The data acquisition layer mainly obtains academic big data resources, including papers and patents, scientific research teams, scientific and technological talents, key laboratories, instruments, and equipment.


### 3.1.2. Edge Server

Edge server adopts edge computing model, which is a distributed network model. Because of its high autonomy, each edge server is located in a specific area and connected to a specific device or data source in that area. The edge server is used to clean the academic resources, remove the redundancy, and submit them to the cloud server for further analysis and processing.


### 3.1.3. Cloud Server

The cloud server layer is composed of a large number of dedicated servers with strong computing power, storage capacity, and stable connection, which can process a large number of data and complex computing in a very short time. Therefore, the most important advantage of the cloud server layer is the aggregation, mining, analysis [39, 40], and storage of massive data, which are beyond the processing capacity of the edge server. The cloud server layer is mainly used to deal with more complex and huge computing intensive data and tasks, such as the training of academic search model based on deep learning.


### 3.1.4. Academic Search Layer

According to the query submitted by users, it aims to search a group of academic Internet of Things resources, including academic resources and equipment. The Internet of Things search engine responds to the query and returns academic resources (such as key laboratories and equipment), academic data (such as papers and patents).The academic search process is shown in Figure2 as follows.Figure 2
Academic search process.


## 3.1.1. Data Acquisition

The data acquisition layer mainly obtains academic big data resources, including papers and patents, scientific research teams, scientific and technological talents, key laboratories, instruments, and equipment.


## 3.1.2. Edge Server

Edge server adopts edge computing model, which is a distributed network model. Because of its high autonomy, each edge server is located in a specific area and connected to a specific device or data source in that area. The edge server is used to clean the academic resources, remove the redundancy, and submit them to the cloud server for further analysis and processing.


## 3.1.3. Cloud Server

The cloud server layer is composed of a large number of dedicated servers with strong computing power, storage capacity, and stable connection, which can process a large number of data and complex computing in a very short time. Therefore, the most important advantage of the cloud server layer is the aggregation, mining, analysis [39, 40], and storage of massive data, which are beyond the processing capacity of the edge server. The cloud server layer is mainly used to deal with more complex and huge computing intensive data and tasks, such as the training of academic search model based on deep learning.


## 3.1.4. Academic Search Layer

According to the query submitted by users, it aims to search a group of academic Internet of Things resources, including academic resources and equipment. The Internet of Things search engine responds to the query and returns academic resources (such as key laboratories and equipment), academic data (such as papers and patents).The academic search process is shown in Figure2 as follows.Figure 2
Academic search process.


## 3.2. Weakly Supervised Academic Search Based on Knowledge-Enhanced Representation

Weakly supervised academic search model is shown in Figure3, including two main processes: knowledge-enhanced feature learning and weakly supervised academic search.Figure 3
Weakly supervised academic search model.For the knowledge-enhanced feature learning, it models entities and their relationships by adding the knowledge extracted from scientific literatures to train word2vec to express rich semantics. The training process makes the learned features not only depend on the cooccurrence information of features in context but also learn the complex relationship between features.Weakly supervised academic search model can be trained on large-scale data with the help of weakly supervision strategy to improve the search performance. In the process of academic search, the representation-based matching model and the interaction-based matching model are deployed to the matching module. The model focuses on not only the accurate matching between words but also the semantic matching between query and document to improve the matching accuracy.


### 3.2.1. Knowledge-Enhanced Feature Representation Learning

In this paper, we propose a knowledge-enhanced feature representation learning method by using word2vec based on skip-gram. The proposed representation learning method is able to express richer semantic information with the knowledge graph embedding model TransE [41]. This paper uses the scientific information extraction tool SCIIE to extract the entities and the relationships in the academic literature. SCIIE defines 6 relationships including “Used-for,” “Feature-of,” “Hyponym-of,” “Part-of,” “Compare,” and “Conjunction.” It can extract entity and interentity relationships in scientific papers and organize them into structured knowledge Tti,r,tj. Here, ti and tj are features extracted from the scientific literature, and r is relationship between ti and tj. This knowledge is added when training word2vec, so that the learning process of feature vectors is based on not only the cooccurrence information of the context but also the relationship between features, thereby improving the quality of semantic expression. Here, we refer to word and entity extracted from the training corpus as features. The objective function of the knowledge-enhanced feature representation learning model is as follows:
(1)L=∑ilogptc∣ti+∑rlogptj∣ti+r,where ti is a feature of the entire corpus and termc is the context of ti. r is the relationship between ti and tj. The left side of Equation (1) is the optimization loss of word2vec based on skip-gram, and the right side is the optimization loss of TransE. ptc∣ti is the probability that tc be predicted as the context of termi. ptj∣ti+r is the probability that ti+r equals to tj. They can be approximately calculated by negative sampling. Given ti and r, tc and tj are specifically positive samples of word2vec model and TransE model. We use negative sampling to extract negative samples of ti.The optimization loss of word2vec based on skip-gram can be calculated as follows:(2)∑ilogptc∣ti=∑ti,tc∈S+logσetc∙eti+∑tc′ϵNtilogσetc′∙eti.Here,ti,tc∈S+ implies that ti is the context of tc, S+ is positive sample sets of ti. tc′ϵNti implies that tc′ is not the context of ti. Nti is the negative sample set of ti. eti, etc,etc′ is the embedding of feature ti,tc, tc′.The optimization loss of TransE can be calculated as follows:(3)∑rlogptj∣ti+r=∑ti,r,tj∈T+logσeti+r∙etj+∑ti,r,tj∈T−log1−σeti+r∙etj.ti,r,tj∈T+implies that ti+r equals to tj. T+ is the positive sample set of ti,r. ti,r,tj∈T− implies that ti+r do not equal to tj. T− is the negative sample set of ti,r.We use stochastic gradient ascent to update parameters. Finally, feature vectorseti can be obtained which express semantic relations among entities and entity relationship.


### 3.2.2. Weakly Supervised Academic Search

The weakly supervised academic search model is implemented as a pairwise ranking model in Figure1. Given a query-document pair q,d1,d2, it tries to learn a model that assigns a larger score to document d1 than document d2 if d1 matches to q better. By combining the ranking-based weak supervision method with the content-based weak supervision method, we propose a novel weak supervision method for generating query document pairs.On the one hand, for a given query and of manually labeled documents, a query-document pair is formed as labeled training data (query,d+, d−). On the other hand, given a query, top-ranked documents can be recalled from candidate documents using BM25 algorithm. Then, human-labeled documents related to the query are labeled as positive samples, and the documents recalled by BM25 are labeled as negative samples, which forms positive and negative sample pairs. At the same time, considering the title of the paper reflects the most critical content of the paper in the most appropriate and concise terms. And the abstract of the paper contains the most important and necessary information of the paper and is a summary of the main content of the paper. So, we choose the title and corresponding abstract of a paper in the document collection as a positive query-document sample. To sample a negative example of the query, we still use the BM25 algorithm to search the corresponding negative example of this query. Then, the pseudopositive and negative sample pairs are formed, denoted by Pse-training data (Pse-query, Pse-D+, Pse-D-). Finally, weakly supervised training data (Query, D+, D−) are formed including labeled training data (query, d+, d−) and pseudolabeled training data (pse-query, pse-d+, pse-d-). The weakly supervised training data are sent to knowledge-enhanced representation layer to map query and document to feature embedding etqt=1m and etidt=1n. Then, etqt=1m and etidt=1n are sent to matching layer to calculate matching score1 and score2. Score1 represents the interaction matching score of the word in query and document, and score2 represents the semantic matching score of query and document.Specifically, KNRM [26] is employed to compute interactive matching score score1. First, the transformation matrix Mij calculates the similarity between the word of query and document.
(4)Mij=coseiq,ejd.Then, kernels are employed to convertM to query-document matching features ∅M.
(5)KKMi=∑jexp−Mij−μk22σk2,K⟶Mi=K1Mi,⋯,KKMi,∅M=∑i=1nlogK⟶Mi.KKMi represents the kth RBF kernel. K⟶Mi employs K kernels to transform translation matrix into a K-dimensional feature vector. The matching features ∅M are sent to a fully connected layer to produce the final ranking score f1q,d.
(6)f1q,d=tanhWs1T∅M+bs1.At the same time,etqt=1m and etidt=1n are sent to a simple Bi-LSTM layer, and we can get the contextual representation. The specific calculation is as follows:
(7)ctq=BiLSTMqct−1q,eiq,ctid=BiLSTMqct−1id,eiid.Then, self-attention is employed to distinguish the importance of different words in the query and sentence of document when calculating the representation of query and document.(8)atq=expWqTtanhWq∙ctq∑i=1mexpWqTtanhWq∙ctq,preq=∑j=1matqcjq,atid=expWdTtanhWd∙ctid∑j=1nexpWdTtanhWd∙ctid,preid=∑t=1natidctid,pred=∑t=1lpretdl.Finally, the similarity scoref2q,d of query and document is calculated through a full connection layer.
(9)f2q,d=tanhWs2Tpreq⨀pred+bs2.Given a pair of weakly supervised training samples, we use the hinge loss [42] of information retrieval as the loss function, which is calculated as follows:
(10)loss=∑q,d+,d−∈Query,D+,D−max0,1−f1q,d++f2q,d++f1q,d−+f2q,d−,where f1q,d represents the interactive matching score between the query and the document. f2q,d represents the semantic matching score between the query and the document to balance human-judged relevance labels and BM25 model scores. The model is fine-tuned using human-labeled relevance judgments after the parameters of the network is pretrained using the weakly supervised data.


## 3.2.1. Knowledge-Enhanced Feature Representation Learning

In this paper, we propose a knowledge-enhanced feature representation learning method by using word2vec based on skip-gram. The proposed representation learning method is able to express richer semantic information with the knowledge graph embedding model TransE [41]. This paper uses the scientific information extraction tool SCIIE to extract the entities and the relationships in the academic literature. SCIIE defines 6 relationships including “Used-for,” “Feature-of,” “Hyponym-of,” “Part-of,” “Compare,” and “Conjunction.” It can extract entity and interentity relationships in scientific papers and organize them into structured knowledge Tti,r,tj. Here, ti and tj are features extracted from the scientific literature, and r is relationship between ti and tj. This knowledge is added when training word2vec, so that the learning process of feature vectors is based on not only the cooccurrence information of the context but also the relationship between features, thereby improving the quality of semantic expression. Here, we refer to word and entity extracted from the training corpus as features. The objective function of the knowledge-enhanced feature representation learning model is as follows:
(1)L=∑ilogptc∣ti+∑rlogptj∣ti+r,where ti is a feature of the entire corpus and termc is the context of ti. r is the relationship between ti and tj. The left side of Equation (1) is the optimization loss of word2vec based on skip-gram, and the right side is the optimization loss of TransE. ptc∣ti is the probability that tc be predicted as the context of termi. ptj∣ti+r is the probability that ti+r equals to tj. They can be approximately calculated by negative sampling. Given ti and r, tc and tj are specifically positive samples of word2vec model and TransE model. We use negative sampling to extract negative samples of ti.The optimization loss of word2vec based on skip-gram can be calculated as follows:(2)∑ilogptc∣ti=∑ti,tc∈S+logσetc∙eti+∑tc′ϵNtilogσetc′∙eti.Here,ti,tc∈S+ implies that ti is the context of tc, S+ is positive sample sets of ti. tc′ϵNti implies that tc′ is not the context of ti. Nti is the negative sample set of ti. eti, etc,etc′ is the embedding of feature ti,tc, tc′.The optimization loss of TransE can be calculated as follows:(3)∑rlogptj∣ti+r=∑ti,r,tj∈T+logσeti+r∙etj+∑ti,r,tj∈T−log1−σeti+r∙etj.ti,r,tj∈T+implies that ti+r equals to tj. T+ is the positive sample set of ti,r. ti,r,tj∈T− implies that ti+r do not equal to tj. T− is the negative sample set of ti,r.We use stochastic gradient ascent to update parameters. Finally, feature vectorseti can be obtained which express semantic relations among entities and entity relationship.


## 3.2.2. Weakly Supervised Academic Search

The weakly supervised academic search model is implemented as a pairwise ranking model in Figure1. Given a query-document pair q,d1,d2, it tries to learn a model that assigns a larger score to document d1 than document d2 if d1 matches to q better. By combining the ranking-based weak supervision method with the content-based weak supervision method, we propose a novel weak supervision method for generating query document pairs.On the one hand, for a given query and of manually labeled documents, a query-document pair is formed as labeled training data (query,d+, d−). On the other hand, given a query, top-ranked documents can be recalled from candidate documents using BM25 algorithm. Then, human-labeled documents related to the query are labeled as positive samples, and the documents recalled by BM25 are labeled as negative samples, which forms positive and negative sample pairs. At the same time, considering the title of the paper reflects the most critical content of the paper in the most appropriate and concise terms. And the abstract of the paper contains the most important and necessary information of the paper and is a summary of the main content of the paper. So, we choose the title and corresponding abstract of a paper in the document collection as a positive query-document sample. To sample a negative example of the query, we still use the BM25 algorithm to search the corresponding negative example of this query. Then, the pseudopositive and negative sample pairs are formed, denoted by Pse-training data (Pse-query, Pse-D+, Pse-D-). Finally, weakly supervised training data (Query, D+, D−) are formed including labeled training data (query, d+, d−) and pseudolabeled training data (pse-query, pse-d+, pse-d-). The weakly supervised training data are sent to knowledge-enhanced representation layer to map query and document to feature embedding etqt=1m and etidt=1n. Then, etqt=1m and etidt=1n are sent to matching layer to calculate matching score1 and score2. Score1 represents the interaction matching score of the word in query and document, and score2 represents the semantic matching score of query and document.Specifically, KNRM [26] is employed to compute interactive matching score score1. First, the transformation matrix Mij calculates the similarity between the word of query and document.
(4)Mij=coseiq,ejd.Then, kernels are employed to convertM to query-document matching features ∅M.
(5)KKMi=∑jexp−Mij−μk22σk2,K⟶Mi=K1Mi,⋯,KKMi,∅M=∑i=1nlogK⟶Mi.KKMi represents the kth RBF kernel. K⟶Mi employs K kernels to transform translation matrix into a K-dimensional feature vector. The matching features ∅M are sent to a fully connected layer to produce the final ranking score f1q,d.
(6)f1q,d=tanhWs1T∅M+bs1.At the same time,etqt=1m and etidt=1n are sent to a simple Bi-LSTM layer, and we can get the contextual representation. The specific calculation is as follows:
(7)ctq=BiLSTMqct−1q,eiq,ctid=BiLSTMqct−1id,eiid.Then, self-attention is employed to distinguish the importance of different words in the query and sentence of document when calculating the representation of query and document.(8)atq=expWqTtanhWq∙ctq∑i=1mexpWqTtanhWq∙ctq,preq=∑j=1matqcjq,atid=expWdTtanhWd∙ctid∑j=1nexpWdTtanhWd∙ctid,preid=∑t=1natidctid,pred=∑t=1lpretdl.Finally, the similarity scoref2q,d of query and document is calculated through a full connection layer.
(9)f2q,d=tanhWs2Tpreq⨀pred+bs2.Given a pair of weakly supervised training samples, we use the hinge loss [42] of information retrieval as the loss function, which is calculated as follows:
(10)loss=∑q,d+,d−∈Query,D+,D−max0,1−f1q,d++f2q,d++f1q,d−+f2q,d−,where f1q,d represents the interactive matching score between the query and the document. f2q,d represents the semantic matching score between the query and the document to balance human-judged relevance labels and BM25 model scores. The model is fine-tuned using human-labeled relevance judgments after the parameters of the network is pretrained using the weakly supervised data.


## 4. Experiments Settings

This section describes our experimental dataset, metrics, baselines, and other implementation details.


### 4.1. Dataset

The experimental datasets come from Semantic Scholar (http://corpus. http://semanticscholar.org/) provided by Xiong et al., including 170,983 candidate document set (https://alleninstitute.org/) with 100 queries. Manual relevance judgement (a 5-level scale, including 4, 3, 2, 1, 0, 4 is the most relevant and 0 is not relevant) are available (http://boston.lti.cs.cmu.edu/appendices/WWW2016/) on the dataset, where both the relevant and irrelevant documents are labeled for each query. In this work, we mainly use the title and abstract of the paper as in [4]. The reasons are as follows: first, the title and abstract of the paper can refine the main points of the paper and summarize the main content of the original text. Second, the original text of the paper is not easy to obtain. Third, the original paper has a large amount of data and slow processing speed. The statistics of experimental datasets is shown in Table 1. Ultimately, we convert this data to a unified format and use this data as an analog for real academic IoT search.Table 1
The statistics of academic datasets.
DetailsNumberNumber of queries100Number of papers170,983Sum of title length1,200,641Sum of abstract length43,241,159


### 4.2. Baselines and Metrics

We compare information retrieval methods that are popular in the field of information retrieval, including K-NRM [26], Conv-KNRM [27], MatchPyramid [24], DRMM [25], MV-LSTM [43], and ArcII [44].KNRM: It first computes cosine similarity between query word and document words using a translation layer, and then, it uses a feed-forward network to perform kernel pooling to compute the relevance score between query and document.Conv-KNRM: Conv-KNRM improves KNRM by using CNN filters to modeln-gram soft matches of queries and documents to capture n-gram such as phrases, concepts, or entities existed in the queries and documents.MatchPyramid: It computes pair-wise dot product between query and document word vectors to compute an interaction matrix. It then passes this matrix through CNN layers with dynamic pooling to compute the similarity score.DRMM: It firstly maps the local interaction matrix of query and document into a fixed length matching histogram. Then, it uses forward matching network to learn hierarchical matching features. These features are calculated by term gate network to get the global relevance score of query and document.MV-LSTM: It uses the word embeddings obtained by passing the sentences through a Bi-LSTM and then computes an interaction vector using cosine similarity or a bilinear operation. It finally passes the interaction vector through a feed-forward network to compute the similarity score.ArcII: ArcII first computes the interaction feature vector between query and document using CNN layers. It then computes the score for the query-document interaction vector using feed-forward network.In academic search, the quality of top-ranked results is critical to improve user satisfaction, so the top rankings of Precision and NDCG are particularly important. In order to evaluate the efficiency of academic retrieval model, we report four standard evaluation indicators: MAP, MRR, Precision of top 5, 10, 15, 20 documents retrieved (P@5, P@10, P@15, P@20), NDCG of top 5, 10, 15, 20 documents retrieved (N@5, N@10, N@15, N@20).


### 4.3. Experiments Setting and Training Details

#### 4.3.1. Knowledge-Enhanced Representation

First, we employ SCIIE to extract the entities and relationships between entities contained in each scientific document. A total of 12,326,751 triples are formed. The detailed knowledge extraction results are shown in Figure4. From the figure, we can see the proportion of the extracted relationships between entities. We use word2vec based on skip-gram and TransE to train the knowledge-enhanced feature representation, which is optimized by negative sampling. We set the embedding dimension to be 300, and the window size to be 5.Figure 4
Proportion of relationship between entities in the structured knowledge.


#### 4.3.2. Weakly Supervised Training Data Preparation

First, for each query in the given dataset, the manually labeled documents are formed into positive and negative sample pairs according to the label level. Then, we use the BM25 algorithm to recall 500 documents with higher scores from the candidate documents. In these recalled 500 documents, the documents which have been manually labeled are marked as positive samples, and other documents are marked as negative samples to form positive and negative sample pairs. Then, we use the title of a paper in the document collection as a pseudoquery, and its corresponding document as a positive example. Then, we use BM25 algorithm to search for the corresponding negative examples of this pseudoquery. Finally, a weakly supervised training set is formed, including original labeled data (query, positive samples, negative samples) and prelabeled data (prequery, prepositive samples, and prenegative samples). The model is pretrained on weakly supervised training data and fine-tuned on human-labeled training data.


#### 4.3.3. Effectiveness Verification Settings of the Proposed Model

Our model is abbreviated as KER-IPM-WS. It includes three modules named as knowledge-enhanced representation learning module (KER), matching module (ISM, including interactive matching IM and semantic matching SM), and weakly supervised training (WS). In order to verify the effectiveness of each module, we designed six model variants as follows:KER-IM: Academic search is implemented based on knowledge-enhanced feature representation, the academic text matching of which is based on interactive matching.KER-SM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.KER-ISM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-IM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching.KER-SM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-ISM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.We implement our model using TensorFlow. For the neural retrieval models CDSSM, KNRM, and Conv-KNRM, we use an open-source implementation MatchZoo (https://github.com/NTMC-Community/MatchZoo). For KNRM and Conv-KNRM, we set the number of bins to 11. We apply gradient descent algorithm and Adam as our optimizer for training the ranking model. And we use dropout technology to prevent overfitting. We set the batch size to 64 and select the learning rate from [1e−1,1e−2,1e−3,1e−4]. The training epoch number is set to 20. We use 5-fold crossvalidation to validate our model. We randomly split all training data into five equal partitions. In each fold, three partitions are used for training, one for validation and one for testing.


## 4.1. Dataset

The experimental datasets come from Semantic Scholar (http://corpus. http://semanticscholar.org/) provided by Xiong et al., including 170,983 candidate document set (https://alleninstitute.org/) with 100 queries. Manual relevance judgement (a 5-level scale, including 4, 3, 2, 1, 0, 4 is the most relevant and 0 is not relevant) are available (http://boston.lti.cs.cmu.edu/appendices/WWW2016/) on the dataset, where both the relevant and irrelevant documents are labeled for each query. In this work, we mainly use the title and abstract of the paper as in [4]. The reasons are as follows: first, the title and abstract of the paper can refine the main points of the paper and summarize the main content of the original text. Second, the original text of the paper is not easy to obtain. Third, the original paper has a large amount of data and slow processing speed. The statistics of experimental datasets is shown in Table 1. Ultimately, we convert this data to a unified format and use this data as an analog for real academic IoT search.Table 1
The statistics of academic datasets.
DetailsNumberNumber of queries100Number of papers170,983Sum of title length1,200,641Sum of abstract length43,241,159


## 4.2. Baselines and Metrics

We compare information retrieval methods that are popular in the field of information retrieval, including K-NRM [26], Conv-KNRM [27], MatchPyramid [24], DRMM [25], MV-LSTM [43], and ArcII [44].KNRM: It first computes cosine similarity between query word and document words using a translation layer, and then, it uses a feed-forward network to perform kernel pooling to compute the relevance score between query and document.Conv-KNRM: Conv-KNRM improves KNRM by using CNN filters to modeln-gram soft matches of queries and documents to capture n-gram such as phrases, concepts, or entities existed in the queries and documents.MatchPyramid: It computes pair-wise dot product between query and document word vectors to compute an interaction matrix. It then passes this matrix through CNN layers with dynamic pooling to compute the similarity score.DRMM: It firstly maps the local interaction matrix of query and document into a fixed length matching histogram. Then, it uses forward matching network to learn hierarchical matching features. These features are calculated by term gate network to get the global relevance score of query and document.MV-LSTM: It uses the word embeddings obtained by passing the sentences through a Bi-LSTM and then computes an interaction vector using cosine similarity or a bilinear operation. It finally passes the interaction vector through a feed-forward network to compute the similarity score.ArcII: ArcII first computes the interaction feature vector between query and document using CNN layers. It then computes the score for the query-document interaction vector using feed-forward network.In academic search, the quality of top-ranked results is critical to improve user satisfaction, so the top rankings of Precision and NDCG are particularly important. In order to evaluate the efficiency of academic retrieval model, we report four standard evaluation indicators: MAP, MRR, Precision of top 5, 10, 15, 20 documents retrieved (P@5, P@10, P@15, P@20), NDCG of top 5, 10, 15, 20 documents retrieved (N@5, N@10, N@15, N@20).


## 4.3. Experiments Setting and Training Details

### 4.3.1. Knowledge-Enhanced Representation

First, we employ SCIIE to extract the entities and relationships between entities contained in each scientific document. A total of 12,326,751 triples are formed. The detailed knowledge extraction results are shown in Figure4. From the figure, we can see the proportion of the extracted relationships between entities. We use word2vec based on skip-gram and TransE to train the knowledge-enhanced feature representation, which is optimized by negative sampling. We set the embedding dimension to be 300, and the window size to be 5.Figure 4
Proportion of relationship between entities in the structured knowledge.


### 4.3.2. Weakly Supervised Training Data Preparation

First, for each query in the given dataset, the manually labeled documents are formed into positive and negative sample pairs according to the label level. Then, we use the BM25 algorithm to recall 500 documents with higher scores from the candidate documents. In these recalled 500 documents, the documents which have been manually labeled are marked as positive samples, and other documents are marked as negative samples to form positive and negative sample pairs. Then, we use the title of a paper in the document collection as a pseudoquery, and its corresponding document as a positive example. Then, we use BM25 algorithm to search for the corresponding negative examples of this pseudoquery. Finally, a weakly supervised training set is formed, including original labeled data (query, positive samples, negative samples) and prelabeled data (prequery, prepositive samples, and prenegative samples). The model is pretrained on weakly supervised training data and fine-tuned on human-labeled training data.


### 4.3.3. Effectiveness Verification Settings of the Proposed Model

Our model is abbreviated as KER-IPM-WS. It includes three modules named as knowledge-enhanced representation learning module (KER), matching module (ISM, including interactive matching IM and semantic matching SM), and weakly supervised training (WS). In order to verify the effectiveness of each module, we designed six model variants as follows:KER-IM: Academic search is implemented based on knowledge-enhanced feature representation, the academic text matching of which is based on interactive matching.KER-SM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.KER-ISM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-IM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching.KER-SM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-ISM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.We implement our model using TensorFlow. For the neural retrieval models CDSSM, KNRM, and Conv-KNRM, we use an open-source implementation MatchZoo (https://github.com/NTMC-Community/MatchZoo). For KNRM and Conv-KNRM, we set the number of bins to 11. We apply gradient descent algorithm and Adam as our optimizer for training the ranking model. And we use dropout technology to prevent overfitting. We set the batch size to 64 and select the learning rate from [1e−1,1e−2,1e−3,1e−4]. The training epoch number is set to 20. We use 5-fold crossvalidation to validate our model. We randomly split all training data into five equal partitions. In each fold, three partitions are used for training, one for validation and one for testing.


## 4.3.1. Knowledge-Enhanced Representation

First, we employ SCIIE to extract the entities and relationships between entities contained in each scientific document. A total of 12,326,751 triples are formed. The detailed knowledge extraction results are shown in Figure4. From the figure, we can see the proportion of the extracted relationships between entities. We use word2vec based on skip-gram and TransE to train the knowledge-enhanced feature representation, which is optimized by negative sampling. We set the embedding dimension to be 300, and the window size to be 5.Figure 4
Proportion of relationship between entities in the structured knowledge.


## 4.3.2. Weakly Supervised Training Data Preparation

First, for each query in the given dataset, the manually labeled documents are formed into positive and negative sample pairs according to the label level. Then, we use the BM25 algorithm to recall 500 documents with higher scores from the candidate documents. In these recalled 500 documents, the documents which have been manually labeled are marked as positive samples, and other documents are marked as negative samples to form positive and negative sample pairs. Then, we use the title of a paper in the document collection as a pseudoquery, and its corresponding document as a positive example. Then, we use BM25 algorithm to search for the corresponding negative examples of this pseudoquery. Finally, a weakly supervised training set is formed, including original labeled data (query, positive samples, negative samples) and prelabeled data (prequery, prepositive samples, and prenegative samples). The model is pretrained on weakly supervised training data and fine-tuned on human-labeled training data.


## 4.3.3. Effectiveness Verification Settings of the Proposed Model

Our model is abbreviated as KER-IPM-WS. It includes three modules named as knowledge-enhanced representation learning module (KER), matching module (ISM, including interactive matching IM and semantic matching SM), and weakly supervised training (WS). In order to verify the effectiveness of each module, we designed six model variants as follows:KER-IM: Academic search is implemented based on knowledge-enhanced feature representation, the academic text matching of which is based on interactive matching.KER-SM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.KER-ISM: Academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-IM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching.KER-SM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on semantic matching.KER-ISM-WS: Weakly supervised academic search is implemented based on knowledge-enhanced feature representation; the academic text matching of which is based on interactive matching and semantic matching.We implement our model using TensorFlow. For the neural retrieval models CDSSM, KNRM, and Conv-KNRM, we use an open-source implementation MatchZoo (https://github.com/NTMC-Community/MatchZoo). For KNRM and Conv-KNRM, we set the number of bins to 11. We apply gradient descent algorithm and Adam as our optimizer for training the ranking model. And we use dropout technology to prevent overfitting. We set the batch size to 64 and select the learning rate from [1e−1,1e−2,1e−3,1e−4]. The training epoch number is set to 20. We use 5-fold crossvalidation to validate our model. We randomly split all training data into five equal partitions. In each fold, three partitions are used for training, one for validation and one for testing.


## 5. Result Analysis and Use Case

### 5.1. Result Analysis

In this section, we first verify the effectiveness of knowledge-enhanced representation by comparing with the retrieval methods in baselines. Then, we further compare the academic retrieval performance of different module of KER-ISM-WS and analyze the impact on retrieval performance of different module of KER-ISM-WS.


### 5.2. Retrieval Performance of Different Retrieval Model

Tables2 and 3, respectively, show the retrieval performance of weakly supervised academic search based on knowledge-enhanced feature representation and the baselines on NDCG@{5, 10, 15, 20} and Precision@{5, 10, 15, 20}. Table 4 shows the retrieval performance of the proposed weakly supervised academic search based on knowledge-enhanced feature representation and the baselines on MAP and MRR.Table 2
Performance comparison on NDCG of different retrieval models.
ModelN@5N@10N@15N@20KNRM0.41600.48070.54060.6077ConvKNRM0.44370.52580.58910.6535MatchPyramid0.36550.44750.50560.5727DRMM0.39350.45360.54110.5996MV-LSTM0.32440.40130.48220.5525ArcII0.42560.47680.55870.6094KER-IPM-WS0.58420.60090.63960.6653Table 3
Performance comparison on precision of different retrieval models.
ModelP@5P@10P@15P@20KNRM0.52410.51030.49870.4653ConvKNRM0.54520.53270.50480.4852MatchPyramid0.45170.43100.43210.3978DRMM0.46470.44140.41870.3831MV-LSTM0.42070.40340..38950.3527ArcII0.46550.42640.42070.3862KER-IPM-WS0.65090.62320.55210.5248Table 4
Performance comparison on NDCG of different module of KER-ISM-WS.
ModelN@5N@10N@15N@20KER-IM0.47530.50750.56720.6157KER-SM0.44270.48690.54060.5993KER-ISM0.52930.56930.59190.6269KER-IM-WS0.52280.54520.58160.6436KER-SM-WS0.48650.51330.57910.6255KER-ISM-WS0.58420.60090.63960.6653It can be seen from Tables2, 3, and 5, the retrieval performance of our model exceeds the baselines. Experimental results show that our method is effective for academic search. It is worth noting that ConV-KNRM employs CNN filters to model n-gram of queries and documents to identify phrases, concepts, or entities. In addition to our model, ConV-KNRM achieves the best retrieval performance in the baselines. These findings indicate that mining phrases, concepts, or entities existed in academic texts is helpful for improving academic retrieval performance.Table 5
Performance comparison on MAP and MRR of different retrieval models.
ModelMAPMRRKNRM0.41730.4929ConvKNRM0.45210.5402MatchPyramid0.38490.4730DRMM0.41580.4775MV-LSTM0.34370.4208ArcII0.44850.5086KER-IPM-WS0.55690.6427


### 5.3. The Impact on Retrieval Performance of Different Module of KER-ISM-WS

This section shows the impact of different module of KER-ISM-WS on retrieval performance Precision, NDCG, MAP, and MRR. The results are shown in Tables4 and 6.Table 6
Performance comparison on precision of different module of KER-ISM-WS.
ModelP@5P@10P@15P@20KER-IM0.57330.54090.49610.4628KER-SM0.54390.50280.48740.4379KER-ISM0.59310.52170.50370.4867KER-IM-WS0.60310.54780.48560.4701KER-SM-WS0.58340.52540.47130.4443KER-ISM-WS0.65090.62320.55210.5248


#### 5.3.1. The Impact on Retrieval Performance of Knowledge-Enhanced Feature Representation

KER_IM is academic search based on knowledge-enhanced feature representation; the academic text matching of which is based on KNRM. Therefore, we can compare it with KNRM to verify the effectiveness of the knowledge-enhanced representation. By comparing the retrieval performance of KNRM in Tables2, 3, and 5 and KER_IM in Tables 4, 6, and 7, we can conclude that the proposed knowledge-enhanced feature representation method can improve retrieval performance.Table 7
Performance comparison on MAP and MRR of different module of KER-ISM-WS.
ModelMAPMRRKER-IM0.47640.5921KER-SM0.44910.5317KER-ISM0.52520.6437KER-IM-WS0.51210.5958KER-SM-WS0.49640.5838KER-ISM-WS0.55690.6427


#### 5.3.2. The Impact on Retrieval Performance of Matching Based on Interactive Matching and Semantic Matching

We can compare KER_IM, KER_SM, and KER_ISM to verify the effectiveness of the matching based on interactive matching and semantic matching. From Tables4, 6, and 7, the results indicate that the retrieval performance of the model KER_ISM is better than KER_IM and KER_SM. The reason is that KER_ISM considers both word interaction matching and sentence semantic matching to guide more accurate search.


#### 5.3.3. The Impact on Retrieval Performance of Weakly Supervised Training

We can compare KER_IM and KER_IM_WS, KER_SM and KER_SM_WS, and KER_ISM and KER-ISM-WS to verify the effectiveness of the weakly supervised training. From Tables5–7, the results indicate that the retrieval performance of the models KER-KER_IM-WS, KER_SM_WS, and KER_ISM_WS is better than that of KER-IM, KER-SM, and KER-ISM. The possible reason is that a large amount of weakly supervised training data can guide the model to learn better parameters.Table 8
Search results of query “object detection”.
KER-PM-WSPaper titleHuman labelEntity relation1Rapidobject detection using a boosted cascade of simple features4Cascade of simple featuresUSED-FOR object detection2Class-specific Hough forests forobject detection5Hough forestsUSED-FORobject detection3Faster R-CNN: towards real-timeobject detection with region proposal networks3R-CNNUSED-FOR object detection4Learning rich features from RGB-D images forobject detection and segmentation2Learning rich featuresUSED-FORobject detectionRGB-D images USED-FOR learning rich featuresRGB-D images USED-FORobject detection5Rich feature hierarchies for accurateobject detection and semantic segmentation3Rich feature hierarchiesUSED-FOR object detectionQuery: object detection.


### 5.4. Use Case—Academic Search

In this section, we use the proposed KER_ISM_WS model to report the academic search results. A real use case for literature retrieval is as follows in Table8. The input query is object detection, which reflects the information needs of users.Table8 shows the retrieval results of our model and the manually labeled level for these retrieval results. Entities in the titles of the retrieved papers are highlighted in green. Our method also gives the entities that exist in the title of the paper and the relationships between them. Entities are marked as green; relationships are marked as red.From the search results and manually labeled tags, we can see that our model has achieved accurate matching. At the same time, we infer from the manually annotated data that the query is trying to find documents that have a certain relationship with the “object detection.” The search results show that our model can retrieve articles which meet the needs. Moreover, all the items in the search results can express the entities related to the query and the relationship information between the entities. This further illustrates the effectiveness of the proposed representation learning model. It not only models the entities but also models the semantic relationships that exist between entities, which helps to improve the performance of the retrieval model.


## 5.1. Result Analysis

In this section, we first verify the effectiveness of knowledge-enhanced representation by comparing with the retrieval methods in baselines. Then, we further compare the academic retrieval performance of different module of KER-ISM-WS and analyze the impact on retrieval performance of different module of KER-ISM-WS.


## 5.2. Retrieval Performance of Different Retrieval Model

Tables2 and 3, respectively, show the retrieval performance of weakly supervised academic search based on knowledge-enhanced feature representation and the baselines on NDCG@{5, 10, 15, 20} and Precision@{5, 10, 15, 20}. Table 4 shows the retrieval performance of the proposed weakly supervised academic search based on knowledge-enhanced feature representation and the baselines on MAP and MRR.Table 2
Performance comparison on NDCG of different retrieval models.
ModelN@5N@10N@15N@20KNRM0.41600.48070.54060.6077ConvKNRM0.44370.52580.58910.6535MatchPyramid0.36550.44750.50560.5727DRMM0.39350.45360.54110.5996MV-LSTM0.32440.40130.48220.5525ArcII0.42560.47680.55870.6094KER-IPM-WS0.58420.60090.63960.6653Table 3
Performance comparison on precision of different retrieval models.
ModelP@5P@10P@15P@20KNRM0.52410.51030.49870.4653ConvKNRM0.54520.53270.50480.4852MatchPyramid0.45170.43100.43210.3978DRMM0.46470.44140.41870.3831MV-LSTM0.42070.40340..38950.3527ArcII0.46550.42640.42070.3862KER-IPM-WS0.65090.62320.55210.5248Table 4
Performance comparison on NDCG of different module of KER-ISM-WS.
ModelN@5N@10N@15N@20KER-IM0.47530.50750.56720.6157KER-SM0.44270.48690.54060.5993KER-ISM0.52930.56930.59190.6269KER-IM-WS0.52280.54520.58160.6436KER-SM-WS0.48650.51330.57910.6255KER-ISM-WS0.58420.60090.63960.6653It can be seen from Tables2, 3, and 5, the retrieval performance of our model exceeds the baselines. Experimental results show that our method is effective for academic search. It is worth noting that ConV-KNRM employs CNN filters to model n-gram of queries and documents to identify phrases, concepts, or entities. In addition to our model, ConV-KNRM achieves the best retrieval performance in the baselines. These findings indicate that mining phrases, concepts, or entities existed in academic texts is helpful for improving academic retrieval performance.Table 5
Performance comparison on MAP and MRR of different retrieval models.
ModelMAPMRRKNRM0.41730.4929ConvKNRM0.45210.5402MatchPyramid0.38490.4730DRMM0.41580.4775MV-LSTM0.34370.4208ArcII0.44850.5086KER-IPM-WS0.55690.6427


## 5.3. The Impact on Retrieval Performance of Different Module of KER-ISM-WS

This section shows the impact of different module of KER-ISM-WS on retrieval performance Precision, NDCG, MAP, and MRR. The results are shown in Tables4 and 6.Table 6
Performance comparison on precision of different module of KER-ISM-WS.
ModelP@5P@10P@15P@20KER-IM0.57330.54090.49610.4628KER-SM0.54390.50280.48740.4379KER-ISM0.59310.52170.50370.4867KER-IM-WS0.60310.54780.48560.4701KER-SM-WS0.58340.52540.47130.4443KER-ISM-WS0.65090.62320.55210.5248


### 5.3.1. The Impact on Retrieval Performance of Knowledge-Enhanced Feature Representation

KER_IM is academic search based on knowledge-enhanced feature representation; the academic text matching of which is based on KNRM. Therefore, we can compare it with KNRM to verify the effectiveness of the knowledge-enhanced representation. By comparing the retrieval performance of KNRM in Tables2, 3, and 5 and KER_IM in Tables 4, 6, and 7, we can conclude that the proposed knowledge-enhanced feature representation method can improve retrieval performance.Table 7
Performance comparison on MAP and MRR of different module of KER-ISM-WS.
ModelMAPMRRKER-IM0.47640.5921KER-SM0.44910.5317KER-ISM0.52520.6437KER-IM-WS0.51210.5958KER-SM-WS0.49640.5838KER-ISM-WS0.55690.6427


### 5.3.2. The Impact on Retrieval Performance of Matching Based on Interactive Matching and Semantic Matching

We can compare KER_IM, KER_SM, and KER_ISM to verify the effectiveness of the matching based on interactive matching and semantic matching. From Tables4, 6, and 7, the results indicate that the retrieval performance of the model KER_ISM is better than KER_IM and KER_SM. The reason is that KER_ISM considers both word interaction matching and sentence semantic matching to guide more accurate search.


### 5.3.3. The Impact on Retrieval Performance of Weakly Supervised Training

We can compare KER_IM and KER_IM_WS, KER_SM and KER_SM_WS, and KER_ISM and KER-ISM-WS to verify the effectiveness of the weakly supervised training. From Tables5–7, the results indicate that the retrieval performance of the models KER-KER_IM-WS, KER_SM_WS, and KER_ISM_WS is better than that of KER-IM, KER-SM, and KER-ISM. The possible reason is that a large amount of weakly supervised training data can guide the model to learn better parameters.Table 8
Search results of query “object detection”.
KER-PM-WSPaper titleHuman labelEntity relation1Rapidobject detection using a boosted cascade of simple features4Cascade of simple featuresUSED-FOR object detection2Class-specific Hough forests forobject detection5Hough forestsUSED-FORobject detection3Faster R-CNN: towards real-timeobject detection with region proposal networks3R-CNNUSED-FOR object detection4Learning rich features from RGB-D images forobject detection and segmentation2Learning rich featuresUSED-FORobject detectionRGB-D images USED-FOR learning rich featuresRGB-D images USED-FORobject detection5Rich feature hierarchies for accurateobject detection and semantic segmentation3Rich feature hierarchiesUSED-FOR object detectionQuery: object detection.


## 5.3.1. The Impact on Retrieval Performance of Knowledge-Enhanced Feature Representation

KER_IM is academic search based on knowledge-enhanced feature representation; the academic text matching of which is based on KNRM. Therefore, we can compare it with KNRM to verify the effectiveness of the knowledge-enhanced representation. By comparing the retrieval performance of KNRM in Tables2, 3, and 5 and KER_IM in Tables 4, 6, and 7, we can conclude that the proposed knowledge-enhanced feature representation method can improve retrieval performance.Table 7
Performance comparison on MAP and MRR of different module of KER-ISM-WS.
ModelMAPMRRKER-IM0.47640.5921KER-SM0.44910.5317KER-ISM0.52520.6437KER-IM-WS0.51210.5958KER-SM-WS0.49640.5838KER-ISM-WS0.55690.6427


## 5.3.2. The Impact on Retrieval Performance of Matching Based on Interactive Matching and Semantic Matching

We can compare KER_IM, KER_SM, and KER_ISM to verify the effectiveness of the matching based on interactive matching and semantic matching. From Tables4, 6, and 7, the results indicate that the retrieval performance of the model KER_ISM is better than KER_IM and KER_SM. The reason is that KER_ISM considers both word interaction matching and sentence semantic matching to guide more accurate search.


## 5.3.3. The Impact on Retrieval Performance of Weakly Supervised Training

We can compare KER_IM and KER_IM_WS, KER_SM and KER_SM_WS, and KER_ISM and KER-ISM-WS to verify the effectiveness of the weakly supervised training. From Tables5–7, the results indicate that the retrieval performance of the models KER-KER_IM-WS, KER_SM_WS, and KER_ISM_WS is better than that of KER-IM, KER-SM, and KER-ISM. The possible reason is that a large amount of weakly supervised training data can guide the model to learn better parameters.Table 8
Search results of query “object detection”.
KER-PM-WSPaper titleHuman labelEntity relation1Rapidobject detection using a boosted cascade of simple features4Cascade of simple featuresUSED-FOR object detection2Class-specific Hough forests forobject detection5Hough forestsUSED-FORobject detection3Faster R-CNN: towards real-timeobject detection with region proposal networks3R-CNNUSED-FOR object detection4Learning rich features from RGB-D images forobject detection and segmentation2Learning rich featuresUSED-FORobject detectionRGB-D images USED-FOR learning rich featuresRGB-D images USED-FORobject detection5Rich feature hierarchies for accurateobject detection and semantic segmentation3Rich feature hierarchiesUSED-FOR object detectionQuery: object detection.


## 5.4. Use Case—Academic Search

In this section, we use the proposed KER_ISM_WS model to report the academic search results. A real use case for literature retrieval is as follows in Table8. The input query is object detection, which reflects the information needs of users.Table8 shows the retrieval results of our model and the manually labeled level for these retrieval results. Entities in the titles of the retrieved papers are highlighted in green. Our method also gives the entities that exist in the title of the paper and the relationships between them. Entities are marked as green; relationships are marked as red.From the search results and manually labeled tags, we can see that our model has achieved accurate matching. At the same time, we infer from the manually annotated data that the query is trying to find documents that have a certain relationship with the “object detection.” The search results show that our model can retrieve articles which meet the needs. Moreover, all the items in the search results can express the entities related to the query and the relationship information between the entities. This further illustrates the effectiveness of the proposed representation learning model. It not only models the entities but also models the semantic relationships that exist between entities, which helps to improve the performance of the retrieval model.


## 6. Conclusion

We design academic Internet of Things search framework based on Internet of Things technology. In order to alleviate the pressure of the cloud server processing massive academic big data, the edge server is introduced to clean and remove the redundancy of the data to form a clean data for further analysis and processing by the cloud server. For academic search, we propose a novel knowledge-enhanced feature representation learning method which can express rich semantics in texts of academic search field. Aiming at the “data hungry” property of deep neural academic retrieval methods, we propose a weakly supervised academic search model based on the knowledge-enhanced representation, which relieves high cost of acquisition of manually labeled data by obtaining a lot of pseudolabeled data in the process of academic search. In the process of text matching for academic retrieval, the proposed model considers both the word-level interactive matching and the sentence-level semantic matching to improve matching accuracy of relevance. Experiment results on real academic search datasets show that the proposed model is effective and greatly improves the academic search performance.



---

*Source: 4411524-2021-09-22.xml*