# Intelligent English Tense Collocation and Evaluation Based on Deep Reinforcement Learning

Authors: Yiling Ding; Tianhua Wang
Journal: Mobile Information Systems
 (2022)

---

## Abstract

The representation of time in sentences is the key problem for tense collocation. Based on the relationship expression among regions in Allen’s interval algebra theory, we propose a vector representation method, i.e., relationship vector, and several operations are defined based on the relationship vector for temporal reasoning in this work. This method transforms the original matrix representation into vector representation, which reduces the amount of computation of temporal reasoning. In addition, we propose a temporal classification and collocation method based on deep learning and deep reinforcement learning. This method uses a bidirectional cyclic neural network and a convolutional neural network for text expression and achieves temporal word classification and temporal collocation based on the deep reinforcement learning model. In the experiments, the proposed method obtains the average accuracy of 92.17% in five datasets, i.e., MPQA, CR, MR, Subj, and TREC, which proves its effectiveness in tense collocation.


---

## Body

## 1. Introduction

Mathematical proof and simulation are the important foundation of modern science and technology, and the development of artificial intelligence (AI) is also based on strong mathematical theories [1]. Symbolization and formulation of human mind on computer have become an important topic in the research of AI. Establishing a unified theoretical system of knowledge-based human intelligence and machine intelligence is the mainstream idea of AI, and the key problem is knowledge representation, knowledge reasoning, and the application of knowledge. In recent years, with the application and development of AI and big data technology, deep learning (DL) technology has made breakthroughs in many fields, such as image recognition, speech recognition, and natural language processing (NLP) [2]. Among them, semantic analysis technology for NLP plays an important role in studying human communication mode. It affects not only the development of AI, but also the mode of human-computer interaction [3]. Using AI semantic analysis technology to analyze the language materials can significantly improve the analysis efficiency of user research results and even dig out important information that is difficult to find by artificial analysis. NLP is a technology that uses computer as a tool to process the information of human unique natural language [4, 5]. Semantic analysis of language materials is the basis of NLP involving linguistics, computational linguistics, machine learning, cognitive language, and other disciplines. The goal of semantic analysis is to realize the automatic semantic analysis of vocabulary, sentences, and chapters and understand the meaning of language text expression by establishing effective computational analysis models. Among them, the semantic analysis of words is the basis, and the commonly used knowledge expression methods include semantic field, semantic network, concept map, and ontology [6].There are a lot of emotions and ideas in natural language that cannot be described by simple logical relations. It is difficult to achieve because of the complexity of recognizing and understanding natural languages by computer. From the development history of NLP, relevant research can be divided into two categories: symbolic approaches and stochastic approaches. The symbolic approaches adopt the analysis method based on linguistic definition rules focusing on the reasoning model and the logical language understanding model, which is called rationalism. The stochastic approaches mainly study random and statistical algorithms to identify and calculate large-scale corpus [7]. Although the symbolic approaches can complete the basic analysis of a single sentence, the understanding of complex sentence patterns and even the analysis of the whole paragraph or chapter are not of high quality. The statistical method contained in the stochastic approaches provides a new idea for the research method of natural language understanding, but the early effect is not satisfactory until the emergence of DL technology in recent years. Relying on statistical methods and DL algorithms, machines can learn from big data and obtain the rule of language usage, which is greatly improved in efficiency and accuracy compared with traditional methods. In particular, in the application of Chinese semantic analysis, the effect improvement is particularly prominent. It uses the ability of automatic feature extraction in the DL model to replace the manual feature construction process in traditional machine learning, which greatly improves the effectiveness of the feature model. From the historical evolution of NLP, we can see that semantic analysis technology has been developed greatly with the intervention of machine learning methods represented by DL. Through the study of literature, it is found that many scholars have studied the application methods of intelligent semantic recognition technology, and their methods and analysis steps are also different according to different application purposes. Allen [8] proposes a temporal reasoning system based on continuous interval representation of time information. It can easily represent the temporal relationship between any two-time intervals, which have strong expression ability and are easy to understand. It is widely used in natural language understanding and other fields. However, in the reasoning operation of interval algebraic representation, repeated table searching is required, which is very time-consuming. It is very impractical for the system with high real-time requirements. Therefore, it is proposed to use interval matrix to represent temporal intervals and convert the traditional table searching into matrix operation, which is very fast. Its effect is very excellent when the real-time requirements are relatively quotient. However, the temporal relationship expressed by matrix is not intuitive.The traditional text representation methods include word bag model and vector space model. The word bag model uses unique heat coding to represent each word as a vector composed of 0 and 1. This method is simple and intuitive, but does not consider word order and semantic information. The vector space model treats text as points in vector space. The term frequency-inverse document frequency (TF-IDF) is commonly used to represent the importance of words and construct text feature representation. However, with the rapid increase in text data on the Internet, traditional text representation methods have some disadvantages, such as high dimension, sparse features, and loss of word order. In this case, many scholars begin to use DL, and there are three types of models for text representation and classification using neural networks [9]. The first type is the serialization model, such as convolutional neural network (CNN), recurrent neural network (RNN), and long-term and short-term memory network (LSTM). They record the text serialization information for training and carry out the text classification task after obtaining the feature representation. The second type is based on attention mechanism. The third category is structured text representation and classification model, such as recursive automatic encoder [10] and short-term and long-term memory network with tree structure. They use the predefined parse tree to divide the text hierarchy and then represent and classify the text. These text representation and classification models based on deep neural networks have been verified on public datasets and have achieved good text representation and classification performance. When constructing text features, traditional text representation methods and classification models based on deep learning often delete some useless words by means of stopping vocabulary or using parsing tree to divide text structure. They cannot delete words irrelevant to classification tasks according to different texts. Parsing trees are also constructed manually, which is time-consuming and laborious. Because reinforcement learning (RL) has the ability of autonomous learning with the rapid improvement of computing performance in recent years, the combination of DL and RL for task decision-making has been widely used. Deep reinforcement learning (DRL) has the advantage of learning knowledge by interacting with the environment [11–14]. Many scholars have applied DRL to text representation and classification tasks and achieved good results. On the basis of DRL, the text representation and classification model can independently learn words related to text classification tasks, divide sentence phrase structures, and make text feature tables. It is worth studying to show and improve the performance of text classification.To realize the accurate collocation of English sentence tenses, we propose a word-level text representation and classification model brand model based on DRL in this work. This method extracts words and divides phrase structures related to text classification tasks from main learning and optimizes text representation and classification tasks for words and phrases, respectively. Then, the Allen interval algebra is used to represent the temporal relationship, and it is applied to the path consistency algorithm of the network graph composed of a group of temporal intervals and the constraint relationship between them. In the method, the temporal constraint consistency of the subnetwork graph composed of three adjacent nodes is used as the preprocessing technology. Furthermore, the calculation result matrix is transformed into a temporal relationship representation, and one or two eigenvalues in the atomic matrix are used to determine its corresponding temporal relationship. Thus, while improving computational efficiency, we can visually represent and understand the temporal relationship and then make interval algebra obtain better representation and reasoning knowledge in our model. The experimental results show that the accuracy of tense collocation in English sentences can reach 92.17%.The rest of this study is organized as follows. Section2 presents the related work of tense collocation methods. Section 3 presents the detailed design on tense collocation method based on DRL. Experimental results and discussion are reported in Section 4. Finally, the conclusion of this study is in Section 5.


## 2. Related Work

Focusing on the text representation and classification model of sentences and documents, Zhang et al. [15] proposed a character-level convolutional neural network (char-CNN) model, which attempted to represent and classify text by constructing alphabet and using one-hot coding and CNN. Kim [16] proposed a word-CNN model that achieves word vector representation of static and non-static channel and English text classification. Mikolov et al. [17] proposed a language model using a cyclic neural network to study text representation and classification. Based on the work, Lai et al. [18] introduced a two-way structure to capture context information and retain a wide range of word sequence information combined with cyclic and CNN models to achieve better text representation and classification results. Tang et al. [19] proposed a gate-based cyclic neural network, which analyzed document-level content through a two-way gated recurrent neural network, and used forward and backpropagation to obtain context feature representation to improve the classification effect.The model of text feature representation and classification using DL has achieved good performance. However, in the task of text representation and classification, CNN, RNN, and other models do not consider the text hierarchy information. For structured DL models, such as tree LSTM, the predefined parsing tree is used for text structured processing, and the parsing tree is constructed by relying on a large amount of manpower and expert experience, which requires a lot of time and energy. Therefore, some scholars began to explore whether there is an interactive way that can make the text representation and classification model extract the vocabulary information related to the text classification task through autonomous learning, divide the text into phrase structures, and improve the accuracy of text feature representation and the performance of text classification. In this demand, DRL has entered the vision of many scholars. RL belongs to the branch of machine learning, and many classical algorithms have been proposed in the early stage, such as Q-learning algorithm and policy gradient algorithm. With the in-depth application of deep neural network, many scholars began to try to combine DL and RL in various fields to solve complex perceptual decision-making tasks. In view of the successful application of DRL in the image field, this method has been introduced in the text field. Yu et al. [20] used the strategic algorithm of DRL in generative adversarial networks (GANs), which provided an idea for solving the task of serializing text.Temporal reasoning includes the formalization of time concept and provides tools for the representation and reasoning of temporal knowledge. The research of temporal reasoning originated in the late 1950s and early 1960s and initially appeared in the form of temporal logic. In the 1980s, the research on temporal logic and reasoning was very active and had never been interrupted. A logical method is one of the main methods to study temporal reasoning. The research on temporal reasoning is mainly reflected in reliability, completeness, expression ability, decidability, and computational complexity. The research of spatial logic could be traced back to the 1950s, but it had not attracted great interest of researchers in AI, spatial database, image processing, and other fields until nearly two decades, mainly including topological spatial logic, metric spatial logic, directional relationship logic, and spatial logic integrating topology and measurement. In 2002, Bennett and Cohn [21] studied the multimodal system for spatiotemporal reasoning and proposed a multidimensional modal logic for representing spatiotemporal information. In 2006, Kudinoy et al. [22] added a difference operator based on logic S4 in topological space to enhance the expression ability of logic and proved its logical completeness after adding modal operator. The azimuth relation is an important spatial relation in spatial reasoning. The topological relation between spatial objects under topological transformation remains unchanged, but the azimuth relation may change. The logic complexity based on the azimuth relation is low, which is very effective for the formal representation of natural space, and has enough ability to represent meaningful spatial knowledge. Therefore, modal logic based on the azimuth relationship can better represent knowledge in natural space.Different from the previous work, this study proposed the DRL-based method to realize the accurate collocation of English sentence tenses. The proposed method shows more effective than baselines in tense collocation.


## 3. The Detailed Methodology

The representation of tenses is an important basis for the accurate collocation of tenses in text generation. In this work, we propose a temporal vector representation, i.e., relationship vector, based on the relationship expression method between regions in Allen’s interval algebra theory. Then, we design an intelligent English tense collocation method, in which several operations are defined, including relationship vector and text representation. This section will describe the details of the algorithm.


### 3.1. Vector Representation of Temporal Relations

The Allen interval algebra defines 13 basic interval relations, and their meanings are shown in Figure1, where x and y represent continuous temporal intervals, x, y ∈ [0, T], and T represents sufficiently large time values. Among the 13 relationships, 6 pairs of relationships are mutually inverse, and the other equal relationships are mutually inverse with itself. Each basic interval relation describes the definite relationship between two temporal intervals. To describe the uncertain temporal relationship, it needs to be represented by a group of basic interval relations, which are called general interval relations.Figure 1
13 relations of basic interval.The temporal relationship represented by matrix has some disadvantages, such as less intuition and high computational complexity. To express the temporal relationship and calculation more simply, we consider using the left and right endpoints of the interval to characterize the relationship between the endpoints of the interval. Let [0,T] be a time interval taken from the time axis, and T is large enough, and the interval to be investigated is always its subinterval. I = [I−, I+] (0 ≤ I− ≤ I+ > T) is one of its subintervals, i.e., I⊆ [0, T]. IL = [0, I−] is the left outer point interval of I, I− is the left endpoint of I, Ii = (I−, I+) is the inner interval of I, IR = [I−, T] is the right outer point interval of I, and I+ is the right endpoint of I. (IL, I−, Ii, I+, IR) elements are called interval I. Therefore, any time interval I can be written as follows:(1)IL∪I−∪Ii∪I+∪IR=0,T.We set upα and β as two subintervals of interval [0, T] and define ΔαβLL, ΔαβRL, ΔαβLR, and ΔαβRR as follows:(2)ΔαβLL=α−−β−,ΔαβRL=α+−β−,ΔαβLR=α−−β+,ΔαβRR=α+−β+,where α−, α+, β–, and β+ are the left and right endpoints of α and β, respectively.Definition 1.
Setting upα and β as two temporal intervals and vector v = (ΔαβLL, ΔαβRL, ΔαβLR, ΔαβRR) is called basic relationship vector, in which each element is −1, 0, or 1. If ΔαβLL > 0, ΔαβLL = 1; if ΔαβLL = 0, ΔαβLL = 0; and if ΔαβLL < 0, ΔαβLL = −1. ΔαβRL, ΔαβLR, and ΔαβRR have similar value rules.
The purpose of Definition1 is to express the basic relationship between intervals through the relationship between the left and right endpoints of intervals. Each vector obtained according to Definition 1 corresponds to a basic relationship, and thus, the basic interval relation of the Allen interval algebra can be expressed by vector, which is called the basic relationship vector. The 13 basic relationship vectors of the Allen interval algebra are listed as follows:(1)
α before β can be denoted as Vb = [−1, −1, −1, −1](2)
α after β can be denoted as Vb = [1, 1, 1, 1](3)
α meets β can be denoted as Vb  = [−1, 0, −1, −1](4)
α met by β can be denoted as Vb = [1, 1, 0, 1](5)
α overlaps β can be denoted as Vb = [−1, 1, −1, −1](6)
α overlapped by β can be denoted as Vb = [1, 1, −1, 1](7)
α finishes β can be denoted as Vb = [1, 1, −1, 0](8)
α finished by β can be denoted as Vb = [−1, 1, −1, 0](9)
α during β can be denoted as Vb = [1, 1, −1, −1](10)
α includes β can be denoted as Vb = [−1, 1, −1, 1](11)
α starts β can be denoted as Vb = [0, 1, −1, −1](12)
α started by β can be denoted as Vb = [0, 1, −1, 1](13)
α equals β can be denoted as Vb = [0, 1, −1, 0]
In the vector representation of relationship, the rules of compound operation and transfer operation of relationship are defined as follows.Definition 2.
Compound operation of relationship (V). If two intervalsx and y satisfy the relationship set P = (p1, p2, …, pk), 1 ≤ k ≤ 13, and Vj (j = 1, 2, …, k) is the atomic relationship vector corresponding to each atomic relation, then the temporal relationship vector Vp between x and y can be linearly expressed by Vj (j = 1, 2, …, k) as follows:(3)Vp=∑j=1kVj=1,1,…1×kV1V2⋮Vk.
It can be known from Definition1 that we should obtain the relationship between the left and right endpoints of α and γ.
Thus, the relationship between the left endpoints ofα and γ can be obtained from the left endpoint of β. The relationship between the right endpoint of α and the left endpoint of γ can be obtained from the right endpoint of β. The relationship between the left endpoint of α and the right endpoint of γ can be obtained from the left endpoint of β. The relationship between the left endpoint of α and the right endpoint of γ can be obtained from the right endpoint of β. From point algebra, the relationship between points {<, >, =} has the following characteristics:(1)
Transitivity, if the pointa < (>, =) b and b < (>, =) c, then a < (>, =) c(2)
Retention, if the pointa < (>) b and b = c, then a < (>) c. If a = b and b < (>) c, then a < (>) c(3)
Uncertainty, if the pointa < (>) b and b > (<) c, then the relationship between a and b is uncertain, which can be denoted as a (<, >, =) b
From the above discussion, the transfer operation of the defined relationship is as follows.Definition 3.
Relation transfer operation (∙). Setting time intervalα, β, and r1, r2 ∈ U, and if α{ r1}β, β{ r2}γ, then Vr1 and Vr2 are their corresponding relationship vectors, and the relationship vector of α between γ can be obtained by Vr1 and Vr2 with operation: Vr1r2 = Vr1∙Vr2, where Vri|i = 1, 2, 3, and 4 are the components of the relationship vector, and the value rules are as follows:(4)Vr1r23=1,if both ofVr13andVr24are0,or one of them is0and the other one is1,−1,if both ofVr13andVr24are1,or one ofthem is0and the other one is−1,0,if both ofVr13andVr24are0,−1,0,1,if one ofVr13andVr24is1andthe other one is−1,(5)Vr1r22=1,if both ofVr14andVr22are0,or one ofthem is0and the other one is1,−1,if both ofVr14andVr22are−1,or one ofthem is0and the other one is−1,0,if both ofVr14andVr22are0,−1,0,1,if one ofVr14andVr22is1andthe other one is−1,(6)Vr1r21=1,if both ofVr11andVr21are0,or one ofthem is0and the other one is1,−1,if both ofVr11andVr21are−1,or one ofthem is0and the other one is−1,0,if both ofVr11andVr21are0,−1,0,1,if one ofVr11andVr21is1andthe other one is−1,(7)Vr1r24=1,if both ofVr14andVr24are0,or one ofthem is0and the other one is1,−1,if both ofVr14andVr24are−1,or one ofthem is0and the other one is−1,0,if both ofVr14andVr24are0,−1,0,1,if one ofVr14andVr24is1andthe other one is−1,
The relationship betweenr1 and r2 in transfer operation can be generalized to relational subsets I1 and I2.Definition 4.
Transitive operation of relation subset (⊙). Setting time intervalα, β, and γ, and relational subsets I1 and I2 ⊆ U, if α{I1}β, β{I2}γ, Vr1, and Vr2 are their relationship vectors, then the relationship vector α between γ can be calculated by Vr1 and Vr2 with the operation ⊙ as follows:(8)VI1I2=VI1T⊙VI2=Vr1Vr2⋮VrmVR1VR2…VRN,where r1, …, rm ∈ I1, R1, …, Rn ∈ I2, and Vr1 and VR1 are their corresponding relationship vectors.


### 3.2. Word-Level Text Representation with Tense Collocation Based on DRL

Considering the slow training speed of strategic network in DRL, it is necessary to pre-train the original DL model without strategic network to accelerate and stabilize strategic network training and verify its effect on word extraction related to classification tasks in text. Therefore, before designing the overall word-level text representation and classification model based on DRL, we determine a DL model with excellent text representation and classification performance as the basic model, which is selected based on bidirectional recurrent neural network and convolution neural network (BRCNN). This model consists of two parts, including bidirectional cyclic neural network and CNN, in which the bidirectional cyclic neural network is used to extract the forward and reverse word order information of the text, and CNN is used to obtain the feature representation of the text.


#### 3.2.1. Input of the Model

We focus on the text representation and classification of sentences in this model, and the network input is the word vector matrix (x1, x2, …, xN)T ∈ RN×D corresponding to the ith word in the sentence, i.e., xi ∈ RD, D represents the dimension of the word vector, and N represents the number of words in the sentence.


#### 3.2.2. Bidirectional Cyclic Neural Network

After the input vector (x1, x2, …, xN)T passes through the bidirectional cyclic neural network, the forward word order information and backward word order information can be obtained, which are (h⟶1, h⟶2, …, h⟶N) and (h↼1, h↼2, …, h↼N), respectively. Attention mechanism contains two groups of weight factors, i.e., forward weighting factors (α⟶1·h⟶1, α⟶2·h⟶2, …, α⟶N·h⟶N) and backward weighting factor (α⟶1·h⟶1, α↼t·h↼2, …, α⟶N·h⟶N), respectively. The word order information vectors output by the cyclic neural network at each forward and backward time are weighted and cascaded to obtain O⟶ and O↼. We cascade O⟶ and O↼ to obtain the context vector ORNN, which is shown as follows:(9)ORNN=O⟶,O↼,where ORNN ∈ RN×2K, in which K is the number of hidden layer neurons.


#### 3.2.3. CNN Model

The input of CNN is the outputORNN of bidirectional cyclic neural network. In BRCNN, the size of convolution kernel f is S × 2K, where S is the window size of convolution kernel corresponding to the number of hidden states ORNN in each convolution. 2K is the convolution kernel width, which is consistent with the vector dimension in ORNN. The output ORNN of bidirectional cyclic neural network can obtain the output vector after convolution operation of convolution kernel. The BRCNN model designed in this work uses three different convolution kernels f1, f2, and f3, and their sizes are S1 × 2K, S2 × 2K, and S3 × 2K, respectively. After the output of the bidirectional cyclic neural network, i.e., ORNN, passes through three convolution kernels f1, f2, and f3, O1 ∈ RN–S1+1, O2 ∈ RN–S2 + 1, and O3 ∈ RN–S3 + 1 can be obtained as shown in Figure 2.Figure 2
Convolution operation of the convolution layer.We assume that the number of convolution kernels in each type isM, and O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) can be obtained after the convolution operation. The pooling operation is often carried out to reduce the dimension of the features extracted by the convolution kernel and prevent the number of features from being too large and the classification model from being too fitted. In this work, maximum pooling is used to process the output of CNN, i.e., taking the maximum value of the second dimension in O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) to obtain the vector, which is the result of pooling. As shown in Figure 2, for the convolution kernel f1, we take the maximum value in the convolution result O1 to obtain the pooled result of a convolution kernel. In the same way, we calculate the results of f1 convolution kernels to obtain the m-dimensional vector O1′ ∈ RM, O2′ ∈ RM, and O3′ ∈ RM, which will be used as input for the next phase of the model.


#### 3.2.4. DRL Model

BRCNN model uses the strategic network of DRL to independently extract and classify temporal related words and delete irrelevant words. By predicting the behavior of each word in the sentence, the strategic network determines the deletion or retention behavior of words, i.e., sampling the behavior of each word to achieve the purpose of purifying text information.In this work, we aim to classify the tenses of a single sentence sampleX and produce collocation results. The word vector matrix of X can be expressed as (x1, x2, …, xN)T, where xt corresponds to the vector representation of the tth word. The status, behavior, and reward of policy network are introduced as follows. We note that the state of the policy network at time t is st (t = 1, 2, …, N). In the BRCNN model, the state st (t = 1, 2, …, N) is composed of the word vector input xt of the tth word and the hidden state information obtained at time t of the bidirectional cyclic neural network. at (t = 1, 2, …, N) denotes the behavior of the policy network at time t. In the BRCNN model, the goal is to extract words related to tenses, and thus, the behavior at is set as the deletion or retention of the tth word. We set at ∈ {0, 1}, where 0 indicates deletion and 1 indicates reservation. During training stage, the strategic network samples the behavior of each word in probability according to the following equation:(10)πatst;Θ=SigmoidWRL⋅st+bRL,where πat|st;Θ denotes the probability of selecting behavior at in state st. The probability calculation of BRCNN model policy network adopts the sigmoid activation function, which can map each input value to the interval from 0 to 1 as the probability of retaining the current word. Then, we subtract this value from 1 to be the probability of word deletion. Θ=WRL,bRL represents the parameters of the BRCNN model policy network. When the policy network is trained and tested, the behavior at is selected according to (11), where t = 1, 2, …, N.(11)at=argmaxaπatst;Θ.The reward value of the strategic network is denoted asRN. In the BRCNN model, the behavior is sampled by the strategic network, the forward and backward word order information is extracted by the bidirectional cyclic neural network, and the word-level text is represented and classified by the CNN. The classification result label can be obtained to calculate the reward value RN of the strategic network. P(y|X) represents the probability of dividing text X into category y, where y represents category information and X is classification text.yreal denotes the real label of sample X, and the probability of correct classification is P(yreal|X). To delete as many independent words as possible and ensure the accuracy of classification, the number of words deleted is considered in the reward value function RN. The number of words deleted by the strategic network is L, and the number of words deleted by X is N, and thus, the definition of reward value function RN is shown as follows:(12)RN=logPyreal|X+λLN.The reward value functionRN of the policy network combines the probability of correct classification with the word deletion ratio “L/N” for obtaining a higher probability of correct classification when deleting as many words irrelevant to the classification task as possible. λ is a super parameter that balances the two factors of correct classification probability P(yreal|X) and word deletion ratio “L/N.”


## 3.1. Vector Representation of Temporal Relations

The Allen interval algebra defines 13 basic interval relations, and their meanings are shown in Figure1, where x and y represent continuous temporal intervals, x, y ∈ [0, T], and T represents sufficiently large time values. Among the 13 relationships, 6 pairs of relationships are mutually inverse, and the other equal relationships are mutually inverse with itself. Each basic interval relation describes the definite relationship between two temporal intervals. To describe the uncertain temporal relationship, it needs to be represented by a group of basic interval relations, which are called general interval relations.Figure 1
13 relations of basic interval.The temporal relationship represented by matrix has some disadvantages, such as less intuition and high computational complexity. To express the temporal relationship and calculation more simply, we consider using the left and right endpoints of the interval to characterize the relationship between the endpoints of the interval. Let [0,T] be a time interval taken from the time axis, and T is large enough, and the interval to be investigated is always its subinterval. I = [I−, I+] (0 ≤ I− ≤ I+ > T) is one of its subintervals, i.e., I⊆ [0, T]. IL = [0, I−] is the left outer point interval of I, I− is the left endpoint of I, Ii = (I−, I+) is the inner interval of I, IR = [I−, T] is the right outer point interval of I, and I+ is the right endpoint of I. (IL, I−, Ii, I+, IR) elements are called interval I. Therefore, any time interval I can be written as follows:(1)IL∪I−∪Ii∪I+∪IR=0,T.We set upα and β as two subintervals of interval [0, T] and define ΔαβLL, ΔαβRL, ΔαβLR, and ΔαβRR as follows:(2)ΔαβLL=α−−β−,ΔαβRL=α+−β−,ΔαβLR=α−−β+,ΔαβRR=α+−β+,where α−, α+, β–, and β+ are the left and right endpoints of α and β, respectively.Definition 1.
Setting upα and β as two temporal intervals and vector v = (ΔαβLL, ΔαβRL, ΔαβLR, ΔαβRR) is called basic relationship vector, in which each element is −1, 0, or 1. If ΔαβLL > 0, ΔαβLL = 1; if ΔαβLL = 0, ΔαβLL = 0; and if ΔαβLL < 0, ΔαβLL = −1. ΔαβRL, ΔαβLR, and ΔαβRR have similar value rules.
The purpose of Definition1 is to express the basic relationship between intervals through the relationship between the left and right endpoints of intervals. Each vector obtained according to Definition 1 corresponds to a basic relationship, and thus, the basic interval relation of the Allen interval algebra can be expressed by vector, which is called the basic relationship vector. The 13 basic relationship vectors of the Allen interval algebra are listed as follows:(1)
α before β can be denoted as Vb = [−1, −1, −1, −1](2)
α after β can be denoted as Vb = [1, 1, 1, 1](3)
α meets β can be denoted as Vb  = [−1, 0, −1, −1](4)
α met by β can be denoted as Vb = [1, 1, 0, 1](5)
α overlaps β can be denoted as Vb = [−1, 1, −1, −1](6)
α overlapped by β can be denoted as Vb = [1, 1, −1, 1](7)
α finishes β can be denoted as Vb = [1, 1, −1, 0](8)
α finished by β can be denoted as Vb = [−1, 1, −1, 0](9)
α during β can be denoted as Vb = [1, 1, −1, −1](10)
α includes β can be denoted as Vb = [−1, 1, −1, 1](11)
α starts β can be denoted as Vb = [0, 1, −1, −1](12)
α started by β can be denoted as Vb = [0, 1, −1, 1](13)
α equals β can be denoted as Vb = [0, 1, −1, 0]
In the vector representation of relationship, the rules of compound operation and transfer operation of relationship are defined as follows.Definition 2.
Compound operation of relationship (V). If two intervalsx and y satisfy the relationship set P = (p1, p2, …, pk), 1 ≤ k ≤ 13, and Vj (j = 1, 2, …, k) is the atomic relationship vector corresponding to each atomic relation, then the temporal relationship vector Vp between x and y can be linearly expressed by Vj (j = 1, 2, …, k) as follows:(3)Vp=∑j=1kVj=1,1,…1×kV1V2⋮Vk.
It can be known from Definition1 that we should obtain the relationship between the left and right endpoints of α and γ.
Thus, the relationship between the left endpoints ofα and γ can be obtained from the left endpoint of β. The relationship between the right endpoint of α and the left endpoint of γ can be obtained from the right endpoint of β. The relationship between the left endpoint of α and the right endpoint of γ can be obtained from the left endpoint of β. The relationship between the left endpoint of α and the right endpoint of γ can be obtained from the right endpoint of β. From point algebra, the relationship between points {<, >, =} has the following characteristics:(1)
Transitivity, if the pointa < (>, =) b and b < (>, =) c, then a < (>, =) c(2)
Retention, if the pointa < (>) b and b = c, then a < (>) c. If a = b and b < (>) c, then a < (>) c(3)
Uncertainty, if the pointa < (>) b and b > (<) c, then the relationship between a and b is uncertain, which can be denoted as a (<, >, =) b
From the above discussion, the transfer operation of the defined relationship is as follows.Definition 3.
Relation transfer operation (∙). Setting time intervalα, β, and r1, r2 ∈ U, and if α{ r1}β, β{ r2}γ, then Vr1 and Vr2 are their corresponding relationship vectors, and the relationship vector of α between γ can be obtained by Vr1 and Vr2 with operation: Vr1r2 = Vr1∙Vr2, where Vri|i = 1, 2, 3, and 4 are the components of the relationship vector, and the value rules are as follows:(4)Vr1r23=1,if both ofVr13andVr24are0,or one of them is0and the other one is1,−1,if both ofVr13andVr24are1,or one ofthem is0and the other one is−1,0,if both ofVr13andVr24are0,−1,0,1,if one ofVr13andVr24is1andthe other one is−1,(5)Vr1r22=1,if both ofVr14andVr22are0,or one ofthem is0and the other one is1,−1,if both ofVr14andVr22are−1,or one ofthem is0and the other one is−1,0,if both ofVr14andVr22are0,−1,0,1,if one ofVr14andVr22is1andthe other one is−1,(6)Vr1r21=1,if both ofVr11andVr21are0,or one ofthem is0and the other one is1,−1,if both ofVr11andVr21are−1,or one ofthem is0and the other one is−1,0,if both ofVr11andVr21are0,−1,0,1,if one ofVr11andVr21is1andthe other one is−1,(7)Vr1r24=1,if both ofVr14andVr24are0,or one ofthem is0and the other one is1,−1,if both ofVr14andVr24are−1,or one ofthem is0and the other one is−1,0,if both ofVr14andVr24are0,−1,0,1,if one ofVr14andVr24is1andthe other one is−1,
The relationship betweenr1 and r2 in transfer operation can be generalized to relational subsets I1 and I2.Definition 4.
Transitive operation of relation subset (⊙). Setting time intervalα, β, and γ, and relational subsets I1 and I2 ⊆ U, if α{I1}β, β{I2}γ, Vr1, and Vr2 are their relationship vectors, then the relationship vector α between γ can be calculated by Vr1 and Vr2 with the operation ⊙ as follows:(8)VI1I2=VI1T⊙VI2=Vr1Vr2⋮VrmVR1VR2…VRN,where r1, …, rm ∈ I1, R1, …, Rn ∈ I2, and Vr1 and VR1 are their corresponding relationship vectors.


## 3.2. Word-Level Text Representation with Tense Collocation Based on DRL

Considering the slow training speed of strategic network in DRL, it is necessary to pre-train the original DL model without strategic network to accelerate and stabilize strategic network training and verify its effect on word extraction related to classification tasks in text. Therefore, before designing the overall word-level text representation and classification model based on DRL, we determine a DL model with excellent text representation and classification performance as the basic model, which is selected based on bidirectional recurrent neural network and convolution neural network (BRCNN). This model consists of two parts, including bidirectional cyclic neural network and CNN, in which the bidirectional cyclic neural network is used to extract the forward and reverse word order information of the text, and CNN is used to obtain the feature representation of the text.


### 3.2.1. Input of the Model

We focus on the text representation and classification of sentences in this model, and the network input is the word vector matrix (x1, x2, …, xN)T ∈ RN×D corresponding to the ith word in the sentence, i.e., xi ∈ RD, D represents the dimension of the word vector, and N represents the number of words in the sentence.


### 3.2.2. Bidirectional Cyclic Neural Network

After the input vector (x1, x2, …, xN)T passes through the bidirectional cyclic neural network, the forward word order information and backward word order information can be obtained, which are (h⟶1, h⟶2, …, h⟶N) and (h↼1, h↼2, …, h↼N), respectively. Attention mechanism contains two groups of weight factors, i.e., forward weighting factors (α⟶1·h⟶1, α⟶2·h⟶2, …, α⟶N·h⟶N) and backward weighting factor (α⟶1·h⟶1, α↼t·h↼2, …, α⟶N·h⟶N), respectively. The word order information vectors output by the cyclic neural network at each forward and backward time are weighted and cascaded to obtain O⟶ and O↼. We cascade O⟶ and O↼ to obtain the context vector ORNN, which is shown as follows:(9)ORNN=O⟶,O↼,where ORNN ∈ RN×2K, in which K is the number of hidden layer neurons.


### 3.2.3. CNN Model

The input of CNN is the outputORNN of bidirectional cyclic neural network. In BRCNN, the size of convolution kernel f is S × 2K, where S is the window size of convolution kernel corresponding to the number of hidden states ORNN in each convolution. 2K is the convolution kernel width, which is consistent with the vector dimension in ORNN. The output ORNN of bidirectional cyclic neural network can obtain the output vector after convolution operation of convolution kernel. The BRCNN model designed in this work uses three different convolution kernels f1, f2, and f3, and their sizes are S1 × 2K, S2 × 2K, and S3 × 2K, respectively. After the output of the bidirectional cyclic neural network, i.e., ORNN, passes through three convolution kernels f1, f2, and f3, O1 ∈ RN–S1+1, O2 ∈ RN–S2 + 1, and O3 ∈ RN–S3 + 1 can be obtained as shown in Figure 2.Figure 2
Convolution operation of the convolution layer.We assume that the number of convolution kernels in each type isM, and O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) can be obtained after the convolution operation. The pooling operation is often carried out to reduce the dimension of the features extracted by the convolution kernel and prevent the number of features from being too large and the classification model from being too fitted. In this work, maximum pooling is used to process the output of CNN, i.e., taking the maximum value of the second dimension in O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) to obtain the vector, which is the result of pooling. As shown in Figure 2, for the convolution kernel f1, we take the maximum value in the convolution result O1 to obtain the pooled result of a convolution kernel. In the same way, we calculate the results of f1 convolution kernels to obtain the m-dimensional vector O1′ ∈ RM, O2′ ∈ RM, and O3′ ∈ RM, which will be used as input for the next phase of the model.


### 3.2.4. DRL Model

BRCNN model uses the strategic network of DRL to independently extract and classify temporal related words and delete irrelevant words. By predicting the behavior of each word in the sentence, the strategic network determines the deletion or retention behavior of words, i.e., sampling the behavior of each word to achieve the purpose of purifying text information.In this work, we aim to classify the tenses of a single sentence sampleX and produce collocation results. The word vector matrix of X can be expressed as (x1, x2, …, xN)T, where xt corresponds to the vector representation of the tth word. The status, behavior, and reward of policy network are introduced as follows. We note that the state of the policy network at time t is st (t = 1, 2, …, N). In the BRCNN model, the state st (t = 1, 2, …, N) is composed of the word vector input xt of the tth word and the hidden state information obtained at time t of the bidirectional cyclic neural network. at (t = 1, 2, …, N) denotes the behavior of the policy network at time t. In the BRCNN model, the goal is to extract words related to tenses, and thus, the behavior at is set as the deletion or retention of the tth word. We set at ∈ {0, 1}, where 0 indicates deletion and 1 indicates reservation. During training stage, the strategic network samples the behavior of each word in probability according to the following equation:(10)πatst;Θ=SigmoidWRL⋅st+bRL,where πat|st;Θ denotes the probability of selecting behavior at in state st. The probability calculation of BRCNN model policy network adopts the sigmoid activation function, which can map each input value to the interval from 0 to 1 as the probability of retaining the current word. Then, we subtract this value from 1 to be the probability of word deletion. Θ=WRL,bRL represents the parameters of the BRCNN model policy network. When the policy network is trained and tested, the behavior at is selected according to (11), where t = 1, 2, …, N.(11)at=argmaxaπatst;Θ.The reward value of the strategic network is denoted asRN. In the BRCNN model, the behavior is sampled by the strategic network, the forward and backward word order information is extracted by the bidirectional cyclic neural network, and the word-level text is represented and classified by the CNN. The classification result label can be obtained to calculate the reward value RN of the strategic network. P(y|X) represents the probability of dividing text X into category y, where y represents category information and X is classification text.yreal denotes the real label of sample X, and the probability of correct classification is P(yreal|X). To delete as many independent words as possible and ensure the accuracy of classification, the number of words deleted is considered in the reward value function RN. The number of words deleted by the strategic network is L, and the number of words deleted by X is N, and thus, the definition of reward value function RN is shown as follows:(12)RN=logPyreal|X+λLN.The reward value functionRN of the policy network combines the probability of correct classification with the word deletion ratio “L/N” for obtaining a higher probability of correct classification when deleting as many words irrelevant to the classification task as possible. λ is a super parameter that balances the two factors of correct classification probability P(yreal|X) and word deletion ratio “L/N.”


## 3.2.1. Input of the Model

We focus on the text representation and classification of sentences in this model, and the network input is the word vector matrix (x1, x2, …, xN)T ∈ RN×D corresponding to the ith word in the sentence, i.e., xi ∈ RD, D represents the dimension of the word vector, and N represents the number of words in the sentence.


## 3.2.2. Bidirectional Cyclic Neural Network

After the input vector (x1, x2, …, xN)T passes through the bidirectional cyclic neural network, the forward word order information and backward word order information can be obtained, which are (h⟶1, h⟶2, …, h⟶N) and (h↼1, h↼2, …, h↼N), respectively. Attention mechanism contains two groups of weight factors, i.e., forward weighting factors (α⟶1·h⟶1, α⟶2·h⟶2, …, α⟶N·h⟶N) and backward weighting factor (α⟶1·h⟶1, α↼t·h↼2, …, α⟶N·h⟶N), respectively. The word order information vectors output by the cyclic neural network at each forward and backward time are weighted and cascaded to obtain O⟶ and O↼. We cascade O⟶ and O↼ to obtain the context vector ORNN, which is shown as follows:(9)ORNN=O⟶,O↼,where ORNN ∈ RN×2K, in which K is the number of hidden layer neurons.


## 3.2.3. CNN Model

The input of CNN is the outputORNN of bidirectional cyclic neural network. In BRCNN, the size of convolution kernel f is S × 2K, where S is the window size of convolution kernel corresponding to the number of hidden states ORNN in each convolution. 2K is the convolution kernel width, which is consistent with the vector dimension in ORNN. The output ORNN of bidirectional cyclic neural network can obtain the output vector after convolution operation of convolution kernel. The BRCNN model designed in this work uses three different convolution kernels f1, f2, and f3, and their sizes are S1 × 2K, S2 × 2K, and S3 × 2K, respectively. After the output of the bidirectional cyclic neural network, i.e., ORNN, passes through three convolution kernels f1, f2, and f3, O1 ∈ RN–S1+1, O2 ∈ RN–S2 + 1, and O3 ∈ RN–S3 + 1 can be obtained as shown in Figure 2.Figure 2
Convolution operation of the convolution layer.We assume that the number of convolution kernels in each type isM, and O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) can be obtained after the convolution operation. The pooling operation is often carried out to reduce the dimension of the features extracted by the convolution kernel and prevent the number of features from being too large and the classification model from being too fitted. In this work, maximum pooling is used to process the output of CNN, i.e., taking the maximum value of the second dimension in O1 ∈ RM × (N–S1 + 1), O2 ∈ RM × (N–S2 + 1), and O3 ∈ RM × (N–S3 + 1) to obtain the vector, which is the result of pooling. As shown in Figure 2, for the convolution kernel f1, we take the maximum value in the convolution result O1 to obtain the pooled result of a convolution kernel. In the same way, we calculate the results of f1 convolution kernels to obtain the m-dimensional vector O1′ ∈ RM, O2′ ∈ RM, and O3′ ∈ RM, which will be used as input for the next phase of the model.


## 3.2.4. DRL Model

BRCNN model uses the strategic network of DRL to independently extract and classify temporal related words and delete irrelevant words. By predicting the behavior of each word in the sentence, the strategic network determines the deletion or retention behavior of words, i.e., sampling the behavior of each word to achieve the purpose of purifying text information.In this work, we aim to classify the tenses of a single sentence sampleX and produce collocation results. The word vector matrix of X can be expressed as (x1, x2, …, xN)T, where xt corresponds to the vector representation of the tth word. The status, behavior, and reward of policy network are introduced as follows. We note that the state of the policy network at time t is st (t = 1, 2, …, N). In the BRCNN model, the state st (t = 1, 2, …, N) is composed of the word vector input xt of the tth word and the hidden state information obtained at time t of the bidirectional cyclic neural network. at (t = 1, 2, …, N) denotes the behavior of the policy network at time t. In the BRCNN model, the goal is to extract words related to tenses, and thus, the behavior at is set as the deletion or retention of the tth word. We set at ∈ {0, 1}, where 0 indicates deletion and 1 indicates reservation. During training stage, the strategic network samples the behavior of each word in probability according to the following equation:(10)πatst;Θ=SigmoidWRL⋅st+bRL,where πat|st;Θ denotes the probability of selecting behavior at in state st. The probability calculation of BRCNN model policy network adopts the sigmoid activation function, which can map each input value to the interval from 0 to 1 as the probability of retaining the current word. Then, we subtract this value from 1 to be the probability of word deletion. Θ=WRL,bRL represents the parameters of the BRCNN model policy network. When the policy network is trained and tested, the behavior at is selected according to (11), where t = 1, 2, …, N.(11)at=argmaxaπatst;Θ.The reward value of the strategic network is denoted asRN. In the BRCNN model, the behavior is sampled by the strategic network, the forward and backward word order information is extracted by the bidirectional cyclic neural network, and the word-level text is represented and classified by the CNN. The classification result label can be obtained to calculate the reward value RN of the strategic network. P(y|X) represents the probability of dividing text X into category y, where y represents category information and X is classification text.yreal denotes the real label of sample X, and the probability of correct classification is P(yreal|X). To delete as many independent words as possible and ensure the accuracy of classification, the number of words deleted is considered in the reward value function RN. The number of words deleted by the strategic network is L, and the number of words deleted by X is N, and thus, the definition of reward value function RN is shown as follows:(12)RN=logPyreal|X+λLN.The reward value functionRN of the policy network combines the probability of correct classification with the word deletion ratio “L/N” for obtaining a higher probability of correct classification when deleting as many words irrelevant to the classification task as possible. λ is a super parameter that balances the two factors of correct classification probability P(yreal|X) and word deletion ratio “L/N.”


## 4. Simulation Experiments

This section compares and analyzes the experimental results of the word-level text representation and tense collocation model based on DRL. In this section, we introduce the content and source of the experimental dataset, and the setting of parameter in the experiment. All the experiments are conducted on 8 GB RAM, Intel Core i7 processor with 3.60 GHz, and Windows 10 operating system. The method is implemented using the MATLAB R2018b Win64.


### 4.1. Dataset Collection

There are five basic text classification datasets used in this experiment. The number of categories, the average length of text, the number of samples, the number of thesauri, and the number of samples in the test set are shown in Table1:(1)
MPQA text dataset: MPQA opinion corpus comes from various news articles and is marked by Wiebe et al., which contain opinions and a series of private states, such as emotion, speculation, and belief(2)
CR text dataset: CR is customer comments on 14 Amazon products, which are divided into positive and negative user emotions(3)
MR text dataset: MR, i.e., movie reviews, is about customers’ emotional evaluation of a series of films, including positive and negative emotions(4)
Subj text dataset: subjectivity (Subj) dataset is a subjective and objective emotion classification dataset(5)
TREC text dataset: TREC is a dataset for classifying problemsTable 1
A summary of the datasets.
DatasetMPQACRMRSubjTRECNumber of categories22226Average length2.918.519.522.410.7Number of samples10606377510662100005952Number of thesauri6246534018765213239592Number of test samples450280360680500The initialization word vector in this experiment comes from the trained word vector of Google, i.e., GoogleNews-vectors-negative300.bin, which is trained by the CBOW model on the dataset Google News. Google News contains about 100 billion English words, and the CBOW model finally obtains the vector representation of 3 million words or phrases, in which the dimension of each word vector is 300. For the words that do not appear in the 3-million-word list, the model generates 300 dimensional vectors randomly by means of uniform distribution:Network weight: the network parameters involved in the BRCNN model are initialized with a normal distribution with a mean value of 0 and a standard deviation of 0.1, and the bias terms are initialized to 0.1. The settings come from the optimal simulation based on different combinations.Hyperparameters: the Adam method is used to train the model in the experiment, and the learning rate is 0.001. L2 regularization parameters with the size of 0.0001 are used in the full-connection layer to prevent overfitting. The data input batch is 64. There are 128 hidden units in the cyclic neural network. The CNN model contains three types of convolution kernels with window sizes of 3, 4, and 5, and 100 convolution kernels are set for each type.λ of reward function RN in policy network is 0.1. Similarly, the setting method is to make different combinations of simulation and select the optimal simulation.


### 4.2. Results and Analysis

The results of comparative analysis of the words with the largest number of deletions or more typical words in the five datasets are shown in Table2. The number of occurrences represents the total number of occurrences of a word in the dataset, the number of retentions is the number of words retained by the BRCNN model, and the retentions percentage is the ratio of retained words to the number of occurrences.Table 2
BRCNN reserved word.
DatasetsWordsNumber of occurrencesNumber of retentionsRetention percentageMPQAAlways2228198288.96MPQABefore3419332497.22MPQAAfter6751662498.11MPQASince2345213491.00MPQANow120581085990.05MPQAUsually6843602988.56CRAlways795587002088.01CRBefore574855698899.14CRAfter487844119484.44CRSince2934275693.93CRNow3050269188.23MRAlways5331512696.15MRBefore2569236692.10MRAfter2302210391.36MRSince5037482995.87MRRecently8975869096.82MRComedy1843165789.90SubjBefore1665142385.47SubjAfter5641556498.63SubjSince4892468895.83SubjRecently9281902497.23SubjYet6574632296.117SubjEver9524936798.35TRECBefore9661955498.98TRECAfter6541631796.58TRECSince5236459487.74TRECRecently8563836497.67TRECYet2566249697.27TRECEver9325886495.05As can be seen from Table2, the BRCNN model can effectively retain tense words in characterization data and make accurate tense collocation based on these words. In the CR dataset, “always” has the highest number of occurrences, but the classification accuracy is lower than “before,” which shows that the temporal feature of “before” is stronger. In the MPQA dataset, the retention percentage of “normally” is relatively low, indicating that its temporal feature is weak. In MR and Subj datasets, the number of occurrences and retention percentage of “recently” are relatively high, which indicates that it is a typical word with strong temporal characteristics. In the TREC dataset, the number of occurrences of “yet” is low, but the retention percentage is high. This means that although the word is used less, the temporal feature is obvious.Table3 shows the accuracy of BRCNN in the English tense collocation proposed in this study. To objectively evaluate the results of temporal collocation, HSASRL [23], SRML [24], and Usr-mtl [25] are introduced as comparison models. It can be seen from the results in Table 3 that the accuracy of BRCNN is higher than HSASRL and Usr-mtl. Compared with the SRML model, the BRCNN model also obtains the highest accuracy in the MPQA, MR, and TREC databases. The reason for experimental results is that BRCNN optimizes the temporal description of sentences using the Allen interval algebra and establishes the corresponding relationship between sentences and tenses by combining the high-precision text abstract with the bidirectional cyclic neural network and the convolution neural network. In addition, to further optimize the temporal expression of the text, BRCNN uses the intelligent decision-making ability of RL to effectively retain the tense words and completes the tense collocation with these words.Table 3
Temporal collocation accuracy of comparison models.
DatasetHSASRL (%)SRML (%)Usr-mtl (%)BRCNN (%)CR88.9292.7183.2691.92MPQA84.5189.4480.8693.76MR85.1887.4989.7392.11Subj83.3794.8890.2687.57TREC85.5690.3388.7195.51Avg accuracy85.5190.9786.5692.17


## 4.1. Dataset Collection

There are five basic text classification datasets used in this experiment. The number of categories, the average length of text, the number of samples, the number of thesauri, and the number of samples in the test set are shown in Table1:(1)
MPQA text dataset: MPQA opinion corpus comes from various news articles and is marked by Wiebe et al., which contain opinions and a series of private states, such as emotion, speculation, and belief(2)
CR text dataset: CR is customer comments on 14 Amazon products, which are divided into positive and negative user emotions(3)
MR text dataset: MR, i.e., movie reviews, is about customers’ emotional evaluation of a series of films, including positive and negative emotions(4)
Subj text dataset: subjectivity (Subj) dataset is a subjective and objective emotion classification dataset(5)
TREC text dataset: TREC is a dataset for classifying problemsTable 1
A summary of the datasets.
DatasetMPQACRMRSubjTRECNumber of categories22226Average length2.918.519.522.410.7Number of samples10606377510662100005952Number of thesauri6246534018765213239592Number of test samples450280360680500The initialization word vector in this experiment comes from the trained word vector of Google, i.e., GoogleNews-vectors-negative300.bin, which is trained by the CBOW model on the dataset Google News. Google News contains about 100 billion English words, and the CBOW model finally obtains the vector representation of 3 million words or phrases, in which the dimension of each word vector is 300. For the words that do not appear in the 3-million-word list, the model generates 300 dimensional vectors randomly by means of uniform distribution:Network weight: the network parameters involved in the BRCNN model are initialized with a normal distribution with a mean value of 0 and a standard deviation of 0.1, and the bias terms are initialized to 0.1. The settings come from the optimal simulation based on different combinations.Hyperparameters: the Adam method is used to train the model in the experiment, and the learning rate is 0.001. L2 regularization parameters with the size of 0.0001 are used in the full-connection layer to prevent overfitting. The data input batch is 64. There are 128 hidden units in the cyclic neural network. The CNN model contains three types of convolution kernels with window sizes of 3, 4, and 5, and 100 convolution kernels are set for each type.λ of reward function RN in policy network is 0.1. Similarly, the setting method is to make different combinations of simulation and select the optimal simulation.


## 4.2. Results and Analysis

The results of comparative analysis of the words with the largest number of deletions or more typical words in the five datasets are shown in Table2. The number of occurrences represents the total number of occurrences of a word in the dataset, the number of retentions is the number of words retained by the BRCNN model, and the retentions percentage is the ratio of retained words to the number of occurrences.Table 2
BRCNN reserved word.
DatasetsWordsNumber of occurrencesNumber of retentionsRetention percentageMPQAAlways2228198288.96MPQABefore3419332497.22MPQAAfter6751662498.11MPQASince2345213491.00MPQANow120581085990.05MPQAUsually6843602988.56CRAlways795587002088.01CRBefore574855698899.14CRAfter487844119484.44CRSince2934275693.93CRNow3050269188.23MRAlways5331512696.15MRBefore2569236692.10MRAfter2302210391.36MRSince5037482995.87MRRecently8975869096.82MRComedy1843165789.90SubjBefore1665142385.47SubjAfter5641556498.63SubjSince4892468895.83SubjRecently9281902497.23SubjYet6574632296.117SubjEver9524936798.35TRECBefore9661955498.98TRECAfter6541631796.58TRECSince5236459487.74TRECRecently8563836497.67TRECYet2566249697.27TRECEver9325886495.05As can be seen from Table2, the BRCNN model can effectively retain tense words in characterization data and make accurate tense collocation based on these words. In the CR dataset, “always” has the highest number of occurrences, but the classification accuracy is lower than “before,” which shows that the temporal feature of “before” is stronger. In the MPQA dataset, the retention percentage of “normally” is relatively low, indicating that its temporal feature is weak. In MR and Subj datasets, the number of occurrences and retention percentage of “recently” are relatively high, which indicates that it is a typical word with strong temporal characteristics. In the TREC dataset, the number of occurrences of “yet” is low, but the retention percentage is high. This means that although the word is used less, the temporal feature is obvious.Table3 shows the accuracy of BRCNN in the English tense collocation proposed in this study. To objectively evaluate the results of temporal collocation, HSASRL [23], SRML [24], and Usr-mtl [25] are introduced as comparison models. It can be seen from the results in Table 3 that the accuracy of BRCNN is higher than HSASRL and Usr-mtl. Compared with the SRML model, the BRCNN model also obtains the highest accuracy in the MPQA, MR, and TREC databases. The reason for experimental results is that BRCNN optimizes the temporal description of sentences using the Allen interval algebra and establishes the corresponding relationship between sentences and tenses by combining the high-precision text abstract with the bidirectional cyclic neural network and the convolution neural network. In addition, to further optimize the temporal expression of the text, BRCNN uses the intelligent decision-making ability of RL to effectively retain the tense words and completes the tense collocation with these words.Table 3
Temporal collocation accuracy of comparison models.
DatasetHSASRL (%)SRML (%)Usr-mtl (%)BRCNN (%)CR88.9292.7183.2691.92MPQA84.5189.4480.8693.76MR85.1887.4989.7392.11Subj83.3794.8890.2687.57TREC85.5690.3388.7195.51Avg accuracy85.5190.9786.5692.17


## 5. Conclusions

In this work, we propose a vector representation, that is, relational vector based on the relational expression of interval in Allen’s interval algebra theory, and define the compound and transfer operations of relations on relational vectors, which provide support for the application of relational vectors in temporal reasoning. In addition, based on this temporal representation method, we propose a BRCNN model for temporal word classification and temporal collocation. The model realizes the text expression based on the bidirectional cyclic neural network and the convolutional neural network, and the classification of temporal words using the DRL model. It carries out the corresponding word collocation according to the calculated tenses. Through the experimental comparison of the five models on the datasets, i.e., MPQA, CR, MR, Subj, and TREC, it is proved that the BRCNN proposed in this work performs well in temporal collocation.As a novel English tense evaluation mechanism, it has some limitations. For example, this study lacks the large-scale experiments. In addition, this study has no sufficient theory analysis. In future, we will address the abovementioned limitations. In particular, we plan to make the practical experiments based on the constructed platform.



---

*Source: 7334686-2022-02-08.xml*