# A Focused Event Crawler with Temporal Intent
## 4 Temporal Intent-Based Focused Crawler
### URL Priority Assignment with Quantified Temporal Distribution  
As mentioned in Section 3, the priority of a URL published at time \(t\) is proportional to the number of relevant web pages, which can be represented by the Google search volume index. Therefore, a natural exponential function, which is an increasing function and is commonly used in time series analysis [43], is constructed and combined into the URLpriority assignment with the Google search volume index. The final equations of the new method are shown in Equations (13) and (14).  
\[P(URL)=\theta\times sim(V_{Dk},V_{Tk})+\gamma\times sim_{anchor} \tag{13}\]  
\[P_{T}(URLL)=\left\{\begin{array}{ll}P(URL)&P(URL)<\delta\\ exp(\frac{\lambda_{i}}{max\left\{\lambda_{1},\ldots,\lambda_{T}\right\}})\times P (URL)&P(URL)\geq\delta\end{array}\right. \tag{14}\]  
where \(P(URL)\) and \(P_{T}(URL)\) represent the URL priority computed by the traditional and proposed methods, respectively. Variable \(sim(V_{Dk},V_{Tk})\) is the same as Equation (12). Variable \(sim_{anchor}\) denotes the anchor relevance. Variables \(\theta\) and \(\gamma\) are weighted factors, and they satisfy \(\theta+\gamma=1\) (\(\theta\geq 0\)_and_\(\gamma\geq 0\)). In Equation (13), the anchor text is a direct description of the URL and is more important for the URL priority than the content of the parent page. Therefore, variables \(\theta\) and \(\gamma\) are set to 0.4 and 0.6, respectively. The _exp_() represents the natural exponential function. The variable \(\lambda_{i}\) is the same as Equation (8). The \(max\{\ \ldots\ \ \}\) denotes the maximum value of \(\lambda_{i}\). The variable \(\delta\) is a priority reduction factor and is set as 0.4 through many experiments. It is used to ensure that the priority of the URL corresponding to a relevant web page is raised and meanwhile the priority of the URL corresponding to an irrelevant web page remains unchanged.  
In Equation (13), the anchor relevance \(sim_{anchor}\) is also calculated by the cosine formula using the anchor vector \(V_{AK}\) and topical vector \(V_{TK}\), as shown in Equations (15) and (16).  
\[V_{AK}=\{(k_{1},w_{AK1}),(k_{2},w_{AK2}),\ldots,(k_{n},w_{Aki})\} \tag{15}\]  
\[sim(V_{Ak},V_{TK})=\frac{\sum_{i=1}^{s}w_{Tki}\times w_{Aki}}{\sqrt{\sum_{i=1} ^{s}w_{Tki}^{2}\times\sum_{i=1}^{s}w_{Aki}^{2}}} \tag{16}\]  
where \(w_{Aki}\) represents the weight of the keyword \(k_{i}\) and is calculated by the frequency of occurrence of keywords \(k_{i}\) in the anchor text. Other variables are the same as in Equation (7).  
In Equation (14), the value of the variable \(\lambda_{i}\) is related to the publication time of the corresponding URL. In this paper, two ways are used to extract the publication time. The first way is to use regular expressions to extract the time from the URL, which contains a time expression. For example, the time of 5 September 2008 can be extracted from the URL of '[[http://news.sohu.com/20080905/n259388056.shtml](http://news.sohu.com/20080905/n259388056.shtml)]([http://news.sohu.com/20080905/n259388056.shtml](http://news.sohu.com/20080905/n259388056.shtml))' (accessed on 23 February 2023). If the URL does not contain a time expression, then its time of publication will be assumed to be the same as the time of publication of its parent web page. In fact, its publication time is equal to or slightly earlier than the publication time of its parent web page. However, each Google search volume index corresponds to a larger time interval, which means that two publication times with a small gap are very likely to have the same Google search volume index. Therefore, the hypothesis of the second way is reasonable.