## 2 Materials and Methods
### Reinforcement Learning  
In our setting, an RL algorithm allows the crawler to determine a strategy (policy) so that it retrieves a fixed number of documents while maximizing the number of related ones. Recently, there have been approaches of focused crawling [17] and biomedical data mining [18] with RL. An agent (the crawler) fetches URLs in an iterative manner. Each iteration is considered a timestep. The agent acts within a crawling environment. The environment has its state per timestep. There is a number of actions that the agent can take at each timestep. These actions lead to rewards over time. Formally, at each timestep (t), the agent fetches a new URL as a result of an action selection (A\({}_{t}\)); then, it transitions from the current state (S\({}_{t}\)) to another state (S\({}_{t+1}\)) and observes a reward (R\({}_{t}\)). We consider the states to be related to the history of information (number of relevant and irrelevant URLs) fetched by the crawler. The actions are related to the URLs (keywords found on the anchor text) extracted from a state transition. The reward is related to the relevance of the current fetched publication with the defined topic. We set the reward equal to 1 for relevant publications and 0 otherwise. For the reward function, at first, we use the LSTM trained by AL in order to decide whether a document is related to ethnopharmacology. Then, we deterministically filter the related predicted ones using the taxonomy of keywords constructed. The goal of the agent is to find a policy (utilizing an RL algorithm) to maximize the discounted cumulative received reward \(G_{t}=R_{t}+\gamma R_{t+1}+\gamma^{2}R_{t+2}+\ldots+\gamma^{T-1}R_{T}\)[19], where T is the fixed number of total documents that the crawler should fetch and \(\gamma\) is the discount factor. In other words, the agent seeks to find a mapping between states and actions in order to get high long-term rewards. For our experiment, we arbitrarily set T = 700 and \(\gamma\) = 0.99. Our evaluation measure for focused crawling is the harvest rate HR(t) [4], which is the cumulative percentage of relevant fetched documents up to timestep t. Formally, it is defined as  
\[\textbf{HR(t)}=\frac{\textbf{Number of Relevant documents fetched since }t}{\textbf{Number of all documents fetched since }t}\]  
Since the RL agent is used to optimize the automatic URL extraction process, taking into account that the reward is \(1\) when the fetched webpage is relevant to our topic, the harvest rate is also an evaluation measure for RL. It actually measures the mean cumulative reward that the agent receives during the whole learning (crawling) process. Thus, optimizing the harvest rate is always equal to optimizing the mean cumulative reward of the RL agent. We employ a Deep Q-learning approach, utilizing the Deep Q-Network (DQN) agent [20], which is based on the TD error [19], R\({}_{t+1}\) + max\({}_{a}\)Q\({}^{\pi^{\prime}}\) (S\({}_{t+1}\), a; \(\theta^{-}\))-Q\({}^{\pi^{\prime}}\) (S\({}_{t}\), A\({}_{t}\); \(\theta\)), where Q\({}^{\pi^{\prime}}\) and Q\({}^{\pi^{\prime}}\) are the action-value functions under the policies \(\pi\) and \(\pi^{\prime}\), respectively. That is, Q\({}^{\pi^{\prime}}\) (S\({}_{t}\), A\({}_{t}\)) = E\({}_{\text{U(D)}}\) [R\({}_{t+1}\) + max\({}_{a}\)Q\({}^{\pi^{\prime}}\) (S\({}_{t+1}\), a; \(\theta^{-}\))] S\({}_{t}\), A\({}_{t}\)]. This reflects the expected cumulative (long-term) rewards, given current state S\({}_{t}\), current action A\({}_{t}\), and immediate reward R\({}_{t+1}\). The DQN agent consists of two neural networks with the same architecture--a Q-Network (\(\theta\)) and a target Q-Network (\(\theta^{-}\))--in order to approximate Q\({}^{\pi}\) and Q\({}^{\pi^{\prime}}\), respectively. Additionally, it has a replay buffer, D, called experience replay, which is important for the uniform sampling of mini-batches of uncorrelated past state transitions. For each Q-Network, we utilize a multilayer perceptron (MLP) with two hidden layers.