# Visual Tracking Using Wang-Landau Reinforcement Sampler
## 3 Proposed Visual Tracking System
### Wang-Landau Reinforcement Sampler for Visual Tracking  
The proposed Wang-Landau sampler can be used to encourage the exploration of reinforcement learning by estimating the DOS [15], in which the DOS value approximates the frequency of visits to each state using Monte Carlo simulations. Based on the DOS, we determine whether a particular state is sufficiently explored. If a state has a small DOS value, the Wang-Landau sampler guides the reinforcement learning to explore that state. Otherwise, the sampler refrains from exploring that state.  
For Wang-Landau sampling, we define \(\mathbf{D}=\{d_{i}\}_{i=1}^{|\mathcal{S}|}\) and \(\mathbf{V}=\{v_{i}\}_{i=1}^{|\mathcal{S}|}\), in which \(d_{i}\) and \(v_{i}\) are the DOS score and the number of visits for the \(i\)-th state, respectively, and \(|\mathcal{S}|\) is the total number of states. Then, we update \(d_{i}\) if the visual tracker visits the \(i\)-th state:  
\[d_{i}\gets d_{i}\times w,\forall i, \tag{12}\]  
where \(w>1\) is a weighting parameter. \(v_{i}\) is updated as follows:  
\[v_{i}\gets v_{i}+1,\forall i, \tag{13}\]  
where \(d_{i}\) and \(v_{i}\) are initialized to 1 and 0, respectively. As the iteration proceeds, the Wang-Landau sampling adopts a coarse-to-fine strategy to attain more accurate DOS values. In the early iteration, we use a large value of \(w\) in (12), which increases the update speed. In the latter iteration, we use a smaller value of \(w\)m to fine-tune the updates. Accordingly, we decrease the value of \(w\), \(w\leftarrow\sqrt{w}\), if a current iteration satisfies the following condition, i.e., the semiflat status:  
\[\forall i,v_{i}\geq 0.8\times\frac{1}{|\mathcal{S}|}\sum_{\forall i}v_{i}, \tag{14}\]  
where we, at least partially, explore all states. After the modification of \(w\), the value \(v_{i}\) is reinitialized to 0.  
Owing to the need to balance the exploration of reinforcement learning with its exploitation, we present a new scheduling approach for reinforcement learning, as follows:  
\[a\sim\begin{cases}\operatorname*{arg\,max}_{a}Q(s,a),&\text{with probability }1-\epsilon^{new}\\ random(\mathcal{A})&\text{with probability }\epsilon^{new},\end{cases} \tag{15}\]  
with  
\[\epsilon^{new}=\max\left[1,\frac{d_{i}}{p_{i}(\mathbf{Y}_{t}|\mathbf{X}_{t})} \right], \tag{16}\]where \(p_{i}(\mathbf{Y}_{t}|\mathbf{X}_{t})\) is the likelihood at the \(i\)-th state, which is defined in (3). In (16), exploration \(d_{i}\) and exploitation \(p_{i}(\mathbf{Y}_{t}|\mathbf{X}_{t})\) compete with each other. For example, if \(d_{i}\) increases with respect to \(p_{i}(\mathbf{Y}_{t}|\mathbf{X}_{t})\), reinforcement learning would explore diverse states with a high probability. If not, it tends to exploit a current state with a high probability.  
Algorithm 1 illustrates the complete process of the proposed method.  
```
1:Input:\(\mathbf{\hat{X}}_{t}\), \(d_{i}=1\), \(v_{i}=0\), \(\forall i\), and \(w>1\).
2:Output:\(\mathbf{\hat{X}}_{t+1}\)
3:for\(j=1\) to \(N\)do
4:\(\bullet\) Reinforcement learning
5: Propose an action \(a\) according to (15).
6: Find the next state \(\mathbf{X}_{t+1}\) using the proposed action in (7).
7: Computer the posterior probability \(p(\mathbf{X}_{t+1}|\mathbf{Y}_{1:t})\) in (2).
8: Estimate the reward \(\mathcal{R}_{t+1}\) using \(p(\mathbf{X}_{t+1}|\mathbf{Y}_{1:t})\) in (6).
9: Update the \(Q\) value using \(\mathcal{R}_{t+1}\) in (9).
10: Determine the optimal policy function \(\pi\) using \(Q\) in (10).
11:\(\bullet\) Wang-Landau Monte Carlo sampling
12: Find the index \(i\) of the state \(\mathbf{X}_{t+1}\).
13: Update the DOS score \(d_{i}\) using (12).
14: Update the number of visits \(v_{i}\) using (13).
15:if\(v_{i}\) reaches the semiflat status in (14) then
16:\(w\leftarrow\sqrt{w}\).
17:\(v_{i}=0\).
18:endif
19:endfor
20: Find the best configuration \(\mathbf{\hat{X}}_{t+1}\) using (1).
```  
**Algorithm 1** Wang-Landau reinforcement sampler