## 3 Proposed Method
### RL with Multi-Policy Movement Strategy  
Based on the observation that human beings usually locate the rough location of the language query and then iteratively adjust the position (move to left/right) and length (zoom in/out) of the window, in this work we employ the RL framework to formulate this task as a sequential decision-making problem. Given the initial \(K\) candidate windows, the agent iteratively adjusts their positions and lengths by learning the movement policy. The essential elements of the RL framework are state, policy, and reward, which are defined as follows. **State:** We use the Bi-directional Gated Recurrent Unit (Bi-GRU) to learn the current state from the cross-modal features in the visual space from both global and local perspectives. Specifically, given the global feature \(V^{\prime\prime}_{global}\) and the local feature \(V^{\prime\prime}_{center}(t)\) at the \(t\)-step  
Figure 3: The dynamic Gaussian mask varies in the training process and allocates dynamic attention values to different positions of the video. Before E epoch, the Gaussian distribution gradually changes from red to blue, and its variance continues to increase. After E epoch, the opposite is true. which are obtained by Equation (3), we encode the concatenation of the two features by a fully connected layer and then feed them into the Bi-GRU. The output hidden vector \(s_{t}\) is used as the current state,  
\[s_{t}=\text{BiGRU}(FC_{5}([V^{\prime\prime}_{global},V^{\prime\prime}_{center}])) \tag{11}\]  
The state vector is updated as the window movement process steps. **Policy:** To subtly adjust the window position, we employ the multi-policy movement strategy whose action space is a tree structure. The root nodes include \(N_{root}=\{\text{moving to left, moving to right, zooming in, zooming out}\}\) and the leaf nodes are \(N_{leaf}=\{\text{moving by }\delta_{1}\text{, moving by }\delta_{2}\text{, zooming by }\eta_{1}\text{, zooming by }\eta_{2}\}\). Note that moving by \(\delta_{1}\) and moving by \(\delta_{2}\) are leaves of moving to left and moving to right in \(N_{root}\), and zooming by \(\eta_{1}\) and zooming by \(\eta_{2}\) are leaves of zooming in and zooming out in \(N_{root}\). We employ the A2C algorithm to model the sequential decision-making process, in which a Bi-GRU and fully connected layer are used to output the policy and state value of the root, \(\pi^{root}(a^{root}_{t}|s_{t})\) and \(v^{root}(s_{t})\), and the leaf, \(\pi^{leaf}(a^{leaf}_{t}|s_{t},a^{root}_{t})\) and \(v^{leaf}(s_{t})\). Then the actions of the root \(a^{root}_{t}\) and leaf \(a^{leaf}_{t}\) are sampled according to their policies, respectively,  
\[ \begin{split} a^{root}_{t}&\sim\pi^{root}\left(a^{ root}_{t}\mid s_{t}\right),\\ a^{leaf}_{t}&\sim\pi^{leaf}\left(a^{leaf}_{t}\mid s _{t},a^{root}_{t}\right)\end{split}  \tag{12}\]  
Specifically, the agent first samples actions from root nodes \(N_{root}\) and then determines the actions in leaf nodes \(N_{leaf}\). **Reward:** The reward aims to encourage the agent to move and adjust the candidate window step by step to a position that is more semantically aligned with the query. We use the alignment score \(O(V^{\prime\prime}_{center})\) of the center part in the current state (see Equation (5)) to compute the reward. For simplicity, we denote the alignment score at the \(t\)th step as \(O_{t}\). To encourage the agent to take better actions and locate the desired position efficiently, we reward the agent if the alignment score at the current step is higher than at the previous step. We give a higher reward if the alignment score is higher than a predefined threshold. On the contrary, we punish the agent if the alignment score at the current step is lower than at the previous step. The punishment is increased if the attenuation is larger than a predefined threshold. In summary, the reward of the \(t\)th step is defined as  
\[r_{t}= \begin{cases}O_{t},&O_{t}\geq thr\\ O_{t}-O_{t-1}+1,&O_{t}<thr\\ -1-O_{t},&O_{t-1}-O_{t}\geq 1-thr\\ -1,&O_{t-1}-O_{t}<1-thr\end{cases}