# VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
## 3 Our Method
### Proposal Generation  
**Computing query-frame similarity.** In Section 3.3, we have articulated the significance of image captioning within VTG-GPT and employed CLIP-T to model debiased queries \(Q\) and image captions \(C\) within the textual domain. Subsequently, taking into account CLIP-T, as a multi-modal model, does not outperform specialized language models in NLP (natural language processing) tasks, as outlined in previous research [36]. Therefore, we explore the use of a language-specific model. We opt for RoBERTa [34] (Sentence-BERT [35]) to extract normalized pooling features of \(Q\in\mathbb{R}^{N_{q}\times L_{c}}\) and \(C\in\mathbb{R}^{N_{v}\times L_{c}}\), denoted as \(f_{q}\in\mathbb{R}^{N_{q}\times d}\) and \(f_{c}\in\mathbb{R}^{N_{v}\times d}\), respectively, where \(d\) represents the dimensionality. We then compute the cosine scores between \(f_{q}\) and \(f_{c}\) as similarities \(S_{s}\in\mathbb{R}^{N_{q}\times N_{v}}\):  
\[S_{s}=\cos(f_{q},f_{c})=\frac{f_{q}\cdot f_{c}}{\|f_{q}\|\|f_{c}\|} \tag{1}\]  
As demonstrated in rows four to five of Table 1, the leverage of expert NLP models yielded significant improvements, which also validates the viewpoints presented in the report [36].  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Similarity Models**} & \multicolumn{2}{c}{**R1**} & \multicolumn{2}{c}{**mAP**} \\ \cline{2-6}  & **@0.5** & **@0.7** & **@0.5** & **@0.75** & **Avg.** \\ \hline CLIP [22] & 45.59 & 26.03 & 45.56 & 23.14 & 24.91 \\ InternVideo [24] & 49.13 & 32.49 & 48.65 & 25.82 & 26.94 \\ \hline CLIP-T [22] & 52.85 & 34.82 & 48.07 & 28.05 & 28.29 \\ RoBERTa [34] & **54.99** & 37.58 & 53.77 & 29.18 & 30.15 \\
**Sentence-BERT**[35] & 54.26 & **38.45** & **53.96** & **29.25** & **30.38** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Preliminary experiment with different similarity models on QVHighlights _val_ split, using proposal generator and proposal scorer but without NMS. CLIP-T is short for using CLIP [22] text encoder only. Please refer to Section 4.1 for a detailed explanation of the evaluation metrics.  
**Proposal generator.** After obtaining query-frame similarity scores \(S_{s}\), we move towards generating temporal proposals \(P\in\mathbb{R}^{N_{p}\times 2}\). A straightforward method would be to apply a fixed threshold, considering frames with similarity scores exceeding this threshold as potential start or end timestamps. However, each query-video pair exhibits a unique similarity distribution. To adaptively obtain proposals, we introduce a dynamic mechanism within our devised proposal generator. For clarity, we denote the similarity between the \(i\)-th debiased query \(Q^{i}\) and video \(V\) as \(S_{s}^{i}\in\mathbb{R}^{N_{p}}\), and the similarity between \(Q^{i}\) and the \(j\)-th frame in \(V\) as \(S_{s}^{i,j}\in\mathbb{R}^{1}\).  
To be specific, the generator begins by computing a histogram of \(S_{s}^{i}\) with \(N_{b}\) bins. It then selects the bins containing the top \(k\) highest similarities as the dynamic threshold \(\theta\):  
\[\theta=\text{top\_k}(S_{s}^{i},N_{b},k), \tag{2}\]  
where \(N_{b}\) and \(k\) are hyperparameters. For their specific values, please refer to the implementation details (Section 4.1) and ablations (Section 4.3). Next, we iteratively assess each frame; if \(S_{s}^{i,j}\) exceeds \(\theta\), its corresponding timestamp is considered the proposal's starting point. When more than \(\lambda\) consecutive frames are all lower than \(\theta\), the last frame with a similarity greater than \(\theta\) is marked as the end timestamp of this proposal. Here, \(\lambda\) denotes the continuity threshold. Finally, we produce proposals for all debiased queries in the same video using this process to form final temporal proposals \(P\in\mathbb{R}^{N_{p}\times 2}\) (representing potentially relevant video segments).