# Video Temporal Grounding with Multi-Model Collaborative Learning
## 3 Methodology
### General Model  
The base architecture of the MMCL framework encompasses a feature extractor, a cross-modal fusion module, and a visual predictor. To address various model requirements under the MMCL framework, the basic model unifies the structure of video temporal grounding architectures, enabling effective multi-model cooperation. The feature extractor obtains features from the input video \(V\) and the natural language query \(Q\), encoding both into high-dimensional representations used by the subsequent cross-modal fusion component to model their semantic interconnections. The video is partitioned into \(N\) equally spaced segments, each of duration \(\Delta t=\dfrac{L}{N}\). A pretrained visual model (e.g., 3D ConvNet [33] or CNN [34]) extracts the visual features for each segment, denoted as follows:  
\[F_{V}^{k}=\text{FeatureExtractor}_{V}(V),\quad F_{V}^{k}\in\mathbb{R}^{N\times d _{v}} \tag{1}\]  
Here, \(d_{v}\) is the dimensionality of the video features, and \(k\) signifies the current model. The query is processed by a pretrained linguistic model (e.g., GloVe [35] or BERT [36]) to obtain features, denoted by the following:  
\[F_{Q}^{k}=\text{FeatureExtractor}_{Q}(Q),\quad F_{Q}^{k}\in\mathbb{R}^{M\times d _{q}} \tag{2}\]  
In this case, \(d_{q}\) represents the dimensional size of the query feature vector. The aforementioned extracted features are subsequently passed to the Cross-Modal Fusion module. The Cross-Modal Fusion module combines the video features \(F_{V}^{k}\) and the query features \(F_{Q}^{k}\) to generate the multimodal fused representation \(X^{k}\), formally given by the following:  
\[X^{k}=\text{Fusion}_{k}(F_{V}^{k},F_{Q}^{k}),\quad X^{k}\in\mathbb{R}^{N \times d_{x}} \tag{3}\]  
Figure 2: Overview of the proposed Multi-Model Collaborative Learning (MMCL) architecture. The framework comprises: (1) two general models; (2) a CLIP-guided module; (3) a pseudo-label generator; (4) an iterative training algorithm.  
In this formulation, \(d_{x}\) signifies the dimensional scale of the fused feature space. The Cross-Modal Fusion module effectively captures the interactive information between video segments and the query semantics, providing high-quality multimodal representations to the visual prediction module. Under the MMCL framework, each model maintains its native multimodal processing approach while leveraging pseudo-labels to learn from the strengths of the other model. MMCL acts as a flexible collaborative layer that supports diverse architectural choices and fosters model synergy. By using pseudo-labels as soft supervision signals, MMCL facilitates knowledge transfer without altering the inherent multimodal interaction strategies of each model, allowing, for example, the 2D-TAN model to learn fine-grained boundary prediction from VSLNet in fast-changing scenarios, and vice versa. The visual predictor estimates the target time range \(P_{k}\) from the fused features \(X_{k}\), with the output format varying by model architecture. Span-based models yield 1D probability distributions for the start and end time points, expressed as follows:  
\[P_{s}^{k},P_{e}^{k}=\text{Predictor}_{k}(X^{k}),\quad P_{s}^{k},P_{e}^{k}\in \mathbb{R}^{N} \tag{4}\]  
Meanwhile, 2D-MAP-based models output a two-dimensional temporal probability matrix, represented by the following:  
\[P_{i,j}^{k}=\text{Predictor}_{k}(X^{k}),\quad P_{i,j}^{k}\in\mathbb{R}^{N\times N} \tag{5}\]  
One dimension \(i\) of the matrix corresponds to the start boundary, and the other dimension \(j\) to the end boundary, with \(i<j\). Each element \(P_{i,j}\) in the matrix represents the probability that the video segment begins at index \(i\) and ends at index \(j\).