SlowGuess/ACL_2020_OCR huggingface dataset:
Hao Zhang $^{1,2}$ , Aixin Sun $^{1}$ , Wei Jing $^{2,3}$ , Joel Tianyi Zhou $^{2,*}$
$^{2}$ Institute of High Performance Computing, A*STAR, Singapore
$^{3}$ Institute for Infocomm Research, A*STAR, Singapore
hao007@e.ntu.edu.sg, axsun@ntu.edu.sg
21wjing@gmail.com, joey_zhou@ihpc.a-star.edu.sg
Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple and yet effective query-guided highlighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through extensive experiments on three benchmark datasets, we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising direction to solve NLVL.<sup>1</sup>
Given an untrimmed video, natural language video localization (NLVL) is to retrieve or localize a temporal moment that semantically corresponds to a given language query. An example is shown in Figure 1. As an important vision-language understanding task, NLVL involves both computer vision and natural language processing techniques (Krishna et al., 2017; Hendricks et al., 2017; Gao et al., 2018; Le et al., 2019; Yu et al., 2019). Clearly, cross-modal reasoning is essential for NLVL to correctly locate the target moment from a video.
Prior works primarily treat NLVL as a ranking task, which is solved by applying multimodal
Figure 1: An illustration of localizing a temporal moment in an untrimmed video by a given language query.
matching architecture to find the best matching video segment for a given language query (Gao et al., 2017; Hendricks et al., 2018; Liu et al., 2018a; Ge et al., 2019; Xu et al., 2019; Chen and Jiang, 2019; Zhang et al., 2019). Recently, some works explore to model cross-interactions between video and query, and to regress the temporal locations of target moment directly (Yuan et al., 2019b; Lu et al., 2019a). There are also studies to formulate NLVL as a sequence decision making problem and to solve it by reinforcement learning (Wang et al., 2019; He et al., 2019).
We address the NLVL task from a different perspective. The essence of NLVL is to search for a video moment as the answer to a given language query from an untrimmed video. By treating the video as a text passage, and the target moment as the answer span, NLVL shares significant similarities with span-based question answering (QA) task. The span-based QA framework (Seo et al., 2017; Wang et al., 2017; Huang et al., 2018) can be adopted for NLVL. Hence, we attempt to solve this task with a multimodal span-based QA approach.
There are two main differences between traditional text span-based QA and NLVL tasks. First, video is continuous and causal relations between video events are usually adjacent. Natural language, on the other hand, is inconsecutive and words in a sentence demonstrate syntactic structure. For instance, changes between adjacent video frames are usually very small, while adjacent word to-
$^{1}$ https://github.com/IsaacChanghau/VSLNet
6543
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6543-6554
kens may carry distinctive meanings. As the result, many events in a video are directly correlated and can even cause one another (Krishna et al., 2017). Causalities between word spans or sentences are usually indirect and can be far apart. Second, compared to word spans in text, human is insensitive to small shifting between video frames. In other words, small offsets between video frames do not affect the understanding of video content, but the differences of a few words or even one word could change the meaning of a sentence.
As a baseline, we first solve the NLVL task with a standard span-based QA framework named VSLBase. Specifically, visual features are analogous to that of text passage; the target moment is regarded as the answer span. VSLBase is trained to predict the start and end boundaries of the answer span. Note that VSLBase does not address the two aforementioned major differences between video and natural language. To this end, we propose an improved version named VSLNet (Video Span Localizing Network). VSLNet introduces a Query-Guided Highlighting (QGH) strategy in addition to VSLBase. Here, we regard the target moment and its adjacent contexts as foreground, while the rest as background, i.e., foreground covers a slightly longer span than the answer span. With QGH, VSLNet is guided to search for the target moment within a highlighted region. Through region highlighting, VSLNet well addresses the two differences. First, the longer region provides additional contexts for locating answer span due to the continuous nature of video content. Second, the highlighted region helps the network to focus on subtle differences between video frames, because the search space is reduced compared to the full video.
Experimental results on three benchmark datasets show that adopting span-based QA framework is suitable for NLVL. With a simple network architecture, VSLBase delivers comparable performance to strong baselines. In addition, VSLNet further boosts the performance and achieves the best among all evaluated methods.
Natural Language Video Localization. The task of retrieving video segments using language queries was introduced in (Hendricks et al., 2017; Gao et al., 2017). Solutions to NLVL need to model the cross-interactions between natural language and
video. The early works treat NLVL as a ranking task, and rely on multimodal matching architecture to find the best matching video moment for a language query (Gao et al., 2017; Hendricks et al., 2017, 2018; Wu and Han, 2018; Liu et al., 2018a,b; Xu et al., 2019; Zhang et al., 2019). Although intuitive, these models are sensitive to negative samples. Specifically, they need to dense sample candidate moments to achieve good performance, which leads to low efficiency and lack of flexibility.
Various approaches have been proposed to overcome those drawbacks. Yuan et al. (2019b) builds a proposal-free method using BiLSTM and directly regresses temporal locations of target moment. Lu et al. (2019a) proposes a dense bottom-up framework, which regresses the distances to start and end boundaries for each frame in target moment, and select the ones with highest confidence as final result. Yuan et al. (2019a) proposes a semantic conditioned dynamic modulation for better correlating sentence related video contents over time, and establishing a precise matching relationship between sentence and video. There are also works (Wang et al., 2019; He et al., 2019) that formulate NLVL as a sequence decision making problem, and adopt reinforcement learning based approaches, to progressively observe candidate moments conditioned on language query.
Most similar to our work are (Chen et al., 2019) and (Ghosh et al., 2019), as both studies are considered using the concept of question answering to address NLVL. However, both studies do not explain the similarity and differences between NLVL and traditional span-based QA, and they do not adopt the standard span-based QA framework. In our study, VSLBase adopts standard span-based QA framework; and VSLNet explicitly addresses the differences between NLVL and traditional span-based QA tasks.
Span-based Question Answering. Span-based QA has been widely studied in past years. Wang and Jiang (2017) combines match-LSTM (Wang and Jiang, 2016) and Pointer-Net (Vinyals et al., 2015) to estimate boundaries of the answer span. BiDAF (Seo et al., 2017) introduces bi-directional attention to obtain query-aware context representation. Xiong et al. (2017) proposes a coattention network to capture the interactions between context and query. R-Net (Wang et al., 2017) integrates mutual and self attentions into RNN encoder for feature refinement. QANet (Yu et al., 2018) lever
Figure 2: An overview of the proposed architecture for NLVL. The feature extractor is fixed during training. Figure (a) depicts the adoption of standard span-based QA framework, i.e., VSLBase. Figure (b) shows the structure of VSLNet.
ages a similar attention mechanism in a stacked convolutional encoder to improve performance. FusionNet (Huang et al., 2018) presents a full-aware multi-level attention to capture complete query information. By treating input video as text passage, the above frameworks are all applicable to NLVL in principle. However, these frameworks are not designed to consider the differences between video and text passage. Their modeling complexity arises from the interactions between query and text passage, both are text. In our solution, VSLBase adopts a simple and standard span-based QA framework, making it easier to model the differences between video and text through adding additional modules. Our VSLNet addresses the differences by introducing the QGH module.
Very recently, pre-trained transformer based language models (Devlin et al., 2019; Dai et al., 2019; Liu et al., 2019; Yang et al., 2019) have elevated the performance of span-based QA tasks by a large margin. Meanwhile, similar pre-trained models (Sun et al., 2019a,b; Yu and Jiang, 2019; Rahman et al., 2019; Nguyen and Okatani, 2019; Lu et al., 2019b; Tan and Bansal, 2019) are being proposed to learn joint distributions over multimodality sequence of visual and linguistic inputs. Exploring the pre-trained models for NLVL is part of our future work and is out of the scope of this study.
We now describe how to address NLVL task by adopting a span-based QA framework. We then present VSLBase (Sections 3.2 to 3.4) and VSLNet in detail. Their architectures are shown in Figure 2.
3.1 Span-based QA for NLVL
We denote the untrimmed video as  $V = \{f_t\}_{t=1}^T$  and the language query as  $Q = \{q_j\}_{j=1}^m$ , where  $T$  and  $m$  are the number of frames and words, re
spectively.  $\tau^s$  and  $\tau^e$  represent the start and end time of the temporal moment i.e., answer span. To address NLVL with span-based QA framework, its data is transformed into a set of SQuAD style triples (Context, Question, Answer) (Rajpurkar et al., 2016). For each video  $V$ , we extract its visual features  $\mathbf{V} = \{\mathbf{v}_i\}_{i=1}^n$  by a pre-trained 3D ConvNet (Carreira and Zisserman, 2017), where  $n$  is the number of extracted features. Here,  $\mathbf{V}$  can be regarded as the sequence of word embeddings for a text passage with  $n$  tokens. Similar to word embeddings, each feature  $\mathbf{v}_i$  here is a video feature vector.
Since span-based QA aims to predict start and end boundaries of an answer span, the start/end time of a video sequence needs to be mapped to the corresponding boundaries in the visual feature sequence  $\mathbf{V}$ . Suppose the video duration is  $\mathcal{T}$ , the start (end) span index is calculated by  $a^{s(e)} = \langle \tau^{s(e)} / \mathcal{T} \times n \rangle$ , where  $\langle \cdot \rangle$  denotes the rounding operator. During the inference, the predicted span boundary can be easily converted to the corresponding time via  $\tau^{s(e)} = a^{s(e)} / n \times \mathcal{T}$ .
After transforming moment annotations in NLVL dataset, we obtain a set of  $(\mathbf{V},Q,\mathbf{A})$  triples. Visual features  $\mathbf{V} = [\mathbf{v}_1,\mathbf{v}_2,\dots ,\mathbf{v}_n]$  act as the passage with  $n$  tokens;  $Q = [q_{1},q_{2},\ldots ,q_{m}]$  is the query with  $m$  tokens, and the answer  $\mathbf{A} = [\mathbf{v}_{a^s},\mathbf{v}_{a^s +1},\dots ,\mathbf{v}_{a^e}]$  corresponds to a piece in the passage. Then, the NLVL task becomes to find the correct start and end boundaries of the answer span,  $a^s$  and  $a^{e}$
3.2 Feature Encoder
We already have visual features  $\mathbf{V} = \{\mathbf{v}_i\}_{i=1}^n \in \mathbb{R}^{n \times d_v}$ . Word embeddings of a text query  $Q$ ,  $\mathbf{Q} = \{\mathbf{q}_j\}_{j=1}^m \in \mathbb{R}^{m \times d_q}$ , are easily obtainable e.g., GloVe. We project them into the same dimension  $d$ ,  $\mathbf{V}' \in \mathbb{R}^{n \times d}$  and  $\mathbf{Q}' \in \mathbb{R}^{m \times d}$ , by two linear
layers (see Figure 2(a)). Then we build the feature encoder with a simplified version of the embedding encoder layer in QANet (Yu et al., 2018).
Instead of applying a stack of multiple encoder blocks, we use only one encoder block. This encoder block consists of four convolution layers, followed by a multi-head attention layer (Vaswani et al., 2017). A feed-forward layer is used to produce the output. Layer normalization (Ba et al., 2016) and residual connection (He et al., 2016) are applied to each layer. The encoded visual features and word embeddings are as follows:
$$
\begin{array}{l} \widetilde {\mathbf {V}} = \text {F e a t u r e E n c o d e r} (\mathbf {V} ^ {\prime}) \\ \tilde {\sim} \end{array} \tag {1}
$$
$$
\widetilde {\mathbf {Q}} = \operatorname {F e a t u r e E n c o d e r} \left(\mathbf {Q} ^ {\prime}\right)
$$
The parameters of feature encoder are shared by visual features and word embeddings.
3.3 Context-Query Attention
After feature encoding, we use context-query attention (CQA) (Seo et al., 2017; Xiong et al., 2017; Yu et al., 2018) to capture the cross-modal interactions between visual and textural features. CQA first calculates the similarity scores,  $S \in \mathbb{R}^{n \times m}$ , between each visual feature and query feature. Then context-to-query  $(\mathcal{A})$  and query-to-context  $(\mathcal{B})$  attention weights are computed as:
$$
\mathcal {A} = \mathcal {S} _ {r} \cdot \widetilde {\mathbf {Q}} \in \mathbb {R} ^ {n \times d}, \mathcal {B} = \mathcal {S} _ {r} \cdot \mathcal {S} _ {c} ^ {T} \cdot \widetilde {\mathbf {V}} \in \mathbb {R} ^ {n \times d}
$$
where  $S_{r}$  and  $S_{c}$  are the row- and column-wise normalization of  $S$  by SoftMax, respectively. Finally, the output of context-query attention is written as:
$$
\mathbf {V} ^ {q} = \operatorname {F F N} \left(\left[ \widetilde {\mathbf {V}}; \mathcal {A}; \widetilde {\mathbf {V}} \odot \mathcal {A}; \widetilde {\mathbf {V}} \odot \mathcal {B} \right]\right) \tag {2}
$$
where  $\mathbf{V}^q\in \mathbb{R}^{n\times d}$  ; FFN is a single feed-forward layer;  $\odot$  denotes element-wise multiplication.
3.4 Conditioned Span Predictor
We construct a conditioned span predictor by using two unidirectional LSTMs and two feed-forward layers, inspired by Ghosh et al. (2019). The main difference between ours and Ghosh et al. (2019) is that we use unidirectional LSTM instead of bidirectional LSTM. We observe that unidirectional LSTM shows similar performance with fewer parameters and higher efficiency. The two LSTMs are stacked so that the LSTM of end boundary can be conditioned on that of start boundary. Then the hidden states of the two LSTMs are fed into the
Figure 3: An illustration of foreground and background of visual features.  $\alpha$  is the ratio of foreground extension.
corresponding feed-forward layers to compute the start and end scores:
$$
\mathbf {h} _ {t} ^ {s} = \operatorname {U n i L S T M} _ {\text {s t a r t}} \left(\mathbf {v} _ {t} ^ {q}, \mathbf {h} _ {t - 1} ^ {s}\right)
$$
$$
\begin{array}{l} \mathbf {h} _ {t} ^ {e} = \operatorname {U n i L S T M} _ {\text {e n d}} \left(\mathbf {h} _ {t} ^ {s}, \mathbf {h} _ {t - 1} ^ {e}\right) \\ \mathbf {G} ^ {s} = \mathbf {W} _ {t - 1} \left[ \mathbf {h} _ {t} ^ {s}, q _ {1} \right] \end{array} \tag {3}
$$
$$
\mathbf {S} _ {t} ^ {s} = \mathbf {W} _ {s} \times \left(\left[ \mathbf {h} _ {t} ^ {s}; \mathbf {v} _ {t} ^ {q} \right]\right) + \mathbf {b} _ {s}
$$
$$
\mathbf {S} _ {t} ^ {e} = \mathbf {W} _ {e} \times \left(\left[ \mathbf {h} _ {t} ^ {e}; \mathbf {v} _ {t} ^ {q} \right]\right) + \mathbf {b} _ {e}
$$
Here,  $\mathbf{S}_t^s$  and  $\mathbf{S}_t^e$  denote the scores of start and end boundaries at position  $t$ ;  $\mathbf{v}_t^q$  represents the  $t$ -th feature in  $\mathbf{V}^q$ . Then, the probability distributions of start and end boundaries are computed by  $P_s = \mathrm{SoftMax}(\mathbf{S}^s) \in \mathbb{R}^n$  and  $P_e = \mathrm{SoftMax}(\mathbf{S}^e) \in \mathbb{R}^n$ , and the training objective is defined as:
$$
\mathcal {L} _ {\text {s p a n}} = \frac {1}{2} \left[ f _ {\mathrm {C E}} \left(P _ {s}, Y _ {s}\right) + f _ {\mathrm {C E}} \left(P _ {e}, Y _ {e}\right) \right] \tag {4}
$$
where  $f_{\mathrm{CE}}$  represents cross-entropy loss function;  $Y_{s}$  and  $Y_{e}$  are the labels for the start  $(a^{s})$  and end  $(a^{e})$  boundaries, respectively. During inference, the predicted answer span  $(\hat{a}^s,\hat{a}^e)$  of a query is generated by maximizing the joint probability of start and end boundaries by:
$$
\begin{array}{l} \operatorname {s p a n} \left(\hat {a} ^ {s}, \hat {a} ^ {e}\right) = \arg \max  _ {\hat {a} ^ {s}, \hat {a} ^ {e}} P _ {s} \left(\hat {a} ^ {s}\right) P _ {e} \left(\hat {a} ^ {e}\right) \tag {5} \\ \mathrm {s . t .} 0 \leq \hat {a} ^ {s} \leq \hat {a} ^ {e} \leq n \\ \end{array}
$$
We have completed the VSLBase architecture (see Figure 2(a)). VSLNet is built on top of VSLBase with QGH, to be detailed next.
3.5 Query-Guided Highlighting
A Query-Guided Highlighting (QGH) strategy is introduced in VSLNet, to address the major differences between text span-based QA and NLVL tasks, as shown in Figure 2(b). With QGH strategy, we consider the target moment as the foreground, and the rest as background, illustrated in Figure 3. The target moment, which is aligned with the language query, starts from  $a^s$  and ends at  $a^e$  with length  $L = a^e - a^s$ . QGH extends the boundaries of the foreground to cover its antecedent and consequent
Figure 4: The structure of Query-Guided Highlighting.
video contents, where the extension ratio is controlled by a hyperparameter  $\alpha$ . As aforementioned in Introduction, the extended boundary could potentially cover additional contexts and also help the network to focus on subtle differences between video frames.
By assigning 1 to foreground and 0 to background, we obtain a sequence of 0-1, denoted by  $Y_{\mathrm{h}}$ . QGH is a binary classification module to predict the confidence a visual feature belongs to foreground or background. The structure of QGH is shown in Figure 4. We first encode word features  $\widetilde{\mathbf{Q}}$  into sentence representation (denoted by  $\mathbf{h}_Q$ ), with self-attention mechanism (Bahdanau et al., 2015). Then  $\mathbf{h}_Q$  is concatenated with each feature in  $\mathbf{V}^q$  as  $\bar{\mathbf{V}}^q = [\bar{\mathbf{v}}_1^q, \dots, \bar{\mathbf{v}}_n^q]$ , where  $\bar{\mathbf{v}}_i^q = [\mathbf{v}_i^q; \mathbf{h}_Q]$ . The highlighting score is computed as:
$$
\mathcal {S} _ {\mathrm {h}} = \sigma \left(\operatorname {C o n v 1 D} \left(\overline {{\mathbf {V}}} ^ {q}\right)\right)
$$
where  $\sigma$  denotes Sigmoid activation;  $S_{\mathrm{h}}\in \mathbb{R}^{n}$ . The highlighted features are calculated by:
$$
\widetilde {\mathbf {V}} ^ {q} = \mathcal {S} _ {\mathrm {h}} \cdot \bar {\mathbf {V}} ^ {q} \tag {6}
$$
Accordingly, feature  $\mathbf{V}^q$  in Equation 3 is replaced by  $\widetilde{\mathbf{V}}^q$  in VSLNet to compute  $\mathcal{L}_{\mathrm{span}}$ . The loss function of query-guided highlighting is formulated as:
$$
\mathcal {L} _ {\mathrm {Q G H}} = f _ {\mathrm {C E}} \left(S _ {\mathrm {h}}, Y _ {\mathrm {h}}\right) \tag {7}
$$
VSLNet is trained in an end-to-end manner by minimizing the following loss:
$$
\mathcal {L} = \mathcal {L} _ {\text {s p a n}} + \mathcal {L} _ {\mathrm {Q G H}}. \tag {8}
$$
We conduct experiments on three benchmark datasets: Charades-STA (Gao et al., 2017), ActivityNet Caption (Krishna et al., 2017), and TACoS (Regneri et al., 2013), summarized in Table 1.
Charades-STA is prepared by Gao et al. (2017) based on Charades dataset (Sigurdsson et al., 2016).
The videos are about daily indoor activities. There are 12,408 and 3,720 moment annotations for training and test, respectively.
ActivityNet Caption contains about 20k videos taken from ActivityNet (Heilbron et al., 2015). We follow the setup in Yuan et al. (2019b), leading to 37,421 moment annotations for training, and 17,505 annotations for test.
TACoS is selected from MPII Cooking Composite Activities dataset (Rohrbach et al., 2012). We follow the setting in Gao et al. (2017), where 10, 146, 4, 589 and 4, 083 annotations are used for training, validation and test, respectively.
Metrics. We adopt “ $\mathrm{R} @ n, \mathrm{IoU} = \mu$ ” and “mIoU” as the evaluation metrics, following (Gao et al., 2017; Liu et al., 2018a; Yuan et al., 2019b). The “ $\mathrm{R} @ n, \mathrm{IoU} = \mu$ ” denotes the percentage of language queries having at least one result whose Intersection over Union (IoU) with ground truth is larger than  $\mu$  in top- $n$  retrieved moments. “mIoU” is the average IoU over all testing samples. In our experiments, we use  $n = 1$  and  $\mu \in \{0.3, 0.5, 0.7\}$ .
Implementation. For language query  $Q$ , we use 300d GloVe (Pennington et al., 2014) vectors to initialize each lowercase word; the word embeddings are fixed during training. For untrimmed video  $V$ , we downsample frames and extract RGB visual features using the 3D ConvNet which was pre-trained on Kinetics dataset (Carreira and Zisserman, 2017). We set the dimension of all the hidden layers in the model as 128; the kernel size of convolution layer is 7; the head size of multi-head attention is 8. For all datasets, the model is trained for 100 epochs with batch size of 16 and early stopping strategy. Parameter optimization is performed by Adam (Kingma and Ba, 2015) with learning rate of 0.0001, linear decay of learning rate and gradient clipping of 1.0. Dropout (Srivastava et al., 2014) of 0.2 is applied to prevent overfitting.
4.3 Comparison with State-of-the-Arts
We compare VSLBase and VSLNet with the following state-of-the-arts: CTRL (Gao et al., 2017), ACRN (Liu et al., 2018a), TGN (Chen et al., 2018), ACL-K (Ge et al., 2019), QSPN (Xu et al., 2019), SAP (Chen and Jiang, 2019), MAN (Zhang et al., 2019), SM-RL (Wang et al., 2019), RWM-RL (He et al., 2019), L-Net (Chen et al., 2019), ExCL (Ghosh et al., 2019), ABLR (Yuan et al.,
6547
Dataset	Domain	# Videos (train/val/test)	# Annotations	Nvocab	\( \bar{L}_{video} \)	\( \bar{L}_{query} \)	\( \bar{L}_{moment} \)	\( \Delta_{moment} \)
Charades-STA	Indoors	5,338/ - /1,334	12,408/ - /3,720	1,303	30.59s	7.22	8.22s	3.59s
ActivityNet Cap	Open	10,009/ - /4,917	37,421/ - /17,505	12,460	117.61s	14.78	36.18s	40.18s
TACoS	Cooking	75/27/25	10,146/4,589/4,083	2,033	287.14s	10.05	5.45s	7.56s
Table 1: Statistics of NLVL datasets, where  $N_{\text{vocab}}$  is vocabulary size of lowercase words,  $\bar{L}_{\text{video}}$  denotes average length of videos in seconds,  $\bar{L}_{\text{query}}$  denotes average number of words in sentence query,  $\bar{L}_{\text{moment}}$  is average length of temporal moments in seconds, and  $\Delta_{\text{moment}}$  is the standard deviation of temporal moment length in seconds.
Model	IoU = 0.3	IoU = 0.5	IoU = 0.7	mIoU
C3D model without fine-tuning as visual feature extractor
CTRL	-	23.63	8.89	-
ACL-K	-	30.48	12.20	-
QSPN	54.70	35.60	15.80	-
SAP	-	27.42	13.36	-
SM-RL	-	24.36	11.17	-
RWM-RL	-	36.70	-	-
MAN	-	46.53	22.72	-
DEBUG	54.95	37.39	17.69	36.34
VSLBase	61.72	40.97	24.14	42.11
VSLNet	64.30	47.31	30.19	45.15
C3D model with fine-tuning on Charades dataset
ExCL	65.10	44.10	23.30	-
VSLBase	68.06	50.23	30.16	47.15
VSLNet	70.46	54.19	35.22	50.02
Table 2: Results  $(\%)$  of  $\mathrm{R}@\mathfrak{n},\mathrm{IoU} = \mu$  and "mIoU" compared with the state-of-the-art on Charades-STA.
Model	IoU = 0.3	IoU = 0.5	IoU = 0.7	mIoU
TGN	45.51	28.47	-	-
ABLR	55.67	36.79	-	36.99
RWM-RL	-	36.90	-	-
QSPN	45.30	27.70	13.60	-
ExCL*	63.00	43.60	24.10	-
DEBUG	55.91	39.72	-	39.51
VSLBase	58.18	39.52	23.21	40.56
VSLNet	63.16	43.22	26.16	43.19