SlowGuess/NAACL_2022_OCR huggingface dataset:
Hung Le†‡§, Nancy F. Chen§, Steven C.H. Ho†‡
$^{\S}$ Agency for Science, Technology and Research (A*STAR)
{hungle,shoi}@salesforce.com，nfychen@i2r.a-star.edu.sg
Designed for tracking user goals in dialogues, a dialogue state tracker is an essential component in a dialogue system. However, the research of dialogue state tracking has largely been limited to unimodality, in which slots and slot values are limited by knowledge domains (e.g. restaurant domain with slots of restaurant name and price range) and are defined by specific database schema. In this paper, we propose to extend the definition of dialogue state tracking to multimodality. Specifically, we introduce a novel dialogue state tracking task to track the information of visual objects that are mentioned in video-grounded dialogues. Each new dialogue utterance may introduce a new video segment, new visual objects, or new object attributes and a state tracker is required to update these information slots accordingly. We created a new synthetic benchmark and designed a novel baseline, Video-Dialogue Transformer Network (VDTN), for this task. VDTN combines both object-level features and segment-level features and learns contextual dependencies between videos and dialogues to generate multimodal dialogue states. We optimized VDTN for a state generation task as well as a self-supervised video understanding task which recovers video segment or object representations. Finally, we trained VDTN to use the decoded states in a response prediction task. Together with comprehensive ablation and qualitative analysis, we discovered interesting insights towards building more capable multimodal dialogue systems.
The main goal of dialogue research is to develop intelligent agents that can assist humans through conversations. For example, a dialogue agent can be tasked to help users to find a restaurant based on their preferences of price ranges and food choices. A crucial part of a dialogue system is Dialogue State Tracking (DST), which is
responsible for tracking and updating user goals in the form of dialogue states, including a set of (slot, value) pairs such as (price, "moderate") and (food, "japanese"). Numerous machine learning approaches have been proposed to tackle DST, including fixed-vocabulary models (Ramadan et al., 2018; Lee et al., 2019) and open-vocabulary models (Lei et al., 2018b; Wu et al., 2019; Le et al., 2020c), for either single-domain (Wen et al., 2017) or multi-domain dialogues (Eric et al., 2017; Budzianowski et al., 2018).
However, the research of DST has largely limited the scope of dialogue agents to unimodality. In this setting, the slots and slot values are defined by the knowledge domains (e.g. restaurant domain) and database schema (e.g. data tables for restaurant entities). The ultimate goal of dialogue research towards building artificial intelligent assistants necessitates DST going beyond unimodal systems. In this paper, we propose Multimodal Dialogue State Tracking (MM-DST) that extends the DST task in a multimodal world. Specifically, MM-DST extends the scope of dialogue states by defining slots and slot values for visual objects that are mentioned in visually-grounded dialogues. For research purposes, following (Alamri et al., 2019), we limited visually-grounded dialogues as ones with a grounding video input and the dialogues contain multiple turns of (question, answer) pairs about this video. Each new utterance in such dialogues may focus on a new video segment, new visual objects, or new object attributes, and the tracker is required to update the dialogue state accordingly at each turn. An example of MM-DST can be seen in Figure 1.
Toward MM-DST, we developed a synthetic benchmark based on the CATER universe (Girdhar and Ramanan, 2020). We also introduced Video-Dialogue Transformer Network (VDTN), a neural network architecture that combines both object-level features and segment-level features in video and learns contextual dependencies between videos
Human Language Technologies, pages 3394 - 3415
(a) Conventional Dialogue State Tracking
(b) Multimodal Dialogue State Tracking
Figure 1: Multimodal Dialogue State Tracking (MM-DST): We proposed to extend the traditional DST from unimodality to multimodality. Compared to traditional DST (a), MM-DST (b) define dialogue states, consisting of slots and slots values for visual objects that are mentioned in dialogues.
and dialogues. Specifically, we maintained the information granularity of visual objects, embedded by object classes and their bounding boxes and injected with segment-level visual context. VDTN enables interactions between each visual object representation and word-level representation in dialogues to decode dialogue states. To decode multimodal dialogue states, we adopted a decoding strategy inspired by the Markov decision process in traditional DST (Young et al., 2010). In this strategy, a model learns to decode the state at a dialogue turn based on the predicted/observed dialogue state available from the last dialogue turn.
Compared to the conventional DST, MM-DST involves the new modality from visual inputs. Our experiments show that simply combining visual and language representations in traditional DST models results in poor performance. Towards this challenge, we enhanced VDTN with self-supervised video understanding tasks which recovers object-based or segment-based representa
tions. Benchmarked against strong unimodal DST models, we observed significant performance gains from VDTN. We provided comprehensive ablation analysis to study the efficacy of VDTN models. Interestingly, we also showed that using decoded states brought performance gains in a dialogue response prediction task, supporting our motivation for introducing multimodality into DST research.
2 Multimodal Dialogue State Tracking
Traditional DST. As defined by (Mrkšić et al., 2017), the traditional DST includes an input of dialogue  $\mathcal{D}$  and a set of slots  $S$  to be tracked from turn to turn. At each dialogue turn  $t$ , we denote the dialogue context as  $\mathcal{D}_t$ , containing all utterances up to the current turn. The objective of DST is for each turn  $t$ , predict a value  $v_i^t$  of each slot  $s_i$  from a predefined set  $S$ , conditioned by the dialogue context  $\mathcal{D}_t$ . We denote the dialogue state at turn  $t$  as  $\mathcal{B}_t = \{(s_i, v_i^t)\}_{i=1}^{i=|\mathcal{S}|}$ . Note that a majority
of traditional DST models assume slots are conditionally independent, given the dialogue context (Zhong et al., 2018; Budzianowski et al., 2018; Wu et al., 2019; Lee et al., 2019; Gao et al., 2019). The learning objective is defined as:
$$
\begin{array}{l} \hat {\mathcal {B}} _ {t} = \underset {\mathcal {B} _ {t}} {\arg \max} P (\mathcal {B} _ {t} | \mathcal {D} _ {t}, \theta) \\ = \arg \max  _ {\mathcal {B} _ {t}} \prod_ {i} ^ {| S |} P \left(v _ {i} ^ {t} \mid s _ {i}, \mathcal {D} _ {t}, \theta\right) \tag {1} \\ \end{array}
$$
Motivation to Multimodality. Yet, the above definition of DST are still limited to unimodality and our ultimate goal of building intelligent dialogue agents, ideally with similar level of intelligence as humans, inspires us to explore multimodality. In neuroscience literature, several studies have analyzed how humans can perceive the world in visual context. (Bar, 2004; Xu and Chun, 2009) found that humans can recognize multiple visual objects and how their contexts, often embedded with other related objects, facilitate this capacity.
Our work is more related to the recent study (Fischer et al., 2020) which focuses on human capacity to create temporal stability across multiple objects. The multimodal DST task is designed to develop multimodal dialogue systems that are capable of maintaining discriminative representations of visual objects over a period of time, segmented by dialogue turns. While computer science literature has focused on related human capacities in intelligent systems, they are mostly limited to vision-only tasks e.g. (He et al., 2016; Ren et al., 2015) or QA tasks e.g. (Antol et al., 2015; Jang et al., 2017) but not in a dialogue task.
Most related work in the dialogue domain is (Pang and Wang, 2020) and almost concurrent to our work is (Kottur et al., 2021). However, (Kottur et al., 2021) is limited to a single object per dialogue, and (Pang and Wang, 2020) extends to multiple objects but does not require to maintain an information state with component slots for each object. Our work aims to complement these directions and address their limitations with a novel definition of multimodal dialogue state.
Multimodal DST (MM-DST). To this end, we proposed to extend conventional dialogue states. First, we use visual object identities themselves as a component of the dialogue state to enable the perception of multiple objects. A dialogue state might have one or more objects and a dialogue system needs to update the object set as the dialogue
carries on. Secondly, for each object, we define slots that represent the information state of objects in dialogues (as denoted by (Fischer et al., 2020) as "content" features of objects memorized by humans). The value of each slot is subject-specific and updated based on the dialogue context of the corresponding object. This definition of DST is closely based on the above well-studied human capacities while complementing the conventional dialogue research (Young et al., 2010; Mrksic et al., 2017), and more lately multimodal dialogue research (Pang and Wang, 2020; Kottur et al., 2021).
We denote a grounding visual input in the form of a video  $\mathcal{V}$  with one or more visual objects  $o_j$ . We assume these objects are semantically different enough (by appearance, by characters, etc.) such that each object can be uniquely identified (e.g. by an object detection module  $\omega$ ). The objective of MM-DST is for each dialogue turn  $t$ , predict a value  $v_i^t$  of each slot  $s_i \in S$  for each object  $o_j \in \mathcal{O}$ . We denote the dialogue state at turn  $t$  as  $\mathcal{B}_t = |\{(o_j, s_i, v_{i,j}^t)\}|_{i=1,j=1}^{i=|S|, j=|\mathcal{O}|}$ . Assuming all slots are conditionally independent given dialogue and video context, the learning objective is extended from Eq. (1):
$$
\begin{array}{l} \hat {\mathcal {B}} _ {t} = \underset {\mathcal {B} _ {t}} {\arg \max } P (\mathcal {B} _ {t} | \mathcal {D} _ {t}, \mathcal {V}, \theta) \\ = \arg \max  _ {\mathcal {B} _ {t}} \prod_ {j} ^ {| \mathcal {O} |} \prod_ {i} ^ {| S |} P \left(v _ {i, j} ^ {t} \mid s _ {i}, o _ {j}, \mathcal {D} _ {t}, \mathcal {V}, \theta\right) P \left(o _ {j} \mid \mathcal {V}, \omega\right) \\ \end{array}
$$
One limitation of the current representation is the absence of temporal placement of objects in time. Naturally humans are able to associate objects and their temporal occurrence over a certain period. Therefore, we defined two temporal-based slots:  $s_{\text{start}}$  and  $s_{\text{end}}$ , denoting the start time and end time of the video segment that an object can be located by each dialogue turn. In this work, we assume that a dialogue turn is limited to a single continuous time span, and hence,  $s_{\text{start}}$  and  $s_{\text{end}}$  can be defined turn-wise, identically for all objects. While this is a strong assumption, we believe it covers a large portion of natural conversational interactions. An example of multimodal dialogue state can be seen in Figure 1.
3 Video-Dialogue Transformer Network
A naive adaptation of conventional DST to MM-DST is to directly combine visual features extracted by a pretrained 3D-CNN model. However, as
Figure 2: Video-Dialogue Transformer Network(VDTN) has 4 key components: (a) Visual Perception and Encoder (Section 3.1) (b) Dialogue Encoder (Section 3.2) (c) Transformer Network (Section 3.3) (d1) State Decoder (Section 3.4) and (d2) Visual Decoder (Section 3.4)
shown in our experiments, this extension of conventional DST results in poor performance and does not address the challenge of MM-DST. In this paper, we established a new baseline, denoted as Video-Dialogue Transformer Network (VDTN) (Refer to Fig. 2 for an overview):
3.1 Visual Perception and Encoder
Visual Perception. This module encodes videos at both frame-level and segment-level representations. Specifically, we used a pretrained Faster R-CNN model (Ren et al., 2015) to extract object representations. We used this model to output the bounding boxes and object identifiers (object classes) in each video frame of the video. For an object  $o_j$ , we denoted the four values of its bounding boxes as  $bb_j = (x_j^1, y_j^1, x_j^2, y_j^2)$  and  $o_j$  as the object class itself. We standardized the video features by extracting features of up to  $N_{obj} = 10$  objects per frame and normalizing all bounding box coordinates by the frame size. Secondly, we used a pretrained ResNeXt model (Xie et al., 2017) to extract the segment-level representations of videos, denoted as  $z_m \in \mathbb{R}^{2048}$  for a segment  $m$ . Practically, we followed the best practice in computer vision by using a temporal sliding window with strides to sample video segments and passed segments to ResNeXt model to extract features. To standardize visual features, we use the same strid
ing configuration  $N_{\text{stride}}$  to sub-sample segments for ResNeXt and frames for Faster R-CNN models.
Visual Representation. Note that we do not finetune the visual feature extractors in VDTN and keep the extracted features fixed. To transform these features into VDTN embedding space, we first concatenated all object identity tokens  $OBJ<class>$  of all frames. An object identity token  $OBJ<class>$  is the code name of the object class (e.g. a class of small blue metal cones) that a visual object can be unique identified (See Figure 2). Frames are separated by a special token  $FRAME<number>$ , where  $<number>$  is the temporal order of the frame. This results in a sequence of tokens  $X_{obj}$  of length  $L_{obj} = (N_{obj} + 1) \times (|\mathcal{V}| / N_{stride})$  where  $|\mathcal{V}|$  is the number of video frames. Correspondingly, we concatenated bounding boxes of all objects, and used a zero vector in positions of  $FRAME<number>$  tokens. We denoted this sequence as  $X_{bb} \in \mathbb{R}^{L_{obj} \times 4}$  where the dimension of 4 is for the bounding box coordinates  $(x^1, y^1, x^2, y^2)$ . Similarly, we stacked each ResNeXt feature vector by  $(N_{obj} + 1)$  for each segment, and obtained a sequence  $X_{cnn} \in \mathbb{R}^{L_{obj} \times 2048}$ .
Visual Encoding. We passed each of  $X_{bb}$  and  $X_{cnn}$  to a linear layer with ReLU activation to map their feature dimension to a uniform dimension  $d$ .
We used a learnable embedding matrix to embed each object identity in  $X_{obj}$ , resulting in embedding features of dimensions  $d$ . The final video input representation is the summation of above vectors, denoted as  $Z_V = Z_{obj} + Z_{bb} + Z_{cnn} \in \mathbb{R}^{L_{obj} \times d}$ .
3.2 Dialogue and State Encoder
Dialogue Encoding. Another encoder encodes dialogue into continuous representations. Given a dialogue context  $\mathcal{D}_t$ , we tokenized all dialogue utterances into sequences of words, separated by special tokens  $USR$  for human utterance and  $SYS$  for system utterance. We used a trainable embedding matrix and sinusoidal positional embeddings to embed this sequence into  $d$ -dimensional vectors.
Flattening State into Sequence. Similar to the recent work in traditional DST (Lei et al., 2018b; Le et al., 2020b; Zhang et al., 2020), we are motivated by the DST decoding strategy following a Markov principle and used the dialogue state of the last dialogue turn  $\mathcal{B}_{t-1}$  as an input to generate the current state  $\mathcal{B}_t$ . Using the same notations from Section 2, we can represent  $B_t$  into a sequence of  $o_j, s_i$ , and  $v_{i,j}^t$  tokens, such as "OBJ4 shape cube OBJ24 size small color red". This sequence is then concatenated with utterances from  $\mathcal{D}_t$ , separated by a special token PRIOR_STATE. We denoted the resulting sequence as  $X_{ctx}$  which is passed to the embedding matrix and positional encoding as described above. As we showed in our experiments, to encode dialogue context, this strategy needs only a few dialogue utterances (that are closer to the current turn  $t$ ) and  $\mathcal{B}_{t-1}$ , rather than the full dialogue history from turn 1. Therefore, dialogue representations  $Z_{ctx}$  have more compressed dimensions of  $|X_{ctx}| \times d$  where  $|X_{ctx}| < |\mathcal{D}_t|$ .
3.3 Multimodal Transformer Network
We concatenated both video and dialogue representations, denoted as  $Z_{VD} = [Z_V;Z_D]$ .  $Z_{VD}$  has a length of  $L_{obj} + L_{ctx}$  and embedding dimension  $d$ . We pased  $Z_{VD}$  to a vanilla Transformer network (Vaswani et al., 2017) through multiple multi-head attention layers with normalization (Ba et al., 2016) and residual connections (He et al., 2016). Each layer allows multimodal interactions between object-level representations from videos and word-level representations from dialogues.
3.4 Dialogue State Decoder and Self-supervised Video Denoising Decoder
State Decoding. This module decodes dialogue state sequence auto-regressively, i.e. each token is conditioned on all dialogue and video representations as well as all tokens previously decoded. At the first decoding position, a special token STATE is embedded into dimension  $d$  (by a learned embedding layer and sinusoidal positional encoding) and concatenated to  $Z_{VD}$ . The resulting sequence is passed to the Transformer network and the output representations of STATE are passed to a linear network layer that transforms representations to state vocabulary embedding space. The decoder applies the same procedure for the subsequent positions to decode dialogue states auto-regressively. Denoting  $b_{k,t}$  as the  $k^{th}$  token in  $\mathcal{B}_t$ , i.e. token of slot, object identity, or slot value, we defined the DST loss function as the negative log-likelihood:
$$
\mathcal {L} _ {d s t} = - \sum \log P (b _ {k, t} | b _ {<   k, t}, X _ {c t x}, X _ {o b j})
$$
Note that this decoder design partially avoids the assumption of conditionally independent slots. During test time, we applied beam search to decode states with the maximum length of 25 tokens in all models and a beam size 5. An END_STATE token is used to mark the end of each sequence.
Visual Denoising Decoding. Finally, moving away from conventional unimodal DST, we proposed to enhance our DST model with a Visual Decoder that learns to recover visual representations in a self-supervised learning task to improve video representation learning. Specifically, during training time, we randomly sampled visual representations and masked each of them with a zero vector. At the object level, in the  $m^{th}$  video frame, we randomly masked a row from  $X_{bb}(m) \in \mathbb{R}^{N_{obj} \times 4}$ . Since each row represents an object, we selected a row to mask by a random object index  $j \in [1, N_{obj}]$  such that the same object has not been masked in the preceding frame or following frame. We denote the Transformer output representations from video inputs as  $Z_V' \in \mathbb{R}^{L_{obj} \times d}$ . This vector is passed to a linear mapping  $f_{bb}$  to bounding box features  $\mathbb{R}^4$ . We defined the learning objective as:
$$
\mathcal {L} _ {o b j} = \sum_ {o} \mathbf {1} _ {\mathrm {m a s k e d}} \times l (f _ {b b} (Z _ {V, o} ^ {\prime}), X _ {b b, o})
$$
where  $l$  is a loss function and  $\mathbf{1}_{\mathrm{masked}} = \{0,1\}$  is a masking indicator. We experimented with both
L1 and L2 loss and reported the results. Similarly, at the segment level, we randomly selected a segment to mask such that the preceding or following segments have not been chosen for masking:
$$
\mathcal {L} _ {s e g} = \sum_ {s} \mathbf {1} _ {\mathrm {m a s k e d}} \times l (f _ {c n n} (Z _ {V, s} ^ {\prime}), X _ {c n n, s})
$$
Dataset. In existing benchmarks of multimodal dialogues such as VisDial (Das et al., 2017a) and AVSD (Alamri et al., 2019), we observed that a large number of data samples contain strong distribution bias in dialogue context, in which dialogue agents can simply ignore the whole dialogue and rely on image-only features (Kim et al., 2020). Another observed bias is the annotator bias that makes a causal link between dialogue context and output response actually harmful (Qi et al., 2020) (as annotator's preferences are treated as a confounding factor). The above biases would obviate the need for a DST task.
To address the above biases, Le et al. (2021b) developed a Diagnostic Benchmark for Video-grounded Dialogues ("DVD"), by synthetically creating dialogues that are grounded on videos from CATER videos (Shamsian et al., 2020). The videos contain visually simple yet highly varied objects. The dialogues are synthetically designed with both short-term and long-term object references. These specifications remove the annotation bias in terms of object appearances in visual context and cross-turn dependencies in dialogue context.
Extension from DVD (Le et al., 2021b). We generated new dialogues following Le et al. (2021b)'s procedures but based on an extended CATER video split (Shamsian et al., 2020) rather than the original CATER video data (Girdhar and Ramanan, 2020). We chose the extended CATER split (Shamsian et al., 2020) as it includes additional annotations of ground-truth bounding box boundaries of visual objects in video frames. This annotation facilitates experiments with Faster-RCNN finetuned on CATER objects and experiments with models of perfect visual perception, i.e.  $P(o_{j}|\mathcal{V},\omega)\approx 1$ . As shown in (Le et al., 2021b), objects can be uniquely referred in utterances based on their appearance by one or more following aspects: "size", "color", "material", and "shape". We directly reuse these and define them as slots in our
Split	# Videos	# Dialogues	# Turns
DVD-DST-Train	9300	9295	92950
DVD-DST-Val	3327	3326	33260
DVD-DST-Test	1371	1371	13710
DVD-DST-All	13998	13992	139920
Table 1: Summary of the DVD-DST dataset
dialogue states, in addition to 2 temporal slots for  $s_{\mathrm{start}}$  and  $s_{\mathrm{end}}$ . We denote the new benchmark as DVD-DST and summarize the dataset in Table 1 (for more detail, please refer to Appendix B).
Baselines. To benchmark VDTN, we compared the model with following baseline models, including both rule-based models and trainable models:
-  $Q$ -retrieval (tfidf), for each test sample, directly retrieves the training sample with the most similar question utterance and use its state as the predicted state;
- State prior selects the most common tuple of (object, slot, value) in training split and uses it as predicted states;
- Object (random), for each test sample, randomly selects one object predicted by the visual perception model and a random (slot, value) tuple (with slots and values inferred from object classes) as the predicted state;
- Object (all) is similar to the prior baseline but selects all possible objects and all possible (slot, value) tuples as the predicted state;
-  $RNN(+Att)$  uses RNN as encoder and an MLP network as decoder. Another variant of the model is enhanced with a vanilla dot-product attention at each decoding step;
- We adapted and experimented with strong unimodal DST baselines, including: TRADE (Wu et al., 2019), UniConv (Le et al., 2020b) and NADST (Le et al., 2020c).
We implemented these baselines and tested them on dialogues with or without videos. When video inputs are applied, we embedded both object and segment-level features (See Section 3.1). The video context features are integrated into baselines in the same techniques in which the original models treat dialogue context features.