SlowGuess/NAACL_2022_OCR huggingface dataset:
Haeju Lee $^{1*}$ , Oh Joon Kwon $^{1*}$ , Yunseon Choi $^{1*}$ , Minho Park $^{3}$ , Ran Han $^{3}$ , Yoonhyung Kim $^{3}$ , Jinhyeon Kim $^{1}$ , Youngjune Lee $^{2,4}$ , Haebin Shin $^{5}$ , Kangwook Lee $^{5}$ , Kee-Eung Kim $^{1,2}$
<sup>1</sup>Kim Jaechul Graduate School of AI, KAIST <sup>2</sup>School of Computing, KAIST <sup>3</sup>ETRI <sup>4</sup>Search CIC, Naver Corp. <sup>5</sup>Samsung Research
The Situated Interactive Multi-Modal Conversations (SIMMC) 2.0 aims to create virtual shopping assistants that can accept complex multi-modal inputs, i.e. visual appearances of objects and user utterances. It consists of four subtasks, multi-modal disambiguation (MM-Disamb), multi-modal coreference resolution (MM-Coref), multi-modal dialog state tracking (MM-DST), and response retrieval and generation. While many task-oriented dialog systems usually tackle each subtask separately, we propose a jointly learned multi-modal encoder-decoder that incorporates visual inputs and performs all four subtasks at once for efficiency. This approach won the MM-Coref and response retrieval subtasks and was nominated runner-up for the remaining subtasks using a single unified model at the 10th Dialog Systems Technology Challenge (DSTC10), setting a high bar for the novel task of multi-modal task-oriented dialog systems.
A task-oriented dialog system aims to assist users to accomplish certain tasks, such as executing actions or retrieving specific information, with natural language conversations. With the rising interest in multi-modal representation learning, the next generation of task-oriented virtual assistants is expected to handle conversations in such contexts, especially in the domain of vision-language (VL). For instance, a multi-modal dialog agent may help the user navigate a virtual clothing store and look for an object meeting the user's criteria. In such cases, a successful dialog agent should be able to parse and understand multi-modal contexts.
To this end, SIMMC 2.0 (Kottur et al., 2021) proposes a situated multi-modal context in the form
of co-observed, realistic scene set in virtual reality (VR) stores to incorporate the complexity of multi-modal task-oriented dialogs. The multimodal subtasks, MM-Disamb and MM-Coref, intend to test the assistant's capability to identify the need for disambiguating reference mentions and to ground them to the scene objects. While challenging, these are all essential to building a successful multi-modal dialog agent.
In this paper, we present our end-to-end, joint-learning approach to address this challenge in SIMMC 2.0. We adopt BART (Lewis et al., 2019) and attach task-specific heads so that the model can make predictions on all subtasks at once. To be more specific, our model performs MM-Disamb, MM-Coref, and response retrieval by the encoder and MM-DST and response generation in a string format by the decoder. We also integrate multimodality into the model by extracting visual features of each object from a convolutional vision backbone and then combining them with nonvisual attributes. Our model is jointly trained on all subtasks and a couple of auxiliary objectives to help the model align the different modalities. For retrieval, we use in-batch negative samples for contrastive metric learning instead of creating a pool of separate training samples.
With modification for the competition setting, our model was ranked at the first place for MM-Coref and response retrieval with  $75.8\%$  coreference F1,  $82.5\%$  MRR,  $72.5\%$  R@1,  $95.0\%$  R@5,  $98.4\%$  R@10, and 1.9 mean rank in the official evaluation of DSTC10. Moreover, our model was nominated runner-up for all other subtasks, in which we achieved  $93.8\%$  disambiguation accuracy,  $90.3\%$  slot F1,  $95.9\%$  intent F1, and 0.295 BLEU-4. The results were obtained with only a single model and consistent with the results on the devtest (i.e. validation) set, demonstrating a robust, common representation on all subtasks learned by the model.
*:These authors contributed equally. Corresponding authors. {hjlee, ojkwon, yschool}@ai.kaist.ac.kr Code is available at https://github.com/KAIST-AILab/DSTC10-SIMMC
Findings of the Association for Computational Linguistics: NAACL 2022, pages 813 - 830 July 10-15, 2022 ©2022 Association for Computational Linguistics
Recent works on (uni-modal) task-oriented dialog systems remove the need for a pipeline composed of NLU (Liu and Lane, 2016), DST (Mrksic et al., 2017), POL (Wen et al., 2017), and NLG (Wen et al., 2015) modules by leveraging pretrained language models (LM) that integrate all the modules in an end-to-end, auto-regressive manner (Ham et al., 2020; Hosseini-Asl et al., 2020; Yang et al., 2021). Given a dialog context, such systems sequentially generates belief state, system action, and response, making predictions based on decisions made by previous modules in the form of tokens and achieving superior results to the pipelined approaches. Some of these systems aim to learn the user preference from dialogs and recommend the object based on external knowledge base (KB) (Zhou et al., 2020).
In a similar context, building cross-modal models has recently gained attention in VL domain. Recent works develop VL models on top of the transformer-based (Vaswani et al., 2017) pretrained LM and vision backbones, focusing on self-supervised pretraining methods to align joint embedding between different modalities. They achieve state-of-the-art performance in downstream tasks such as visual question answering (VQA), as shown in (Chen et al., 2020) and (Li et al., 2020). However, there are only a handful of works focusing on situated VL task-oriented dialog systems (Liao et al., 2018), where visual modality of the task is provided in a sanitized setting rather than a natural, situated scene.
3 SIMMC 2.0 Description
SIMMC 2.0 (Kottur et al., 2021) follows the setting of SIMMC 1.0 (Moon et al., 2020), which assumed conversations occurring between a user and an assistant in a situated, co-observed VR scene. SIMMC 2.0 improves on its predecessor by providing a far richer visual context with 19.7 objects on average that are often occluded, cluttered, or even out of view. An example dialog is shown in Figure 1.
The SIMMC 2.0 dataset consists of 11,244 dialogs split into train (65%), dev (5%), devtest (15%), and teststd (15%) sets. Each dialog includes multiple turns where each turn has grounded multi
Figure 1: An instance of dialog and the corresponding scene in SIMMC 2.0. Here, the assistant asks the user to disambiguate between the blue hoodie jacket (denoted 1) and the beige jacket (denoted 2), grounding its mentions to the scene via multi-modal context  $M_0 = \{1,2\}$ . Once the user chooses the blue one, the system retrieves the information on the disambiguated object. The multimodal context in this case would be  $M_1 = \{1\}$ .
modal context and an accompanying scene with referential indices. We shall denote a SIMMC dialog with  $r$  rounds as  $\mathcal{D} := \{(U_t, A_t, M_t, S_t, B_t)\}_{t=1}^r$ , where  $U_t$  is user utterance,  $A_t$  system utterance,  $M_t$  multi-modal context,  $S_t$  scene context, and  $B_t$  user belief state at turn  $t$ . Here,  $M_t$  is a set of object indices mentioned by the system and  $S_t$  contains the corresponding attributes and locations of all the objects in a scene. User belief state  $B_t$  is composed of dialog act (i.e. user intent) and slot (i.e. a tuple of (slot name, value), for instance ("price", "$11.99")). We also define the dialog history at some turn  $T \leq r$  as  $H_T := \{U_0, A_0, M_0, \ldots, U_{T-1}, A_{T-1}, M_{T-1}\}$ .
The assistant needs to make predictions conditioned on history  $H_{T}$ , current user utterance  $U_{T}$  and the scenes up to the current turn  $S_{t\leq T}$ . The object set consists of fashion and furniture domain, where each domain has 288 and 57 items respectively. The system is allowed to look up which item is present in a scene at all time along with its bounding box information. As a side information, the metadata of each object are provided: its non-visual attributes such as brand, size, customer rating and price are available for both training and inference, but looking up the visual attribute (e.g. color, pattern, materials, sleeve length) is prohibited for inference so as to make the agent reason with multi-modal information.
$^{2}$ Dataset is publicly available at https://github.com/facebookresearch/simmc2
Figure 2: Overview of the jointly learned multi-tasking BART. For  $H_{T}$ , we show only the last turn without user utterance due to space limit. The details on the loss functions are provided in model specifics. Each scene object is represented by the concatenation of scene canonical object ID token (e.g.  $<11>$ ) and features from a vision encoder. It is then passed through MM-Coref and attribute classification head. MM-DST and response generation subtasks are approached in terms of auto-regressive LM.
3.2 Subtasks
Multi-modal disambiguation (MM-Disamb)
The first subtask is to identify whether the assistant should disambiguate mentions in the next turn given the dialog and multi-modal context. For instance, given user utterance "How much is the pair on the left?", there may be more than two pairs of pants on the left. In this case, ambiguity in reference should be resolved. This can be cast into a binary classification task, and the performance is measured by accuracy.
Multi-modal coreference resolution (MM-Coref) The second subtask is to map the referential mentions of the user utterance to the object indices in the scene. These mentions should be resolved through the linguistic context and the multi-modal context. The performance is measured by object slot F1 score.
Multi-modal dialog state tracking (MM-DST)
The third subtask extends the traditional uni-modal DST to ground user belief state on the multi-modal objects. This will measure the assistant's under
standing throughout each dialog, which includes disambiguation and coreference resolution. The performance is measured by the F1 score for dialog act and slots.
Response retrieval & generation The last subtask is to retrieve or generate appropriate system utterance. Response generation is evaluated with BLEU-4 (Papineni et al., 2002). For response retrieval, the system is expected to choose the most relevant response from a pool of 100 candidate responses. Recall@  $k$  ( $k \in \{1,5,10\}$ ), mean rank, and mean reciprocal rank (MRR) are used for retrieval evaluation.
4 Multi-Modal Transformer Model
The setting of the dataset is similar to that of VQA where finetuning the pretrained VL models is prevalent; however, these models are usually pretrained on natural images (Lin et al., 2014; Krishna et al., 2017) and require a large number of training samples of 3D rendered images that are aligned properly with text. Hence, we decide to work primarily with pretrained LM and convolutional vision en
coder to suit the setting of SIMMC 2.0. In particular, we integrate the visual modality by encoding each object with finetuned ResNet-34 (He et al., 2016). We also index each object in the scene by its referential ID (canonical object ID), which are concatenated with corresponding visual representations for subtasks.
In order to further align the different modalities, we provide additional supervision signals at train time by looking up the object metadata. We note that all of the subtasks are related to each other. For example, if the assistant decides that the user utterance needs to be disambiguated, then the appropriate system action is to respond along the line of "Which one are you referring to?" Once disambiguated, the user may ask for the price of "blue striped shirt", where representations learned from MM-Coref prediction subtask (and/or attribute classification) can help the model predict the correct slot values for MM-DST and response generation. We expect that the latent representation of the multimodal dialog learned from other subtasks will translate readily to other subtasks. Hence, we utilize hard parameter sharing (Caruana, 1993) on the encoder to jointly learn on all subtasks. This reduces not only the number of network parameters, but also the risk of overfitting (Baxter, 1997).
Moreover, we decide to view MM-Coref as a type of set prediction (Zaheer et al., 2017), where joint learning of set cardinality and state distribution has been shown effective (Rezatofighi et al., 2018). Hence, we define an additional empty coreference target prediction (Empty-Coref), a simplified cardinality prediction task that outputs whether the current user utterance has no MM-Coref targets. Moreover, we perform supervised learning on object attributes to help align object-language modalities.
We adopt BART (Lewis et al., 2019) as the pretrained language backbone. Our preliminary experiments suggested that performing certain subtasks such as MM-Disamb by a bi-directional encoder (e.g. BERT) proved more effective than doing so by an uni-directional decoder (e.g. GPT-2). To harness both the NLU capabilities of the encoder and the NLG capabilities of the decoder, we choose a transformer encoder-decoder to handle all subtasks at once. We attach classification heads for MM-Disamb and MM-Coref subtasks to the encoder and LM head for MM-DST and response generation to the decoder. We also perform retrieval by
computing the dot product between representation vectors of response candidates and multi-modal dialog context. Figure 2 provides an overview of the model.
4.1 Input Representation
For all of the subtasks, we define our input to be a simple concatenation  $x \coloneqq [H_T; U_T; S_{t \leq T}]$  with separators. We define  $H_T$  to be the dialog history up to 2 turns to limit the length of input, i.e.  $\{U_{T-2}, A_{T-2}, M_{T-2}, U_{T-1}, A_{T-1}, M_{T-1}\}$ . SIMMC 2.0 assumes that utterances may mention objects that are not in the current scene  $S_T$  but in the previously observed scene  $S_{t < T} \neq S_T$ . Hence, our model integrates the objects from the previous scene that are not in the current scene. An exemplar input is provided in Table 1.
4.1.1 Canonical object ID token
A canonical object ID token takes the form of  $<\backslash d+>$  (e.g.  $<32>$ ). This provides a relational context of the object within the scene, grounding each object to its scene object index provided in the dataset. This scheme was also used in the baseline (Kottur et al., 2021), but without any association to object attributes. In our method, this token intends to provide contextual information about the object alongside its visual attributes, allowing the assistant to make connections between different modalities.
For the assistant to understand the spatial information, we must incorporate the location of each object. We follow the commonly used techniques in VL models (Li et al., 2020; Chen et al., 2020; Zhang et al., 2021) for encoding object locations with the bounding box information. Given a bounding box represented by its upper-left and lower-right vertices,  $(x_{1},y_{1})$  and  $(x_{2},y_{2})$ , with height  $h$  and width  $w$ , we encode its location as a normalized tuple  $(x_{1} / w - 0.5,y_{1} / h - 0.5,x_{2} / w - 0.5,y_{2} / h - 0.5,(x_{2} - x_{1})(y_{2} - y_{1}) / (h\cdot w))$ . This is passed through a location embedding layer (a linear layer followed by layer norm) to be added with the canonical object ID token encoding.
4.1.2 Representation of objects
Each item is represented by its visual and non-visual attributes. The visual attributes are provided by the hidden features of the cropped image encoded by ResNet-34 (He et al., 2016). Once finetuned by classifying the objects from the train split scenes to their corresponding visual attributes,
Table 1: Example input representations for our model. We show only up to last 1 turn due to space limit. Thus, the common input  $x$  is a concatenation  $\left\lbrack  {{H}_{T};{U}_{T};{S}_{t \leq  T}}\right\rbrack$  where  ${H}_{T} = \left\{  {{U}_{T - 1},{A}_{T - 1},{M}_{T - 1}}\right\}$  . Here,we separate the previous scene history  ${S}_{t < T}$  to show how we handle out-of-view objects. The generation target is a concatenation  $\left\lbrack  {{B}_{T};{A}_{T}}\right\rbrack$  ,which is used by the decoder. The response candidate is  ${A}_{T}$  with speaker identifier  $< \operatorname{SYS} >$  prepended. Here,we denote the visual feature of the  $i$  -th object extracted from the vision encoder as  $< \operatorname{vis}\_ \{ \mathrm{i}\}  >$  .
the vision encoder is fixed throughout the training of the actual dialog system. The non-visual attributes are randomly initialized learnable embeddings. Both types of attributes are mapped by a linear layer then concatenated to represent an object to match the dimension of BART. For the competition setting (or deployment within virtual environment) where the object identity is readily available at inference, we replace then train the entire object representation with randomly initialized learnable embeddings.
4.1.3 Separator tokens
We define several separator tokens to delimit different components of the multi-modal dialogs. We use  $\langle \mathrm{SOM} \rangle$ ,  $\langle \mathrm{EOM} \rangle$  for the start and the end of multi-modal context and  $\langle \mathrm{SOO} \rangle$ ,  $\langle \mathrm{EOO} \rangle$  for the start and the end of scene objects. Within the scene context,  $\langle \mathrm{OBJ} \rangle$  token is used as a marker between current scene objects. We also mark those from the previous scene with  $\langle \mathrm{PREV\_OBJ} \rangle$ . For generation target, we mark the start and the end of the user belief state with  $\langle \mathrm{SOB} \rangle$ ,  $\langle \mathrm{EOB} \rangle$ .
4.2 Model Specifics
4.2.1 Binary prediction for MM-Disamb and MM-Coref
We formulate MM-Disamb as a binary classification on the pooled output of the encoder from the pooling token  $\langle \text{DISAMB} \rangle$ . The binary head for MM-Disamb should predict true if the current user utterance  $U_{T}$  needs to be disambiguated and false otherwise.
For MM-Coref, we make binary predictions on all objects in  $S_{t \leq T}$ . We do so by passing the concatenated canonical object (e.g.  $<11>$ ) and the
representation of each object through a binary classification head. The MM-Coref head will predict true if the current user utterance mentions that object and false otherwise. We use a simple cross-entropy loss for both MM-Disamb and MM-Coref, denoted  $\mathcal{L}_{\mathrm{mm - disamb}}$  and  $\mathcal{L}_{\mathrm{mm - coref}}$ .
4.2.2 Auto-regressive LM for MM-DST and response generation
We also approach MM-DST and response generation subtasks with auto-regressive LM following the recent approaches in end-to-end dialog systems. For MM-DST and response generation, we use the standard left-to-right LM loss (Bengio et al., 2003).
$$
\mathcal {L} _ {\mathrm {L M}} = \sum_ {i = 1} ^ {L} - \log P (\omega_ {i} \mid \omega_ {1}, \dots , \omega_ {i - 1}),
$$
where  $\omega_{i}$  is the  $i$ -th target token and  $L$  the total length of the target.
4.2.3 In-batch negative samples for retrieval
For response retrieval task, we make use of in-batch negative samples for contrastive learning on similarity metrics, following (Karpukhin et al., 2020) except that we use a single-tower architecture. We treat the system responses of the other samples in the batch (formatted according to Table 1) as in-batch negatives. We then pool the input and the response candidate representations via bos token to compute their dot product from which cross-entropy is applied, i.e.,
$$
\mathcal {L} _ {\mathrm {r e t r i e v a l}} = - \log \frac {\exp (\mathbf {x} \cdot \mathbf {a} ^ {+})}{\sum_ {\mathbf {a} ^ {-} \in B ^ {-} (\mathbf {x}) \cup \{\mathbf {a} ^ {+} \}} \exp (\mathbf {x} \cdot \mathbf {a} ^ {-})},
$$
where  $\mathbf{a}^{+}$  is the positive response sample of the input  $\mathbf{x}$  and  $B^{-}(\mathbf{x})$  the set of in-batch negative
responses (assume  $\mathbf{x}$ ,  $\mathbf{a}^+$ , and  $\mathbf{a}^{-}$  are pooled representations from the encoder). We formulate the task loss  $\mathcal{L}_{\mathrm{task}}$  as a linear combination of losses from each subtask.
$$
\begin{array}{l} \mathcal {L} _ {\text {t a s k}} = \lambda_ {\mathrm {L M}} \mathcal {L} _ {\mathrm {L M}} + \lambda_ {\mathrm {m m - d i s a m b}} \mathcal {L} _ {\mathrm {m m - d i s a m b}} \\ + \lambda_ {\text {m m - c o r e f}} \mathcal {L} _ {\text {m m - c o r e f}} + \lambda_ {\text {r e t r i e v a l}} \mathcal {L} _ {\text {r e t r i e v a l}} \\ \end{array}
$$
4.3 Auxiliary Tasks
4.3.1 Binary prediction for Empty-Coref
We define an additional Empty-Coref task, in which the assistant predicts whether the current dialog turn has MM-Coref targets. We find this additional signal for coreference resolution, denoted  $\mathcal{L}_{\mathrm{empty - coref}}$  , is advantageous in boosting MM-Coref performance, a type of set prediction task. Moreover, MM-Coref sometimes predicts targets when there is actually none, so we override any MMCoref predictions if the Empty-Coref prediction is true (i.e. there is no coreference target). For this, we use  $<  \mathrm{EMPTY\_COREF}>$  for pooling. At training time, we use cross-entropy loss for  $\mathcal{L}_{\mathrm{empty - coref}}$
4.3.2 Encoding object attributes
We encode object attributes by providing additional supervision signal during training. We do so by simply training to classify each object to its corresponding visual and non-visual attributes such as color, price, and customer ratings. Each object is represented as a concatenation of its canonical object ID and object features as in MM-Coref (refer to Figure 2). Each attribute head predicts a categorical class for each corresponding object, for example, if an object is a grey jacket, the color-attribute head should predict grey and the type-attribute head jacket.
Let  $\mathcal{O}_{t\leq T}$  be the set of objects in the scene history,  $S_{t\leq T}$ . We denote attribute multi-class classification loss  $\mathcal{L}_{\mathrm{att}}$  for all objects in  $\mathcal{O}_{t\leq T}$ ,
$$
\mathcal {L} _ {\mathrm {a t t}} = \sum_ {j \in \mathcal {O} _ {t \leq T}} \sum_ {k = 1} ^ {K} \sum_ {c \in \mathcal {C} _ {k}} - \mathbb {1} \{c = y _ {j k} \} \log P (c),
$$
where  $K$  is the number of attributes,  $\mathcal{C}_k$  the set of all classes of the  $k$ -th attribute,  $y_{jk}$  the label of the  $k$ -th attribute of the  $j$ -th object, and  $\mathbb{1}\{\cdot\}$  is an indicator function.
As a result, the auxiliary loss  $\mathcal{L}_{\mathrm{aux}}$  is defined as the weighted sum of attribute loss and empty-coreference prediction loss:
$$
\mathcal {L} _ {\text {a u x}} = \lambda_ {\text {a t t}} \mathcal {L} _ {\text {a t t}} + \lambda_ {\text {e m p l y - c o r e f}} \mathcal {L} _ {\text {e m p l y - c o r e f}} \tag {2}
$$
In summary, we minimize the sum of the task loss  $\mathcal{L}_{\mathrm{task}}$  (Equation 1) and the auxiliary loss  $\mathcal{L}_{\mathrm{aux}}$  (Equation 2).
The details on training hyperparameters are provided in Appendix A. For model selection, we evaluate the model on the devtest split at every 1000 training steps. We give priority to the left-most metric for each subtask (Table 2) and early stop on those winning the most among 5 subtasks (counting response generation and retrieval separately).
The dataset organizers provided two baseline models: an end-to-end GPT-2 (Radford et al., 2019) and multi-modal transformer networks (MTN) (Le et al., 2019). The baselines handle each subtask separately, except for MM-Coref, MM-DST, and response generation. The GPT-2 baseline generates the user belief state, coreference objects indices, and response in an end-to-end manner given a dialog history with multi-modal context provided in terms of object indices. For retrieval, a generated response is compared against the available pool of response candidates, from which the candidate with the most likelihood is chosen. MTN baseline conditions on the scene image and dialog history then generate the user belief state and response using a multi-modal transformer. The MTN baseline only implements MM-DST and response generation.