SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 9 and 10 display the ablation results by component predictions, using precision and recall metrics. We still noted consistent observations as described in Section 4. Notably, we found that current VDTN models are better in tuning the correct predictions (as shown by high precision metrics) but still fail to select all components as a set (as shown by low recall metrics). This might be caused by the upstream errors coming from the visual perception models, which may fail to visually perceive all objects and their attributes.
Results by turn positions. Table 11 reported the results of VDTN predictions of states that are separated by the corresponding dialogue positions. The results are from the VDTN model trained with both  $\mathcal{L}_{dst}$  and  $\mathcal{L}_{seg}$ . As expected, we observed a downward trend of results as the turn position increases.
Impacts of dialogue context encoder. In Table 12a, we observed the benefits of using the Markov process to decode dialogue states based on the dialogue states of the last turn  $\mathcal{B}_{t-1}$ . This strategy allows us to discard parts of dialogue history that is already represented by the state. We noted that the
optimal design is to use at least 1 last dialogue turn as the dialogue history. In a hypothetical scenario, we applied the oracle  $\mathcal{B}_{t - 1}$  during test time, and noted the performance is improved significantly. This observation indicates the sensitivity of VDTN to a turn-wise auto-regressive decoding process.
Impacts of frame-level and segment-level sampling. As expected, Table 12b displays higher performance with higher object limits  $N_{obj}$ , which increases the chance of detecting the right visual objects in videos. We noted performance gains when sampling strides increase up to 24 frames. However, in the extreme case, when sampling stride is 300 frames, the performance on temporal slots reduce (as shown by "Joint State IoU@p"). This raises the issue to sample data more efficiently by balancing between temporal sparsity in videos and state prediction performance. We also observed that in a hypothetical scenario with a perfect object perception model, the performance improves significantly, especially on the predictions of discrete slots, although less effect on temporal slots.
Impacts of object-level representation. Table 13 reported the results when only segment-level features are used. We observed that both VDTN and  $RNN(V + D)$  are affected significantly, specifically by  $24\%$  and  $3.1\%$  "Joint Obj State Acc" score respectively. Interestingly, we noted that  $RNN(V)$ , using only video inputs, are not affected by the removal of object-level features. These observations
Turn Position	Obj Identity F1	Obj Slot F1	Obj State F1	Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7
1	88.8%	84.0%	82.4%	74.0%	40.5%	34.6%
2	86.9%	81.1%	77.2%	60.0%	37.5%	33.6%
3	84.9%	77.6%	71.0%	41.6%	22.8%	19.5%
4	84.2%	75.6%	66.5%	29.0%	15.2%	12.5%
5	84.0%	74.0%	63.1%	21.3%	11.3%	9.4%
6	84.3%	73.0%	60.2%	17.1%	9.6%	8.2%
7	83.9%	71.6%	57.1%	12.7%	6.1%	5.3%
8	84.1%	70.6%	54.9%	10.2%	4.7%	3.9%
9	84.0%	69.1%	51.8%	7.9%	3.6%	2.6%
10	84.1%	68.0%	49.5%	6.0%	2.3%	1.7%
Average	84.9%	74.5%	63.4%	28.0%	15.3%	13.1%
Table 11: Ablation results by dialogue turn positions
Bt-1	Max turns	Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7
✓	10	22.5%	11.5%	10.1%
✓	7	22.0%	11.8%	10.4%
✓	1	24.8%	13.8%	11.8%
✓	0	22.3%	12.3%	10.5%
-	10	18.5%	9.4%	8.6%
-	7	19.0%	9.5%	8.7%
-	1	7.8%	4.5%	4.1%
-	0	1.3%	0.7%	0.7%
✓*	1	29.3%	18.6%	16.4%
(a) dialogue encoding by prior states and dialogue sizes:  $\ast$  denotes using oracle values.
Nobj	Nstride	Joint Object State Acc	Joint State IoU@0.5	Joint State IoU@0.7
10	12	24.8%	13.8%	11.8%
7	12	18.0%	10.1%	9.0%
3	12	4.9%	2.9%	2.6%
0	12	1.5%	0.7%	0.7%
10	300	28.2%	6.0%	3.7%
10	24	27.8%	14.8%	12.6%
10	15	26.3%	14.4%	12.4%
10	12	24.8%	13.8%	11.8%
10*	12	29.2%	15.6%	13.4%
(b) video encoding by number of objects and sampling strides: * denotes perfect object perception.
Table 12: Ablation results by encoding strategies: All models are trained only with  $\mathcal{L}_{dst}$ .
indicate that current MM-DST requires object-level information. We expected that existing 3DCNN models such as ResNeXt still fail to capture such level of granularity.
Qualitative analysis. Table 14 and 15 display 2 sample dialogues and state predictions. We displayed the corresponding video screenshots for these dialogues in Figure 5. To cross-reference between videos and dialogues, we displayed the bounding boxes and their object classes in video screenshots. These object classes are indicated in ground-truth and decoded dialogue states in dialogues. Overall, we noted that VDTN generated temporal slots of start and end time such that the resulting periods better match the ground-truth temporal segments. VDTN also showed to maintain the dialogue states better from turn to turn.
Synthetic datasets result in overestimation of real performance and don't translate to real-world usability. We agree that the current state accuracy seems to be quite low at about  $28\%$ . However, we want to highlight that state accuracy used in this paper is a very strict metric, which only considers a prediction as correct if it completely
matches the ground truth. In DVD, assuming the average 10 objects per video with the set of attributes as in Figure 4 (+ 'none' value in each slot), we can roughly equate the multimodal DST as a 7200-class classification task, each class is a distinct set of objects, each with all possible attribute combinations. Combined with the cascading error from object perception models, we think the current reported results are reasonable.
Moreover, we want to highlight that the reported performance of baselines reasonably matches their own capacities in unimodal DST. We can consider Object State F1 as the performance on single-object state and it can closely correlate with the joint state accuracy in unimodal DST (remember that unimodal DST such as MultiWOZ (Budzianowski et al., 2018) is only limited to a single object/entity per dialogue). As seen in Table 2, the Object State F1 results of TRADE (Wu et al., 2019), UniConv (Le et al., 2020b), and NADST (Le et al., 2020c) are between  $46 - 50\%$ . This performance range is indeed not very far off from the performance of these baseline models in unimodal DST in the MultiWOZ benchmark (Budzianowski et al., 2018).
Finally, we also want to highlight that like other synthetic benchmarks such as CLEVR (Johnson
Model	Xbb + Xcnn	Xcnn only
Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7	Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7
VDTN	28.0%	15.3%	13.1%	4.0%	2.2%	2.0%
RNN(V)	1.0%	0.1%	0.1%	1.5%	0.4%	0.4%
RNN(V+D)	6.8%	2.6%	2.3%	3.7%	1.8%	1.6%
Table 13: Results with and without object representations
et al., 2017), we want to use DVD in this work as a test bed to study and design better multimodal dialogue systems. However, we do not intend to use it as a training data for practical systems. The DVD-DST benchmark should be used to supplement real-world video-grounded dialogue datasets.
MM-DST in practical applications e.g. with videos of humans. While we introduced MM-DST task and VDTN as a new baseline, we noted that the existing results are limited to the synthetic benchmark. For instance, in the real world, there would be many identical objects with the same (size, color, material, shape) tuples, which would make the current formulation of dialogue states difficult. In such object-driven conversations, we would recommend a dialogue agent not focus on all possible objects in each video frame, but only on a "focus group" of objects. These objects, required to be semantically different, are topical subjects of the conversations.
Say we want to scale to a new domain e.g. videos of humans, the first challenge from the current study is the recognition of human objects, which often have higher visual complexity than moving objects as in DVD. We also noted that it is impossible to define all human object classes as in CATER object classes, each of which is unique by its own appearance. To overcome this limitation, we would want to explore multimodal DST with the research of human object tracking, e.g. (Fernando et al., 2018), and consider human object identities uniquely defined per video. Another limitation is the definition of slots to track in each human object. While this requires careful considerations, for both practical and ethical reasons, we noted several potential papers that investigate human attributes in dialogues such as human emotions (Wang et al., 2021). Along these lines, we are excited to see interesting adaptations of multimodal dialogue states grounded on videos of humans.
Table 14: Example predictions from a sample dialogue (CATER video ID #002660)
Table 15: Example predictions from a sample dialogue (CATER video ID #001441)
(a) Video #002660
(b) Video #001441
Figure 5: Example screenshots of CATER videos for dialogues in Table 14 (Video #002660) and 15 (Video #001441). We showed example bounding boxes and their object classes in each video.