SlowGuess/NAACL_2022_OCR huggingface dataset:
- We observe that when the predicted programs are correct, the output responses generally match the ground-truth (See the  $1^{st}$  and  $2^{nd}$  turn in Figure 6, and the  $1^{st}$  and  $4^{th}$  turn in Figure 8) or close to the ground-truth responses ( $1^{st}$  turn in Figure 7).
- When the output responses do not match the ground truth, we can understand the model mistakes by interpreting the predicted programs. For example, in the  $3^{rd}$  turn in Figure 6, the output response describes a room because the predicted video program focuses on the entity "what room" instead of the entity "an object" in the question. Another example is the  $3^{rd}$  turn in Figure 8 where the entity "rooftop" is missing in the video program. These mismatches can deviate the information retrieved from the video during video program execution, leading to wrong output responses with wrong visual contents.
- We also note that in some cases, one or both of the predicted programs are incorrect, but the predicted responses still match the ground-truth responses. This might be explained as the predicted module parameters are still close enough to the "gold" labels (e.g.  $4^{th}$  turn in Figure 6). Sometimes, our model predicted programs that are more appropriate than the ground truth. For example, in the  $2^{nd}$  turn in Figure 7, the program is added with a where module parameterized by the entity "the shopping bag" which was solved from the reference "them" mentioned in the question.
- We observe that for complex questions that involve more than one queries (e.g. the  $3^{rd}$  turn in Figure 8), it becomes more challenging to decode an appropriate video understanding program and generate responses that can address all queries.
- In Figure 9, we demonstrate some output examples of VGNMN and compare with two baselines: Baseline (Hori et al., 2019) and MTN (Le et al., 2019b). We noted that VGNMN can include important entities relevant to the current dialogue turn to construct output responses while other models might miss some entity details, e.g. "them/dishes" in example A and "the magazine" in example B. These small yet important details can determine the correctness of dialogue responses.
Turn	Question	Dialogue Understanding Program	Video Understanding Program	Response
1	how many people are in the video?	Predicted: summarize()
✔
Gold: summarize()	Predicted: describe (how many people)
✔
Gold: describe (how many people)	Predicted: there is only one person in the video.
✔
Gold: there is one person in the video.
2	does this person ever leave the field of view?	Predicted: find (this person) → summarize()
✔
Gold: find (this person) → summarize()	Predicted: where (one person in the video) → when (leave the field of view) → exist()
✔
Gold: where (one person in the video) → when (leave the field of view) → exist()	Predicted: no, he does not leave the view.
✔
Gold: no, he is in the video the whole time.
3	in the door way to the next room, there is an object. what is that?	Predicted: find (the room), find (the door) → summarize() ×
Gold: summarize()	Predicted: where (what room) → when (what is that) → exist()
✔
Gold: describe (what)	Predicted: it looks like he is in a living room.
✔
Gold: he went to the doorway for a vacuum.
4	when he gets up, does he have anything in his hand?	Predicted: find (he), find (his) → summarize()
✔
Gold: find (he), find (his) → summarize()	Predicted: where (one person in the video) → when (get up, have anything in his hands) → describe (when) ×
Gold: where (one person in the video) → when (get up), when (have anything in his hands) → describe (when)	Predicted: he has a vacuum in his hands.
✔
Gold: he goes for the vacuum.
Figure 6: Example output programs and responses from a dialogue in the test split of the AVSD benchmark.
Figure 7: Example output programs and responses from a dialogue in the test split of the AVSD benchmark.
Turn	Question	Dialogue Understanding Program	Video Understanding Program	Response
1	what is holding in his hand in the beginning?	Predicted: find (his) →summarize() Gold: find (his) →summarize()	Predicted: where (a man) →when (holding in his hand in the beginning) → describe (what) Gold: where (a man) → when (holding in his hand in the beginning) → describe (what)	Predicted: he is holding a shopping bag in his hand. X Gold: looks like a bag of beans
2	does he put them down?	Predicted: find (he) , find (them) →summarize() X Gold: find (he) → summarize()	Predicted: where (a man), where (a shopping bag) → when (put them down) → exist() X Gold: where (a man) → when (put them down) → exist()	Predicted: yes he puts them down on the table. X Gold: yes he puts it down after looking at it
Figure 8: Example output programs and responses from a dialogue in the test split of the AVSD benchmark.
Figure 9: Interpretability of example outputs from VGNMN and baselines models (Hori et al., 2019; Le et al., 2019b)
Turn	Question	Dialogue Understanding Program	Video Understanding Program	Response
1	hi how many people are in the video from start to finish?	Predicted: summarize() Gold: summarize()	Predicted: describe (how many people) Gold: describe (how many people)	Predicted: there are two people in the video. Gold: hello, there are two men
2	are they carrying on a conversation between themselves?	Predicted: find (they) →summarize() Gold: find (they), find (themselves) →summarize()	Predicted: where (two men in the video) →when (carrying on a conversation between themselves) →exist() Gold: where (two men) → when (carrying a conversation between themselves) →exist()	Predicted: yes, they are carrying a conversation X Gold: no, background noise is birdsong and shuffling noises
3	what are they doing in the scene? are they on a rooftop?	Predicted: find (they), find (the scene) →summarize() Gold: find (they) →summarize()	Predicted: where (two men in the video), where (the scene) → when (doing in the scene) →describe (what) X Gold: where (two men), where (rooftop) → when (doing in the scene) →describe (what)	Predicted: one is washing a chair and the other is taking pictures X Gold: yes, on a second floor roof deck, one man is washing a chair, another man is either filming or taking pictures
4	does he sit in the chair after washing it?	Predicted: find (he), find (the chair) →summarize() Gold: find (he), find (the chair) →summarize()	Predicted: where (one man in the video), where (a chair) → when (sit in the chair after washing it) →exist() Gold: where (one man), where (a chair) → when (sit in the chair after washing it) →exist()	Predicted: no, he does not sit in the chair Y Gold: no he does not, there is a pipe with water running all over
Video
(Sample Frame)	Dialog History and Question	Predicted Dialogue & Video Understanding Program	Predicted and Ground-truth Responses
A	Dialog History: ...what is going on when the video starts? a man is looking out a doorway...what is he doing in there then? he starts to wash dishes ...
Question: does he wipe them with a towel?	Dialogue Program (ViINMN): find(he), find(them) → summarize()
Video Program (ViINMN): where(a man), where(the dishes) → when(wipe them with a towel) → exist()	Predicted (ViINMN): no he doesn't wipe them with a towel.
✓
Predicted (Baseline): no he does not take a picture of the camera.
✗
Predicted (MTN): no he does not do anything with with hands.
✗
Gold: looks like he just uses his hands.
B	Dialog History: ...what is he reading?
He is reading a magazine and he is laughing...
Question: is he laughing because of the magazine?	Dialogue Program (ViINMN): find(he), find(the magazine) → summarize()
Dialogue Program (ViINMN): where(one young boy in the video), where(a magazine) → when (laughing because of the magazine) → exist()	Predicted (ViINMN): yes, he is laughing at the magazine.
✓
Predicted (Baseline): yes he is talking to the camera.
✗
Predicted (MTN): no he is not laughing at the end of the video.
✗
Gold: maybe, because then he throws the magazine aside.
C	Dialog History: ...how many people are in the video? There is one person in the video...
Question: in the door way to the next room, there is an object. What is that?	Dialogue Program (ViINMN): find(the room), find(the door) → summarize()
Dialogue Program (ViINMN): where(what room) → when (what is that) → exist()	Predicted (ViINMN): it looks like he is in a living room.
✗
Predicted (Baseline): i m not sure what it is.
✗
Predicted (MTN): he walks into the room.
✗
Gold: he went to the doorway for a vacuum.