Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
Authors: Hung Le
Nancy F. Chen
Steven Hoi
Venue: ICLR 2021 Poster (2021)
Keywords: video-grounded dialogues
reasoning paths
semantic graphs

Abstract: Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. In this paper, we propose a novel framework of Reasoning Paths in Dialogue Context (PDC). PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. PDC model then learns to predict reasoning paths over this semantic graph. Our path prediction model predicts a path from the current turn through past dialogue turns that contain additional visual cues to answer the current question. Our reasoning model sequentially processes both visual and textual information through this reasoning path and the propagated features are used to generate the answer. Our experimental results demonstrate the effectiveness of our method and provide additional insights on how models use semantic dependencies in a dialogue context to retrieve visual cues.

Published as a conference paper at ICLR 2021
LEARNING REASONING PATHS OVER SEMANTIC GRAPHS FOR VIDEO-GROUNDED DIALOGEES
Hung Le Nancy F. Chen', Steven C.H. Hoi
i Singapore Management University
hungle. 201 smm.. edu .g
& A*STAR, Institute for Infocomm Research
n nfychen stssssarr.. edu. sg
8 Salesforce Research Asia
shoi@salesforeee
ABSTRACT
Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi- turn setting. Previous approaches to geleo-grounde dialogues mostly use dialogue context simple text input without modelling the inherent information flows at the turn level. In this paper, we propose novel framework of Reasoning Paths Dialogue Context (PDC). PDC model discovers information flows among dialogue turns through a semantic graph constructed based on lexical components each question and answer. PDC model then learns predict reasoning paths over this semantic graph. Our path prediction model predicts a path from the current turn through past dialogue turns that contain additional visual cues to answer the current question. Our reasoning model sequentially processes both visual and textual information through this reasoning path and the propagated features used generate the answer. Our experimental results demonstrate the effectiveness of our method and provide additional insights on how models use semantic dependencies in a dialogue context to retrieve visual cues.
INTRODUCTION
Traditional visual question answering (Antol al. 2015 Jang et al. 2017) involves answering questions about a given image. Extending from this line of research, recently Das et al. (2017); Alamri et al. (2019) add another level of complexity by positioning each question and answer pair in multi-turn conversational setting (See Figure I for example). This line of research has promising applications to improve virtual intelligent assistants in multi-modal scenarios (e.g. assistants for people with visual impairment). Most state-of ofte--of-the-pin the-part approaches in this line of research (Kang ct al. 2019; Schwwrtz et al., 2019b Le et al.) 2019) tackle the additional complexity in the multi-turn setting by learning to process dialogue context sequentially turn by turn. Despite the success of these approaches, they often fail to exploit the dependencies between dialogue turns long distance, e.g. the 2nd and 5th turns in Figure | long dialogues. this shortcoming becomes more obvious and necessitates an approach for learning long-distance dependencies between dialogue turns.
To reason over dialogue context with long-distance dependencies, recent research in dialogues discovers graph-based structures at the turn level to predict the speaker's emotion (Ghosal et al. 2019) or generate sequential questions aaor--auoorrrsssiveey (Chai & Wan 22200). Recently Zheng et al. (2019) incorporate graph neural models to connect the textual cues between all pairs of dialogue turns. These methods, however, involve fixed graphical structure of dialogue turns, in which only small number of nodes contains lexical overlap with the question of the current turn, e.g. the 1st 3rd and 5th turns in Figure 1 These methods also fail to factor in the temporality of dialogue turns as the graph structures do not guarantee the sequential ordering among turns. In this paper, propose a novel framework of Reasoning Paths in Dialogue Context (PDC). PDC model learns a reasoning path that traverses through dialogue turns to propagate contextual cues that are densely related to the semantics of the current questions. Our approach balances between a sequential and graphical process to exploit dialogue information.

---

Published as a conference paper at ICLR 2021
Sequential propegasion
Is just one person in video There one visible porson, yes.
(2) Graph-based propagation
C Quaff is he carrying his hand ? he is lokinng down at his calpponn and while walking forward
room.
Q there any nokee the video No theme nokee the video
can you watching video on phone A:l
he's watching
he walks into lable from not paying attendon.
5
C dosis walk back and fn the viceo?
Path based propagation
A: he walks lowards the book of the living room and walks rIGh into the table
Figure I: Sequential reasoning approaches fail to detect long-distance dependencies hetween the current turn and the 2nd turn. Graph-based reasoning approaches signals from all turns are directly forwarded to the current turn but the 1s and 3rd contain little lexical overlap to the current question.
Our work is related to the long-studied research domain of discourse structures, e.g. (Barzilay & Lapata [008) Feng & Hirst, 2011; Tan et al. 2016; Habernal & Gurevych] 2017). A form of discourse structure argument structures, including premises and claims and their relations. Argument structures have been studied to assess different characteristics in text, such as coherence, persuasiveness, and susceptibility to attack. However, most efforts are designed for discourse study in monologues and much less attention is directed towards conversational data. In this work, we investigate a form of discourse structure through semantic graphs built upon the overlap of component representations among dialogue turns. We further enhance the models with a reasoning path learning model to learn the best information path for the next utterance generation.
To learn reasoning path, we incorporate method with bridge entities, concept often seen in reading comprehension research, and earlier used in entity- discourse analysis (Barzilay & Lapata 2008). In reading comprehension problems, bridge entities denote entities that common between two knowledge bases e.g. Wikipedia paragraphs in HotpotQA (Yang et al., 2018b). In discourse analysis, entities and their locations in text are used learn linguistic patterns that indicate certain qualities of document. In our method, we first reconstruct cach dialogue turn (including question and answer) into set of component sub-nodes (e.g. entities, action phrases) using common syntactical dependency parsers. Each result dialogue turn contains sub-nodes that can be used as bridge entities. Our reasoning path learning approach contains phases: (1) first, at each dialogue turn, a graph network is constructed at the turn level. Any two turns are connected if they have an overlapping sub-node or if two of their sub-nodes are semantically similar. (2) secondly, a path generator is trained to predict a path from the current dialogue turn to past dialogue turns that provide additional and relevant cues to answer the current question. The predicted path is used as a skeleton layout propagate visual features through each step of the path.
Specifically, in PDC, we adopt non parammetttteeee approaches (e.g. cosine similarity) to construct the edges in graph networks and each sub-node is represented by pre-trained word embedding vectors. Our path generator is a transformer decoder that regressively generates the next turn index conditioned on the previously generated turn sequence. Our reasoning model is combination of vanilla graph convolutional network (Kipf & Welling 2017) and transformer encoder (Vaswani et al., (017). In each traversing step, we retrieve visual features conditioned by the corresponding dialogue turn and propagate the features to the next step. Finally, the propagated multimodal features are used as input transformer decoder to predict the answer.
Our experimental results show that our method can improve the results on the Audio- Visual Scene- Aware Dialogues (AVSD) generation settings (Alamri et al., (019)), outperform previous of the-art methods. We evaluate our approach through comprehensive ablation analysis and qualitative study. PDC model also provides additional insights on how the inherent contextual cues in dialogue context are learned in neural networks in the form of reasoning path.
2 RELATED WORK
Discourses in monologues. Related to our work is the research of discourse structures. A long- studied line of research in this domain focuses argument mining to identify the structure of argument, claims and premises, and relations between them (Feng & Hirst 2011 Stab & Gurevych] Peldszus & Stede 2015 Persing & Ng 2016 Habernal & Gurevych 2017). More recently,
2

---

Published as a conference paper at ICLR 2021
Ghosh et al. (2016) Duthie & Budzynska 2018); Jiang al. (2019) propose to learn argument structures in student essays and official debates. In earlier approaches, Barzilay & Lapata (2008);
Lin et al. 201ID; p: Feng et al. (2014) study discourses to derive coherence assessment methods through entity-based representations of text. These approaches are proposed from linguistic theories surrounding entity patterns in discourses, i.e. how they are introduced and discussed (Grosz et al.)
1995). Guinaudeau& & Strube (2013): Putra & Tokunaga (2017) extend prior work with graphical structures in which sentence similarity is calculated based on semantic vectors representing those sentences. These lines of research show that studying discourse structures is useful in many tasks, such as document ranking and discrimination. However, most of these approaches are designed for monologues rather than dialogues.
Discourses in dialogues. More related to our problem setting discourse research on text in a multi-turn setting, MMrakami & Raymond (2010); Boltužić & Snajder (2014]; Swanson al. (2015); Tan et al. () Niculae et (2017); Morio & Fujita (2018); Chakrabarty et al. (2019) introduce new corpus and different methods to mine arguments in online discussion forums. Their models are trained to extract claims and premises in each user post and identify their relations between argument components in each pair of user posts. More recently, Li et al. (2020a); Joo et al. (2020) extend argument mining in online threads to identify attackability and persuasiveness in online posts.
Niculae
In this work, we address the problem of video-grounded dialogue, in which dialogue turns are often semantically connected by common grounding information source, video. In this task, a discourse-based approach enables dialogue models to learn to anticipate the upcoming textual information in future dialogue turns. However, directly applying prior work on discourse argument dialogues is not straightforward due to the inherent difference between online discussion posts and video-grounded dialogues. In video-grounded dialogues, the language is often closer to spoken language and there are fewer clear argument structures to be learned. Moreover, the presence of video necessitates the interaction between multiple modalities, text and vision. Incorporating traditional discourse structures to model cross cross-modalee cosssmtyyl interaction is not straightforward. In this work, we propose model dialogue context by using compositional graphical structures and constructing information traversal paths through dialogue turns.
structures into rideo-groundee
Graph-based dialogue models. Related our work is research study that investigates different types of graph structures in dialogue. Hu et al. (2019); Shi & Huang (2019): et al. (2020) address the "reply_to" _t_" relationship among multi-party dialogues through graph networks that incorporate conversational flows in comment threads on social networks, e.g. Reddit and Ubuntu IRC, and online games. Zheng et al. (2019) propose a fully connected graph structure at the turn level for visual dialogues. Concurrently, Ghosal et al. (2019) also propose a fully connected graph structure with heterogeneous edges to detect the emotion of participating speakers. All of these methods discover graph structures connecting pairs of dialogue turns of little lexical overlap, resulting in sub-optimal feature propagation. This drawback becomes more significant in question answering problems in multi-turn settings. Our approach constructs graph networks based on compositional similarities.
Reasoning path learning. Our method is also motivated by the recent research of machine reading comprehension, e.g. WikiHop (Welbl al. 2018) and HotpotQA (Yang et al.) 2018a). De Cao et al. (2019): Qiu et al. (2019) construct graph networks of supporting documents with entity nodes that are connected based on different kinds of relationships. Tu et al. (2019); Tang et al. (2020) enhance these methods with additional edges connecting output candidates and documents. Extended from these methods are path- bash-based approaches that learn to predict a reasoning path through supporting documents. Kundu et al. (2019); Asai et al. (2020) score and rank path candidates that connect entities question to the target answer. A common strategy among these methods the use of bridge entities. However, unlike reading comprehension, dialogues are normally not entity-centric and it is not trivial to directly adopt bridge entities into dialogue context.
Cross-modalis feature learning. Our work is related to study that integrates visual and linguistic information representation. A line of research in this domain is the problem of visual QA, e.g. (Minh Le et al. 2020 Gao al. 2019). Closer to method are methods that adopt compositionality in textual features. Specifically, Socher et al.] (2014) introduce image and language representation learning by detecting the component lexical parts in sentences and combining them with image features. The main difference between these approaches and our work is the study of cross-modalities in a multi-turn setting. Our approach directly tackles the embedded sequential order in dialogue utterances and examines how cross cross-modality features are passed from turn to turn.
3

---

Published as a conference paper at ICLR 2021
Answer: walks towards the back of the living room. and walks right into the table.
Answer Decoder
Hadden
living room
he
walk
Multimodal Reasoning Model (Section 2.4]
walk eround
PS
Compositional Semantic Graph (Bection 82)
walk
Reasoning Path Model (Suciion
laugh
down
look
back and forth...
eop
Dependency Trees
Dialogue- Vi-Vedd---dd Attention
Graph Adjacency
Dialogue Encoder
-Turni(t-4): ...he looking down cellphone laughing while walking around in a living room. ...Turnt TT-untt((t2))- ...he walks into a table from not paying attention...
Turn-to-Dialogue
Mean Pooling
Embedding Position Encoding
Turn Poeltion Embedding
just
Video Encoder
66
Pretrained Embedding
jaumee
Figure 2: An overview of our PDC method.
3 METHOD
To describe our PDC model, we introduce a new graph-based method (Section 3.2 graph structure connect turn-level representations in dialogue context based on their compositional semantics. The compositional semantics consists of sub-nodes detected through syntactical depen- dency parsing methods. We enhance our approach with a path-based propagation method (Section 3.3 to narrow down the contextual information that facilitates question answering of the current turn. Our approach integrates a strong strategy to model dialogue flows in the form of graphical and path-based information such that contextual linguistic information is exploited to propagate relevant visual features (Section 3.4) Figure 2 demonstrates an overview our method.
that constructs a
3.1 PROBLEM DEFINITION
The inputs question answering problem in multi-tur setting consist of dialogue D and the visual input of a video Z. Fach dialogue contains a sequence of dialogue turns, each of which is pair of question Q and answer A. At each dialogue turn f, denote the dialogue context C, as all previous dialogue turns C, {(Q, (=) Since it is positioned in dialogue, the question of turn & Qt might be dependent on a subset of the dialogue context C,. The output the answer of the current turn A.. Each textual component, i.e. Q and A, is represented as sequence of token or word indices (w, Im= [V), where T. is the sequence length and V is the vocabulary set. The objective of the task is the generation objective that output answers of the current turn:
A arg max P(A.II. 2): = arg max A,
II m=1
Pm 1mm 2):
(I)
A
3.2 COMPOSITIONAL SEMANTIC GRAPH OF DIALOGUE CONTEXT
The semantic relations between dialogue turns are decomposed to semantic relations between sub- nodes that constitute each These composition relations as strong clues determine how dialogue turn related to another. We first employ a co-reference resolution system, e.g. (Clark & Manning 2016), to replace pronouns with the original entities. We then explore using the Stanford parser system! to discover sub-nodes. The parser decomposes each sentence into grammatical components, where word and its modifier are connected in a tree structure. For each dialogue turn, we concatenate the question and answer of that turn as input to the parser. The output dependency tree is pruned to remove unimportant constituents and merge adjacent nodes to form semantic unit.
v..9.2 retrieved wwwpttttp////m. //tpp///ppp///// htw.:/ppsttaaoodeedd//////eeeeee wawpaaaaeeeeeeeereeeeeeaea shtml

---

Published as a conference paper at ICLR 2021
A graph structure & is then constructed. Any two turms connected if one of their corresponding sub-nodes are semantically similar. To calculate the similarity score, we obtain their pre-trained word2vee embedding:" and compute the cosine similarity score. Algorithm provides the details of the procedure to automatically construct a semantic graph. Note that our approach can also be applied with other co-reference resolution systems, parser, pre-trained embeddings. Unlike graph structures in machine reading comprehension such as Wikipedia graph, the semantic graph G is not fixed throughout the sample population but is constructed for each dialogue and at each turn.
Algorithm 1: Compositional semantic graph dialogue context
Data: Dialogue context Ct, question of the current turn Q Result: Semantic graph g = -V,E) begin
T + - - 0 - - 0; S - 0;
H Coreferencee Citeerennc_Reoouutoo(((((. 2r)):
for each dialogue turn h E H do
Tí +- Merge Nodes(Prune Tree(Dependency Parse(h))); + TU {Tnl: [h +- E EU Position(h)), Turn Position(h))}
for each dependency tree T= = (Vr, ET) E do - SU( for each sub-node 8; E do for each sub-node 8, E S do if not In Same Turn(s,, sy) and Is Similar(s,, sy) then & ー & (Get Dial (s)n(s;) Get Dial Turn(s))) E E ( Dial Turn(s)). Get_Dial (s))
12
13 return g
3.3
LEARNING TO GENERATE REASONING PATHS
Our proposed compositional approach construct semantic graph in dialogue context ensures lexical overlaps with the question, but the graph structure does not guarantee the temporal order of dialogue turns. To ensure this sequential information is maintained, we train generator to predict reasoning paths that traverse through current dialogue turn to past dialogue turns.
We use a Transformer decoder to model the reasoning paths from the current turn t. The first position of the path, 2p is initialized with the turn-level position embedding of t. The next turn index is generated auto-regressively by conditioning on the previously generated path sequence:
Embed(t) E IR
Zo:m-1 1 ft; (m)))
(3)
where i; denotes a predicted dialogue turn index. The dialogue context and question of the current turn are represented by embedding vectors of their component tokens. Following Vaswani et al. (017)) their representations are enhanced with the sine-cosine positional encoding PosEncode.
Q: Embed Q,) + PosEncode( Q+) E R xd
(4)
C (m,ed(C,) PosEncode((i) e IR
(5)
Note that the dialogue context representation C₁ is the embedding of dialogue turns up to the last turn /. 1, excluding answer embedding of the current turn At-
We denote Transformer attention block Transformer key, value). The path generator incorporates contextual information through attention layers on dialogue context and question.
(
path = Transfromer Zo:m Zo.m- Zomm-1) E pmdd
(6)
(2)
path = Transfromer (D()) path Qt, Q+) E Rmxd
(7)
(3)
path = Transfromer path: C₁ Ct) E m
(8)

---

Published as a conference paper at ICLR 2021
At the m-th decoding step (m 2 1), our model selects the next dialogue turn among the set of dialogue turns that are adjacent to one at (m - 1) decoding step in the semantic graph. This is enforced through masking the softmax output scores in which non-adjacent turn indices are assigned to a very low scalar Smasked. We denote the adjacency matrix of semantic graph 9 = (V. E) as a
square matrix A of size [V] x [V] where A., 1 if (1 & and Ai,; = 1A4 1,,., [V. The
probability of decoded turns at the m-th decoding step is:
Pm = softmax (D) path,m W path.) E IR. VI, Pm. $masked Vil Apm-1,
where W path E Idd The decoding process is terminated when the next decoded token is an [EOP] (end-of-path) token. During inference time, we adopt greedy decoding approach. Due to the small size of V, we found that greedy approach can perform as well as beam search methods. The computational cost of generating reasoning paths in dialogue context is, thus, only dependent on the average path length, which is bounded by the maximum number of dialogue turns.
Data Augmentation. We train our path generator in supervision manner. At each dialogue turn t with semantic graph G, we use graph traversal method, c.g. BFS, find all paths that start from the current turn to any past turn. We maintain the ground-truth paths with dialogue temporal order by keeping the dialogue turn index in path position m lower than the turn index in path position m 1. We also narrow down ground-truth paths based on their total lexical overlaps with the expected output answers. Using the dialogue in Figure [ as an example, using BFS results three potential path candidates: 5 - 4, 5 + 2, and 5 - + 2. We select 5 - 4 + 2 as the ground truth path because it can cover the most sub-nodes in the expected answers. two paths have the same number of lexical overlaps, we select one with shorter length. If two paths are equivalent,
randomly sample one path following uniform distribution at each training step. Ground-truth reasoning paths are added with [EOP] token the final position for termination condition. The objective train the path generator is the generation objective of reasoning path at each dialogue turn:
Lyath
R₂ arg max P(R, (C,, 26: arg max R.
II m=1
Pm (0.im/RR,1m-1..... 21;
(10)
R2
3.4 MULTIMODAL REASONING FROM REASONING PATHS
The graph structure G and generated path R₂ are used as layout propagate features of both textual and visual inputs. For each dialogue turn from V. we obtain the corresponding embeddings and apply mean pooling to get vector representation. We denote the turn-level representations of V as V E ddd/VV We use attention to retrieve the turn-dependent visual features from visual input.
M Trrnsformer IT) dV
(11)
where is two-dimensional feature representation of visual input I. We define new multi- modal graph based on semantic graph G: Grom = (Vmm, Emm) where Vmm = MM and edges (m, my) Emmvi Vi, j. ji E E. We employ a vanilla graph convolution network (Kipf & Welling 2017) to update turn-level multimodal representations through message passing along all edges.
ek
1 [
Σ myERA
Σ Eki m. g(ms, Ck. e)
(12)
where 2g is the set of adjacent nodes of my and f(.) and g(.) are non-linear layers, e.g. MLP and their inputs are just simply concatenated. To propagate features along reasoning path R., we utilize the updated tum-leeel multimodal representations M e [V] and traverse the path sequentially through the representation of the corresponding turn index I' in each traversing step. Specifically, We obtain G (m "... Lpan The traversing process can be done through a recurrent network or a transformer encoder.
T Transformer(G..o G. E IR Apash Xd
(13)
To incorporate propagated features into the target response, we adopt ofate-of-the-art decoder model from (Le et al.) 2019) that exploits multimodal attention over contextual features. Specifically, We integrate both M and G at each response decoding step through two separate attention layers.

---

Published as a conference paper at ICLR 2021
Models
H-1
B-2
B-3
B-4
R
Baseline (Hori et al.
2019)
0,621
0.480
0.379
0,305
0.217
0.481
0.733
TopicEmb Kumar al. 2019
0.632
0.499
0.402
0.329
0.223
0.488
0.762
FGA (Schwartz al. 2019b]
0.806
JMAN CChu el al. 2020)
0.667
0.521
0.413
0.334
0.239
0.533
0.941
FA+HRED Nuuenn al.
2019
0,695
0,533
0.444
0.360
0.249
0.544
0.997
MTN (e al.
2019)
0.715
0.581
0.476
0.392
0,269
0.559
1.066
VideoS um
Sanabria et al.
2019
0.718
0.584
0,478
0.394
0,267
0.563
1.094
MSTN Lee el al. 2020)
0.377
0.275
0.566
1.115
Student-Teacher Hori et al.
2019
0.727
0.593
0.488
0.405
0.273
0.566
1.118
PDC (Ours)
0.747
0.616
0.512
0.429
0.282
0.579
1.194
VidcoSum Sanabria et al.
2019
0.723
0.586
0.476
0.387
0,266
0,564
1.087
VGD-GPT2 Le & Hoi 2020 13"
0.749
0.620
0.520
0.436
0.282
0.582
1.194
RLM (Li et 20206)
0.765
0.643
0.543
0.459
0.294
0.606
1.308
PDC (Ours) GPT2
0.770
0.653
0.539
0.449
0.292
0.606
1.295
Table 1: AVSD@DSTE test results: uses visual features other than I3D, e.g. ResNeXt, scene graphs. t incorporates additional video background audio inputs. indicates finetuning methods additional data pre-trained language models. Metric notations: B-n: BLEU-n, M: METEOR, R: ROUGE-L. CIDEr.
Besides, we also experiment with integrating propagated features with decoder as Transformer language models. Transformer language models have shown impressive performance recently in generation tasks by transferring language representations pretrained in massive data (Radford et al.) 2019). To integrate, we simply concatenate M and G to the input sequence embeddings as input to language models, similar as
(Le Hoi 2020 et al., 20206).
Optimization, The multimodal reasoning model is learned jointly with other model components. All model parameters are optimized through the objectives from both Equation [ and TO, We use the standard cross-entropy loss which calculates the logarithm of each softmax score at each decoding position of Ae and R..
4 EXPERIMENTS
Dataset. We use the Audio-Visure Visual Sene-Aware Dialogue (AVSD) benchmark developed by Alamri ct al. (2019). The benchmark focuses on dialogues grounded on videos from the Charades dataset (Sigurdsson al. 2016). Each dialogue can have up 10 dialogue turns, which makes it appropriate choice to evaluate our approach reasoning paths over dialogue context. We used the standard visual features 13D to represent the video input. We experimented with the test splits used in the 7th Dialogue System Technology Challenge (DSTC7) (Yoshino et al. 2019) and DSTCS (Kim al. 2019). Please see the Appendix A for experimental setups.
Train
Val
Test@ DSTC7
Test DSTC8
#Dialogs /QQessions/Anwwers
7,659
1,787
1,710
1,710
153,180
35,740
13,490
18,810
#Words
1,450.754
339,006
110,252
162,226
Table 2: Dataset Summary of the AVSD benchmark with both test splits @ @DTCC7 and @DSTC8.
The dialogues in the AVSD benchmark focuses on question answering over multiple turns and entail less semantic variance than open-domain dialogues. Therefore, we report the objective scores, including BLEU (Papineni al., 2002). METEOR (Banerjee Lavie 2005), ROUGE-L (Lin) 2004), and CIDEr /Vedantam et al. (215)) which are found to have strong correlation with human subjective scores (Alamri et al.) 2019). D. In Table D and 3 we present the test results our models in comparison with previous models in DSTC7 and DSTC8 respectively. In both test splits, our models achieve very strong performance against models without using pre-trained language models. Comparing with models using pre- pre-trained models and additional fine-tuning, our models achieve competitive performances both test splits. The performance gain of our models when using GPT2 indicates current model sensitivity to language modelling generator. A unique benefit of
Overall Results.
7

---

Published as a conference paper at ICLR 2021
Models
B-1
H-2
B-3
B-4
M
Baseline Hori et al.
2019
0.614
0.467
0.365
0.289
0.210
0,480
0.651
DMN Xie & lacobacci
2020
0.296
0.214
0.496
0.761
Simple
Schwartz el al.
2019
0.311
0.224
0,502
0.766
JMAN Chuu ct al 2020
0.645
0.504
0.402
0.324
0.232
0,521
0.875
STSGR (Geng et a.. 22220
0.357
0.267
0,553
1.004
MSTN (Le et al.
2020
0.385
0.270
0,564
1.073
PDC(OUrr)
0.723
0.595
0.493
0.410
0.270
0,570
1.105
RLM (Li et al. 2020b 13
0.746
0.626
0.528
0.445
0.286
0.598
1.240
PDC (Ours)+ + GPT2
0.749
0.629
0.528
0,439
0.285
0,592
1.201
Table 3: WNSDDDSTCS test results: uses visual features other than I3D, e.g. ResNeXt, scene graphs. t incorporates additional video background audio inputs. indicates finetuning methods on additional data or pre-trained language models. Metric notations: B-n: BLEU-n, M: METEOR, R: ROUGE-I, C: CIDEr.
Semanties
Direction
GraphProp
PathProp
B-1
B-2
B-3
B-4
M
R
Comp.
BiDirect
0.747
0.616
0.512
0.429
0.282
0.579
1.194
Comp.
BiDireet
0.746
0.611
0.503
0.418
0.281
0.580
1.179
Comp.
TODirect
0.748
0.613
0.505
0.420
0.279
0.579
1.181
Comp.
BiDirect
0.745
0.611
0.504
0.419
0.282
0.577
1.172
Global
BiDirect
0.743
0.609
0.504
0.421
0.279
0.579
1.178
Global
BiDirect
0.744
0.610
0.502
0.416
0.280
0.577
1.169
Global
TODirect
0.743
0.609
0.501
0.416
0.279
0.579
1.161
Global
BiDirect
0.749
0.613
0.505
0.421
0.279
0.578
1.172
Fully
BiDirect
0.745
0.607
0.500
0.414
0.277
0.576
1.169
Fully
TODirect
0.743
0.605
0.497
0.411
0.277
0.573
1.163
Table 4: Ablation of AVSD DSTC7 test results: We experiment with graphs that are compositional semantics, global semantics, and fully-com colle-conee with bidirectional or temporally ordered edges, and with graph-based or path-based feature propagation. Metric notations: B-n: BLEU-n, M: METEOR, R: ROUGE-L, C: CIDEr
models from prior approaches is the insights of how the models exploit information from dialogue turns in the form of reasoning paths (Please see example outputs in Figure 3
Ablation Analysis. In Table 4 we report the results of path learning in global semantic graph. In these graphs, we do not decompose each dialogue turn into component sub-nodes (line in Algorithm D but directly compute the similarity score based on the whole sentence embedding. In this case, to train the path generator, we obtain the ground-truth path by using BFS to traverse to the node with the most sentence-level similarity score to the expected answer. We observe that: (1) models that learn paths based on component lexical overlaps results in better performance than paths based on global lexical overlaps in most of the objective metrics. (2) Propagation by reasoning path alone without using GCN does not result in better performance. This can be explained as the information in each traversal step is not independent but still contains semantic dependencies other turns. It is different from standard reading comprehension problems where each knowledge base is independent and it is not required to propagate features through a graph structure to obtain contextual updates. Please the Appendix B for additional analysis of Table 4
Impacts of Reasoning Path Learning. We compare models that can learn reasoning paths against those that use a fixed propagation path through the past dialogue turns. From Tā we observe that: (1) learning dynamic mnssd reasoning paths outperforms all models that propagate through default path. This is achieved by using the reasoning path as a skeleton for feature propagation as well as adopting the joint training strategy. We can consider dynamically learned paths as an ideal traversal path to propagate visual cues among all possible paths within the semantic graph of the dialogue context. (2) our path generator can generate reasoning paths well and the model with learned paths can perform well as using the oracle paths. (3) due to the short length of reasoning paths (limited by the maximum dialogue length), either beam search or greedy decoding approach is good enough to generate paths. The greedy approach has the advantage of much lower computational cost.
Qualitative Analysis. In Figure 3 we demonstrate some examples of our predicted responses and the corresponding reasoning paths. Specifically, we showcase samples in which the reasoning paths are 2-hops (Example A and B) and 3-hops (Example C and D), and the distance in each hop can be over one dialogue turn (Example B and D) or more (Fxample A and C). The example reasoning paths
8

---

Published as a conference paper at ICLR 2021
Reasoning Path
B-1
B-2
B-3
B-4
M
Learned Path (beam search)
0.747
0.616
0.512
0.429
0.282
0.579
1.194
Learned Path (greedy)
0.747
0.616
0.512
0.430
0.282
0.580
1.195
Oracle Path
0.748
0.617
0.512
0.430
0.282
0.580
1.195
Random Path
0.552
0.437
0.345
0.274
0.194
0.420
0.684
Path through last 10 turns
0.744
0.607
0,500
0.415
0.278
0.576
1.166
Path through last 9 turns
0.743
0.607
0.500
0,416
0.277
0.574
1.161
Path through last 8 turns
0.749
0.615
0.509
0.423
0.277
0.578
1.168
Path through last 7 turns
0.754
0.618
0.510
0.422
0.282
0.579
1.170
Path through last 6 turns
0.746
0.608
0.498
0.412
0.278
0.575
1.150
Path through last S turns
0.744
0.607
0,500
0.415
0.278
0.576
1.169
Path through last 4 turns
0.745
0.610
0.502
0,417
0.278
0.576
1.165
Path through last 3 turns
0.744
0.607
0,500
0,414
0.278
0.576
1.163
Path through 2 turns
0.748
0.615
0.508
0,423
0.278
0.579
1.171
Path through last I turns
0.740
0.603
0.494
0.408
0.276
0.575
1.149
Table 5: Comparison of AVSD@DSTC. test results between learned paths and paths sequences of the last turns: experiment with paths predicted by our path generator and paths as sequence the last turns, ie. (t, max O, t -n)). Metric notations: B-n: BLEU-n, M: METEOR, R: ROUGE-L, C: CIDEr.
what does lhe with the bag when she is done picking Irash A: sho laee the big and looka hersel in the mimor of the bwltroam
Q woat phects does be intereet with P.A. ha opene a dloset deng and bakon mit plattic bag.
shs jalew
any point no, video ends
Q. are therg www.hher.oii oliscta Ioi interacte with that you have not lold me about the only obaet the broom and standy hara
looking at the
wher- rolding hand old ouatioon
she holding
videcuends
10
dhorrdww the do with mushio
hold
Predicted path: 10 - 7
Predicted
Answer (predicted) the puts the outhion the bag
Answer (predicted) Answer (ground-truth) frooughou the whole viown ths Frnding the broom
hoks whole time
Answer (gr ground-tnuthy the puts it white bag. addition the custion sooms dusty bocause she sneozes.
whellimee the roam intillyy A Bi men and the momen both Intux the room f the first framo, but as-sson as Eatx oator the hatmezr the woman go0s pack the room. batoina Zie man alansiro the hulhott
this 1 soom 7 linng room
yes. waits patiently
Qr dosc he ook cameea be tooks at when be goos to Jum
houck
Predicted path: 10
Predicted
Answer (predicted) (ground-eruth) twendos mayoe quess
eary
out room. Answer (ground--nne durutn video the going through the door would first.
(prediesed):
Figure 3: Example outputs of reasoning paths and dialogue responses. We demonstrate 4 cases of reasoning paths with 2 to 3 hops and with varied distances between two ends of the reasoning path.
show to be able to connect a sequence of dialogue turns that are most relevant to questions of the current turn. For instance, in Example A, the reasoning path can connect the 7th and 9th turn the current turn as they contain lexical overlaps, i.e. "the bag", and "the cushion". The path skips the 8th turn which is not relevant to the current question. Likewise, in Example C. the path skips the 8th turns. All examples show that dialogue context can be used to extract additional visual clues relevant to the current turn. Information from dialogues, thus, deserves more attention than just being used as background text input. Please see the Appendix for additional analysis.
CONCLUSION
We proposed PDC, novel approach to learning a reasoning path over dialogue turns for video- grounded dialogues. Our approach exploits the compositional semantics in each dialogue turn to construct a semantic graph, which is then used to derive optimal path for feature propagation. Our experiments demonstrate that our model can learn retrieve paths that are most relevant the current question. We hope our approach can motivate further study to investigate reasoning over multiple turns, especially in complex settings with interconnected dialogue flows (Sun al. 2019)

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank all reviewers for their insightful feedback on the manuscript of this paper. The first author of this paper is supported by the Agency for Science, Technology and Research (A *SAAR) Computing and Information Science scholarship.
REFERENCES
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. Audio-visual scene- aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrencc Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Akari Asai, Kazuma Hashimoto, Hannanch Hajishirzi, Richard Socher, and Caiming Xiong. Learn- ing to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations, 2020. / www.:enetffoumm
id (d-SJggHHrYDDi
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Regina Barzilay and Mirella Lapata. Modeling local coherence: An entity- entity-based approach. Computational Linguistics, 34(1):1-34, 2008. doi: 10.1162/coli.2008.4411 3.1 URL. https
7/doi.org/10. 116.///011.1..0..///
2008.34.11
Filip Boltužić and Jan Snajder. Back up your stance: Recognizing arguments in online discussions.
In Proceedings of the First Workshop on Argumentation Mining, pp. 49-58, 2014.
Zi Chai and Xiaojun Wan. Learning to ask more: Semiiauttorerresivve sequential question generation under dual- -uaph interaction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 225-237, Online, July 2020. Association for Computational Linguis-
tics. doi: 18.18/ 122553/w11220...mmaia.... URL https: wwww ac web. org/ antho logy 2020. acl ma mwin....n.....m
Tuhin Chakrabarty; Christopher Hidey, Smaranda Muresan, Kathy McKeown, and Alyssa Hwang. AMPERSAND: argument mining for persuasive online discussions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pp. 2933-2943, Association for Computational Linguistics, 2019. doi: 10.18653/w1/D199 12211 URL https 1/40l.oog/10.11555/01100/0
Yun-Wei Chu, Kuan- Yen Lin, Chao-Chun Hsu, and Lun- Wei Ku. Multi-step joint- modality attention network for scene-aware dialogue system. DSTC Workshop @ AAAI, 2020.
Kevin Clark and Christopher D. Manning. Deep reinforcement learning for contion-ranking corefer- ence models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2262,, Austin, Texas, November 2016. Association for Computational Lin- guistics. doi: 10.18653/v1/D16-1122 1245. URL https: www. ac www.ww.wwbb......... org/
anthology/ wwt..e/oooo..ttoooogooo/o//oo///
D16 216-1245
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruy Batra. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 326-335. 2017.
10

---

Published as a conference paper at ICLR 2021
Nicola De Cao. Wilker Aziz, and Ivan Titov. Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Logg and Short Papers), pp. 2306-2317, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 1118665/11/119-- 1240. URLhtttss: www ac wwtp://wwwww.www...rog.....
anthology/N19 1240
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Associa- lion for Computational Linguistics, pp. 2694-2703, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/119-11 1259, anthol anthology/P1 1259.
/www www.ww.www............. org/
Rory Duthie and Katarzyna Budzynska. A deep modular rnn approach for ethos mining. In
Proceedings of the Twenty-Seventh International Joint Conference Artificial Intelligence, IJCAI- 18, pp. 4041-4047. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 2004966/jaaii22118/66666.
otg/ .2.00.24999 249 //tee/ijaaaaa//// .ataa.221180 2018
Vanessa Wei Feng and Graeme Hirst. Classifying arguments by scheme. In Proceedings of the 49th anmual meeting of the association for computational linguistics: Human language technologies, pp. 987-99899999 2011.
Vanessa Wei Feng, Ziheng Lin, and Graeme Hirst. The impact of deep hierarchical discourse structures in the evaluation of text coherence. In Proceedings of COLING 2014, the 25th International Conference Computational Linguisties: Technical Papers, pp. 940-949, 2014.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6639-6648, 2019.
Shijie Geng, Peng Gao, Chiori Hori, Jonathan Le Roux, and Anoop Cherian. Spatio-temporal scene graphs for video dialog. arXiv preprint arXiv:2007.03844 2020.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. DialogueGCN: A graph convolutional neural network for emotion recognition conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), pp- 154-164, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D11-11
URL https::///// www. a. web. http://wwww.l.lll..o.//tatttaoo/
1015
Debanjan Ghosh, Aquila Khanam, Yubo Han, and Smaranda Muresan. Coarse- grasse-graindd argumentation features for scoring persuasive essays. In Proceedings of the 54th Annual Meeting the Association the
for Computational Linguistics (Volume 2: Short Papers), pp. 549-554, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 6-2299 URL https
www.aclwve www............/////ttooooooooo. htw.o//www..ooo//aaaaooooo///-/a
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Barbara Grosz, Scott Weinstein, and Aravind K. Joshi. Centering: A framework for modeling the local coherence of discourse. Comput. Linguist., 21(2):203-23 June 1995. ISSN 0891-2017.
Camille Guinaudeau and Michael Strube. Graph-based local coherence modeling. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 93- 93-103, 2013.
Ivan Habernal and Iryna Gurevych. Argumentation mining in eser-generated web discourse. Compu- fational Linguistics, 125-)11 2017.
11

---

Published as a conference paper at ICLR 2021
C. Hori, H. Alamri, J. Wang, G. Wichern, T. Hori, A. Cherian, T. K. Marks, V. Cartillier, R. G. Lopes, A. Das, I. Essa, D. Batra, and D. Parikh. End-to-end audio visual scene-aware dialog using multimodal Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2352-2356, May 2019. doi: 10.1 (0000//CAASSP2118888888788
irention-baseer video features. In ICASSP 2019 2019 IEEE International
Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. Joint student-teacher learning for audio-visual scene-aware dialog. Proc. Interspeech 2019, pp. 1886-1890, 2019.
Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, and Rui Yan. Gsn: A graph- graph-structured network for multi-party dialogues. In Proceedings of the Twenty-Eighe Interna- tional Joint Conference on Artificial Intelligence, LSCAI- 19, pp. 5010-5016. International Joint Conferences on Artificial Intelligence Organization, 2019, doi: 10.24963/jcca..0199/66 URL 2019/696
https: orgg 21..0.....1111
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2758-2766, 2017.
Shiyan Jiang, Kexin Yang, Chandrakumari Suvarna, Pooja Casula, Mingtong Zhang, and Carolyn Rosé. Applying Rhetorical Structure Theory to student essays for providing automated writing feedback. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019. pp. 163-168, Minneapolis, MN, June 2019. Association for Computational Linguistics. doi: N0.118633/1/WW19-277 URL htt.tts:::/ www. acl web. org www.:/ee....//hoooooooeooo
27220
Yohan Jo, Seojin Bang, Emaad Manzoor, Eduard Hovy, and Chris Reed. Detecting attackable sentences in arguments. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1-23, Online, November 2020. Association for Computational Linguistics. URL https::///// www. ac www.wwww.bbb......... wtt.eee..rggggttooogggogoeo/e///
ant http:/annoooooog///ooo emnlp main.
Gi-Cheon Kang, Jaeseo Lim, and Byoung- Tak Zhang. Dual attention networks for visual reference resolution in visual dialog. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2024-2033, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (t.18653/v1/D199 1209. URL https: //ww www. ac lwe
org/ wwwthhoogy/DDS1300
Scokhwan Kim, Michel Galley, Chulaka Gunasekara. Sungjin Lee, Adam Atkinson, Baolin Peng, Hannes Schulz, Jianfeng Gao, Jinchao Li, Mahmoud Adada, et al. The eighth dialog system technology challenge. arXiv preprint arXiv: 10911.0633 1,06394, 2019.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, and Lama Nachman. Leveraging topics and audio features with multimodal attention for audio visual scene-aware dialog. 3rd Visually Grounded Interaction and Language (ViGIL) Workshop, NeurIPS, 2019.
Souvik Kundu, Tushar Khot, Ashish Sabharwal, and Peter Clark. Exploiting explicit paths for multi-hop reading comprehension. In Proceedings of the 57th Annual Meeting of the Associa- fion for Computational Linguisties, pp. Computational Linguistics. doi: 10.18653/w1/P19- 1263. URL httthttp:://// hottps::/www.... http:://wwwwwwwww..../ org/
Florence, Italy, July 2019. Association for
lantholoy/PP9- 1263
Hung Le and Steven C.H. Hoi. Video-grounde dialogues with pretrained generation language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5842-5848, Online, July 2020. Association for Computational Linguistics. doi: 10. w..18653/v120020 acl-main.518, URL httpt:::p/// / www. web. wwa.boorg/attttt wgwp/yy2g220/0 -m 518
12

---

Published as a conference paper at ICLR 2021
Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. Multimodal transformer networks for end- to-end Aido-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5612-5623, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18533/11/19- 1564. URL https: / /ww.. ac web
brg/anthology/ httplolyy//p///oog/
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. Dstc8-avsd: Multimodal semantic transformer network with retrieval style word generator. DSTC Workshop @ AAAI 2020, 2020.
Jialu Li, Esin Durmus, and Claire Cardie. Exploring the role of argument structure in online debate persuasion. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 8905 8912, Online, November 2020a. Association for Computational Linguistics. URL https: / www. ac htwpeb...ogaaattooogg//////////
emnlp main. 716
Zekang Li, Zongjia Li, Jinchao Zhang. Yang Feng. Cheng Niu, and Jie Zhou. Bridging text and video: universal multimodal transformer for video-audio scene-aware dialog. DSTC Workshop @ AAAI, 2020b.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Sunmarization Branches Out, 2004.
Ziheng Lin, Hwee Tou Ng, and Min- Yen Kan. Automatically evaluating text coherence using dis- course relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 997-1006, 2011.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Dynamic language binding in relational visual reasoning. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth Inter- national Joint Conference on Artificial Intelligence, IJCAI-20, pp. 818-824. International Joint Conferences Artificial Intelligence Organization, 7 2020. doi: 10.24963/ijcai.22200 2020/1 14. URL 2020/114
Main track.
Gaku Morio and Katsuhide Fujita. End-to-end argument mining for discussion threads based par- allel constrained pointer architecture. Noam Slonim and Ranit Aharonov (eds.), Proceedings of the 5th Workshop Argument Mining, ArgMining @ EMNLP 2018, Brussels, Belgium, November 2018, pp. 11-21. Association for Computational Linguistics, 2018. doi: 10. 11111//////111-555 URL https: /ttp///oiioooo http///oo.orgg//// wttpr///.1.1113/1////o//.o
5202
Akiko Murakami and Rudy Raymond. Support or oppose? classifying positions in online debates from reply activities and opinion expressions. In Coling 2010: Posters, pp. 869-8768 1199-87,225 2010.
Dat Tien Nguyen, Shikhar Sharma, Hannes Schulz, and Layla Asri. From film video: Multi-turn question answering with multi-modal context. In AAAI 2019 Dialog System Technology Challenge (DSTC7) Workshop, 2019.
Vlad Niculae, Joonsuk Park, and Claire Cardie. Argument mining with structured SVMs and RNNs. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 985-995, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/11866/111111100111 URL https::////. www. ac 1 web. org/ hrg/aante anthology P17-109 099
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. Association for Computational Linguisties, 2002.
Andreas Peldszus and Manfred Stede. Joint prediction in mst-style discourse parsing for argumenta- tion mining. In Proceedingy of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 938-948, 2015.
Isaac Persing and Vincent Ng. End-to-end argumentation mining in student essays. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1394, 2016.
13

---

Published as a conference paper at ICLR 2021
Jan Wira Gotama Putra and Takenobu Tokunaga. Evaluating text coherence based on semantic similarity graph. In Proceedings of TexIGraphs-11. 1:: the Workshop on Craph- based Methods for Natural Language Processing, pp. 76 76-85,22 2017.
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dynamically fused graph network for multi-hop reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6140 6150, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/w1/P19-1660 1617. URL https: www aclweb
org/antho hogp//tthooooyyPPooooooo
ogy /P 9-119-166 6
Alec Radford, Jeffree Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenA/ Blog, 1(8)99 2019.
Ramon Sanabria, Shruti Palaskar, and Florian Metze. Cmu sinbad's submission for the dstc7 avsd challenge. In DSTC7 at AAAI2019 workshop, volume 6, 2019.
Idan Schwartz, Alexander G Schwing, and Tamir Hazan. A simple baseline for audio-visual scene- aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12548- 12558, 2019a.
Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. Factor graph attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2039-2029200 2048, 2019b.
Zhouxing Shi and Minlie Huang. A deep sequential model for discourse parsing on multi-party dialogues. In AAAI, 2019.
Gunnar A Sigurdsson, Güi Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Crowdsoureing data collection for activity understanding. In European Conference on Computer Vision, pp. 510-526. Springer, 2016.
Hollywood in homes:
Richard Socher, Andrej Karpathy, Quoe V. Le, Christopher D. Manning, and Andrew Y. Y. Grounded compositional semantics for finding and describing images with sentences. Trunsactions of the doi: 10.1162/tacl_ Lattiaclla__011111 URL
Association for Computational Linguistics, 2:207-218, 2014.
https: /wwww ac web. org/ antho wwaethhooogy/yoooo 00tp//114-11111
Christian Stab and Iryna Gurevych. Identifying argumentative discourse structures in persuasive essays. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 46-56, 2014.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardic. Dream: A challenge data set and models for dialogue- based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217-231, 2019.
Reid Swanson, Brian Ecker, and Marilyn Walker. Argument mining: Extracting arguments from online dialogue. In Proceedings of the 16th Annual Meeting of the Special Interest Group Discourse and Dialogue, pp. 217-226, Prague, Czech Republic, September 2015. Association for
Computational Linguistics. doi: 10.18653/11WW15-4633 URL https:t:::/ www www.wb.www..oooo. org/
aantholoy/WIIL 211//115111 4631
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-288888 1-22866, 2016.
Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu- Mizil, and Lillian Lee. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pp. 613-624, 2016.
Zeyun Tang, Yongliang Shen, Xinyin Ma, Wei Xu, Jiale Yu, and Weiming Lu. Multi-hop reading comprehension across documents with path-based graph convolutional network. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3905-3911. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 01.24663/jcaii2222002/ URL https: / //di. wrtp/o.........3 ijcai. 2020/540 Main track.
14

---

Published as a conference paper at ICLR 2021
Ming Tu, Guangtao Wang, Jing Huang. Yun Tang. Xiaodong He, and Bowen Zhou. Multi-hop hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2004-27022211 13, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/P19-11 1260. URL htt.http:/// www c: web httpb//rrg/aaoooogggoooooo//o///
9- 9266
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1998-60006 Curran Associates, Inc., 2017. URLhttp: 7/papers Dips. wtt.pppcccpppp////ee//paapa / 7181 attention aa-annnss-l-------- you need. pdf
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Johannes Welbl, Pontus Stenetorp. and Sebastian Riedel. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguisticss, 6:287-302, 2018. doi: 10.1 10.1162/tacl_a a _01_0222l___000111..111111 CRLhttps: www acl web org/ anthologyy/Q1--100
Huiyuam Xie and Ignacio lacobacci. Audio visual scene-aware dialog system using dynamic memory networks, 02 2020.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing. pp. 2369-2380, Brussels. Belgium, October-Novembor 2018a. Association for Computational Linguistics. doi: 10.18653/v1DD18-112 1259. URL https: //tp:://wwww... www. acwweb..c.bb.... 118-1259
org/ anthology/
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018b.
Koichiro Yoshino, Chiori Hori, Julien Perez, Luis Fernando D' D'Haro, Lazaros Polymenakos, Chulaka Gunasekara, Walter S Lasecki, Jonathan K Kummerfeld. Michel Galley, Chris Brockett, al. Dialog system technology challenge 7. arXiv preprint arXiv:
2019.
Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. Reasoning visual dialogs with structural and partial observations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6669-6678. 2019.
Henghui Zhu, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. Who did they respond to? conversation structure modeling using masked hierarchical transformer. In The Phirty-Fourty AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9741 9748. AAAI Press, 2020. URL https: / aaai. org/ oj8 index php Taap/AAAI/aee / niew/6524
15

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP
We experiment with the Adam optimizer (Kingma & Ba 2015). The models are trained with a warm-up learning rate period of 5 epochs before the learning rate decays and the training finishes up to 50 epochs. The best model selected by the average loss in the validation set. All model parameters, except the decoder parameters when using pre-trained language models, are initialized with uniform distribution (Glorot & Bengio, 2010). The Transformer hyperr-aaaaeterss are fine-tuned by validation results over [128. 256), h h= (1. 4,8, 16), and a dropout rate from 0.1 to 0.5. Label smoothing (Szegedy et al. 2016) is applied on labels of A, (label smoothing does not help when optimizing over R2 as the labels are limited by the maximum length of dialogues, i.e. 10 in AVSD).
B IMPACTS OF COMPOSITIONAL SEMANTIC GRAPH
We experiment with model variants based on different types of graph structures. Specifically, we compare our compositional semantic graph against a graph built upon the turn-level global semantics. In these graphs, we do not decompose each dialogue turn into component sub-nodes (line 5 in Algorithm [I) but directly compute the similarity score based on the whole sentence embedding. We also experiment with a fully connected graph structure. In each graph structure, we experiment with temporally ordered edges (TODirect). This is enforce by adding a check whether Get Dial_TTarassss the results in Table 4 we observe that: (1) based on the CIDEr metric, the best performing graph structure is the compositional semantic graph while the global semantic graph and fully connected graph structure are almost equivalent. This is consistent with the previous insight in machine reading comprehension research that entity lexical overlaps between knowledge bases are often overlooked by global embeddings (Ding et al. (2019) and it is not reliable to construct a knowledge graph based on global representations alone. (2) regarding the direction of edges, bidirectional edges and temporally ordered edges perform similarly, indicating that processing dialogue turns following temporal orders provides enough information and backward processing is only supplementary.
Turn(s;) Turn(s)) > Get Dial_Turn(s) in line 11 and removing line 12 in Algorithm I From
C ADDITIONAL QUALITATIVE ANALYSIS
In Figure 4 demonstrate examples outputs of reasoning paths and dialogue responses and have the following observations:
For questions that do not involve actions and can be answered by single frame, there is typically no reasoning path, i.e. the path only includes the current turn (Example A and B). These questions are usually simple and they are rarely involved in multiple dialogue turns.
In many cases, the dialogue agent can predict an appropriate path but still not generate the correct answers (Example and G). These paths are able to connect turns that are most relevant to the current turns but these past turns do not contain contain very limited clues to the expected answers. For example, in Example F, the 2nd and 4th turn are linked by the lexical component for "the woman". However, they do not have useful information relevant to the current turn, i.c. her clothes.
Finally, our approach shows that the current benchmark, AVSD, typically contains one-hop (Example C, D. E) to two-hop (Example F, G, H) reasoning paths over dialogue context. We hope future dialogue benchmarks will factor in the complexity of dialogue context in terms of reasoning hops to facilitate better research of intelligent dialogue systems.
Discussion of failure cases. From the above observations, we identify the following scenarios that our models are susceptible to and propose potential directions for improvement.
Long complex utterances. One limitation of our methods is its dependence on syntactical parser methods to decompose a sentence into sub-nodes. In most dialogues, this problem is not too serious due to the short length of utterances, usually just single sentence. However, in cases that the utterance contains multiple sentences/clauses or exhibits usage of spoken language with loose linguistic syntax, the parser may fail to decompose it properly. For
16

---

Published as a conference paper at ICLR 2021
instance, in Example G in Figure 4 the ground-truth answer contains causality- based clause ("because")), ) making it harder to identify sub-nodes such as "sneeze" or "dusty".
Contextualized semantic similarity. Another area we can improve upon this method to inject some forms of sentence-level contextual cues into each sub-node to improve their semantic representations. For instance, hypothetical dialogue that involves question utterances such the 2nd turn in Example A and the 6th turn in Example E in Figure 4 our method might not detect the connection between these two as they do not have overlap component sub-nodes. However, they both related the audio aspect of the video and reasoning path between these two turns appropriate.
D STATISTICS OF LOCAL VS. GLOBAL SEMANTIC GRAPHS
In Table 6 report the statistics of graph structures constructed by local and global semantics in all data splits of the AVSD benchmark. We observe that constructing graphs with local semantics result in lower number of instances with no reasoning paths than making graphs with global semantics. This is due to compositionality in our method, resulting in higher lexical overlap between dialogue turns. With our method, the number of sub-nodes per dialogue turn more than on average, making it easier to connect dialogue turns. This also leads to a larger and more diverse set of reasoning paths for supervision learning. In local semantic graphs, the average number of reasoning paths per dialogue turn is 2 to 3 on average, higher than this number in global semantic graphs. Although method requires additional computational effort to constructing these graphs, it is scalable to the size of the dialogue, i.e. number of the dialogue turns. To efficiently construct these graphs in dialogue, the semantic graph of a dialogue turn can be built on top of the semantic graph the last turn. This done by simply adding the sub-nodes the last turn's 's semantic graph and defining new edges adjacent to these sub-nodes only. In this way, the complexity of our graph construction method is linear to the number of dialogue turns.
Local Semantic Graphs
Global Semantic Graphs
train
val
test
train
val
test
# sub-nodes total sub-nodes per dialogue turn
341,186
88,617
31,936
76,590
17,870
6,745
4.45
4.96
4.73
1.00
1.00
1.00
# edges in total # dialogue turns with no reasoning paths
619,428
147,588
32,765
270,356
63,141
40,470
27.704
6,420
3,048
35,048
8,152
3,696
# reasoning paths per dialogue avg. reasoning paths per dialogue turn
35
33
24
10
9
5
3.10
3.10
1.99
1.21
1.10
1.04
Table 6: Comparison of dialogue context graphs built by local semantics and global semantics.
17

---

Published as a conference paper at ICLR 2021
there just zow preson the video A: yeu just vounger box
balding
Q: there acurd video 2
and havr the boy buahh
Q what dising on the sodee" A: tas slarts nut laving on then gata up
La caadle Lond..
video ? only
the
the betrrom
is gay nisht A: appears be 2tx
Predicted path::
Answer (predicted) you opppaars bedrcom. Answer (ground-truthi. appeara la
many
Predicted
Answer (predicted) theme
the
Answer (ground--ruth) lnnko (Bvn them window the diorr
and
Q: what happens first in the video Ubiaid coutine autim
the firtt frame. but as poon as Ruzz lie halbaaz woran
beit ad comm leaxing this man stansing in the belliwe
happens
plastic
does the man Hol for her yos the man ealt patiently in hall for hor.
wh does the git de neet the takes ausa hercric
dhen women nntum your she mitumn back hanttan and Gives to the man
do Bay lelk to each otlerr ?.A. your, Tiex are speaking to each other as Sittz wnte dowm nataaay together.
Predicted path: -- Answer (predictedy puts shell Answer (gr (rouud--uth)
alis aossa Nim be lock, happons tha sail dowe
Q: here ancone the video Ac there 00.000 sisa shown the video
wno oover the housd housd tim??
Predicted path: --3
Answer (predicted) mat wndex
(ground truth): stops the going through
brakkt
he
dony jusl into room Bann cheil you.
is hhee
watchiin somathine
and
Q. what dies the woman A: there auvommn voice out don't
where the sound is caming from aounds
amotent sound
sbes
know. lon't
Qr the man the mom the whele time on the dumtoo.
dowe ever baok up
Predicted path:
Answer (predicted] musi not appni ot all. Answer (ground fruthy: no. doeen't speak at
Predicted path:
(predicted) does intintann bock
(ground-tnuth))
Q. looks like there is bas, whal dose slse da with A alte picka the Ireed from the trashcan and guts In the bog
Q: what is happoning the video A: woman deaning indwww....
whene is the ecomani located A: front of window
show any emotion
displays mostly emotion.
time day
during daytime
Q. dose shs look lowards the carrea no als does not look at the camera.
does the over spoak A: no. she does not speak.
done picking
bag
Q the room decorated A: no. |Ijust has plain white walls.
Q: cocs the joove the room any point A: no. the video ends with her
thene seything whow in the vidwo Eie wirdow
the women and
look iig ei the minor
adhat- riding
beginning video holling
what is Sit wearing
Predicted path: 7
Arsewer (predicted) weveing and peints
(ground wearing blue black pams, and
Predicted path: 10
Answer (predicted] the puts the cushion the bag.
Answer (ground-euthh: the outs I the white addition the oushion
swems lo be dusty becouse she smeizes.
Figure 4: Example outputs of reasoning paths and dialogue responses.
18


Reviews:
[{"review_id": "f-ovP7fU1O7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the problem of video-grounded multi-turn QA and adopts reasoning paths to exploit dialogue information.\n\nSequential: fail to exploit long turn dependencies\nGraphical: fixed structure, fail to factor temporal dependencies\nThe proposed reasoning path method: balanced between sequential and graphical\n\nIt first constructs a turn-level semantic graph based on overlapping lexical span:\n- Extract lexical spans from each turn (<Q, A> pair) using a (Stanford) parser\n- Two turns are connected if one of their corresponding lexical spans are similar (in terms of word2vec embedding).\n\nThen it trains a path generator to predict paths from each turn to its preceding turns:\n- It starts from the current turn and auto-regressively finds the most dependent preceding turn with Transformers\n- The turn-level semantic graph is used to mask the dependencies.\n- It is trained with supervised loss where the target paths are constructed by running BFS on the semantic graph\n\nFinally, the proposed paths are used to employ multimodal reasoning:\n- Visual features are combined with turn level attention\n- Multi-model turn-level embeddings are propagated using GCN\n- Then use SOTA decoder to generate language response\n\nThe author conducts experiments on a benchmark, and the proposed method achieves better QA performance than SOTA without a pre-trained language model and achieves comparable performance when the pre-trained language model is involved.\n\nThe author further studies different variations of graph structures and show that using graphs constructed based on lexical spans is better than fully connected graphs or graphs based on whole sentence embedding. And it also shows that including bidirectional edges does not necessarily improve the performance.\n\nA nice feature of the method is that the generated reasoning path can serve as extra explanations for the answer.\n\nSome concerns:\n\n- The model is graph-based thus is restricted to scenarios with a small number of turns, and becomes computationally expensive for long-conversation scenarios.\n- Need a more detailed explanation of how the message passing part (section 3.4) is trained.\n- Each pair of turns may share multiple pairs of lexical spans that are identical, e.g. Figure 3-A, “she” in turn 10, but there are 2 “she”s in turn 9. Does the frequency influence the similarity?\n- It would be more convincing if it gives an analysis of failure cases.\n- Section 3.3: Eq.(4), what is the initial $D_0$ correspond to $Z_0$?\n- The reasoning path generator uses $C_t$ as input, does it include $A_t$ during inference?\n\nMinor concerns:\n\n- Many symbols are used before their definition:\n    > Explanation of $\\mathcal{V}$ is first given in Algorithm 1 (section 3.2) but first used in Eq.1 (section 3.1).\n    > Section 3.3, 2nd paragraph, 4th line: undefined symbols $\\hat{r}_1,\\dots,\\hat{r}_{m-1}$. They are later mentioned as turn indices in section 3.4, last line of page 5.\n- Page 5, line 2: “incorporate” —> “incorporates”.\n- Index m is used as a word position in Eq.(1) but becomes a decoding step from section 3.3.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NUphSR-bqni", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper addresses the visual question answering in a multi-turn or conversational setting. Given a video (series of frames or images), a model has to reason across space and time to arrive at a correct answer for a given question. This task involves understanding the content and context of dialogue turns, i.e., given a question and N dialogue turns, only M<<N of the dialogue turns are strongly related to the question posed. This paper proposes to simulate the dependencies between dialogue turns, forming a reasoning path, to answer a given question. In a way, the proposed approach selects relevant dialogue turns that are useful to answer the question.  \n\nThere are two steps to make the reasoning path:\n(1) At each dialogue turn, a graph network is constructed at the turn level. Any two turns are connected if they have an overlapping lexical span or if their lexical spans are semantically similar.\n(2) Secondly, a path generator is trained to predict a path from the current dialogue turn to past dialogue turns that provide additional and relevant cues to answer the current question. \n\nUltimately, the main idea to create a reasoning path is based on compositional semantic similarities.\n\n\n\nComments (Technical, Major Flaws of this paper): \n\n(A) I am not sure whether the author(s) is aware, but from the NLP perspective, the current method (step 1) is trying to simulate the discourse structure of dialogues. I believe that this is an important direction, and the uniqueness of this works lies in the multi-modality of the input, i.e., possibility of the interplay between texts and images (using the information in both modalities). \n- The claimed novelty in this paper is in the construction or usage of reasoning graph, i.e., to construct a graph structure to connect turn-level representations in dialogue. However, in Step 1, the use of entity and/or compositional similarity to create a graph structure out of a text is not new at all, and the paper fails to cite related works, as if it is the first one to propose this. In fact, the idea has been used in NLP for a long time (albeit mostly in the monologue). \n- I am not sure whether combining entity with action phrases (called \"lexical spans\" in the paper) is new. Can you confirm whether the proposed \"lexical spans\" is indeed new to construct/simulate the discourse structure?\n- Regarding step 1, perhaps the main contribution of this paper is applying the idea to dialogues, instead of monologues? Another possible contribution is \"filtering out\" unimportant semantic relations. In normal discourse structure, all parts of texts are connected in a single structure. However, in the context of this paper, only edges that are relevant to the posed question are used. \n- Unless the paper can discuss the related NLP works for step 1, I can only treat this paper as the extension of the corresponding NLP method in a multi-modal setting. There is an engineering contribution, but not from the methodological (theoretical) perspective.\n- I think the author(s) will benefit much by surveying papers on discourse structures (or the ``shallow\" construction of them), instead of machine reading comprehension. Many studies tried to establish discourse structure (albeit in a monologue) using entities that are mentioned and their semantic representations. A few of such works are:\n  R. Barzilay and M. Lapatta. 2008. Modelling Local Coherence: An Entity-based Approach. https://www.aclweb.org/anthology/J08-1001.pdf\n  C. Guinaudeau and M. Strube. 2013. Graph-based Local Coherence Modeling. https://www.aclweb.org/anthology/P13-1010/\n  J.W.G. Putra and T. Tokunaga. 2017. http://www.aclweb.org/anthology/W/W17/W17-2410.pdf \n- The currently proposed method step 1 seems to be the combination of entity graph + semantic similarity graph in these related works, but the current paper \"filters\" only edges relevant to the posed question. \n- A related work to construct the discourse structure in dialogues:\n+ G. Morio and K. Fujita. 2018. End-to-end Argument Mining for Discussion Threads Based on Parallel Constrained Pointer Architecture. https://www.aclweb.org/anthology/W18-5202.pdf\n\n(B) The reasoning model, which is a combination of GCN + transformer can be interesting. However, the idea of cross-modality representation refinement is somewhat similar to what has been studied in VQA.\n  Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020). Dynamic Language Binding in Relational Visual Reasoning. In IJCAI 2020.\n  Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C., Wang, X., & Li, H. (2019). Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In CVPR 2019.\n\n(C) After constructing the reasoning path (in response to the given question), the next step is to decode such representation to generate the answer. This paper proposes to use the transformer model to do that. I believe the use of the transformer model to generate text is not new. In fact, the author(s) mentions this in the paper (the last paragraph of Section 3.4). \n\n(D) In overall, if we look at the pipeline (system) level, the proposed pipeline is new (the whole process). However, I seriously concern about the step (1) of the proposed method (page 1). My main concern about this paper is its lack of awareness of related works in text processing (step 1 of their method). In fact, it fails to cite relevant works (that are very similar to this work). I might appreciate this paper in terms of engineering contribution (in a multi-modal setting), but I cannot acknowledge that step 1 is novel. Having that said, I think the authors need to provide a comparison to related works, proving the novelty of the current method. I am willing to increase the rating if the authors can properly address my concerns during the rebuttal phase.\n\n(E) The content from 3.3 to 3.4 is very hard to follow. \nCorrection of terms:\n\t- linguistic dependency parsers --> \"syntactical\" dependency parsers (this is the correct term)\n\t- linguistically, the term \"lexical span\" is weird. A span is a series of continuous lexicons (in the text surface). I suggest using a better term, as the \"lexical span\" in this paper might be discontinuous (do I misunderstand?).", "extra_scores": {}, "extra_text": {}}, {"review_id": "ChZf8L9MBS_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes creating a semantic graph connecting the multiple turns in a dialogue and subsequently learning reasoning paths in that graph to find the most relevant nodes for answering a given question in a dialogue context.\n\nStrengths:\n* This paper proposes to learn non-linear information flows in a sequential data which is a well-motivated problem.\n* The proposed method is novel. Training a transformer decoder to learn reasoning paths and using BFS supervision to find the ground-truth paths is very interesting (however empirical support that this is effective is lacking, see Weaknesses below).\n* The proposed approach significantly and consistently outperforms existing benchmarks on the AVSD dataset.\n* The illustration in Fig 3 demonstrates the benefit of using the selected nodes from the graph.\n\nWeaknesses:\n* From the results in Table 1, 3, it is clear that multimodal reasoning over \"relevant\" nodes in the semantic graph (termed as reasoning path in the paper) helps the model by reducing the noise. However, whether there is any benefit to learning that path is unclear from the ablation study present in Table 5 where there is no noticeable difference in the results between learned v/s fixed paths. In fact, \"Path through last 7 turns\" performs comparable to the \"Learned Path\" which raises question about the usefulness of transformer decoder based path learner.\n* Similarly results in Table 4 seem within noise error from each other, hence making the arguments \"component lexical overlap is better than global lexical overlap\" weak.\n* It is unclear if this approach of creating the graph (by finding pairwise semantic similarity) is scalable to real world datasets containing several turns with potentially large number of tokens. There is no analysis of information redundancy among the nodes of the graph which can help prune the graph.\n* The results are only presented on one dataset. The transformer decoder learning seems to be tuned to this particular dataset (\"greedy approach works better due to small size of V\"). This questions the generality of the proposed approach.\n\nQuestions / Suggestions\n* It is unclear what (if) makes this method specific to video-grounded Q/A. Can this methodology be applied to other problems involving dialogues? Can this be applied to any problem containing time series data?\n* There needs to be some discussion as to why RLM performs better/comparable (Table 1) when using pretraining.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of visual question answering in multi-turn dialogues.
The proposed method is to identify relevant dialog turns as a path in a semantic graph that connects the dialogue turns. Empirical performance of the proposed method is strong. Reviewers concerns have been compressively addressed. Overall, the paper has novelty, and explores an interesting direction in this line of work.
Decision: Accept (Poster)