Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues

1 INTRODUCTION . Traditional visual question answering ( Antol et al. , 2015 ; Jang et al. , 2017 ) involves answering questions about a given image . Extending from this line of research , recently Das et al . ( 2017 ) ; Alamri et al . ( 2019 ) add another level of complexity by positioning each question and answer pair in a multi-turn or conversational setting ( See Figure 1 for an example ) . This line of research has promising applications to improve virtual intelligent assistants in multi-modal scenarios ( e.g . assistants for people with visual impairment ) . Most state-of-the-part approaches in this line of research ( Kang et al. , 2019 ; Schwartz et al. , 2019b ; Le et al. , 2019 ) tackle the additional complexity in the multi-turn setting by learning to process dialogue context sequentially turn by turn . Despite the success of these approaches , they often fail to exploit the dependencies between dialogue turns of long distance , e.g . the 2nd and 5th turns in Figure 1 . In long dialogues , this shortcoming becomes more obvious and necessitates an approach for learning long-distance dependencies between dialogue turns . To reason over dialogue context with long-distance dependencies , recent research in dialogues discovers graph-based structures at the turn level to predict the speaker ’ s emotion ( Ghosal et al. , 2019 ) or generate sequential questions semi-autoregressively ( Chai & Wan , 2020 ) . Recently Zheng et al . ( 2019 ) incorporate graph neural models to connect the textual cues between all pairs of dialogue turns . These methods , however , involve a fixed graphical structure of dialogue turns , in which only a small number of nodes contains lexical overlap with the question of the current turn , e.g . the 1st , 3rd , and 5th turns in Figure 1 . These methods also fail to factor in the temporality of dialogue turns as the graph structures do not guarantee the sequential ordering among turns . In this paper , we propose a novel framework of Reasoning Paths in Dialogue Context ( PDC ) . PDC model learns a reasoning path that traverses through dialogue turns to propagate contextual cues that are densely related to the semantics of the current questions . Our approach balances between a sequential and graphical process to exploit dialogue information . Our work is related to the long-studied research domain of discourse structures , e.g . ( Barzilay & Lapata , 2008 ; Feng & Hirst , 2011 ; Tan et al. , 2016 ; Habernal & Gurevych , 2017 ) . A form of discourse structure is argument structures , including premises and claims and their relations . Argument structures have been studied to assess different characteristics in text , such as coherence , persuasiveness , and susceptibility to attack . However , most efforts are designed for discourse study in monologues and much less attention is directed towards conversational data . In this work , we investigate a form of discourse structure through semantic graphs built upon the overlap of component representations among dialogue turns . We further enhance the models with a reasoning path learning model to learn the best information path for the next utterance generation . To learn a reasoning path , we incorporate our method with bridge entities , a concept often seen in reading comprehension research , and earlier used in entity-based discourse analysis ( Barzilay & Lapata , 2008 ) . In reading comprehension problems , bridge entities denote entities that are common between two knowledge bases e.g . Wikipedia paragraphs in HotpotQA ( Yang et al. , 2018b ) . In discourse analysis , entities and their locations in text are used to learn linguistic patterns that indicate certain qualities of a document . In our method , we first reconstruct each dialogue turn ( including question and answer ) into a set of component sub-nodes ( e.g . entities , action phrases ) using common syntactical dependency parsers . Each result dialogue turn contains sub-nodes that can be used as bridge entities . Our reasoning path learning approach contains 2 phases : ( 1 ) first , at each dialogue turn , a graph network is constructed at the turn level . Any two turns are connected if they have an overlapping sub-node or if two of their sub-nodes are semantically similar . ( 2 ) secondly , a path generator is trained to predict a path from the current dialogue turn to past dialogue turns that provide additional and relevant cues to answer the current question . The predicted path is used as a skeleton layout to propagate visual features through each step of the path . Specifically , in PDC , we adopt non-parameterized approaches ( e.g . cosine similarity ) to construct the edges in graph networks and each sub-node is represented by pre-trained word embedding vectors . Our path generator is a transformer decoder that regressively generates the next turn index conditioned on the previously generated turn sequence . Our reasoning model is a combination of a vanilla graph convolutional network ( Kipf & Welling , 2017 ) and transformer encoder ( Vaswani et al. , 2017 ) . In each traversing step , we retrieve visual features conditioned by the corresponding dialogue turn and propagate the features to the next step . Finally , the propagated multimodal features are used as input to a transformer decoder to predict the answer . Our experimental results show that our method can improve the results on the Audio-Visual SceneAware Dialogues ( AVSD ) generation settings ( Alamri et al. , 2019 ) , outperform previous state-ofthe-art methods . We evaluate our approach through comprehensive ablation analysis and qualitative study . PDC model also provides additional insights on how the inherent contextual cues in dialogue context are learned in neural networks in the form of a reasoning path . 2 RELATED WORK . Discourses in monologues . Related to our work is the research of discourse structures . A longstudied line of research in this domain focuses on argument mining to identify the structure of argument , claims and premises , and relations between them ( Feng & Hirst , 2011 ; Stab & Gurevych , 2014 ; Peldszus & Stede , 2015 ; Persing & Ng , 2016 ; Habernal & Gurevych , 2017 ) . More recently , Ghosh et al . ( 2016 ) ; Duthie & Budzynska ( 2018 ) ; Jiang et al . ( 2019 ) propose to learn argument structures in student essays and official debates . In earlier approaches , Barzilay & Lapata ( 2008 ) ; Lin et al . ( 2011 ) ; Feng et al . ( 2014 ) study discourses to derive coherence assessment methods through entity-based representations of text . These approaches are proposed from linguistic theories surrounding entity patterns in discourses , i.e . how they are introduced and discussed ( Grosz et al. , 1995 ) . Guinaudeau & Strube ( 2013 ) ; Putra & Tokunaga ( 2017 ) extend prior work with graphical structures in which sentence similarity is calculated based on semantic vectors representing those sentences . These lines of research show that studying discourse structures is useful in many tasks , such as document ranking and discrimination . However , most of these approaches are designed for monologues rather than dialogues . Discourses in dialogues . More related to our problem setting is discourse research on text in a multi-turn setting . Murakami & Raymond ( 2010 ) ; Boltužić & Šnajder ( 2014 ) ; Swanson et al . ( 2015 ) ; Tan et al . ( 2016 ) ; Niculae et al . ( 2017 ) ; Morio & Fujita ( 2018 ) ; Chakrabarty et al . ( 2019 ) introduce new corpus and different methods to mine arguments in online discussion forums . Their models are trained to extract claims and premises in each user post and identify their relations between argument components in each pair of user posts . More recently , Li et al . ( 2020a ) ; Jo et al . ( 2020 ) extend argument mining in online threads to identify attackability and persuasiveness in online posts . In this work , we address the problem of video-grounded dialogue , in which dialogue turns are often semantically connected by a common grounding information source , a video . In this task , a discourse-based approach enables dialogue models to learn to anticipate the upcoming textual information in future dialogue turns . However , directly applying prior work on discourse or argument structures into video-grounded dialogues is not straightforward due to the inherent difference between online discussion posts and video-grounded dialogues . In video-grounded dialogues , the language is often closer to spoken language and there are fewer clear argument structures to be learned . Moreover , the presence of video necessitates the interaction between multiple modalities , text and vision . Incorporating traditional discourse structures to model cross-modality interaction is not straightforward . In this work , we propose to model dialogue context by using compositional graphical structures and constructing information traversal paths through dialogue turns . Graph-based dialogue models . Related to our work is research study that investigates different types of graph structures in dialogue . Hu et al . ( 2019 ) ; Shi & Huang ( 2019 ) ; Zhu et al . ( 2020 ) address the “ reply_to ” relationship among multi-party dialogues through graph networks that incorporate conversational flows in comment threads on social networks , e.g . Reddit and Ubuntu IRC , and online games . Zheng et al . ( 2019 ) propose a fully connected graph structure at the turn level for visual dialogues . Concurrently , Ghosal et al . ( 2019 ) also propose a fully connected graph structure with heterogeneous edges to detect the emotion of participating speakers . All of these methods discover graph structures connecting pairs of dialogue turns of little lexical overlap , resulting in sub-optimal feature propagation . This drawback becomes more significant in question answering problems in multi-turn settings . Our approach constructs graph networks based on compositional similarities . Reasoning path learning . Our method is also motivated by the recent research of machine reading comprehension , e.g . WikiHop ( Welbl et al. , 2018 ) and HotpotQA ( Yang et al. , 2018a ) . De Cao et al . ( 2019 ) ; Qiu et al . ( 2019 ) construct graph networks of supporting documents with entity nodes that are connected based on different kinds of relationships . Tu et al . ( 2019 ) ; Tang et al . ( 2020 ) enhance these methods with additional edges connecting output candidates and documents . Extended from these methods are path-based approaches that learn to predict a reasoning path through supporting documents . Kundu et al . ( 2019 ) ; Asai et al . ( 2020 ) score and rank path candidates that connect entities in question to the target answer . A common strategy among these methods is the use of bridge entities . However , unlike reading comprehension , dialogues are normally not entity-centric and it is not trivial to directly adopt bridge entities into dialogue context . Cross-modality feature learning . Our work is related to study that integrates visual and linguistic information representation . A line of research in this domain is the problem of visual QA , e.g . ( Minh Le et al. , 2020 ; Gao et al. , 2019 ) . Closer to our method are methods that adopt compositionality in textual features . Specifically , Socher et al . ( 2014 ) introduce image and language representation learning by detecting the component lexical parts in sentences and combining them with image features . The main difference between these approaches and our work is the study of cross-modalities in a multi-turn setting . Our approach directly tackles the embedded sequential order in dialogue utterances and examines how cross-modality features are passed from turn to turn .

Summary: This paper addresses the visual question answering in a multi-turn or conversational setting. Given a video (series of frames or images), a model has to reason across space and time to arrive at a correct answer for a given question. This task involves understanding the content and context of dialogue turns, i.e., given a question and N dialogue turns, only M<<N of the dialogue turns are strongly related to the question posed. This paper proposes to simulate the dependencies between dialogue turns, forming a reasoning path, to answer a given question. In a way, the proposed approach selects relevant dialogue turns that are useful to answer the question.