AutoGraph: Enabling Visual Context via Graph Alignment in Open Domain Multi-Modal Dialogue Generation

Authors: deji  zhao
donghong  han
bo  ning
mengxiang  li
china  telecom
corp  ltd
shuangyong  song
ye  yuan
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 04:43:35

Abstract: Open-domain multi-modal dialogue system heavily relies on visual information to generate contextually relevant responses. The existing open-domain multi-modal dialog generation methods ignore the complementary relationship between multiple modalities, and are difficult to integrate with LLMs. To address these issues, we propose an automatically constructed visual context graph method, called AutoGraph. We aim to structure complex information and seamlessly integrate it with large language models (LLMs), aligning information from multiple modalities at both semantic and structural levels. Specifically, we fully connect the text graphs and scene graphs, and then trim unnecessary edges via LLMs to automatically construct a visual context graph. Next, we design several graph sampling grammar for the first time to convert graph structures into sequence which is suitable for LLMs. Finally, we propose a two-stage fine-tuning method to allow LLMs to understand graph sampling grammar and generate responses. The AutoGraph method is a general approach that can enhance the visual capabilities of LLMs. We validate our proposed method on text-based LLMs, and visual-based LLMs, respectively. Experimental results show that our proposed method achieves state-of-the-art performance on multiple public datasets.

Main paper: 
 introduction: 
 Open-domain multi-modal dialogue generation has garnered increased attention in recent years due to its ability to closely mimic real-life scenarios and generate contextually appropriate responses [8, 30] . Dialogue systems are no longer limited to textual forms, and visual information plays a crucial role in dialogue agent. Multimodal dialogue systems can comprehend not only textual but also other modal information to generate appropriate responses. This integration of multi-modal information into traditional text-based dialogue systems, known as open-domain multimodal dialogue systems, has attracted increasing research interest [22, 32, 35, 36, 52] .Unlike previous Visual Question Answering (VQA) tasks [1] that focus on a single, or small number of images related to the context, the open-domain multi-modal dialog generation task has a lot third-person viewpoint multi-modal information. Although existing models have shown promising performances, they still suffer from two problems. Firstly, recent image-grounded dialogue models [17, 33, 45] endeavor to enhance dialogue generation capabilities by integrating relevant images into the dialogue models. Video-grounded dialogue models [7, 19, 20] aim to combine video modal information with dialogue systems. These methods encode multi-modal infromation through different encoders respectively, while ignoring the complementary relationship between the different modalities. As shown in Figure 1 , if the visual contextual information is ignored, it is difficult to clarify the specific coreference relationship between 'this', 'it' and 'you'. Encoding visual context information separately involves a lot of noise. And it is difficult to map information from multiple modalities to a unified multi-modal vector space, which leads to the lack of factual and incoherent responses.Secondly, the multi-modal community has sparked new interest in enhancing Large Language Models (LLMs) with visual information [14] , with representative models such as LlaVA [20] , Vis-CPM [7] , and Monkey [16] . But existing image-grounded or videogrounded models' features are difficult to combine with LLMs due to the different vector spaces, which limits the ability of LLMs to uniformly model multiple modal features. It has been demonstrated that leveraging scene graphs can effectively enhance the understanding of image modalities [51] . The GlaMM model [31] try to push LLMs to generate scene graphs sentence to understand of the image. Structure-CLIP proposed by Huang et al. [9] employ scene graph knowledge to fit image-text matching tasks via multimodal language models. However, these methods focus on VQA task and still struggle to handle multi-turn conversations or video modal information with a large number of frames present. The GraphGPT [39] model attempts to integrate text-based graph structures with LLMs and takes the index of nodes in the graph as direct input to the context. However, GraphGPT's comprehension ability is poor for graph structures with numerous nodes because LLMs cannot accurately establish the correspondence between indexes and nodes.To address the aforementioned two issues, text is used as a cue to align multiple modalities in this paper. LlaVA's [20] approach inspires the idea that we can use images to augment text-based LLMs. To establish complementary relationships between different modalities and incorporate them with LLMs, we design an automatically constructed multi-modal context graphs method and several graph sampling grammar, called AutoGraph. We aim to structure complex information and seamlessly integrate it with large language models (LLMs), aligning information from multiple modalities at both semantic and structural levels. The AutoGraph method is a general approach that can enhance the visual capabilities of LLMs.More specifically, in order to obtain aligned multi-modal context graphs, we employ the semantic dependency graph parsing to extract the structure of the utterance and obtain a textual graph. And the scene graph parsing method is used to convert the video modalities into image graphs. We fully connect the text graph and image graphs to form a holistic heterogeneous graph, then design a pruning strategy to align the graph structures of the two modalities at the semantic level. In order to combine the aligned multi-modal graph with LLMs, we devise three types of graph sampling method to transform the graph structures into a sequential structure. We argue that while the shift to sequentialization may change the word order, for LLMs it amounts to learning a new grammar to establish a mapping relationship with the target responses. During the finetuning stage, we propose a two-stage fine-tuning strategy to enable the LLMs to better comprehend the graph sampling grammar we designed. We validate our proposed method on text-based LLMs, and visual-based LLMs, respectively. Experimental results show that our proposed AutoGraph method can effectively enhance the performance of different types of LLMs and achieve the best results on multiple public datasets.The main contributions are summarized as follows:• We align multiple modal information at the semantic and structural levels through an automatically constructed visual context graph. Open-domain dialogue datasets based on image-grounded include MMChat [52] , DialogCC [13] , MMDialog [4] , Image-Chat [34] , and others. Maria [17] and VisAD [33] Zhang et al. [46] propose the ZRIGF model, which includes contrastive and generation pretraining modules, mapping different modalities to the same vector space and generating appropriate responses. VisCPM [7] and Monkey [16] are Multi-modal Large Language models (MLLMs), which are pre-trained by image and context information. These MLLMs establish mapping relationships between multiple modalities through extensive data.2.1.2 Video-grounded approaches. Video-grounded dialogue can be seen as a more complex extension of image-grounded dialogue, as videos contain numerous frames that drastically increase computational complexity. Open-domain dialogue datasets for videogrounded dialogue include MELD [26] , OpenViDial [22] , Open-ViDial 2.0 [43] , Tiktalk [19] , and others. Lin et al. [19] [11, 47] improve the performance of dialogue systems through coreference relationships across multiple modalities. Zhao et al. [50] construct dialogue graphs to model the speaker's cognitive shifts during the conversation. A expand strategy is proposed by Zhao et al. [49] to enlarge the constructed dialogue graph. But the dialogue graph constructed by aforementioned methods is based on utterance level, the AutoGraph model in this paper is focus on words level. 
fusion with llms. incorporating structured information into: 
 LLMs can effectively enhance the model's performance. ERNIE 3.0 [38] simply converts graph triplets into a tokenized text passage as input. K-BERT [21] injects knowledge triplets into sentences through visibility matrices to reduce the sequence length, with only knowledge entities being included as part of the sequence. To further distill knowledge, CoLAKE [37] proposes a unified word knowledge graph, where tokens from input sentences form a fully connected word graph. GraphGPT [39] directly uses the index of nodes in the graph as input to the context window, enabling LLMs to comprehend the graph's topology. However, these methods are merely sequentialization approaches for different knowledge triplets, rather than a method for converting a large connected dialogue graph into a sequence. Our proposed AutoGraph method aimd to design a graph sampling grammar to reduce the knowledge noise and seamlessly integrate context graph with LLMs. 
method 3.1 task definition: 
 The goal of the open-domain multi-modal dialog generation task is to generate appropriate responses based on contextual information with multi-modality. We formulate the this task as follows.Given text modal 𝐶𝑜𝑛𝑡𝑒𝑥𝑡𝑠𝑇 = {𝑈 1 , 𝑈 2 , ..., 𝑈 𝑛 } and visual modal 𝐶𝑜𝑛𝑡𝑒𝑥𝑡𝑠𝑉 = {𝑉 1 , 𝑉 2 , ..., 𝑉 𝑛 }, 𝑛 is the number of turns in a dialogue. 𝑉 𝑖 and 𝑈 𝑖 are videos and utterances at the 𝑖th turn. The target response is 𝒀 = (𝑦 1 , 𝑦 2 , ..., 𝑦 𝑚 ), where 𝑚 is the number of words.In this paper, we propose the AutoGraph model to automatically comprehend and align text and video modal information, the structure of the model is shown in Figure 2 . The AutoGraph model consists of three modules, §3.3 Visual Context Graph Construction, §3.4 Graph Sampling Grammar and §3.5 Two-stage Fine-tuning. We start with semantic dependency graph parsing and scene graph parsing to obtain structured relations from text and images, respectively. Then, we fully connect the text graph and the scene graph, and let LLMs trim meaningless edges to obtain the Visual Context Graph via the Few-shot and In-Context Semantic Alignment Prompt. We design graph sampling grammar to transform graph structures into sequence structures, and finally use a two-stage fine-tuning strategy to prompt the LLMs to understand this new graph sampling grammar.To align textual and video modalities in visual context, we construct a structured visual context graph based on semantic alignment. The visual context graph G 𝑉𝑇 is a heterogeneous graph, consisting of the text graph G 𝑇 and the image graph G 𝑉 .EQUATIONIt is worth mentioning that our proposed AutoGraph approach focuses on the word level rather than the sentence level. We believe that achieving better alignment with other modalities and constructing the visual context graph requires more fine-grained knowledge. In the field of context graph construction, there has been a lot of work. In the MuSE model, Zhao et al. [49] split sentences of speakers using punctuation marks to build the context graph. Zhao et al. [50] individually constructed dialogue context graphs from the perspective of different speakers. These graphs focus on the sentence level, treating the utterances of speakers as nodes in the graph.In the part of text-to-graph transformation, as highlighted in the green box in Figure 2 , we use semantic dependency graph parsing [2] to obtain the dependency graph between words. We split the speaker's context 𝐶𝑜𝑛𝑡𝑒𝑥𝑡𝑠𝑇 into multi-turns {𝑈 1 , 𝑈 2 , ..., 𝑈 𝑛 } and then separately parse the semantic dependency graph for each utterance 𝑢 𝑖 , obtaining the text graph G 𝑇 𝑖 , where 𝑖 represents the 𝑖th turn of the context.EQUATIONTaking 𝑈 1 in Figure 2 as an example, the result after parsing with the semantic dependency graph is shown in Figure 3 . In Figure 3 , the edges between words represent some form of dependency relationship in the semantic dependency graph parsing. However, to avoid introducing redundant information, these relationships are not used in this paper, and only the edges are employed as connections between nodes. Finally, we get the text graph extract keyframes from video [19, 22, 43] , we extract each video clip 𝑉 𝑖 into multiple keyframes 𝑓 𝑗 .G 𝑇 = {G 𝑇 1 , G 𝑇 2 , ..., G 𝑇 𝑛 },Text Graph Image Graph Aligned Graph 1 VT  1 V  n V  1 T  n T  1 V  1 T  Few-EQUATIONEQUATION𝑗 is the number of total key frames in video clip 𝑉 𝑖 .For each keyframe 𝑓 𝑗 , we employ scene graph parsing [44] to construct the scene graph at the 𝑖th turn.EQUATIONThe graphs of multiple keyframes are merged as subgraphs to form the image graph G 𝑉 𝑖 .EQUATION𝑗 is the number of total frames in video clip 𝑉 𝑖 . Finally, the same method is applied to construct image graph G 𝑉 for each video clip and After traversing both ends of all fully connected edges between word A and word B, we finally obtain the aligned visual context graph G 𝑉𝑇 .G 𝑉 = {G 𝑉 1 , G 𝑉 2 , ..., G 𝑉 𝑛 }.EQUATIONAt present, most LLMs can only draw knowledge from sequential structures and cannot directly understand graph-based structure information. Inspired by structured knowledge-enhanced pretrained models [21, [37] [38] [39] , we attempt to transform the visual context graph G 𝑉𝑇 into a sequential structure, allowing us to sample the topology of the graph. We design three different graph sampling grammar based on graph-level sampling and node-level sampling, which are §3.4.1 𝐺𝑟𝑎𝑝ℎ𝑆𝑎𝑚𝑝𝑙𝑒, §3.4.2 𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 and §3.4.3 𝐷𝑒𝑒𝑝𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒.In graph-level sampling, similar to the node feature update mechanism in Graph Convolutional Networks (GCN) [12] , for the target node 𝑁 𝑖 , the 𝐺𝑟𝑎𝑝ℎ𝑆𝑎𝑚𝑝𝑙𝑒 function can sample all neighboring nodes within 1 -ℎ𝑜𝑝 (including 𝑁 𝑖 ).The graph-level sampling sequence 𝐺𝑟𝑎𝑝ℎ𝑆𝑎𝑚𝑝𝑙𝑒 of the entire visual context graph G 𝑉𝑇 can be calculated as follows:EQUATIONwhere 𝑀 represents the number of nodes and 𝐴 represents the adjacency matrix.In the graph-level sampling, all neighboring nodes of the target node 𝑁 𝑖 are sampled, resulting in many nodes being repeatedly sampled after traversing the entire graph G 𝑉𝑇 . Inspired by GraphSAGE [6] , we modify the sampling grammar that only 𝐾 neighboring nodes of the target node 𝑁 𝑖 will be sampled in node-level sampling. If the target node does not have 𝐾 neighboring nodes, we will not pad to 𝐾 nodes.Building upon this, we sample first-order and second-order neighbor nodes to sample graph topology to a sequence. The node-level sampling sequence 𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 of the entire visual context graph G 𝑉𝑇 can be calculated as follows:EQUATIONEQUATIONwhere 𝐴 2 represents the square of the matrix 𝐴 and 𝑀 represents the number of nodes. To avoid redundant sampling of the target node, we set the 𝐴 2 𝑖 𝑗 = 0 in the adjacency matrix when 𝑖 = 𝑗, and to prevent repeated sampling of other nodes, we set the values of the remaining nodes in 𝐴 2 𝑖 𝑗 to 1 .Based on 𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒, we further explore deep node sampling methods. We continuously sample first, second, and third-order of the target node 𝑁 𝑖 . The deep node-level sampling sequence 𝐷𝑒𝑒𝑝𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 of the entire visual context graph G 𝑉𝑇 is calculated as follows:EQUATIONEQUATIONwhere 𝐴 3 represents the cube of the matrix 𝐴, 𝐴 2 is calculated via equation 10, 𝐾 represents the 𝐾 neighbors of target node 𝑁 𝑖 and 𝑀 represents the number of nodes. We do not explore deeper-level sampling as it would significantly increase the length of the graph sampling sequence. To the best of our knowledge, this is the first attempt to devise a grammar to facilitate LLMs in acquiring graph structure information. Given that the LLMs have the ability to learn other forms of grammar [10] , we believe that LLMs can also handle this type of graph sampling grammar.In order to facilitate the understanding of entirely new grammar structures by the LLMs, we devise a two-stage fine-tuning strategy.The objective of the first stage is to enable the LLMs to comprehend this grammar, restoring utterance 𝑈 𝑖 via the visual context graph G 𝑉𝑇 𝑖 of the 𝑖th turn. Through prompting, the LLMs reconstruct the sentence under graph sampling grammar into the original sentence, establishing a connection between the new grammar and human-familiar grammar. The goal of the second stage is to enable the LLMs to predict the response to the next utterance 𝑈 𝑖+1 based on the visual context graphG 𝑉𝑇 = {G 𝑉𝑇 1 , G 𝑉𝑇 2 , ..., G 𝑉𝑇 𝑖 }.Taking the 𝑈 1 in Figure 2 as an example and two-stage finetuning prompt is shown as follows. The Relations in stage 1 and stage 2 is fixed, and the Sentence is the target sentence that the model needs to predict and participates in the loss calculation during the fine-tuning process.  
experiment 4.1 datasets: 
 We conduct experiments on two publicly available datasets, the MELD dataset [26] and the large-scale OpenViDial dataset [22] . The dataset statistics are summarized in Table 1 . To stay close to open-domain conversation scenarios, we chose these two datasets. OpenViDial has a larger scale compared to the MELD dataset.• MMChat [52] : A multi-modal dialogue model based on a multi-layer text encoder, Fast-RCNN encoder and GPT-2 decoder.• RESEE [41] : RESEE enhances the text-based conversational abilities via visual knowledge. The visual and text encoders are composed of CLIP [28] and T5 [29] , respectively. • Llama2-7B [40] : An open-source, high-performance, text only large language model for English. We fine-tune it using the textual modality of the MELD and OpenViDial datasets. • LLaVA-7B [20] : The large language model based on Llama2-7B, fine-tuned by visual instructions, achieved excellent performance in several multi-modal tasks. After fine-tuning with the MELD and OpenViDial dataset, we use it as the baseline model.We reproduce the results based on the source code provided in the original paper. All experiments are trained with the same parameters. The learning rate of QLoRA and Full fine-tuning is 1e-4 and 1e-5, respectively. All baseline models based on the LLMs are available through the corresponding open source projects. QLoRA rank is set to 128. In the AutoGraph model, 𝐾 is set to 3, and 𝑀 depends on how many neighbors the target node has. Experiments on the MELD dataset are accelerated by 8 * NVIDIA 32GB V100 GPUs, and experiments on the OpenViDial dataset are accelerated by 4 * NVIDIA 40GB A100 GPUs. Our code can be found through https://github.com/DericZhao/AutoGraph.4.4.1 Automatic Evaluation. Automatic evaluation is efficient and fair. Following previous work, we adopt mainstream evaluation metrics, which include Distinct-𝑛 [15] , BLEU [23, 27] , Rouge-L [18] , F-BERT [48] and CHRF [25] . The Distinct-𝑛 aims to encourage the dialogue system to generate diverse responses, while the BLEU, Rouge-L, F-BERT and CHRF seek to produce contextually relevant responses through different approaches.Human evaluation is to evaluate responses quality from a human perspective. We conduct the aspect-based pairwise preference test for human evaluation. (1) Coherence (Coh.): which measures the relevance and coherence of the generated responses to the context. ( 2) Informativeness (Inf.): which response conveys more information related to context. (3) Grammar (Gra.): which measures whether the grammar of responses is correct. We randomly sample 200 response pairs of each model and recruit 10 evaluators to judge. 
results and analysis 5.1 automatic evaluation results: 
 With AutoGraph method, different LLMs achieve state-of-the-art automatic evaluation results. We evaluate the effectiveness of our AutoGraph method on both the MELD dataset and the OpenViDial dataset. The results of automatic evaluation are shown in Tables 2, 3, 4, and 5. We employ QLoRA [3] and Full fine-tuning methods to fine-tune various LLMs. Our experimental objectives include three main goals: 1. Validate the performance of the AutoGraph method on the Llama2 text-based LLMs. 2. Validate the performance of the AutoGraph method on the LLaVA multi-modal LLMs. 3. Verify the sampling capability of Compared to other general multi-modal LLMs, leveraging the best sampling methods in AutoGraph can enable the text-based Llama2-7B model to outperform the majority of baseline models. This demonstrates that employing the AutoGraph method can effectively enable text-based LLMs to comprehend multi-modal information and enhance model performance.2. AutoGraph on Multi-modal LLMs. The LLaVA-7B model has been fine-tuned with visual prompts, and after fine-tuning on the MELD dataset and OpenViDial dataset, its performance surpasses that of models specifically designed for open-domain dialogue generation, such as MMChat and RESEE. We also try to incorporate the AutoGraph method into the LlaVA-7B model to enhance the understanding of scene information. Experimental results indicate that the model's performance is further improved after adding the visual context graph.Compared to other general LLMs, LLaVA-7B exhibits superior performance after being enhanced by the AutoGraph method. Since LLaVA-7B already possesses visual capabilities, we believe that the AutoGraph method further provides the complementary relationship information. The experimental results indicate that our proposed AutoGraph method can also enhance multi-modal LLMs.3. Graph Sampling Ability. We verify the sampling capabilities of different graph sampling methods for constructed visual context graphs. The 𝐺𝑟𝑎𝑝ℎ𝑆𝑎𝑚𝑝𝑙𝑒 graph sampling method shows minimal enhancement in model performance. We attribute this to the noise caused by a significant number of duplicated nodes when traversing the entire visual context graph. The 𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 graph sampling method alleviates the issue of repeated sampling. Models enhanced by the 𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 sampling method tend to perform better on diversity metrics, such as Distinct-1 and Distinct-2. We believe this may stem from the insufficient understanding of the overall topology of the graph, as only 𝐾 nodes around the target node are sampled, potentially leading to incomplete sampling. The 𝐷𝑒𝑒𝑝𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 graph sampling comprehends the topology of the graph by sampling nodes at deeper levels. Models with the 𝐷𝑒𝑒𝑝𝑁𝑜𝑑𝑒𝑆𝑎𝑚𝑝𝑙𝑒 graph sampling grammar exhibit excellent performance on context relevance metrics, such as BLEU, Rouge-L, F-BERT, and CHRF scores.Moreover, different fine-tuning methods also lead to different results. The experimental results of Full fine-tuning are significantly better than those of QLoRA. QLoRA is an efficient finetuning method as it only adjusts some layers' parameters in LLMs, while Full fine-tuning makes adjustments to all parameters. Both In human evaluation, we are more concerned with whether the proposed graph sampling grammar can effectively map to the standard grammar in real-life. The human evaluation data on the MELD dataset is presented in Table 7 . We evaluate the grammar of the response sentences, and LLMs demonstrate a full understanding of this graph sampling grammar from a human perspective. In terms of coherence and informativeness, the Llama2-7B model enhanced with the AutoGraph method has surpassed the text-based Llama2-7B model. In the human evaluation of the multi-modal LlaVA-7B model, there is a certain performance improvement after incorporating the AutoGraph method. Human evaluations further demonstrate that the AutoGraph method effectively enhances the visual dialogue capabilities of LLMs.We conduct ablation studies to verify the effectiveness of the twostage fine-tuning strategy, as shown in Table 6 . The results indicate that, after eliminating the first-stage restore prompts, the model's performance is affected to varying degrees. This indicates that the two-stage fine-tuning can assist the model in initially understanding the new grammar and providing support during the predict prompt process in the second stage. And the performance of LLMs with only predict prompt still surpasses most baseline models. Furthermore, the most significant performance change is observed on the MELD dataset, which we attribute to its smaller data size compared to the OpenViDial dataset. Figure 4 shows the case study of different models. 
conclusion: 
 In this paper, we propose an automated method for constructing visual context graphs to address the task of open-domain multi-modal dialogue generation. We first construct the visual context graph based on semantic and structural alignment. Then, to integrate the visual context graph with LLMs, we design several graph sampling grammar. Finally, we propose a two-stage fine-tuning strategy to enable the LLMs to comprehend the new grammar and generate responses. The experiments on text-based and multi-modal large language models validate the effectiveness of the AutoGraph method.In the future, we will explore how to automatically construct better dialogue graphs to integrate more structured information and develop dynamic graph sampling methods based on different edge weights. 
 
 Appendix:  


Decision: This paper introduces a method for generating visual context graphs to improve open-domain multi-modal dialogue generation, demonstrating effectiveness through graph construction, sampling, and two-stage fine-tuning validated by experiments on various datasets.

Reasons to accept: 
The motivation is clear and sound with corresponding method design. 
Experiments are well organized with good performance. 
Some core concerns have been addressed in the rebuttal, e.g., time efficiency and scalability.

Reasons to reject:
Multiple reviewers still have concern on innovation and usefulness (Reviewer GVcn, cff8).
Reviewers have concern on the paper presentation, including writing (Reviewer PLP4), core method introduction (Reviewer asem), and experiment details (Reviewer GVcn).
Reviewers ask for comparisons with dialog methods using LLM (Reviewer asem).

The paper receives the initial rating of weak reject, two borderline reject, and weak accept. After rebuttal and discussions, reviewers raise the rating to three borderline reject, and weak accept. After reading the paper, review comments, and discussion, AC recommends accept, and strongly encourages authors to incorporate the content from the rebuttal into the final version, listed in the above “reasons to reject.”

Reviews: review: paper_summary: paper_summary: This paper proposes a novel approach for creating visual context graphs to enhance open-domain multi-modal dialogue generation. It can first construct the visual context graph, then perform graph sampling operations, and finally generate suitable responses through two-stage fine-tuning. Experimental results on multiple datasets validate the effectiveness of the scheme.
strength_weakness: strength_weakness: 1.The structure of the article is clear.  
2.The motivation is sound.
limitations: limitations: 1.LLMs and graph networks tend to be less efficient. Does the fact that the authors have used both structures lead to extremely slow reasoning and thus no practical application?  
2.The whole process is somewhat cumbersome and seems to be finely engineered. This can limit the generalizability and usefulness of the model.  
3.The whole work is somewhat weakly innovative, it feels like a stacking of some existing work, like scene graphs + graph sampling + prompt engineering.  
4.Many experimental details are not given, such as hyperparameter settings (K, M, etc.).  
5.Several works in related fields should be cited, e.g.,  
[1] Multimodal Dialog System: Relational Graph-based Context-aware Question Understanding.  
[2] Structured Co-reference Graph Attention for Video-grounded Dialogue.
paper_summary: paper_summary: The paper presents AutoGraph, a method that automatically constructs visual context graphs, addressing the challenge of integrating complementary multi-modal information in open-domain dialogue systems. By enhancing LLMs with graph alignment, it improves response relevance and coherence, achieving state-of-the-art performance across various datasets, demonstrating efficacy in both text-based and multi-modal LLMs.
strength_weakness: strength_weakness: Compared to existing approaches, AutoGraph demonstrates superior performance, effectively enhancing both text-based and visually equipped LLMs, as evidenced through rigorous experimentation and notable improvements on diverse public datasets.
limitations: limitations: 1.Unfortunately, the paper suffers from numerous grammatical errors and typos that detract from the clarity and professionalism of the research presentation. 

2.While the study introduces and evaluates three different graph sampling methods, the results show that DeepNodeSample, a third-order node sampling technique, outperforms lower-order methods in terms of contextual relevance.  So why not explore higher-order sampling methods?

3.Why the authors decide to employ an older models such as GPT-2? Since the more advanced models like GPT-3.5 or GPT-4 is also available.

4. In Eq.6, how are multiple subgraphs fused, in a fully connected form?
paper_summary: paper_summary: The paper introduces "AutoGraph," a method aimed at enhancing open-domain multi-modal dialogue systems by constructing visual context graphs. These graphs align semantic and structural information from multiple modalities (text and images/video), integrating this complex data with large language models (LLMs) to improve dialogue response quality. The technique involves creating connections between text and scene graphs, trimming irrelevant connections through LLMs, and converting these graphs into a sequential format that LLMs can process. The authors validate their approach through experiments with text-based and visual-based LLMs, showing improved performance on public datasets.
strength_weakness: strength_weakness: Innovative Integration: The paper effectively addresses the challenge of integrating multimodal data with LLMs by introducing a novel method for constructing and utilizing visual context graphs.

Detailed Methodology: The methodological approach, including graph sampling and fine-tuning strategies, is well-articulated, offering clear insights into the process and its novelty.

Robust Validation: The model is tested across different datasets and compared with several baselines, providing a thorough evaluation of its performance.
limitations: limitations: In this paper, although three distinct graph sampling strategies (GraphSample, NodeSample, DeepNodeSample) are proposed, each addressing specific issues independently, there is a lack of consideration for combining these strategies, which may limit flexibility and efficiency when handling complex graph structures. A Mixture of Sample (MoS) approach could potentially optimize model performance by integrating the advantages of each strategy, such as through hierarchical sampling or conditional sampling adapted to different graph structures and characteristics. The absence of such an integrated approach could be seen as a limitation of the current method. Future research could explore how to effectively combine these sampling methods to further enhance the model's generalization ability and manage complexity.
paper_summary: paper_summary: This paper aims to integrate LLM and existing open-domain multi-modal dialog generation techniques for deep relationship utilization. It designs AutoGraph method where the visual context graph aligns multi-modal information at multiple levels and it is transformed into sequences to use LLM.
strength_weakness: strength_weakness: 1.	The analysis of the motivation is clear.
2.	The experiments are well organized and the case study is interesting with some inspiring observations.
limitations: limitations: 1.	The task definition is unclear, is the target response supposed to match the visual modal context of i+1th turn? If so, V_n+1 should be given as input. The concepts mentioned in subsections of chapter3 do not correspond very well with the Figure 2, making it hard to understand. It would be better to adjust the structure of Figure 2, and present a more comprehensive case in the appendix.
2.	The experiments only compare with original LLM methods but not some dialog methods using LLM.  
3.	The text graph is constructed using parsing tool, yet the parsing labels of the edges are not used, then how could the LLM know what the remaining empty edges means? It is quite unconvincing.
4.	Is the “Semantic Alignment Prompt” mentioned in the text applicable to all samples? If so, according to this prompt, any concepts of the same type will have connections, what is the basis for this? For instance, if there are only one personal pronoun and multiple names in the sentence, it is obviously not possible to connect the correct reference.
5.	The scene graph parsing focuses solely on nouns, would this result in a significant amount of image information being overlooked? For example, attributes like color, size, etc., and actions between items.
6.	It is difficult to understand which graphs are directed and which are undirected based solely on the provided figures and explanations, and the arrows are drawn somewhat casually. This will result in the misunderstanding of the subsequent graph operation.