An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:53:34

Abstract: With the rise of large-scale language models(LLMs), it is currently popular and effective to convert multimodal information into text descriptions for multimodal multi-hop question answering. However, we argue that the current methods of multi-modal multi-hop question answering still mainly face two challenges: 1) The retrieved evidence containing a large amount of redundant information, inevitably leads to a significant drop in performance due to irrelevant information misleading the prediction. 2) The reasoning process without interpretable reasoning steps makes the model difficult to discover the logical errors for handling complex questions. To solve these problems, we propose a unified LLMs-based approach but wihout heavily relying on them due to the LLM's potential errors, and innovatively treat multimodal multi-hop question answering as a joint entailment tree generation and question answering problem. Specifically, we design a multi-task learning framework with a focus on facilitating common knowledge sharing across interpretability and prediction tasks while preventing task-specific errors from interfering with each other via mixture of experts. Afterward, we design an iterative feedback mechanism to further enhance both tasks by feeding back the results of the joint training to the LLM for regenerating entailment trees, aiming to iteratively refine the potential answer. Notably, our method has \textbf{won the first place} in the official leaderboards of WebQA (since April 10, 2024), and achieving competitive results on MultimodalQA.

Main paper: 
 related work: 
 Multimodal Multi-Hop Question Answering Multimodal multihop question answering is a task based on multimodal question answering, but it requires multi-hop reasoning to generate the final answer. VQA [1] is first proposed to answer questions from visualonly inputs. Later, WebQA [3] and MultimodalQA [21] require integrating information across free text, images, or semi-structured tables, to answer multi-hop reasoning question. To address the challenge of finding answers from multiple sources of information, MuRAG [5] designs a multi-modal transformer architecture to accept both text and image feature inputs, and builds a million-scale dataset for pretraining the model. [17, 17, 25, 27] unified multimodal information into text using image caption model and table linearization method, they proposed a new multimodal question answering paradigm, but there is no restriction during transfer, resulting a lot of information redundancy and affecting the performance of the model. Also, there are many recent works on multimodal question answering using large models. [11] train an image caption model to generate image caption for gpt-3 to understand images then generate responses; [15] use multimodal large model LLaVA to generate more accurate image caption, then construct different in-context learning templates according to each modalities, enabling GPT-3 to leverage its powerful performance in this task. Both approaches need to generate image caption for the large language model to understand question, but there are no conditional restrictions during the image caption generation stage; or when generating image captions directly based on multi-hop questions, the questions contain information cannot be asked by a single image, which causes errors during the image caption generation stage.For the above problems we found, we proposed an approach to filtered redundant information through the logical structure of the entailment tree, ensure the simplicity of information and the rationality of reasoning. Entailment Tree Generation The task of entailment tree generation currently serves NLP question answering systems primarily. [6] introduce EntailmentBank, a dataset specifically designed for the task of entailment tree generation. Each multi-step entailment tree in EntailmentBank serves as an explanation, clearly demonstrating the reasoning process behind a hypothesis based. Recent methods [10, 14, 20] have presented multi-step generation approaches, which iteratively select premise facts and generate intermediate conclusions.At present, the all correct score(only if all of the leaves, steps, and intermediates are all correct) of entailment tree structure generation in the field of NLP is very low(2.9% in full corpus) and cannot be directly applied to other fields; however, in the current multimodal question answering datasets, the questions are relatively simple. According to the statistics of two datasets [3, 21] , for the MultimodalQA dataset, the proportion of complex questions (with reasoning hops greater than or equal to 3) is only 11.3%. After removing the simple multiple comparison questions, the proportion of complex questions only accounts for 1% of the total dataset. For the WebQA dataset, the proportion of complex questions (with reasoning hops greater than or equal to 3) is only 1%. These two datasets are currently the most complex in multimodal multi-hop question answering, suitable for evaluating our methods. Multi-Task Mixture-of-Experts Recent rumors suggest that GPT-4's internal structure employs a mixture-of-experts (MoE) approach, which has been influential in the development of large-scale models [12, 23, 28] . The use of MoE as an architectural foundation has become prevalent in recent large models, propelling the advancement of both the models themselves and the MoE concept.Moreover, multi-task MoE models have seen significant development prior to their integration into large-scale models. [18] propose a multi-layer gated network based on different tasks, allowing each task to have its independent experts, thereby enabling the model to better capture the inter-task correlations. Additionally, based on the previous method, [22] propose a method which retains the shared experts, allowing for interaction between different experts. [9] devise a task-aware gating mechanism within sparse MoEs to route the input (tokens from different tasks) to specialized experts conditioned on the task.We combine the methods of multi-task MoE from previous research with the current MoE training approaches based on large models, enabling the multi-task MoE model to be suitable for multitask learning with data generated by large-scale models. 
method: 
 As shown in Figure 2 , our method is divided into two stages: (a) entailment tree initialization stage and (b) iterative mixture-of-experts optimization stage. The goal of the first stage is to initialize the entailment tree for the use of small model in assisting with answering and providing interpretability. We decompose the original question to build a fact base, and use LLM to initialize the structure of the entailment tree based on the fact base. The goal of the second stage is to correct the leaf node and structural errors in the initialized entailment tree through joint learning of fact retrieval generation task and question answering task, and to iteratively optimize the entailment tree through the feedback of the results of joint learning to the LLM.3.1.1 Fact Base Construction. In the fact base construction module, we need to decompose the multi-hop question into several subquestions based on different evidence and process them differently according to the modality of the corresponding evidence. Decompose Multi-Hop Question First, we need to retrieve the multimodal evidence required to answer the multi-hop question. However, since our method generates image captions by decomposing the question, we use the global image caption and image attribute features to retrieve evidences according to the method in [27] , and finally retrieve the required multimodal evidence set E:EQUATIONwhere "n" represents the number of evidence in the evidence set. '𝑇 𝑒𝑥𝑡 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 ' refers to all the evidence obtained after converting 'images, text, tables' into text, which is referred to as '𝑇 𝑒𝑥𝑡 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 '.After obtaining all the retrieved multimodal evidence 𝐸 = [𝐸 1 , 𝐸 2 , ..., 𝐸 𝑛 ], we prompt GPT-3.5 to decompose the original question based on all the evidence 𝐸 and generate n sub question 𝑞 𝑠 with their corresponding evidence. Suppose that the k-th question of 𝑞 𝑠 has L tokens, denoted as 𝑞 𝑠 𝑘 = (𝑦 1 𝑘 , 𝑦 2 𝑘 , ..., 𝑦 𝐿 𝑘 ), the decoding process can be formulated as:EQUATIONwhere 𝑝 𝑞 is the instruction prompt. The outline of the prompt 𝑝 𝑞 for LLM is as shown in Figure 3 : After obtaining all the decomposed sub-questions based on each evidence, we process them differently according to the modality of the corresponding evidence and convert them into facts stored in the fact base. Image Fact For the image modality, we further decompose each sub-question into atomic questions 𝑞 𝑖 using GPT-3.5. Suppose that the r-th atomic question of 𝑞 𝑖 decomposed from 𝑞 𝑠 𝑘 has 𝐿 𝑖 tokens, denoted as 𝑞 𝑖 𝑘 = (𝑦 1 𝑘 , 𝑦 2 𝑘 , ..., 𝑦 𝐿 𝑖 𝑘 ), the decoding process can be formulated as:EQUATIONThen, we input the decomposed atomic question 𝑞 𝑖 and corresponding image 𝐸 𝑖 𝑚 into the VQA model to obtain answers. We use LLaVA-1.5 [13] as our VQA model. it uses Clip [19] as the visual feature extractor: Suppose that the r-th atomic question's answer 𝑎 𝑟 has 𝐿 𝑎 tokens, denoted as 𝑎 𝑟 𝑘 = (𝑦 1 𝑘 , 𝑦 2 𝑘 , ..., 𝑦 𝐿 𝑘 ), the decoding process can be formulated as:EQUATIONEQUATIONThen we use GPT-3.5 to refine the obtained (atomic question, answer) pairs into facts stored in the fact base. Table Fact After we decompose the origin question, and obtained sub-question 𝑞 𝑡 𝑏 for table evidences, we use [27] which use simple natural language templates to transform tables into sentences that sound natural to humans. As an example, we can turn the table into a sentence by arranging the cells in a linear fashion, like this: "row one's seaon is ..., Date is ..., location is ... ", and then feed into GPT-3.5 to obtain (sub-question,answer) pairs into facts stored in the fact base. Suppose that the t-th table's sub-question 𝑞 𝑡 𝑏 𝑡 has 𝐿 𝑡𝑏 tokens, denoted as 𝑞 𝑡 𝑏 𝑡 = (𝑦 1 𝑡 , 𝑦 2 𝑡 , ..., 𝑦 𝐿 𝑡 ), the decoding process can be formulated as:EQUATION3.1.2 Entailment Tree Generation.Definition 3.1. The entailment tree generation task input consists of a corpus of premises C (facts from fact base) and a hypothesis h (original question). The objective is to generate an entailment tree T that explains the hypothesis h by using a subset of the premises in C as building blocks. Entailment trees are represented as a tuple 𝑇 = (ℎ, 𝐿, 𝐸, 𝑆), where leaf nodes 𝑙 𝑖 ∈ 𝐿 are retrieved from the corpus (𝑖.𝑒.𝐿 ⊆ 𝐶), internal tree nodes 𝑒 𝑖 ∈ 𝐸 are intermediate conclusions (new sentences not present in corpus C, note that intermediate conclusions are generated by LLM), and 𝑠 𝑖 ∈ 𝑆 is a list of entailment steps that can explain the hypothesis h, which is always the tree root and the final conclusion.Entailment Tree Structure Generation The main function of the entailment tree structure generation module is to select leaf nodes and generate the entailment tree structure. At this stage, we use predefined symbols for the entailment tree structure generation task which introduced by [6] , where the facts in the fact base serve as a set of possible leaf nodes, and the original question serves as the conclusion. The entailment tree structure is generated using GPT-3.5, where the facts are the leaf nodes, and several leaf nodes are combined to form intermediate nodes. In the entailment tree structure generation module, only facts and the original question are given, and the intermediate nodes generated do not contain any specific information. In the initialization phase of the entailment tree, we do not directly predict the answer. Instead, we continuously fill the entailment tree with leaf nodes by predicting the structure of the entailment tree, and constantly improve the corresponding intermediate nodes through the combination of leaf nodes.After predicting the structure of the entailment tree based on the question and perfecting the entailment tree, since the root node (question) does not contain the answer, we use the "answer" placeholder to replace the root node and input it into the second phase to perfect the entailment tree.We use the symbol "&" to denote "and", and "->" to denote "entails", Suppose that the j-th origin question 𝑠 𝑗 and fact base 𝐹 𝐵 = [𝑓 𝑎𝑐𝑡 1 , 𝑓 𝑎𝑐𝑡 2 , ..., 𝑓 𝑎𝑐𝑡 𝑚 ] are input into GPT-3.5 and generate entailment tree structure 𝑡 𝑠 which has 𝐿 𝑡 tokens, denoted as 𝑡 𝑠 𝑗 = (𝑦 1 𝑗 , 𝑦 2 𝑗 , ..., 𝑦 𝐿 𝑗 ), the decoding process can be formulated as:EQUATIONwhere 𝑝 𝑡 is the instruction prompt. The template of the prompt 𝑝 𝑡 for LLM is as follows:/* Instruction for the decompose question task */ Please generate entailment tree according to the given facts and target-question.\n /* n in-context examples */ TARGET-QUESTION: origin question q \n Facts: fact 1: ... , fact 2: ... , ... Entailment Tree: fact & fact2 -> int1; ... The algorithm we use is shown below: The meaning of "Split the entailment tree structure into multiple sub-trees set T" is that we start from the root node of the constructed entailment tree structure, continuously obtain subtrees composed of non-leaf nodes 𝑁 𝑟 and their child nodes 𝑁 𝑐 (𝐷𝑒𝑝𝑡ℎ(𝑁 𝑐 ) = 𝐷𝑒𝑝𝑡ℎ(𝑁 𝑟 ) + 1), and continuously add them to the set to form a set of subtrees T. we split the set of all subtrees in the entailment tree structure. If the number of subtrees in the subtree set is greater than 1, we input all the subtrees one by one into the large model in order and deduce the root (intermediate node) of each subtree based on the leaf nodes (facts). Finally, we merge them into a complete entailment tree 𝑇𝑟𝑒𝑒 𝑖𝑛𝑖𝑡𝑖𝑎𝑙 .Due to the possible leaf node selection errors and entailment tree structural errors in the previously mentioned initialized entailment Algorithm 1 Algorithm of Entailment Tree Refinement.Require: Fact Base, Entailment Tree Structure. 1: Split the entailment tree structure into multiple sub-trees set 𝑇 . for 𝑖 = 0 to 𝑙𝑒𝑛𝑔𝑡ℎ(𝑇 ) do 4:Split sub-tree 𝑇 𝑖 into node set 𝑁 5:for 𝑗 = 0 to 𝑙𝑒𝑛𝑔𝑡ℎ(𝑁 ) do Complete Entailment Tree = T 16: end if tree, we use a hybrid expert model to jointly learn the fact retrieval generation task (retrieving leaf nodes) and question answering tasks, and correct the entailment tree structure through an iterative feedback mechanism.Answering. First, we use T5 encoder to extract the features of each fact in the fact base as 𝐹 𝑓 𝑎𝑐𝑡 = [𝑓 1 [𝑚𝑒𝑎𝑛] , 𝑓 2[𝑚𝑒𝑎𝑛] , ..., 𝑓 𝑛 [𝑚𝑒𝑎𝑛] ], Afterwards, we convert the entailment tree into a natural language description denote as 𝑇𝑟𝑒𝑒 𝑡𝑒𝑥𝑡 𝑖𝑛𝑖𝑡𝑖𝑎𝑙 and input 𝑇𝑟𝑒𝑒 𝑡𝑒𝑥𝑡 𝑖𝑛𝑖𝑡𝑖𝑎𝑙 into the shared encoder of T5, and then concat it with origin question 𝑞 to obtain the final feature 𝐹 𝑒𝑡 :EQUATIONOnce we have obtained the features, we input 𝐹 𝑒𝑡 into the Mixtureof-Experts model. Multi-Task Mixture-of-Experts our mixture-of-experts model primarily consists of three parts: two gating networks, two taskspecific expert networks, and one shared expert network. The taskspecific expert networks are dedicated to the tasks of fact retrieval generation and question answering, respectively, while the shared expert network can be utilized for both tasks. The gating network is responsible for selecting the appropriate experts. Specifically, gating network A selects from the fact retrieval generation task and shared experts, while gating network B selects from the question answering task and shared experts.Top-2 Selection. According to the formulation above, when g(•) is a sparse vector, only part of the experts would be activated and updated by back-propagation during training. We set the gating layer as a top-K selection as:EQUATIONwhere 𝑓 (𝐹 𝑒𝑡 ) is routing linear transformation 𝑅 𝐷 → 𝑅 𝐸 . Token-choice Routing We generally follow [23] for our routing design to ensure training stability. Given E trainable experts and input representation 𝐹 𝑒𝑡 ∈ 𝑅 𝑙𝑒𝑛 𝑒𝑡 * 𝐷 , the output of MoE model can be formulated as:EQUATIONwhere 𝑒 𝑖 (𝐹 𝑒𝑡 ) is a non-linear transformation 𝑅 𝐷 → 𝑅 𝐷 of the ith expert, and 𝑔(𝐹 𝑒𝑡 ) 𝑖 is the i th element of the output of the trainable router 𝑔(𝐹 𝑒𝑡 ), a non-linear mapping 𝑅 𝐷 → 𝑅 𝐸 . Usually, both 𝑒 (𝐹 𝑒𝑡 ) and 𝑔(𝐹 𝑒𝑡 ) are parameterized by neural networks. Please note each expert is an FFN layer instead of a complete Transformer model in most MoE-based Transformer models, including ours. After we have selected the experts for the fact retrieval generation task and the question answering task through the gated network, we add 𝑀𝑜𝐸 (𝐹 𝑒𝑡 ) with 𝐹 𝑒𝑡 to obtain the final MoE output:EQUATIONthen we input 𝑀𝑜𝐸 𝑓 into the decoder of their respective tasks. Decoders for Multi-Tasks We handle the two decoders differently. The FRG Decoder performs cross-attention with all fact features and the entailment tree description passed through the Shared Encoder, while the QA Decoder only performs cross-attention with the entailment tree description passed through the Shared Encoder. The reason for doing this is that we hope the QA model can get the answer based solely on the entailment tree. If it cannot, then optimize the entailment tree through the FRG model and the Iterative Feedback Mechanism until the QA model can answer the question based solely on the entailment tree.The decoder for fact retrieval generation 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑓 𝑟𝑔 performs cross attention with the facts feature 𝐹 𝑓 𝑎𝑐𝑡 and shared encoder output 𝐹 𝑒𝑡 : 𝑤 𝑡 , 𝑔 𝑡 = 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝐷𝑒𝑐𝑜𝑑𝑒𝑟 (𝑞), 𝑀𝑜𝐸 𝑓 ), 𝐹 𝑓 𝑎𝑐𝑡 ) (12) where 𝑤 𝑡 denotes the cross-attention weights at time step t. then the decoder 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑓 𝑟𝑔 begins to retrieval at time step |𝑄 |, i.e., the length of the question 𝑞, and then we utilize cross-entropy loss for it:EQUATIONwhere 𝛼 𝑡,𝑖 denotes the cross-attention scores of the i-th fact at time step t, 𝛼 𝑡,𝑡 + denotes the score of the target source(The fact index sequence extracted from the entailment tree generated by LLM) at time step t, M is the number of retrieval steps.The decoder for the question answering task 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑞𝑎 performs cross attention with the overall features 𝑀𝑜𝐸 𝑓 :EQUATIONthe decoder 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 𝑞𝑎 begins to retrieval at time step |𝑄 |, i.e., the length of the question 𝑞, and then we utilize cross-entropy loss for it:EQUATIONThe overall loss function is as follows:EQUATION3.2.2 Iterative Feedback Mechanism. After we complete the tasks of fact retrieval generation and question answering, we first replace the fact index obtained from the fact retrieval generation task in the fact base with the corresponding facts. Then, we concatenate it with the final answer and input it into GPT-3.5 as additional information to correct the entailment tree and conduct a second round of training. We add a prompt 𝑝 𝑖 𝑓 𝑚 to the original prompt template which we use to generate the entailment tree structure: "Given the following potentially relevant facts and the potentially correct answer, please generate entailment tree in n words. Facts:f Answer:a Question:q" Algorithm 2 Algorithm of Iterative Feedback Mechanism.Require: facts set 𝑓 from fact retrieval generation task, answer 𝑎 from question answering task. 1: iterative number 𝑘 = 𝑛; 𝑓 𝑖 = 𝑓 ; 𝑎 𝑖 = 𝑎 2: for 𝑖 = 0 to 𝑘 do 3:Fill the prompt with 𝑓 𝑖 and 𝑎 𝑖 to get a complete prompt 𝑝 𝑖 𝑐 .Concat 𝑝 𝑖 𝑐 with 𝑝 𝑡 (shown in Figure 4 )5:Continue the processing of stage 1 to get a new entailment tree 𝑇 𝑖 .Continue with the steps of stage 2 through 𝑇 𝑖 , and get 𝑓 𝑖 𝑛𝑒𝑤 and 𝑎 𝑖 𝑛𝑒𝑤 respectively from the fact retrieval generation task and the question answering task. We conducted experiments on two of the most representative MMQA datasets: WebQA and MultimodalQA.WebQA [3] is a multimodal and multi-hop question answering dataset that contains QA pairs that require one or two images and text snippets to answer. Each question has a set of distractors that the model must consider along with the correct clues to provide an answer. WebQA uses BARTScore to measure both the fluency and keyword accuracy of the answer denote as QA-FL and QA-Acc in Table 2 . These two scores are multiplied together to obtain the QA score. The clue retrieval can be easily evaluated using F1 score.MultimodalQA [21] involves answering multi-hop complex questions by combining information from text, tables, and images. Each question also includes visual and text distractors. The performance is measured by F1 score at the word level and the Exact Match (EM) of the predicted answer. 
implementation details: 
 We conduct experiments on two datasets: WebQA, and MultimodalQA. The information source for WebQA includes both text and image modalities, while MultimodalQA focuses on text, images, and tables. For WebQA and MultimodalQA, a candidate clue list is given, and the model needs to find the most relevant clue to evaluate the accuracy of the clue retrieval. The backbone for retrieval is BERT [7] . We use LLaVA-1.5 as our VQA model, and use T5 as our FRG and QA model. We utilize the Transformers library and pretrained parameters from HuggingFace 4 and conduct experiments using 24G GPU cards. Further, AdamW [16] is used as the optimization algorithm with a learning rate of 1e-4. The batch sizes for retrieval and qa are 32, 12. 1 shows all the methods that use large models on the webqa dataset. promptcap [11] trained an image caption model to generate image caption to let GPT-3 have more information about the images, however, the image captions generated by promptcap are too coarse-grained. Our method decomposes question by GPT-3.5 and filters the question to generate image caption, which let image caption model more focused on specific areas.Retr 3 : WebQA official test-set results on QA-Accuracy and Retrieve F1. Our method significantly exceeds other current methods in terms of QA-Accuracy.Table 2 shows all results on WebQA offcial test-set result, MuRAG [5] design a multimodal transformer, SKURG [24] design a entity fusion method, solar [27] Also, as shown in Table 4 , we surpass sota result on Multi-modalQA in Single-Modal and Multi-modal set. This demonstrates that our method possesses superior reasoning abilities.The WebQA and Multimodalqa dataset both requires measuring the accuracy of the final answer. Our method achieves much higher accuracy than other methods on the final answer, because in the first step of decomposing the question, GPT-3.5 can comprehensively decompose the question and match the final multimodal clues. This leads to a more focused generation of image descriptions based on the decomposed questions, providing reasoning steps for the final generation. Therefore, our method achieves a good result in terms of accuracy. In this experiment, we ablate the question decomposition, subquestion image caption modules, fact retrieval generation module, multi-task mixture-of-experts module, Iterative feedback module. When ablating the question decomposition module, we directly use the original question as input, and directly use the original question as the prompt for LLaVA to generate image captions; when ablating the sub-question image caption module and entailment tree generation modules, we directly concatenate the retrieved evidence and sub-questions and input them to GPT-3.5. The results of the ablation experiments are shown in Table 5 . It can be seen that both question decomposition and the final sub-question image caption have a positive impact on the results, with the sub-question image description being particularly significant.As shown in Figure 5 and Figure 6 , we present two case studies in the Multimodal datasets. Figure 5 shows the entailment tree generated in the data sample of complex reasoning. Our method can generate effective entailment trees for a small amount of complex reasoning to guide the reasoning process. Figure 6 shows that when the first stage LLM generates an incorrect entailment tree (with errors in leaf node selection and structure), our proposed second stage can correct these errors through the joint learning of fact retrieval generation task and question answering task. 6 : Quality Analysis and Explainability on WebQA dev set. p1-p10 is different evaluator,q-True is an indicator counting whether the question decomposition is correct. r-True is an indicator that counting whether the entailment tree's reasoning path is correct.In this section, we show quality analysis of decomposed questions and explainability of final reasoning path, including the refined GPT-3.5 sub-questions and their specific image captions generated by LLaVA in our method. To evaluate the quality of question decomposition, we recruited 10 volunteers to conduct human evaluation. Each evaluator was randomly provided with 50 original questions, each origin question has corresponding sub-questions and reasoning path. We evaluate two indicators which are the accuracy of the decomposed question, the accuracy of the reasoning path. The accuracy of the decomposed question and the reasoning path is 89% and 86.2% respectively. We stipulate that when counting whether the question decomposition is correct, only when all the decomposed questions are correct can it be considered correct. When counting the explainable reasoning path, we require that the reasoning path be considered correct only when the evaluator thinks that the reasoning path can reason out the final answer. 
conclusion: 
 We follow the current popular method [4, 8, 27] , of unifying multimodal information into text information. In this text-driven multimodal paradigm, to the best of our knowledge, this is the first time that an explanatory improvement has been made from the perspective of entailment tree generation. In this paper, we construct an entailment tree through LLMs, and iteratively correct the entailment tree by proposing a multi-task MoE and iterative feedback mechanism. During the process, we can generate entailment trees based on the set of facts to assist the model in complex problem reasoning. Different from previous methods of entailment tree generation, our method not only has interpretability, but also helps to improve the accuracy of question answering. Our experiments demonstrate the potential of this approach. 
limitations: 
 While our method has demonstrated its superior performance on two benchmarks, it still has several limitations. First, this method might not fully use the sub-question answers, for successively prompting next sub-question, although it may reduce error propagation. Second, the majority of data in current multimodal multi-hop question answering datasets does not entail complex reasoning. While the question decomposition module primarily contributes to performance enhancement when employing an entailment tree, we excel in handling a small fraction of complex problems by generating entailment trees, which can direct the model's reasoning path.  
what animal race in the race of a.p. warrior at the churchill downs track?: 
 Figure 1 : Examples of current methods that converting multimodal information to text on the MultiModalQA dataset.In the figure, much redundant multimodal information has been converted into text(black font). Meanwhile, the key information has a strong logical relationship with each other, and current methods also have not utilized this relationship.generate accurate and complete answers. Currently, most multimodal multi-hop question answering methods adopt the approach of converting multimodal information into textual descriptions (by transforming images through image caption models, and tables through natural language descriptions), and then using large-scale language models (LLMs) to generate answers [8, 11, 26, 27] . The salient advantage of this method is that it can leverage the powerful language understanding and generation capabilities of LLMs, as well as the interpretability of textual descriptions. However, this method indiscriminately converts all multimodal information into textual descriptions, inevitably producing a large amount of redundant information. As shown in Figure 1 , the current general method in the field of multimodal question answering only requires a small amount of key information to answer the multi-hop question. For instance, to find the "churchill down truck", only the fifth row of the table is needed. The other information in the table is not helpful to answer the question. Similarly, in the picture, only the "racing brown horse" needs to be paid attention to. The other entities are of little help to answer the question. However, the previous method generates too much redundant textual information, which may mislead the model into generating incorrect answers. More importantly, there are an inherent reasoning relationships between these key pieces of information, which has been ignored by the current methods for interpreting and utilizing this logical relationship between them, avoiding the logical and factual errors when dealing with complex questions [17] .Different from previous methods with redundancy and having no interpretability, we innovatively treat multimodal multi-hop question answering as a joint entailment tree generation and question answering problem. To the best of our knowledge, this is the first work introducing entailment tree generation from a mutual-beneficial perspective, bridging small model and large model generation in multimodal multi-hop question answering.In the field of Natural Language Processing(NLP), entailment tree generation is an important sub-task in the question-answering task [6, 10, 20] while it has not yet been applied in the multimodal domain. The reason is that it is nontrivial to directly apply the existing method in NLP to our cases for benefiting the answering beyond the interpretability. During our attempt to directly transfer entailment tree generation methods to the multimodal multi-hop question-answering domain, we discovered that the current entailment tree generation methods have very low accuracy in generating entailment trees. Even predicting the structure of entailment trees yields similarly low accuracy. Therefore, we propose a new LLMsbased method that introduces the task of entailment tree generation into the multimodal multi-hop question answering task, to address both prediction and interpretability problems, and allows the introduced entailment tree to be iteratively refined.Although LLMs show promising performance in many tasks, we still have observed that the entailment trees generated by LLM mainly contain two types of errors: 1) incorrect selection of leaf nodes. 2) incorrect structure of the entailment tree. Therefore, we propose the LLM and smaller model interactively based framework to carry out the fact retrieval generation task and the questionanswering task. We use multi-task learning with smaller models, and use an iterative feedback mechanism to re-predict the structure of the entailment tree based on the leaf nodes and the answers predicted by the small model. Inspired by the idea of [9, 22, 23] , to facilitate mutual enhancement among the small models' multi-task learning, we employ a shared multi-task mixture-of-experts model, allowing interactions between the fact selection and supervised QA tasks as guidance for LLM.Specifically, in entailment tree initialization stage, we iteratively use large-scale language models [2] to decompose an existing multihop question into sub-questions that need to be solved, and completes them based on existing evidence (question, answer) for as facts to construct a fact base. For entailment tree generation [6] , since even predicting the structure of the entailment tree can only achieve a very low accuracy, which is much harder for our case, we propose to first generate entailment tree structure without the details, using the existing method of entailment tree generation [6] in NLP, then use large models to continuously fill in the values of the missing intermediate nodes in the entailment tree.Different from those in previous entailment tree generation tasks, where the leaf nodes and answers were provided simultaneously [6, 10, 20] , for multi-modal multi-hop question answering tasks, the answers are not visible during testing, so our definition of entailment tree generation tasks is different from the past [6] . When we build the entailment tree, the input is a set of leaf nodes (facts from the fact base) and a question as hypothesis, while previous methods require inputting a set of leaf nodes and answer. After initializing the entailment tree in our method, both the leaf nodes and intermediate nodes are filled (the intermediate nodes are generated by the LLM through the collection of their child nodes), and we do not predict the final answer during the initialization phase of the entailment tree, but predict the answer in the second stage. For a detailed definition of our entailment tree generation, please refer to section 3.1.2.Our method introduces entailment tree generation into the field of multi-modal multi-hop question answering, filters facts by generating entaiment tree, models logical relationships between different modalities, eliminates irrelevant information in multi-modal contexts, and maintains logical consistency.We conduct experiments on two public MMQA datasets, namely WebQA [3] and MultiModalQA [21] . We use accuracy, F1 score and reasoning path quality as evaluation metrics. Our experimental results show that our method achieve sota result on WebQA dataset. We also show the entailment trees generated by our method, demonstrating the effectiveness and explainability of our method. To the best of our knowledge, this is the first attempt to improve Multimodal Multi-Hop QA that uses entailment trees to constrain the process of converting multimodal information into text. The main contributions of our paper are as follows:• By constructing a fact base, we reduce information redundancy, and use the fact base to build an entailment tree to generate explicit reasoning steps, which assist the model in generating more accurate answers. • We introduce entailment tree generation into multi-modal multi-hop question answering. In order to correct potential errors in the entailment tree generated by the LLM, we proposed a multi-task mixture-of-experts model and iterative feedback mechanism. • We achieve state-of-the-art results on WebQA dataset, and achieve competitive results on MultimodalQA dataset, demonstrating the effectiveness of our method. 
 
 Appendix:  


Decision: This paper innovatively addresses Multimodal Multi-Hop Question Answering with an entailment tree generation approach. The reviewers noted that the introduction of entailment tree generation to multimodal QA is both novel and interesting, potentially pioneering a new path in this research area. The proposed method has secured first place on the official leaderboard of WebQA (multimodal and multihop QA) and demonstrated strong performance on other multimodal QA datasets.

However, a significant drawback of this paper lies in its writing and presentation. As noted by Reviewer dbca and other reviewers, the paper contains some typos and formatting issues. Additionally, the AC observed that the language is somewhat verbose and clarity could be improved.

Despite these issues, the AC believes the novelty and effectiveness of this paper outweigh its drawbacks in presentation and recommends acceptance. The authors should carefully revise the paper according to Reviewer dbca's comments, proofread to identify and correct potential errors, and incorporate rebuttal discussions into the final version.

Reviews: review: paper_summary: paper_summary: The paper presents an innovative approach to multimodal multi-hop question answering (MMQA), a complex task that necessitates reasoning across different modalities such as text, images, and tables. The authors identify two key challenges in existing MMQA methods: the presence of redundant information and the lack of interpretable reasoning steps. To address these, the paper proposes a novel method that integrates entailment tree generation with a multi-task learning framework and an iterative feedback mechanism. This approach is particularly noteworthy for its application of a mixture-of-experts model to enhance both the interpretability and accuracy of the predictions.
strength_weakness: strength_weakness: 1.The authors demonstrate the effectiveness of their approach through experiments on two public MMQA datasets, WebQA and MultiModalQA.
2. The introduction of entailment trees in the multimodal domain is novel. The use of a multi-task mixture-of-experts model to facilitate knowledge sharing and prevent task-specific errors is another innovative aspect of the proposed method.
limitations: limitations: 1. Some typos in Line 21.
paper_summary: paper_summary: The paper addresses challenges in multimodal multi-hop question answering (MMQA) by proposing a novel approach that integrates entailment tree generation with question answering, enhancing both interpretability and accuracy. The paper proposes a unified approach that treats MMQA as a joint problem of entailment tree generation and question answering. This is achieved through a multi-task learning framework that shares common knowledge across tasks while preventing interference between them via a mixture of experts.
strength_weakness: strength_weakness: - The paper innovatively introduces entailment tree generation into MMQA, which is a novel and groundbreaking attempt in this field.
- The proposed framework is both effective and interpretable, achieving SOTA performance on the WebQA dataset.
limitations: limitations: A significant drawback of this paper is its disorganized presentation. Specifically:

1. Some figure captions lack punctuation, such as in Figure 4.
2. All equations are missing punctuation.
3. There are issues with citation formatting, failing to distinguish between cite and citet usage, as seen in line 229 and line 223 with double citations [17].
4. Inconsistent use of quotation marks, as observed in lines 318, 319, and 647.
5. Numerous capitalization errors, such as "it" instead of "It" in line 344, and "gpt-3" instead of "GPT-3" in line 230.
6. References to figures and tables do not enable proper hyperlink navigation.
7. The symbols used throughout the paper are inconsistent and confusing, as noted in lines 404 and 442. The authors should reorganize the use of symbols.
8. The example text in Figure 2(a) is very small and difficult to read.
9. Table 6 is not referenced in the text, potentially corresponding to Section 4.6. The percentages 89% and 86.2% in Section 4.6 are not clearly sourced.
10. The paper’s header has a title and conference information overlapping.
11. The footnote in the caption of Table 2 is incorrectly formatted as "0".
12. There should be a space before citations, such as in line 713.
13. Line 269 inappropriately uses the term "rumors" to support the author's point in an academic paper.

Although I find the paper's approach innovative, the poor presentation makes me inclined to reject it.
paper_summary: paper_summary: This paper introduces an approach that combines multimodal multi-hop question answering with generative inference trees to improve the accuracy and interpretability of the reasoning process.
Through a hybrid expert model and iterative feedback mechanism, this method won the first place in the WebQA competition and achieved competitive results on MultimodalQA.
strength_weakness: strength_weakness: The advantages of this article include:

The joint method of inference tree generation and question answering is introduced to solve the problems of redundant information and unexplainable reasoning process in multimodal multi-hop question answering. By building an inference tree and filtering redundant information, the model can generate answers more accurately.

Combining the hybrid expert model and iterative feedback mechanism, it fully utilizes the advantages of large-scale language models and improves the performance and interpretability of the model.
limitations: limitations: The method of this paper is the same as other methods. In the end, it is also converted into text to infer the answer, but it is said that redundant information is filtered out. So the idea of ​​the article is confusing.

1. Is it necessary to use the tree generation method? The article does not compare some straight forward methods of filtering information.

2. Combining the tree generation and moe methods, is the performance improvement worth the extra computing cost and computing efficiency?
paper_summary: paper_summary: This work focuses on multi-modal multi-hop question answering and treats it as a joint entailment tree generation and question answering problem. Firstly, the author design a multi-task learning framework to facilitate common knowledge sharing between interpretability and prediction tasks. Subsequently, an iterative feedback mechanism is designed to iteratively refine potential answers by feeding the joint training results back to the LLM to regenerate the potential tree.
strength_weakness: strength_weakness: - For the first time, the paper combines entailment tree generation and question answering tasks from the perspective of mutual benefit to bridge the generation of large language models and small language models.
- The authors also designed a multi-task mixture-of-expert model and an iterative feedback mechanism to iteratively refine the generated answers.
- Experimental results show that competitive results are achieved on WebQA and MultiModalQA datasets.
limitations: limitations: - This work provides a very detailed introduction to the method, but **lacks a comprehensive and in-depth analysis in the experimental results**. The analysis of this work in 4.3 Results and 4.4 Ablation Study is too general and lacks in-depth analysis of the proposed module. In addition, since the author only used two datasets for experiments, it is recommended that the author add ablation results on the WebQA dataset to better prove the role of the proposed module.
- The paper uses an iterative feedback mechanism, but **the paper does not report the iteration round k and the number of tokens consumed** (input tokens and output tokens) during the experiment, which are very important for evaluating the actual usability of the method.
- Is Table5 w/o MMOE _directly concatenate the retrieved evidence and sub-questions and input them to GPT-3.5_ method mentioned in th paper on line 802? If so, does this indicate that **the improvement of the model may come from the capabilities of GPT3.5 itself, rather than the modules proposed in the paper?** Because on the MultimodalQA dataset, the improvement of the method in the paper compared to w/oMMOE is not obvious compared to other modules.
- **Have the authors tried replacing GPT3.5 with an existing open source model?** For example, LLaMA series and mistral series. Can open source models also achieve better improvements?
- Some formula errors: line 404 $q_tb$, line 610 $F_fact$, line 645 $p_ifm$
paper_summary: paper_summary: This paper constructs a fact base to reduce information redundancy. It uses this fact base to build an entailment tree, which helps generate explicit reasoning steps, improving the model's accuracy in answering questions. The study introduces entailment tree generation into multi-modal, multi-hop question answering. To correct potential errors in the entailment tree generated by the LLM, the authors propose a multi-task mixture-of-experts model and an iterative feedback mechanism. This approach achievesSOTA performance on twoMMQA datasets.
strength_weakness: strength_weakness: Novelty: This paper is the first to introduce entailment tree generation from a mutual-beneficial perspective, effectively bridging small model and large model generation in multi-modal, multi-hop question answering. This novel idea adds significant value to the field.

Technical Approach: The use of a MoE architecture combined with an iterative feedback mechanism is innovative. This approach addresses potential errors in entailment tree generation by leveraging the strengths of multiple models, thereby enhancing overall accuracy.

Clarity: The paper is well-written and easy to follow. The authors provide clear explanations and logical reasoning, making the complex concepts accessible to the reader.

Adequate Evaluation: The authors demonstrate the effectiveness of their approach by achieving state-of-the-art performance on two MMQA datasets. This strong empirical evaluation justifies the practicality and robustness of their proposed method.
limitations: limitations: Parameter Details: The use of the MoE architecture likely increases the number of parameters significantly. It would be beneficial to include detailed information on the parameter count and compare it with baseline models to understand the trade-offs between performance gains and computational costs.

Table 5 Clarity: The paper lacks clarity in Table 5, as there is no corresponding line that specifically addresses the ablation of the entailment tree generation modules. 

Iterative Feedback Mechanism: The paper does not specify how many iterations the iterative feedback mechanism performs. Including an ablation study on the number of iterations would provide valuable insights into the effectiveness and efficiency of this mechanism, helping to determine the optimal number of iterations needed for best performance.