Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models
Authors: Zhenyu Pan
Haozheng Luo
Manling Li
Han Liu
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
question answering
chain-of-thought
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We present a Chain-of-Action (CoA) framework for multimodal and retrieval-augmented Question-Answering (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is a novel reasoning-retrieval mechanism that decomposes a complex question into a reasoning chain via systematic prompting and pre-designed actions.  Methodologically, we propose three types of domain-adaptable `Plug-and-Play'  actions for retrieving real-time information from heterogeneous sources. We also propose a multi-reference faith score to verify conflicts in the answers.
In addition, our system demonstrates that detecting the knowledge boundaries of LLMs can significantly reduce both LLM interaction frequency and tokens usage in QA tasks. Empirically, we exploit both public benchmarks and a Web3 case study to demonstrate the capability of CoA over other methods.

Published as a conference paper at ICLR 2025
CHAIN-OF-ACTION FAITHFUL AND MULTIMODAL QUESTION ANSWERING THROUGH LARGE LANGUAGE
MODELS
Zhenyu Pan Haozheng Luo Manling Li Han Liu
Department of Computer Science, Northwestem University, Evanston, 60208, USA 4 Department of Statisties and Data Science, Northwestern University, Evanston, IL 60208, USA
(zhenyupan, hluo]@u. aluo)@u.northwese northnestthween...m edu (manling.1k li, munliujenorthwestere
ABSTRACT
We present a Chain-of-Action (CoA) framework for multimodal and retrieval- augmented Question-Answerigg (QA). Compared to the literature, CoA overcomes two major challenges of current QA applications: (i) unfaithful hallucination that is inconsistent with real-time or domain facts and (ii) weak reasoning performance over compositional information. Our key contribution is novel reasoning retrieval mechanism that decomposes a complex question into a reasoning chain via system- atic prompting and pre-designed actions. Methodologicallly, actions for retrieving real-time information from heterogeneous sources. We also propose multi-reference faith score verify conflicts in the answers. In addition, our system demonstrates that detecting the knowledge boundaries of LLMs can significantly reduce both system latency and LLM usage in QA tasks. Empirically, we exploit both public benchmarks and a Web3 case study demonstrate the capability of CoA over other methods.
propose three types
of domain-adaptable "Plug-and-2
INTRODUCTION
This work proposes new reasoning-retrieval framework to enhance the quality of Large Language Models (LLMs) question answering without additional training and querying costs. As exemplified in Figure this work overcomes three major drawbacks applying LLMs to answer complex questions: (i) unfaithful generation, where the response may not align with real-time or domain-specift facts (e.g. failing to localize relevant facts in Figure [()).. (ii) weak reasoning, where LLMs struggle to aggregate heterogeneous information sources, resolve their conflicts, adequately reason over the information to provide useful, tailored responses (such as the failure of the stopped analysis in Figure T Te) despite having successfully localized relevant search results), and (iii) inefficient process, where a large number interactions with LLMs and token usage are costly.
currently a good time buy
Chain Actions
Chain Thought
ReAct Agent
Token Name: BTC
can provide you with a step-by-step analysis to help you make informed decision
Current Price: $42426.71 Market Sentiment Analysis
mixed
Revearch and Undersand Biccon Defore investing Biicoin, it's essential good undersiadding
Sup I:
The search results provide information on the current state of Bitcoin, including recent news regulatory developments. However, does not provide clear answer whether it good time to Bitzoin. So the final is cannot currently good time to buy Bitzoin.
and negative cutlooks..
/
Technical Amalysis
Suep 2: Assess Your F'iunciall Situation your financial situaton tolerznce. Only invest money that you can affoed to loss, eryptocurreieiss can be
RSI currently 38.55 28, sugpesting
RTC not yet oversold...
Recommendation and Trade Strategy
beanish indicators market sentment, a short position...
Step & Stay informed Contimousll monitar the cryptocurenncy market and adjust your strategy neafal. Market conditions change mpldly.
Figure I: Chain-of-action prompting empowers LLMs to generate (1) faithful, informative, concrete analysis grounded in heterogeneous sources (open web, domain knowledge, tabular data, etc.) as well as (2) well-reasoned chains for complex questions better interpret human goals and intents. This stands superior to previous approaches that yield generic, ambiguous, high-level responses.

---

Published as a conference paper at ICLR 2025
Inccontent Learning
(i) (AC] qub--qussion
Monitoring
Missing Fllg
Final
Q1 now a good Jenu Io Diivest biocur?
CHAIN ACTIONS
Subri What is Bllooin
MF. Fasse
A1: Shooin.....
Model
Node?
SUBlZ Newa Bilcoin
Weeb-guery.ng
Newsr about
Conerruc meserring ang ypuedlen)) 10
Ncda8
Node5
Cumrent price
Tous
Al: Cureent price
is
and proo.
trww movd ti)o For wech dlep you action.....
SLBA Perfor mance Mankel
Tous
Krowicdge srooding
KSI Trernsss with
people has
poofive
No00G
Nodc4
Hatorical franda
MF: Fasse
Price DE...
Experts
Expert dlo
BLUA: Export opiniors
Trus
AS
have
Figure 2: Overview of offinonn framework. We use in-context learning to prompt LLM to generate the action chain. The chain has many nodes consisting of sub- sub--uestions (Sub), missing flags (MF), and L.L. L---eeerrtted guess answers (A). Then, the actions address multimodal retrieval of the nodes in three steps: (i) retrieving related information, (ii) verifying whether the L.M --geeerrreeed answer needs correction by retrieval, and (iii) checking if we need to fill in missing contents with the retrieval. Finally, we generate the final answer by the LLM based on the processed action chain.
To enhance faithfulness and multi-step reasoning, previous approaches such as chain-of tiiii--ff-hhuuhh based work [28 21 [32 330 encourage LLMs to think step-by-step to break down complex questions. However, only pushing models continue thinking may not be ideal. Models are expected to learn to pause to verify results and decide if they need more information before continuing to generate. into the reasoning chain.
Recent work, thereby, explores integrating information retrieval [3 [2] However, we argue that seeking external information is not only retrieval, but should manifest as configurable 'Plug-and-Play' actions: querying web text, encoding domain knowledge, analyzing tabular and numerical data, etc. The term "plug-and-play refers to the ability to freely add or remove pre-designed actions, such as the three different actions implemented in our work. However, for any new action to be integrated in the future, careful design and adjustment will be required to ensure compatibility with the framework's 's input and output formats. The key challenge of such heterogencous data is to automatically decide when to cease generation to solicit information, what types of external sources to leverage, and how to cross-vas valsddae conflicting insights. In addition, previous efforts to improve system efficiency have primarily focused accelerating the retrieval process. However, we observe that most of the cost lies in summarizing the retrieved information. By detecting the knowledge boundary-refering during its training on a high-quality dataset-we can reduce the need for frequent summarization. Since LLMs with extensive parameters have undergone costly training, leveraging this comprehensive knowledge can minimize unnecessary efforts during the filtering and summarization phases.
the parametric knowledge that the model has acquired
To that end, we propose universal framework CoA equipping LLMs to proactively initiate information-seeking actions. We design three 'Plug-and-Play" actions in this paper: (i) web-querying to extract real-time information as discrete text. tokens, (ii) knowledge-encoding to embed domain- specific knowledge concepts as continuous vectors, and (iii) data-analyzing for accessing and interpreting numeric tabular sources. A key advantage of this framework is the extensibility to diverse modalities, e.g., images in the future. Beyond adapting across data modalities, new actions can be introduced to handle emerging domains or data processing techniques. Additionally, our direct prompting strategy for detecting knowledge boundaries reduces both system latency and LLM usage.
In detail, as illustrated in Figure 2 the CoA first inject the question and action descriptions into the pre-designed prompting template through in-context learning. Then LLMS construct action chains (ACs), where each action node represents a sub-question, a missing-dat flag indicating the need for additional information, and an initial answer. After that, we perform action execution and monitoring to address retrieval demands in three steps: (i) retrieving related information, (ii) verifying conflict between the initial answer and the retrieved information, and (iii) inferring missing content with retrieved information when necessary. To verify information conflicts, we design a verification module utilizing our multi-reference faith score (MRFS). If the generated answer confidence is below threshold, the corresponding action incorporates the retrieved information for answer correction. In this way. L.LMs can generate the final answer that is sound and externally- grounded. A key feature of CoA is automatically solicit external information that forms as tokens, vectors, or numbers for integration into model reasoning. Rather than hard-coding their connections, actions are designed as datnset-agnostic modules that L.MMs invoke selectively.
The significant improvement of CoA is not only showed in experiments on multiple QA datasets, but also is validated from the success of the real-world deployment. Upon integration into a Web3 QA application, key metrics including active users and positive feedback volumes increased remarkably within a few months. This performance highlights CoA's effectiveness in applications.
2

---

Published as a conference paper at ICLR 2025
When launchad
Web-querying
2010
Q1: Did lphone and Samsung 510 leunch the same year?
Sub2: When S10 launched
Wab-querying
2019
Final Answeer
2018 is not the come as 2010, the answer
(CHAIN) ACTIONS
Sub1: What Beccin
Web-quaryinn
Bitcoin is digital Currency.
Sub2: News Blicoln
Web-queryiing
News Bricoin..
Sub3: price Bicco
Date-anslyzing
35000S one
Is now good Time invast biooo??
Sub4: Performance Market
Data-analyzzng
Info Market
Irends
Knowledge enooding
means..
Subé: Expert opinions Bitoonn
Web-querying
Options
Based on current news and peoe, people has pasilive view to Bitsoin Exparts aloo howe.
Figure 3: Two samples from our Chain-of-Action Framework.
In summary, our main contributions are as follows:
We present CoA, which integrates a novel caasoning-retrieve mechanism decompose complex questions into reasoning chains of configurable actions via systematic prompting. It can retrieve heterogeneous information and reduce information conflicts.
We propose three types of Plug-and-Playy domain-adaptable actions to address retrievals for real-time information, domain knowledge, and tabular data. The actions are flexible to incorporate additional sources.
We propose a novel metric, multi-reference faith score (MRFS), to identify and resolve conflicts between retrieved information and LLM ALM-gennrated answers, enhancing reliability. It also significantly reduces both LLM interaction frequency and token usage.
Our Weh3 QA product shows significant user engagement and positive feedhack, validating the CoA framework's ' effectiveness and practicality in real-world scenarios.
2 METHODOL.OGY
In this work, we propose 'Plug-and-Pae framework adaptable to different data modalities, currently capable of handling text and tabular data. The current focus is to validate the framework's efficacy in these two modalities, laying a solid foundation for further integration of additional modalities. This intention is also a significant direction for our future work involving Vision Language Models. As shown in Figure 2 we first introduce how to generate the action chain by LLM (Sec. 2.1 Then, the actions address multimodal retrieval demands of the chain's 's nodes in three processes: (i) retrieving related information, (ii) verifying whether the (LM-generated answer is good enough in demand of more information from retrieval, and (iii) checking if the initial answer of each node's sub-question is missing that fill in missing contents with the retrieved information. Finally, get the final answer by the LLM based on this refined and processed action chain.
2.1 ACTION CHAIN GENERATION
We use in-context learning to generate an action chain by LLM. As shown in Figure 2 (a), we design prompt template to decompose the user's question into many sub-questions, as well as the corresponding Missing Flags (MF) and guess answers shown in Figure (b). Then, we assign one of the actions to solve each sub-question.
Prompt design. We design prompt template as shown in Figure 4 starting with "Construct an action reasoning chain for [questions]..." to prompt LLM generate an Action Chain AC not answer our question Q directly.
ACQ (Action,, Sub,, MF A1); (Action2, Sub2, MF2, A2),....
(I)
(Action,, Sub,, MF An.)
Each action node represents four elements, including Action; the sub-questions Sub;, the missing flag MF; the guess answer from LLMs A,, where i (1,..., When the inner-knowledge of LLM is enough answer the sub-questions, LLM generates initial answer as the answer". Otherwise, the value of "missing flag" becomes "True". followed by a blank "guess answer".
3

---

Published as a conference paper at ICLR 2025
Construct an action reasoning chain for this complex [Question]: 'SQUESTION' in JSON format. For each step of the reasoning chain, choose an words), Knowledge-encodrng Engine (search existing damain calculate some results)
action from three choices: querriggrying Engine(search real-lime analyzing Engine (query value also generate sub question for each action to get one of |web search keywords, needed information desorption, data features descrption] as the value element 'Sub". Also, generate an Initial answer for each Sub the value of the element Guusss answer" if you make sure it correct. In the element "Miaaing fleg" value "False", otherwise, generate the final enswer for the [Question] by referning to the "Action" "Su""""""""""" "Quess_ answer" "Missing flagg in "Chain", as the value of the element "Final answer". For example:
information
knowlodge base),
aloment' "Action', and
addition, you cannot
sub-questions,
"True" You
try
(QQeesion": "Is it good invest in Bicoin now? is good time. B. is good time.".
"Chain":
("Action": "Knolledee encooding. "Sub": "what is bitcoin", "Guess, answerr: "Bitcoin one of the eryptocurrencies.. ", "Missing.
"Faise"),
("Action": "Deta-nnalyzinng" "Sub": "What the
trend bitcoin?", "Guess_ answer": "The Bitcoln increases from XX
Yy..,, "Misaing flag": "False").
("Action": "Web-querying", "buttinbitooin "bitcoin news", "Guess answer"; "", "Missing, flag": "True" 'H..
"Final snswer": "Bitcinn is one the eyplocurrencces that is risky invest And its price become more and more high recendl Also, there is of news to promote Bitcoin. So, good invest in Bitoonn now.")
Figure 4: Prompt to Generate Action Chain in Chain-of-Action (CoA). This template integrates the user's question along with a description of each available action. The resulting action chain comprises elements such as actions, subs, guess answers and missing flags. This prompt not only decomposes complex questions into multiple sub-questions, guided by the features of the actions but also allows the LLM to answer certain sub-questions using its existing know-k.oo.eege. This process exemplifies our proposed reasoning- retrivval mechanism.
2.2 ACTIONS IMPLEMENTATION
We propose three types of actions to address multimodal retrieval demands (text and tabular data): (1) Web-querying for searching real-time text data from websites, (2) chooledge-encoding for retrieving related text data from local domain-specific corpus datasets, and (3) Data-analyzing for extracting tabular data from domain-specific databases via generated SQL codes. Each action has three steps to execute: (i) Information Retrieval, (ii) Answering Verification, and (iii) Missing Detection. We first introduce the design of the actions. Then, we describe the details of three common steps. When combined with textual data, we want to demonstrate that tabular data forms critical part of heterogeneous imputs, significantly enhancing system understanding and response capabilities.
2.2.1 DATA COLLECTION
Action 1: Web-querying. Web-querying action utilizes the existing search engines (c.g.. Google Search) and follows our query strategy get the relevant content from the Internet. In detail, first searches for the keywords of the given sub-question Sub,, to obtain the result list. If the corresponding "Missing flag" is "True", we choose the top-k results and extract their contents from their page sources. Otherwise, we combine their titles and snippets Sn of the top M pages. Then, we transfer each pair of title and snippet Sn, } into a 1536-dimension vector Brub(Tm/Snna the embedding model (fext-embeddding--aa--l)
by from OpenAl (17). Meanwhile, we also transfer the sub-question and guess answer (Sub,,, Ar into Emb(Sub,, J. Next, we calculate the similarity between each Emb(T m [Snm } and ( |A to filter the pages whose similarities are lower than 0.8. Then, we extract the contents of high-similarity pages and calculate the similarity between them and Emb(Sub,,|A,, } to rank and get the top-k final pages. Those contents of the k final pages are the final information that we retrieve by the action.
Knowledge-encoding action utilizes the vector database (c.g.. ChromaDB) as data storage to store the domain information and corresponding embedded vectors. For example, we collect web3 domain information from different sources (X, experts' blogs, white papers, and trending strategies) to support our QA case study. After data collection, we split each document into many chunks based on the length. Then, we encode each chunk of content into an embedded vector and store it in our vector database with its index. When we need to execute this engine to retrieve domain information, we could forward the Emb(Sub,,lA,) to compute the similarity between the input and each chunk to obtain the top-k results.
Action 2: Knowledge-encodiing
Action 3: Data- Data-analyzing. Data-analyzing action aims to retrieve the data information from some real-value data sources (e.g., market data of digital currencies). In some special situations, we could directly retrieve the relevant values from our deployed API when some sub-questions demand up-to-date or historical value data. Furthermore, we can also use L.MM to compute more sophisticated features by generating Python or SQL. codes to execute. It is flexible and compatible with various situations. In this paper, we only design it to retrieve the market data for the Web3 case.
2.2.2 DATA VERIFICATION
In the action chain, the framework executes the data collection for each node until it finishes the whole chain, as shown in Algorithm []
4

---

Published as a conference paper at ICLR 2025
Information Retrieval. In the information retrieval stage, need find the most relevant and similar contents from different knowledge/data sources. At first, we choose both sub- sub-questions and guess the answer of each node as a query section, QS,- Then, with the encoding of LLM's embedding model, we transfer our query QS, (Sub,, A,) into a I 1536-dimension vector Emb(QS,) With this embedded vector, can perform information retrieval and then rank the results by calculating the similarity. Finally, actions return the top-k results Riqsy;
RiQs) = (ri r2 /r.....17/ Tr.)
(2)
Answering Verification. After the information retrieval, we verify the information conflicts be- tween guess answer A, and retrieved facts R[os)- Inspired by the ROUGE [13), we propose the MRFS. To get the MRFS, we compute the pairwise faith score between candidate summary and every reference, then take the maximum of faith scores. S is composite metric computed based on three individual components: Precision (P), Recall (Rcl), and Average Word Length (AWL) in the Candidate Summary. The mathematical representation of the score is given by:
= a x P+ Rel + X AWL
Where:
a, B are weights corresponding to the importance of Precision, Recall, and Average Word Length, respectively. Their values can be adjusted based on specific requirements but should sum up to for normalization purposes.
P (Precision) is the ratio of relevant tokens among the retrieved tokens:
number of relevant items retrieved
P total number of items retrieved
Rel (Recall) is defined as the ratio of relevant tokens that were retrieved:
number of relevant items retrieved
Rd total number of relevant items
(5)
AW L (Average Word Length in Candidate Summary) represents the mean length of the words present in the summarized content:
sum of lengths of all words
AWL = total number of words
(6)
Adjusting the weights a, B, will allow for emphasizing different aspects (Precision, Recall, or Word Length) depending the specific evaluation criteria or context. After getting the MRFS through: MRS argı max Srr,, Az), we setup a threshold T to decide whether the answer A; is faithful. If MRFS is greater than T. we keep the answer; otherwise, we change the answer A, to reference contents. As shown in Fig given generated text A, ("david had an apple and a banana") and a reference text rk ("david is good person, and he got apple, banana, and oranges.") "), we first tokenize both texts: A, tokens: ["david", "had", "an", "apple", "and", "a", "banana"] and Tk tokens: ["david", "is", "a", "good", "person", "and", "he". "got", "an", "apple", "a", "banana", "and", "oranges"]. Then, for precision (P), we calculate the ratio of tokens in A, that are also found in rı. the intersection of tokens are ["david", "an", "apple", "and", "a", "banana"] and the result is P = & R 0.857. For recall (Rcl), we calculate the ratio of tokens in ry that are correctly predicted by A; and get Rdl = ~ 0.429. For average word length (AWL), we calculate by AWL 5+3+245433 141666 3.57.
Missing Detection. The last stage of each action is detecting whether the guess answer A; is complete. When sub-question needs some special real-time information, the corresponding guess answer A; could be incomplete with a Missing Flag F; being "true". If guess answer's MF is "True", we inject the retrieved information into the A; to fill in the blank "Guess_ane _ansser..
2.3 FINAL ANSWER GENERATION
After all actions' executions, we use a prompt template shown in Figure to to integrate all corrected answers and sub-questions of the AC. Then, we prompt LLM with retrieved information and generate the final answer starting with "[Final Content]" through the corrected reasoning chain.
5

---

Precision Calculation Published as a conference paper at ICLR 2025
A (generated text) "david had an apple and a banana*
(reference text) "david is good person,
got apple, banana, oranges."
We need calculate Precision (P), Recall (Rel), and Average Word Length (ANL) using tokens units.
# Break down the sentences into tokens
tokenss
apple
banana". split()
R_tokens *david is good person, and he got an apple, banana, and oranges. i spllit()
Calculate Precision and Recall
Precision: Proportion of relevant instances among the retrieved instances
Recall: Fraction relevant instances that were retrieved
appear in beth and Relevant are (ttaeem))iieereeetiinn_..oo.
relevant_token
precision 1entzelevent_tokens) / len(A_tokens)
e Recall Calculation
recall len(zelevant_tokems) 1en(R_token)) tokens)
Average Word Length (AKL) Candidate Summary ANL is calculated as the sum of the lengths all words divided by the total number of words in awl sum(len(word) for word in A_tokens) / [en(A_tokans)
precision, recall, awl
Figure 5: The pseudo codes about how calculate the MRFS.
Table 1: The functional comparison of Chain-of-Thought baselines with our method CoA.
Method
shot
CoT
SC
ToT
Auto
CaT
lo-
Least-
ToolFormer
Self-Ask
React
DSP
SearchChain
CoA
Multistep Reasoning
Retricval
Multinuoall
Verification
3 EXPERIMENTS
In this section, we compare the performance of our Chain-of-Aciio framework with state-of-the-ar baselines across public benchmarks. Subsequently, we provide a detailed analysis of our launched case study: Question Answering (QA) application in the Web3 domain.
3.1 EXPERIMENTS WITH BENCHMARKS
Datasets We select 4 classic, form, and open-domain QA task. Four classic QA tasks that include web-based QA (WQA) (2) general QA (DATE, General Knowledge, Social QA (SoQA)), Truth QA (24), Strategy QA (SQA) 6 and Fact Checking (FEVER (26)) Long-form QA task is the first long-form QA dataset focusing on ambiguous factoid questions, ASQA (25]. Open-domain QA task is QReCC [1], testing the ability to handle context-dependent queries across different domains.
Metrics Cover-EM 20 are often used to represent whether the generated answer contains the ground truth. However, it has many limitations inherent in token- based exact-mater scoring methods, which may capture model's understanding of complex queries and can lead misjudgements. So, we propose GPT-EM to overcome the limitations in our evaluation process, aiming to provide accurate and nuanced assessment of performance across different prompting and agent framework. The prompt of GPT-4 that we use is shown in Appendix D Besides GPT-EM, we also use ROUGE-L in ASQA. Unlike the ROUGE metric in Table which serves bascline for our MRFS verification metric, ROUGE-L in ASQA focuses on the Longest Common Subsequence (LCS) assess fluency and coherence in diverse long-fore answers.
Baselines We have two types of baselines: the first type focuses on reasoning, prompting LLM to solve complex questions (Few-shot Prompting, Chain-of-Thought (CoT) [29]. Self Consistency (SC) 28). Tree of Thought (ToT) [32] Least-to-Most [38], and Auto-Chain-oe of ohouhhtht (Auto-CoT) 37). and the second Retrieval-A Saw.raal--uummeeeeGeeeeee
(RAG) type that integrates Information Retrieval to
enhance reasoning capabilities (ToolFormer [22]. Self-Ask [18), React (34), SearchChain (SeChain)
31 and DSP We conduct thorough functional comparison between these baseline methods
https://github.comtt https///gtttuubbcccoooggg/////e/

---

Published as a conference paper at ICLR 2025
We conduct an evaluation of accuracy for 7 classical QA. 1 fact-checking dataset, and long-form QA. Our study involves the baseline methods alongside our CoA method. assess the performance of these methods across seven tasks, considering both information retrieval and non- retrieval scenarios. The results average over three runs, are presented with variance values omitted (all 2%). Our presentation format involves bolding the best results and underlining the second-best results. Our findings highlight the superior performance of CoA, which achieved the highest accuracy in 12 out of 14 test scenarios. Notably, CoA consistently outperforms all baseline methods, even when external memory was not employed, demonstrating its robust and top-tier performance. Black
Table 2:
GPT-EM, and Red means ROUGE-L
Question Answering
Fact Checking FEVER
Long Form ASQA
Method
Web
DATE
GK
Social
Truth
Strategy
QReCC
Without
Information
Retrieval
Zero-shot
43.6
91.0
73.8
65.9
50.0
30.2/17.4
Few-shot
44.7
49.5
91.1
74.2
68.9
65,9
18.4
50.7
34.5/20.9
CoT (29
42.5
43.7
88.1
71.0
66.2
65.8
30.6
40.4
47.4/21.1
SC (28
36.5
50.0
87.5
60.0
66.7
70.8
53.3
34.8/20.3
ToT 32
32.3
47.1
85.1
68.5
66.6
43.3
20.4
41.2
32.5/10.4
Cot 37
42.1
52.3
89.7
39.1
61.6
65.4
21.0
32.5
36.3/21.0
Lest-to-Most A8
44.0
42.1
80.8
68.1
59.5
22.4
43.4
39.1/23.7
SeChain IR
44.7
75.0
64.9
54.1
75.6
35.9
35.7/16.4
CoA w/o actions
64.7
55,3
91.4
80.2
63.3
70.6
37.2
54,2
47.9/22.6
Interaction with Information
Retrieval
ToolForer 22
34.5
$3.9
72.3
48.1
57.5
69.4
50.0
60.2
37.1/20.5
Self-Ask 18
31.1
55.1
79.7
52.1
60.5
64.2
32.5/15.3
React 34
38.3
85.1
65.8
59,9
70.4
37.3
43,9
45.0/18.8
DSP 9
59.4
48.8
85.1
68.2
58.4
72.4
38.1
62.2
55.0/13.1
SearchChuin
87.6
61.7
65.9
45.2/21.9
CoA (MRFS verification)
70.7
57.4
98.6
83.1
67.3
79.2
69.7
68.9
60.9/29.1
-w/o verification
66.9
56.8
95.7
81.5
65.0
75.2
66.3
65.7
55.7/24.0
-w/o imputation
67.4
56.3
97.1
82.9
65.8
76.5
63.8
65.3
54.9/24.3
ROUGE
96.2
81.9
65.7
76.3
67.2
56.2/26.8
-w/o Action 1
65.0
55.9
89.3
81.5
64,2
75.8
51.5
65.3
55.2/25.0
-w/o Action 2
68.1
56.3
95.2
82.5
65.5
76.2
66.8
67.0
55.9/26.0
Here is the corrected reasoning chain for this complex [Question]: "(how's buying bitcoin}". Each step of the reasoning chain has [Sub-question] and [Solved Answer]. Answer the [Question]: "(how's buying bitcoin)" starting with [Final Content] through the reasoning chain.
For example:
[Question]:" Is good to invest bitcoin now?"
[Sub 1]: What is bitcoin?
[Solved Answer]: Bitcoin is one of the digital concurrencies...
[Sub 2]: What is the recent price trend of bitcoin?
[Solved Answer]: the price of Bitcoin increases by 3018 dollars in...
[Sub3]: J: news of bitcoin
[Solved Answer]: One news shows that ... And it promotes Bitcoin a lot... [Final Content): Bitcoin is one of the cryptocurrencies that is risky to invest [1]. And its price become more and more high recently [2]. Also, there is a lot of news to promote Bitcoin such as... [3]. So the answer It is good time to invest Bitcoin now, but you need to consider the risk of investing in cryptocurrency.
Figure 6: Prompt for final answer generation. We use the processed chain prompt LLM to reanswer.
and our Chain-of-Action (CoA), as presented in Table The "-w/o imputation" means that do not use retrieved information to fill the blank answers sub-questions that LLM cannot answer.
Our experimental framework integrates data preprocessing inspired by Big-Bench and Auto-COT B7]. In generating process, all baselines (including CoA) use gpt-3.5-turbo 16 the backbone. To address challenges in controlling response formats when working with black-box models, we developed an advanced evaluation pipeline that leverages GPT-4 3. GPT-4 is used exclusively in the evaluation stage assess the alignment of the generated answers from all baselines (including CoA) with the ground truth. Details of the evaluation prompt used can be found in Appendix [D] This setting ensures fairly comparison among all baselines while utilizing GPT-4's advanced evaluation capabilities to ensure consistent and reliable assessment.
Implementation.

---

Published as a conference paper at ICLR 2025
Table 3: Usage comparison across benchmarks. Each block labeled as A+B/C represents the number of tokens used for the input (A), the output (B), and the time consumed in seconds (C) by the LLM.
SQA
WQA
SoQA
FEVER
Self-ask ReAct
811+15/0.87
805+19/1.03
881+21/1.03
860+19/0.88
7365142920/99.8
20273+1331/65.3
479514954/16666
37409+1858/67.7
SeChain
82120+3215/132.99
61097+1190/239.1 4
45561+2072/105.3
CoA (ours)
30485+1120/43.1
11873+605/35.6
26011+429/58.3
18824+1021/30.5
3.1.1 EXPERIMENTAL ANALYSIS
Overall Performance. Table 2 compares the effectiveness of our CoA framework and 11 baseline methods across 7 classical QA, fact-checking. and long-form QA datasets. We evaluate the performance both information retrieval and non-retrieval scenarios, separately. The sole exception pertains to React, implemented by Langchian [27]. It exhibits an unresponsive behavior in the DATE dataset. As result, we omit the comparison involving React within the DATE dataset. Our CoA framework demonstrates superior performance metrics in 12 of 14 test scenarios. Our method achieves a significant 3,42% improvement in the test tasks without information retrieval compared to the state-of-the-art baseline (SearchChain without IR), and a 6. 14% increase in the test tasks with information retrieval over its state-of-the-ar arrt baseline (SearchChain), This is significant outcome, as it underscores the effectiveness of our framework. It also demonstrates that CoA is for various puestion-answerte is consistent regardless of the integration of IR. This indicates that our framework has intrinsic robustness and comprehensive understanding that is not reliant external information. We also find our CoA without IR is impressive. After deeply explore how the process of generating and answering sub-questions contributes to these improvements in Appendix we get:
tasks. In particular, the enhancement in performance
Comparative Insights: Our CoA offers a richer analysis by integrating multiple aspects of the scenario into comprehensive reasoning chain. This approach addresses the direct question and contextualizes Alex's 's actions within his duties and the prison's 's operational protocols, providing multidimensional understanding. In contrast, CoT tends to a more straightforward, surface-level interpretation. This explanation underscores how CoA's 's approach provides deeper, more contextually enriched answers compared to COT, making particularly effective for complex scenarios requiring a nuanced understanding of actions within their broader social and procedural context.
Complexity of reasoning processes. In further analysis in Table we delve into the complexity of reasoning processes in baselines. Our framework exhibits higher average number of reasoning steps when decomposing complex questions. This metric vital, highlighting the framework's 's capability to engage in multi-step inference process. capability that is essential for solving intricate problems that require more than surface-leve understanding. The fact that our framework outperforms others in this measure suggests that it can better understand and navigate the layers of complexity within questions, which is a testament to the sophisticated reasoning algorithms it employs.
Efficiency of current framework. Table 3 and Table 4 explores the average system latency and LLM usage per question. We choose these 4 datasets compared to 8 datasets in Table 2 because they represent more complex tasks compared to the others. These datasets require higher number of reasoning steps and exhibit a greater mislcading ratio by IR, which presents a more challenging benchmark. We believe focusing on complex datasets allows us to more distinctly demonstrate the superior performance of CoA in scenarios that are not only more demanding but also more indicative of real-world applications. CoA shows reduced cost, reflecting the CoA's efficiency in minimizing system latency and LLM usage. It is a vital attribute for addressing complex issues with lower expenditure. suggests that CoA surpasses others with detecting knowledge boundaries of LLM.
Misleading of external knowledge. Table scrutinizes the methods in terms of their susceptibility to being misled by external knowledge. This nuanced aspect of framework evaluation, as speaks to the framework's 's ability to discern relevant from irrelevant information, nontrivial task in the age of information overload. Our framework emerges as the most resistant to misinformation, maintaining high accuracy even when interfacing with external data sources. This reveals not only the advanced data parsing and filtering capabilities of CoA but also its potential to mitigate the risks associated with the proliferation of false LLM L.M generated information.
Robustness of current framework. The verification of the correctness and relevance of decomposed sub-queries and model decisions is important. We employ rigorous evaluation Actions framework.
8

---

Published as a conference paper at ICLR 2025
Table 4:
We perform thorough analysis to pare the average number of interactions with LLM across four datasets. The results, obtained through three separate runs, are displayed without includ- ing variance values (all < 0.4%)
Table 5: We perform analysis showing that
ternal knowledge leads LLM astray in answering using baseline methods. Our study takes place in context involving information retrieval tasks.
WQA
SQA
SoQA
FEVER
WQA
SQA
SoQA
FEVER
Self-Ask
14.3
10.3
14.1
10.7
Self-Ask
5.3
5.0
5.0
5.1
React
16.1
10.0
15.8
11.2
React
5.2
5.2
5.3
5.5
DSP
13.5
9.2
14.3
10.1
SeChain
6.4
6.7
6.0
5.6
SeChain
7.2
5.3
9.4
8.5
CoA
4.0
4.0
4.4
4.2
CoA
1.9
2.6
6.1
3.4
Table 6: Comparison of SOTA Daselines-Reat (Rt) and Self-Ask' (SA)-with our CoA method in the Web3 case, evaluated on coverage, non- redundancy, and readability. The results, aver- aged over three runs, are displayed without in- cluding variance values (all 0.4%). For the CoA results, the left side is the performance with- out tabular action, while the right side is the full version.
Table 7: We conduct analysis of the average number reasoning steps to demonstrate the intricacy of test tasks. Our study takes place in a non-information retrieval context. The re- sults, obtained through three separate runs, are displayed without including variance values (all S 0.1%).
WQA
SQA
SoQA
CaT SC Auto-CoT
2,2
Rt
SA
DSP
CoA
2.1
2.1
2.8
Coverage
1.5
1.8
1.7
2.0/2.9
3.2
2.9
3.0
Least-to-Most
1.2
1.2
1.8
Non-redundancy
2,0
1.9
2.1
2.2/2.3
Self-Ask w/o IR
2.1
2.9
Readability
2.1
2.1
2.0
2.5/2.7
SeChain w/o IR
3.4
Overall
1.9
2.0
2.0
2.0/2.6
CoA Actions
3.9
To ensure robust assessment, randomly selected 200 questions from the Social QA dataset, which represents a maximum average number of reasoning steps. We then generated action chains for each question and assessed the correctness and relevance of the resulting sub-questions. We employed human annotation to evaluate the results, which yielded a high ratio correctness at 95.5% and relevance 97.0%. These results indicate a strong alignment between the decomposed sub- queries, the model decisions, and the expected outcomes. Furthermore, recognizing the importance of scalability and efficiency evaluations, we are exploring the potential for automating the process of assessing relevance and correctness. This includes thorough literature review to identify and integrate more explicit methods into our experimental framework, enhancing our ability verify sub-questions and actions.
In conclusion, the empirical evidence from our assessments presents a compelling case for the superiority of our framework. It excels in understanding and answering complex queries, demonstrates advanced reasoning capabilities, and exhibits resilience against the pitfalls of external misinformation. These findings position our framework as new benchmark in the realms of question-answering and fact-checking, underscoring its comprehensive superiority.
3.2 CASE STUDY WITH WEB3 QA APPLICATION
We also apply our framework to develop QA application in the real-world Web3 domain. Users can ask this QA system up-to-date questions about the Web3 domain. Our system automatically decomposes the user's question into many sub-questions and solves them one by one. In the solving sub-questions process, the system considers injecting knowledge from different sources, such as search engines, existing domain knowledge, and even market databases. Figure @ illustrates system's website interface. Despite having a substantial user base and positive user feedback, we rely expert evaluation assess
study and showcase the framework' 's real-world performance.
We design an expert evaluation to assess the quality of explanations and reasoning. Our experts rate explanations a to scale (with 3 being the best) based criteria:
Expert Evaluation.

---

Published as a conference paper at ICLR 2025
The explanation and reasoning should cover all essential points important for the fact-checking process.
Coverage:
Non-redundancy: The explanation and reasoning should include only relevant information necessary to understand and fact-check the claim, avoiding unnecessary or repeated details.
Readability: The explanation and reasoning should be clear and easy to read.
Overall Quality: This is general assessment of the overall quality of the generated explanation and reasoning.
We design an expert evaluation to assess the quality of explanations and reasoning trajectories. Our experts rate these explanations on a to 3 scale (with 3 being the best) based on several criteria: We randomly sample the 100 questions from real users' question history and use React, Self-Ask, and our CoA to answer these questions. Details about expert evaluators selection are shown in Appendix [ Table shows the averaged scores of the expert evaluation. It reveals that CoA outperforms others in expert evaluations, demonstrating its ability to deliver responses that are both more readable and less redundant compared to baseline methods. In summary, these results demonstrate that our framework can get the best performance the real-world scenario.
4 RELATED WORK
We review the literature about prompting methods, agent frameworks, tool learning, and hallucination methods. Owing to page constraints, the contents of tool learning and hallucination methods are relegated to the appendix C
Prompting methods. The key to prompting is to lead LLMs' behavior to follow the instructions. The generic way few-shot prompting & enables in-context learning, guiding LLMs to follow instructions and answer questions with only few examples. CoT [29] and its improved prompting versions [28 21 performance. However, they still only support the text information and can not generate the newest information, which is not included in training data.
try to lead the LLMs to decompose a complex task into a reasoning chain and get better
Agent frameworks. Many frameworks aim expand both the ability and knowledge edges of LLMs. ReAct [33 allows LLMs to interact with external tools to retrieve additional information. Self-ask [18] repeatedly prompts the model to ask follow-ur up questions construct the thought process through the search engine. However, these frameworks do not fully harness LLMs' intrinsic knowledge to solve any inner question in the answering process. And they also do not consider the conflicts between LLM-generated content and retrieved information. Search-in Beaec--itthh-Chaain (31) relying on the Dense Passage Retrieval (DPR) tries to verify information in the reasoning chain. However, its processing is so complex and sequential that it costs inevitable L.M usages and causes corresponding high latency. Moreover, still cannot support multimodal data processing. While Chain-of Knowledge [1] augments LLMs by incorporating grounding information from heterogeneous sources, it highly relies on the fine-tuning of one more LLMs to generate queries sequentially. addition, it cannot support real-time information. Therefore, we propose more efficient CoA framework that needs no training cost and supports real-time information. Most importantly, our CoA framework solves sub-questions parallelly, ensuring efficiency.
5 CONCLUSIONS AND FUTURE WORK
We introduces the Chain-of-Action (CoA) framework, an innovative approach designed to enhance LLMs capabilities in handling complex tasks, particularly in scenarios where real-time or domain- specific information is crucial. We also propose a efficient verification module utlizing our MRFS to correct the L.LM- L..--genertted answer by retrieved information. The system successfully demonstrates that detecting the knowledge boundaries of LLMs can improve the efficiency a lot in QA tasks (tokens usage and number of interactions with LLMs). notable application of CoA is in a Web3 Question Answering product, which demonstrates substantial success in user engagement and satisfaction. It exemplifies the framework's potential in specialized, real-world domains.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Han Liu is partially supported by NIH ROILM1372201, AbbVie and Dolby. Haozheng Luo is partially supported by the OpenAl Researcher Access Program. Manling Li partially supported by the Stanford Institute for Human-Centered Artificial Intelligence (HAI). NSF CCRI #2120095, AFOSR YIP FA9550-23-1-0122 ONR MURI N00014-22-1 1--22-1-2740, ONR YIP N00014-24-1-211 Amazon, and Microsoft. This research utilized computational resources and staff contributions from the Quest High-Performance Computing Facility at Northwestern University, supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. The content remains the sole responsibility of the authors and does not necessarily reflect the official views of the funding agencies.
REFERENCES
[1] Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. Open-domain question answering goes conversational via question rewriting. arXiv preprint arXiv:2010.04899, 2020.
[2] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answe answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544 Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[3] Marialena Bevilacqua, Kezia Oketch, Ruiyang Qin, Will Stamey, Xinyuan Zhang, Yi Gan, Kai Yang, and Ahmed Abbasi. When automated assessment meets automated content generation: Examining text quality in the era of gpts. arXiv preprint ar3iv:2309. 14488, 2023.
[4] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 12712, 2023.
[5] Yirui Chen, Xudong Huang, Quan Zhang, Wei Li. Mingjian Zhu, Qiangyu Yan, Simiao Li, Hanting Chen, Hailin Hu, Jie Yang, et al. Gim: A million-scale benchmark for generative image manipulation detection and localization. arXiv preprint arXiv::4066. 16531, 2024.
[6] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use Laptop? Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL), 2021.
[7] Guangzeng Han, Weisi Liu, Xiaolei Huang, and Brian Borsari. Chain-of--intracce Enhaneing large language models for psychiatric behavior understanding by dyadic contexts, arXiv preprint arXiv:2403.1 13786, 2024.
[8] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
[9] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demoosttaatessaacch-preeee Composing retrieval and language models for knowledge-intensiee NLP. arXiv preprint trXiv:2212.14024 2022.
[10] Zhixin Lai, Xuesheng Zhang, and Suiyao Chen. Adaptive ensembles of fine-tuned transformers for -enegreerteed text detection. arXiv preprint arXiv:2403.13335 2024.
[11] Miaoran Li, Baolin Peng, and Zhu Zhang. Self-checker: Plug-and play modules for fact- checking with large language models. arXiv preprint Stt.:2305...62.. 2023.
[12] Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Poria. Chain of knowledge: A framework for grounding large language models with structured knowledge bases. arXiv preprint arXiv:2305. /3269, 2023.

---

Published as a conference paper at ICLR 2025
[13] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[14] Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schomstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robol Learning, pages 1084-1110. PMLR, 2023.
[15] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint (xXi:23303.0899 2023.
[16] OpenAl, Aug 2023.
[17] OpenAl. Gpt-4 technical report, 2023.
[18] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
[19] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint 0305:22304.08354 2023.
[20] Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, and Saurabh Tiwary. Knowledge-aware language model pretraining. arXiv preprint arXiv:2007.00653 2020.
[21] Abulhair Saparov and He He. Language models greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.0/240) 2022.
[22] Timo Schick, Jane Dwivedi-Yu, Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda. and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. 2023.
[23] Timo Schick, Jane Dwivedi-Yis Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprini arXiv:2302.04700 2023.
[24] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprini ar arXiv:2206.046155 2022.
[25] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092, 2022.
[26] James Thorne, Andreas Vlachos, Christos Christodoulopoulls,, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In NAACL-HLT, 2018.
[27] Oguzhan Topsakal and Tahir Cetin Akinci. Creating large language model applications utilizing langchain: A primer on developing IIm apps fast. In Proceedings of the International Conference on Applied Engineering and Natural Sciences, Konya, Turkey, pages 10-12, 2023.
(28) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistnncy improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
[29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S. Mohamed, Agarwal, D. Belgrave, Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824-24837 Curran Associates, Inc., 2022.
[30] Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.0683 2024.
12

---

Published as a conference paper at ICLR 2025
[31] Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua. Search-in-the -itthe chain: Towards the accurate, credible and traceable content generation for complex knowledge- intensive tasks. arXiv preprint 2rXiv:230..14322 2023.
[32] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.
2023.
[33] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0362,, 2022.
[34] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[35] Quan Zhang, Xiaoyu Liu, Wei Li, Hanting Chen, Junchao Liu, Jie Hu, Zhiwei Xiong, Chun Yuan, and Yunhe Wang. Distilling semantic priors from sam efficient image restoration models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 25409-25419, June 2024.
[36] Quan Zhang, Yuxin Qi, Xi Tang, Rui Yuan, Xi Lin, Ke Zhang, and Chun Yuan. Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction. arXiv preprint arXiv:2501.11122 2025.
[37] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting large language models. In The Eleventh International Conference on Learning Representa- tions (ICLR 2023), 2023.
[38] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Lc, et al. Least-to-most prompting enables complex reasoning in large language models, arXiv preprint arXiv: 2022.
13

---

Published as a conference paper at ICLR 2025
Supplementary Material
A
FUTURE WORK
Future work includes explorations on the information extraction and analysis on more data modalities, such vision data [5 36 35). Additionally, plan to extend CoA tasks requiring intricate rea- soning paths involving recursive or nested logic by implementing an iterative generation mechanism. This approach will involve generating initial action chain and iteratively refining it based newly retrieved information, up to a maximum of 10 iterations or until no further retrieval is needed. Such an iterative process aims minimize irrelevant sub-questions and dynamically adapt the reasoning path, enhancing CoA's applicability to complex scenarios.
Looking forward, plan construct real-time benchmark using back-testing in the Web3 investment market. This benchmark will enable the community to evaluate performance in fast- evolving scenarios and further validate the effectiveness of approaches like CoA. The ultimate goal is to enhance faithfulness and multi-step reasoning for real-world question answering, where comprehensive analysis must syne with external data.
B
ALGORITHMS
Algorithm Description of Actions Workflow
Initialize: Actions Chain: AC; Question: Q: LLM Model: M: Query Section: QS; Sub-question:
Sub; Guess Answer: A; Faith Score: S; Multi-reference Faith Score: MRFS; Retrieved Results: R;
Missing Flag: MF:
Output: Final Generated Answer.
Function (R(uu, A,, MM P.):
Q Concat [Suby A,];
R = Retrieval QS,);
M RFS arg, max (rk. A.);
if M F; = True then
A AC.add(Sub,, r1); Add Top-1 data
end if
if
M RFF < then AC..orrec(Suub,ra)
end
AC.add r1);
end Function
Function MAIN(Q, .)::
AC = ChainGeneratee
M);
for each (Sub,, A,, M F.) in AC do IR(Sub, A,, MF);
end for GemanAnswerGenerate M);
return "Finish": end Function
C RELATED WORK
Tool learning. Recently, tool learning combines the strengths of specialized tools and foundation models to achieve enhanced accuracy and efficiency of problem solving [19] Toolformer 23 tries to train models to execute APIs for solving problems. Lang2LTL [14] utilizes LLM to ground temporal navigational commands to LTL specifications. However, they mainly focus specific tasks and domains with delicate algorithm designs. [4] finds that the otate-of-the-art methods do not know when they should use tools and when they should simply respond based their parametric knowledge. [19] also finds that information conflict between Model Knowledge and Augmented Knowledge is vital challenge to the accuracy and reliability of model generation and planning.
14

---

Published as a conference paper at ICLR 2025
Hence, CoA framework is designed to teach LLMs when to request external help and when to solve tasks by themselves with decreasing information conflicts.
Hallucination methods. Some work try solve the hallucination problem by ensemble algorithms [O]. But they are only based on training process without obtaining the real-time information. Retrieval augmentation and verification the main approaches for mitigating hallucination (7]. Self- Checker [1] comprises many modules including retrieval and veracity prediction for fact-checking by prompting LLMs only. SelfCheckGPT (I5) is black-box zero-resource hallucination verification scheme, which operates by comparing multiple sampled responses and measuring consistency. However, both methods need lots of interactions with LLMs be inefficient. This drawback motives us to propose the efficient and effective verification module utilizing our MRFS to decrease the hallucination without lots of interactions with L.MSs.
D PROMPTS
Here the prompts used in this work:
Evaluation Prompt of GPT-4
Given (question, ground truth answer, LLM-generated answer), you need to check whether the generated answer contains the ground truth by their meaning, not individual word only. If correct, the output is 1, otherwise, 0. For example:
[Question]: What should do when drink spoiled milk? (A) drink more (B) drink coffee (C) take some medicine.
[Ground truth]: (C) take some medicine
[Generated answer]: when you drink spoiled milk, you not drink more even drink coffee. You should go to the hospital and check if you need to take some medicines not. [Output]: 1
[Question]: (QUESTION)
[Ground truth]: {GROUND_TRUUS
[Generated answer]: OGENERATED_ANSWER
[Output]:
MORE EVALUATION DETAILS
Our comparative analysis clearly shows that our approach, which utilizes global reasoning chain, is more effective than traditional methods that rely intermediate reasoning step-by-step question generation and answering. This global perspective allows us to maintain a cohesive view throughout the reasoning process, leading more accurate and insightful answers. Let's consider question from the Social QA dataset and compare the responses generated by the CoT method and our CoA approach shown Figure 7
Chain of Thought (CoT) Analysis: It linearly processes the events: Alex guiding Robin is directly connected to her final meal. It views the action straightforwardly. focusing primarily on the imme- diate and most obvious context without deeper integration of Alex's 's roles or broader procedural implications.
Chain of Action (CoA) Analysis: (I)Questioning: CoA starts by questioning why Alex would take Robin the execution chamber, indicating an exploration beyond the immediate action. Inves- tigations: A. 'Work at the jail': CoA explores and affirms Alex's 's duty to adhere to jail protocols, recognizing that his actions are part broader job responsibilities. B. 'So Robin can eat': acknowl- edges this the immediate reason but looks further into the procedural context that mandates such actions. C. 'Release her': It dismisses this option based on scenario inconsistency.
(3) Final Answer: CoA combines these insights, concluding that Alex's 's actions governed by his professional duties to follow protocol, which includes ensuring that condemned prisoners receive their last meal as part of the humane treatment standards.
15

---

Published as a conference paper at ICLR 2025
answer the question Robin
step-by-step analysis:
Alex walked Robin towards the execution chamber This indicates that Alex is guiding Robin to the
location where she will have her last meal before execution.
for her last meal This further clarifees that the purpose of walking Robin towards the execution
chamber connected to her final meal.
reason for Alex walking execution chamber. Alex's Job at the Jall is not relevant to this specific action.B. "so Robin can eat" This is the correct reason for Alex's action. The purpuse of walking Robin towards the execution chamber is to allow her to have her last meal.C. "Relcase her" This does not align with the context provided. Robin beling led towards the execution chamber, the intention not to release her. Therefore, the most reasonable relevant reason for Alex walking Robin towards the execution chamber for her last meal is "so Robin can eat."
Why did Alex do this?A. "work at jail" This
towards
""Question": "Alex walked Robin towards execution chamber her last meal. Why did Alex do this? A. 'work at the jail' B. "so Robin can eat" C. 'Release her"",
"Chaln":
("Action": "Questioning", "Sub": "Why did Alex walk Robin towards the execution chamber for her last
meal?", "Guess_answer" "To follow jail protocols for condemned prisoners have last meal before execution"; "missing_flag" "False"),
("Action": "Invostigation", "Sub": "work at the jail", "Gucss_answer": "As jall guard, Alix's job is to carry out jall procedures for condemned prisoners", "maossins,"mlssngg... "missing_flag "False"),
answer";
(" ("Action": "Investigation", "Sub": "sc robin can eat", eate,' "Quess_answer "To fulfill the prisoner's last request for final meal before execution", "missing_flaag" flag": "False"),
("Action": "Investigation", "Sub": "Release her", "Guess_answers "To release Robin from the sentence grant her clemency"; "missing__niii flag": "True"ll,
"Final_answer anawer": "Alex walked Robin towards the execution chamber for her last meal as part of his job as jall guard to follow the jail protocols for condemned prisoners to have a last meal before execution. Additionally, it fulfills the prisoner's last request
final meal before execution.")
Figure 7: Responses of baseline and our CoA for the same question. The top and bottom ones are from Chain-of-Thought Thought and our Chain-of-Action respectively.
Comparative Insights: Our CoA offers richer analysis by integrating multiple aspects of the scenario into a comprehensive reasoning chain. This approach addresses the direct question and contextualizes Alex's 's actions within his duties and the prison's operational protocols, providing multidimensional understanding. In contrast, CoT tends to a more straighfforwal,, surface-level interpretation. This explanation underscores how CoA's 's approuch provides deeper, more contextually enriched answers compared to CoT, making it particularly effective for complex scenarios requiring nuanced understanding of actions within their broader social and procedural context.
F
CASE STUDY
We first introduce that our expert evaluators were selected from pool of professionals actively working in the Web3 domain. During the initial stages of product development, we conducted targeted survey distributed to well-known Web3 practitioners. The survey included 20 questions, with 10 focusing on foundational concepts in Web3 and the remaining 10 being open-ended questions designed assess their understanding and vision of the Web3 field. Responses senior Web3 investors evaluating the open-ended answers based on their expertise and perspective. From this process, selected the top 20 candidates with the highest overall scores to evaluation experts. As an incentive and to ensure continued engagement, these experts were granted free carly-stage access to the product. This rigorous selection process was designed to ensure that the evaluators possessed both technical expertise and a nuanced understanding of the Web3 domain. The 20 questions are follow:
scored, with three
as
Conceptual Questions of Web3 practitioners:
The following multiple-choice questions test the foundational knowledge
1. What is blockchain?
(a) A type of database
(b) A distributed ledger technology
(c) A centralized system for storing transactions
16

---

Published as a conference paper at ICLR 2025
Is it currently a good time to buy Bitcoin?
Token Name: BTC
Current Price: $42261.38
1M1
Market Sentiment Analysis
The news sentiment around BTC is predominantly positive, with bullish predictions and significant movements from dormant wallets, which could indicate increased interest and potential upward price pressure.
The negative sentiment from the underperformance and selloffs in the Grayscale BTC Trust may create some caution among investors, but overall market sentiment leans towards the positive side.
Technical Analysis
The current RSI value 52.79 suggests that BTC is neither overbought nor oversold, indicating a relatively balanced market condition.
Recommendation and Trade Strategy
Given the mixed signals from the technical indicators, a cautious long position could be considered, with close monitoring of the MACD and KDJ for changes in momentum.
My technical analysis based on the candlestick chart indicators recommended users who prefer short-term trading.
Are you satisfied with the answer? Help us improve by clicking like/dislike button.
Figure 8: Example Web3 QA application interface. our application, the agent responds to questions and retrieves relevant information for the response.
(d) None of the above
2. Which of the following best describes smart contracts?
(a) Legally binding digital agreements
Self-executing programs stored on the blockchain
(c) Al-powered decision-making tools
(d) Cloud-hosted contracts
3. What is the main purpose of consensus mechanisms in blockchain systems?
(a) To ensure security and prevent fraud
(b) store data efficiently
(c) To optimize transaction speed
(d) To encrypt private keys
4. Which of the following is an example of a Layer 2 scaling solution?
(a) Bitcoin
(b) Ethereum
(c) Polygon
(d) IPFS
5. What is the role of tokens in decentralized finance (DeFi)?
(a) Representation of digital assets or rights
(b) Payment for centralized services
Replacement of blockchain miners
(d) None of the above
6. What does gas fee" refer in Ethereum transactions?
17

---

Published as a conference paper at ICLR 2025
The cost of storing data centralized server
The incentive for nodes validate transactions
(c) The fee for storing a smart contract
(d) The cost of staking ETH
7. What is the key difference between Proof-of-Worf (PoW) and Proof-of-Stake (PoS)?
(a) PoW requires miners; PoS uses validators based on stakes
(b) PoW is faster than PoS
(c) PoS consumes
energy than PoW
(d) PoW supports NFTs; PoS does not
8. Which of the following technologies is commonly used to enable Web3 storage?
(a) IPFS
(b) Redis
(c) Firebase
(d) MySQL
9. What does "interoperability" mean in the context of Web3?
(a) The ability of blockchain scale efficiently
The compatibility of different blockchain networks
(c) The execution of transactions in real-time
(d) The use of Al in blockchain operations
10. What is the primary purpose of decentralized autonomous organizations (DAOs)?
(a) govern blockchain systems automatically
(b) To enable community-driven decision-making without centralized control
(c) To improve transaction speeds blockchain networks
(d) To create legal entities on the blockchain
Open-Ended Questions The following open-ended questions evaluate understanding, creativity, and vision in the Web3 domain:
1. What are the biggest challenges facing blockchain scalability, and how can they be ad- dressed?
2. How a decentralized identity system could revolutionize user privacy and security.
3. What do you think is the most promising application of NFTs beyond art and collectibles?
4. How can blockchain technology improve transparency and trust in traditional financial systems?
5. Explain the potential of Web3 transforming industries outside of finance, such as health- care supply chain management.
6. In your view, what factors determine the success of Layer blockchain compared its competitors?
7. How would you design tokenomics model to incentivize user engagement in decentralized application (dApp)?
8. What are the trade-off's between decentralization and scalability in current blockchain designs? How would you prioritize them?
Discuss the ethical implications of decentralization in Web3 systems. Do you think complete decentralization is desirable? Why or why not?
10. If you were advising startup launching DAO, what key considerations would you recommend to ensure its success?
Then, we compare the effectiveness of various web3 case solutions, including Chain-of-Actions Chain-of-Thouue Thought, and the ReAct agent, in addressing complex financial questions.
18

---

Published as a conference paper at ICLR 2025
What la the sigrticance the dris Ve upgrade?
Me Chain Aciorr
Crain of Thought
Int Pielet Apent
The drick ynu signas bor Aparica ang onlone forte Celll aeneytien And
Eehaoedd Tradballe batrumeee The canad oili yok dleu No'
As i'e soseaig uadin X01, larnt waw sucallo walion sTick sapade...
selver renoseal theile lakas lnving oepaens 41A4 usw
Berieinn Nathetie frambr be soure lehle..
The sigall lanen of the douti is segegaah out 1oaeed gloess cocrli altirna hiading rolerv and glout oppart mires for sridk
wp betae dic will de 14 ryganic
daneeu oitt yis you doell temoal morserts adii, and Bucci
memesoy mid Viteprii only repuaded ti plotherh modtin
ou Doll ccapecc.
currently good line buy Biteser?
li dellere
Sen prvide Not stop by dlip analys help you mabe on metamned
Pholet
kay 1: Rou Unrlerial Rilice
Market Sentieeee kaabil
Tnckkaal Asaayyuk
Sap undi Slualoo
Conaleyen Danccial
selerace Only torot
recart have regalabry cavelis nants Tomeees 5-dee nit prodde Rizers Formoo
The solach powils anomatinn ditaii indlddng
The BS comunnt 7.333 iuppoting the BDC in ivcredd
mecil ankii decoppe aneucy dladet jajed aldlg medel.
charp
Figure 9: Case studies 1 and 2. Case involves question that necessitates up-to-date information. Our Chain-of-Actions (CoA) framework efficiently gathers domain knowledge about dYdX and the associated upgrade documentation from the web, subsequently synthesizing this information into a definitive answer. Conversely, the Chain-of- ohoTuhth (CoTT) approach solely offers guidance on reading the white paper, lacking the capability to access real-time data. The ReAct agent, while successful in locating relevant content via search engines, offers only a broad overview, falling short of providing detailed insights. In case 2, our CoA stands out for providing real-time market price and technical analysis, offering multi-dimensional market insights through a combination of sentiment analysis and technical indicators such as RSI, along with specific trading strategy recommendationss which are invaluable for investors seeking actionable advice. In contrast, the Co'i' lacks integration of real-time data, focusing more the peeision-making process rather than immediate market actions, while the ReAct, despite aggregating relevant information, falls short by not offering specific guidance and failing to provide a clear conclusion.
Should sell all my degeooin noar?
Hd Of Actions
to Chain of Thought
IN Pedket Agent
Takan Nama:
Priec amant
CART grovd peronelleed foandal advck as the caciadon to or your
na sands, your lrnirne gon and da dacislennc limn doine key
Degunum
kabkey, adlng
Andlys
The moca san actlicar unard DCGE pectelaasdyy piddii....
Ancia Your Inmanve Oecs: Qnaiter ancerd Doppcote the rew plact
Marken Sentimee Performene Depocier tes soowe rignt fene voleliiyy end
Frakes dhe qusiia
dee caner of Degas raiut look
lnnetare Riok Specdddiio Drgaccie, other cogle coaicheed
Tirketaa kowtyak
Bermmoede Trade Henker
Cariie and juj ras Itss se the ride cuukine band at nal culliiing
dlaa an ln cesi cill pai's and slick liesaa Raticaatur that in
bearia tade
relee and weri.
Why did Bitapin experionse di dégreficant drop and reach low point of $48,844 on the Bisttapp exsherge on Jonsery 11, 2004 Po Chain of Theugin
IN Crain Actions
ict Revict Agant
The alpricae drep in Stooi's pics SHB d44 Statamp on Jenuary 11.
expeoeed pain
Shoperary Pheds Ballie Mang lansnas acdall ioeyle Bluckk
lug Pheean stae ETE Aupeneak alts to of tilees Eas lage-
hodicn loiklig
the
sileess or coutuuueent expert de in peoele anghatl laaee ca reneits teleeees moceeerin diran
cocerve ane on hhigrnu uppii Junn any 2012 Nol stane Fo need oud
recnral cur dediiroo memile drount repulitrr romo. Willou more much resatr drop E1 cansary 2004 ueclirr sendine t. right atsy reverpmedt. me cechecc comi nends and heding actitty Devestoe dutuuavee
bahed
reva eourta
Kisune But thre menior
These dataes sobeer oant wo rggn dag.eeerrrogooooo thhees po 503
unde saand pica tovemaras the dry monnmno maree
Figure 10: Case study 3 and 4. In case 3, our CoA for DOGE provides a comprehensive snapshot by analyzing current market sentiment, which is predominantly positive, and technical indicators like the RSI, which is neutral at 40.66. The recommendation takes into account both these aspects, suggesting caution due to the bearish technical indicators despite the positive sentiment, which is a nuanced approach for traders considering DOGE transactions. However, the CoT and ReAct fall short by not providing specific or updated information, offering general investment advice, and emphasizing personal research and judgment instead of actionable insights. For case 4, the Bitcoin price drop case, our CoA gives a detailed account of factors leading to the price decline, including selling pressure post-ETF approval and temporary profit-taking by investors. presents clear narrative of events contributing to the drop. showing a deep dive into the cause-and-effect within the market. 'he ReAct, on the other hand, acknowledges the lack of clear reasons for the price drop and suggests a variety of possible influences, such as market conditions and investor sentiment, highlighting the complexity and multifaceted nature of cryptocurrency markets but not providing a definitive explanation.
19

---

Published as a conference paper at ICLR 2025
Kinea Pinenco more stable than other lending protessta? Chain Thooge
Oh Actino
Sucar thet fevertor otee mrome
ent aling trpuut doi heghver progeee..
thre vor spoe vuly an bevicmmett momeraaiion rouke lo Thendees forme
stble
the informatin cath toanr tar Thance panationd protand lollaeeon
adhunce
Aiperkuned Leadmnigg The platin lod sus causain cupeloua
protooas have anarged
vony slo oftee son peltie afhr ligle airuda reedors and lindes. eoecil deakk anou sading knda Finace saed other leainng herures, auihis thechallia and mrcuatio tobee acordiry
Einore, rallity persool rulies booee. stop
fire
yonconn onfor euto erpeiiccal lnakenii, sest
suderin preccee odlorrl yalls
What la Autonoomss World?
Cfuir Actions
Chain Thouge
SI PeACY Agee
The Nulrnmma Wolk dual mineererta paming June seeediiuu sulnoiigg tivrigt
oo)
Ser patro sno abou hannetana Mord safude
cioi srin of man with also iounrer can provos come
As lask markeegegg sodea Jinuary 2030 "Artnn World"
weyy
nor edil bocaena and ae gave by digha coreeaa toan ingue
oring pataraal el Autooooaa as sal Aravr Aki tomoue World coocriad by cagaa lenoris de not to report of pukoe dlon
Advercan Setailag: acae culcoss "utt sccano Rodif cap o81 brow
Blacuiaai inagratic.... Acaica accas vadi as and
Figure 11: Case study 5 and 6. In case 5, CoA for Kinza Finance highlights its stability lending protocol, detailing its robust security measures, experienced leadership, and strategic partnerships. This comprehensive approach emphasizes the unique strengths that contribute to Kinza's stability and resilience in the market. On the contrary, the Co' lacks current updates on Kinza, providing only general factors that affect lending protocol stability, while the ReAct, although it gathers relevant information, only partially addresses why Kinza might be more stable, pointing to security measures and token economy without a thorough analysis. For case 6, discussing Autonomous World, our CoA provides clear definition and outlines the impact of such virtual environments on gaming and blockchain technology, citing features, challenges, and the potential for revolutionizing digital interaction. This detailed overview presents newarrd-loking view of the integration of blockchain in gaming. However the COT, with its last update in January 2022, does not offer a precise definition, indicating that the concept can vary widely. Similarly, the ReAct retrieves general information but lacks focused perspective on the practical implications.
G SYSTEM
All experiments are carried out on a cluster, with the exception of the distributed compute node experiment. Each node within the cluster is equipped with 1 NVIDIA GEFORCE RTX 2080 Ti GPUs and 6 8-core Intel XEON Silver 4214 processors running at 2.20GHz. The combined RAM capacity across the cluster nodes amounts to 755GB, and the operating system employed is Ubuntu 18.04.
H HYPERPARAMETER
In experiment, exclusively require the hyperparameters for LLM, with the exception of Auto-CoTl For the Auto-CoT method, we utilize the KNN model for clustering. Below, we provide a list of all the hyperparameters used in our experiments.
Table 8: Hyperparameter used in the task.
parameter
values
temperature
0.0
max length
1000
top_p
1.0
n e_cllsserr
5
nurmer
3
seed
1
20


Reviews:
[{"review_id": "6HbAS1PEI9", "reviewer": "Reviewer_mjpZ", "summary": "The paper introduces the Chain-of-Action (CoA) framework, a novel approach to multimodal and retrieval-augmented question answering that enhances the faithfulness and reasoning quality of large language models (LLMs). CoA addresses key challenges in QA, such as unfaithful responses and weak reasoning, by decomposing questions into a series of reasoning steps or actions that systematically retrieve and verify information from various sources. The framework introduces three \"Plug-and-Play\" actions—web querying, knowledge encoding, and data analyzing—that support multimodal data integration. Additionally, a multi-reference faith score (MRFS) is proposed to resolve inconsistencies and improve response accuracy. Experimental results demonstrate CoA’s effectiveness in handling complex questions across QA benchmarks and in real-world applications, particularly in the Web3 domain.", "questions": "1. Could you clarify what “imputation” refers to in Table 2? Are there results available for CoA without MRFS, and what does “w/ ROUGE” mean? My understanding was that ROUGE is used only in ASQA.\n\n2. In Table 3, could you provide separate statistics for input and output tokens, as well as the average token usage per action? This would help readers better understand the specific cost details.\n\n3. Could you elaborate on what is meant by the term “knowledge boundary”?\n\n4. Are the results of the Chain-of-Action framework directly comparable to previous studies? I noticed that this study used GPT-4, while DSP and SearchChain relied on older-generation LLMs (text-davinci-002 and gpt-3.5-turbo, respectively).\n\n5. Would it be fair and perhaps clearer to rename Sections 2.2.1 and 2.2.2 as \"Data Collection\" and \"Data Verification,\" instead of “Actions Design” and “Actions Workflow”? These alternative terms seem easier to understand and align well with the content of the corresponding subsections.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This study introduces a framework embodying the divide-and-conquer approach, effectively breaking down complex tasks into manageable components that are tackled sequentially. This structure enhances the model's ability to handle multifaceted queries with improved precision.\n\n2. The empirical results demonstrate notable improvements in both performance and efficiency, as reflected in reduced API calls and token usage compared to prior methods. These gains underscore the framework’s effectiveness and potential for cost-saving in real-world applications.\n\n3. The introduction of the multi-reference faith score (MRFS) is a contribution, which effectively identifies and mitigates information conflicts, and improves answer reliability and trustworthiness in real-time applications.", "weaknesses": "1. The paper’s primary weakness lies in how it presents its key concepts and narrative. Many claims, such as \"multimodal,\" \"plug-and-play,\" and \"action-based\" elements, lack direct evidence or clear definitions, making it challenging to follow the core contributions. Though the pipeline is straightforward, understanding the study's actual workflow is hindered by (1) inaccurate terminology, (2) loosely connected methodology descriptions, and (3) a mix of abstract workflows and technical details.\n\n2. Certain terms are uncommon or seem misapplied, which leads to confusion. For example, terms like \"multimodal\" (when referring to text and tabular data), \"chain-of-action\" (more of a \"chain-of-data-collection-and-verification\"), \"actions design\" (data collection), \"actions workflow\" (data verification), \"node\" (sub-question), and \"knowledge boundary\" (what a model actually knows) lack clarity and could benefit from more precise definitions or alternatives.\n\n3. Question decomposition appears critical to this framework, yet there is limited discussion on decomposition strategies or comparisons with existing baselines. Further elaboration here would strengthen the paper's contributions.\n\n4. The \"plug-and-play\" feature is presented as a low-cost prompting strategy; however, integrating retrieval for each data type (e.g., web, internal knowledge) may not be straightforward. It may be worth reconsidering or refining this claim to better reflect its implementation complexity.\n\n5. The paper’s claim of multimodal data handling is unclear. If the input consists of real-time information, domain knowledge, and tabular data, it may be more accurately described as handling heterogeneous data rather than multimodal data. Additionally, if tabular data is linearized as text for LLM input, the fundamental multimodal claim weakens.\n\n6. The study does not include ablations to show the specific contribution of tabular data. Providing such analyses could clarify its impact on the framework's performance.\n\n7. Section 3.2 mentions expert evaluation on a 1 to 3 scale based on three criteria, but it lacks details on the expert recruitment process, qualifications, and any inter-rater reliability metrics. Adding these details would increase the transparency and credibility of the evaluation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8GPw6TbPlV", "reviewer": "Reviewer_wYMh", "summary": "The authors propose a new QA retrieval mechanism called Chain of Action(CoA). When a question is asked to an LLM, there is a prompt which generates a list of actions the LLM needs to take first to effectively answer the questions. They introduce a Plug and Play approach where in case of Multimodal LLMs, the actions taken can be integrated into the application. The actions can be web query or data analysis. The paper integrates 3 such actions. The LLM then performs each of the individual action generated and then there is another query which combines information from all the actions. The LLM then gives an answer based on the newly injected information", "questions": "1. Can you elaborate on the key differences between \"thoughts\" in CoT and \"actions\" in CoA? How does this change improve the overall performance? It would also be helpful if you can discuss the limitations and trade-offs between them.\n2. If the system doesn't have the ability to add additional actions like web query, does CoA still perform better than CoT. \n3. Does CoA add significant latency to QA process?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors utilize newer multimodal LLM abilities to perform actions such as web query and data analysis. The authors come up with a new QA mechanism for LLMs which uses the actions The method is called Chain of Action(CoA).\n2. The authors demonstrate that this method significantly outperforms other reasoning and QA methods on many QA datasets. \n3. The improvement of using actions over thoughts does seem to be the natural way of solving a question. This approach has significant potential for improving QA capabilities of LLMs.", "weaknesses": "1. Based on the number of actions to be taken and what kind of \"plug\" is used for the action, the time taken to finish all actions and send out an answer might become significant. It would have been good to see the study on latency(eg. average response time) of the system because of the new method.\n2. It would be helpful to conduct an ablation study when you remove specific action types to isolate their impact on performance. This would provide clearer insights on how much this method relies on additional capabilities. \n3. Comparing CoA with and without the ability to perform additional \"plugs\" across different types of questions can be useful in understanding the impact of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YxgLKGEVmo", "reviewer": "Reviewer_VgSf", "summary": "The paper presents the Chain-of-Action (CoA) framework designed to improve large language models' (LLMs) performance in multimodal and retrieval-augmented question-answering (QA). CoA addresses challenges such as hallucination and weak compositional reasoning by decomposing questions into reasoning chains. This method incorporates plug-and-play actions for retrieving diverse data sources and uses a novel multi-reference faith score for verification. Empirical results show CoA outperforms other methods in public benchmarks and real-world applications.", "questions": "1. Can the authors provide more details on how the CoA framework could be adapted for tasks involving visual or mixed data modalities?\n2. How does the framework handle discrepancies or conflicts when sources provide contradictory information?\n3. Are there plans to explore CoA's performance in real-time, fast-evolving information retrieval scenarios where data may change rapidly (e.g., live news events)?\n4. Could the use of CoA extend to tasks requiring intricate reasoning paths that involve recursive or nested logic?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Innovative Framework**: The CoA's structured decomposition into sub-questions and its use of domain-adaptable plug-and-play actions represent a significant advancement in enhancing the faithfulness and accuracy of LLM responses.\n- **Empirical Validation**: Demonstrated strong performance on benchmarks and real-world applications, notably outperforming existing baselines in multimodal QA tasks.\n- **Verification Mechanism**: The multi-reference faith score is an effective metric for cross-validating LLM-generated answers against external sources, enhancing reliability.\n- **Practical Impact**: Real-world implementation in a Web3 QA system showed increased user engagement and positive feedback, validating the method's applicability.", "weaknesses": "- While the CoA approach shows strong empirical performance, its adaptability to more diverse or unstructured data modalities beyond text and tabular data remains to be proven.\n- The scalability and efficiency when integrating more complex or real-time data sources require further exploration, especially in scenarios with rapidly changing information.\n- The approach, despite its modular design, may face challenges in tasks involving higher-order reasoning or complex multi-step dependencies that are not purely fact-based.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)