AUTOACT: Automatic Agent Learning from Scratch for QA via Self-Planning (2024)

Abstract: Language agents have achieved considerable performance on various complex questionanswering tasks by planning with external tools. Despite the incessant exploration in this field, existing language agent systems still struggle with costly, non-reproducible data reliance and face the challenge of compelling a single model for multiple functions. To this end, we introduce AUTOACT, an automatic agent learning framework for QA that does not rely on largescale annotated data and synthetic planning trajectories from closed-source models (e.g., GPT-4). Given limited data with a tool library, AUTOACT first automatically synthesizes planning trajectories without any assistance from humans or strong closed-source models. Then, AUTOACT leverages a division-of-labor strategy to automatically differentiate based on the target task information and synthesized trajectories, producing a sub-agent group to complete the task. We conduct comprehensive experiments with different LLMs, which demonstrates that AUTOACT yields better or parallel performance compared to various strong baselines. Further analysis demonstrates the effectiveness of the division-of-labor strategy, with the trajectory quality generated by AUTOACT generally outperforming that of others1.

heading: 1 Introduction
text: Language agents (Wang et al., 2023a; Xi et al., 2023; Guo et al., 2024), which leverage the powerful reasoning capabilities (Qiao et al., 2023b; Zhang et al., 2023) of Large Language Models (LLMs) to interact with executable tools, have emerged as essential components of AI systems designed to address complex question-answering tasks (Torantulino, 2023; Osika, 2023; Nakajima, 2023; Tang et al., 2023; Xie et al., 2023). The process of endowing LLMs with such interactive capabilities is referred to as Agent Learning wherein
∗ Corresponding Author. 1Code: https://github.com/zjunlp/AutoAct.
planning (Huang et al., 2024b) plays a pivotal role, which is responsible for decomposing complex questions into simpler ones (Wei et al., 2022; Yao et al., 2023; Team, 2023; Qian et al., 2023), invoking external tools (Shen et al., 2023; Lu et al., 2023; Qin et al., 2023), reflecting on past mistakes (Shinn et al., 2023; Madaan et al., 2023), and aggregating information from various sources to reach the final answer. There have been a lot of works (Li et al., 2023; Shen et al., 2023; Hong et al., 2023; Talebirad and Nadiri, 2023; Chen et al., 2023d,b) that directly prompt closed-source off-the-shelf LLMs to plan on particular tasks. Despite their convenience and flexibility, closed-source LLMs inevitably suffer from unresolved issues, as their accessibility often comes at a steep price and their black-box nature makes the result reproduction difficult. In light of this, some recent endeavors have shifted their focus towards imbuing open-source models with planning capabilities through fine-tuning (Chen et al., 2023a; Zeng et al., 2023; Yin et al., 2023).
However, despite the achievements of the existing fine-tuning-based methods, they are not without limitations. On the one hand, training opensource models necessitates a substantial amount of annotated QA data pairs and still relies on closedsource models to synthesize planning trajectories. However, fulfilling these requirements in many
3003
real-world scenarios, such as private personal bots or sensitive company business, often proves to be rocky. On the other hand, from the perspective of agent framework, fine-tuning-based methods compel one single language agent to learn all planning abilities, placing even greater pressure on them. These contradict Simon’s principle of bounded rationality (Mintrom, 2015), which states that “precise social division-of-labor and clear individual tasks can compensate for the limited ability of individuals to process and utilize information”.
To this end, we introduce AUTOACT, an automatic agent learning framework for QA, which does not rely on large-scale annotated data and synthetic trajectories from closed-source models while incorporating explicit individual tasks with precise division-of-labor (see Fig. 1). Given a limited set of user-provided data examples, AUTOACT starts with a META-AGENT to obtain an augmented database through self-instruct (Wang et al., 2023b). Then, armed with a prepared tool library, the METAAGENT can automatically synthesize planning trajectories without any assistance from humans or strong closed-source models. Finally, we propose the division-of-labor strategy which resembles cell differentiation based on the self-synthesized trajectories (genes), where the META-AGENT acts as a stem cell (Colman, 2008) and differentiates into three sub-agents with distinct functions: task decomposition, tool invocation, and self-reflection, respectively. Our differentiation process is essentially a parameter-efficient training process on the self-synthesized trajectories with low-consumption resources. We list the differences between AUTOACT and prior works in Tab. 3.
Experiments on complex question-answering tasks with different LLMs demonstrate that AUTOACT yields better or parallel performance compared to various strong baselines. Extensive empirical analysis demonstrates the effectiveness of our appropriate division-of-labor strategy.

heading: 2 AUTOACT

heading: 2.1 Critical Components of AUTOACT
text: META-AGENT. The META-AGENT is responsible for all the preparatory work before selfdifferentiation and serves as the backbone model for all sub-agents. Given limited target task information and a pre-prepared tool library, the METAAGENT can differentiate into an agent group capable of collaborating to accomplish the target task.
In AUTOACT, the META-AGENT can be initialized with any kind of open-source model.
Target Task Information. In this paper, we mainly focus on agent learning from scratch, which means the task information at hand is quite limited, primarily encompassing three aspects: task name M, task description P , task data examples C. Concretely, P represents a detailed description of the task’s characteristics. C = {qi, ai}|C|i=1 indicates |C| question-answer example pairs of the task, where |C| is very small which users can effortlessly provide (e.g., a few demonstrations). For a more in-depth view of task information, please refer to Appx. E. Note that the task information serves as the only user-provided knowledge of the task for AUTOACT to conduct automatic agent learning.
Tool Library. To facilitate our agents in automatic task planning, we provide a comprehensive tool library at their disposal. The tool library can be denoted as T = {mi, di, ui}|T |i=1, where m represents the tool name, d defines the tool functionality, u details the tool usage instruction, and |T | stands for the tool amount of the library. In our automatic procedure, the META-AGENT has the autonomy to select appropriate tools from the tool library based on the task information. Users also have the option to expand the tool library according to their specific needs, allowing for more flexible utilization. We list the details of our tool library in Appx. F.

heading: 2.2 Starting from Scratch via Self-Instruct
text: To acquire a sufficient amount of task data and provide an ample training resource, it is necessary to augment the data based on the examples at hand. We accomplish this process through selfinstruct. Initially, the database D is set to be equal to the task data examples C, with C as the seed for data generation. In each round, the META-AGENT generates new question-answer pairs by few-shot prompting, and the few-shot prompt examples are randomly sampled from D. The generated data will be added to D followed by filtering, with the exclusion of format erroneous and duplicate data before its inclusion. Eventually, we obtain a database D = {qi, ai}|D|i=1, where the number of data |D| satisfies |D| ≫ |C|. The prompt we use for selfinstruct can be seen in Appx. G.1 and we list some cases generated through self-instruct in Appx. H.

heading: 2.3 Automatic Agent Learning via Self-Planning
text: Automatic Tool Selection. With the tool library at hand, we ask the META-AGENT to select applicable tools for each task automatically. Specifically, we put T = {mi, di, ui}|T |i=1 in the form of a tool list as part of the prompt. Along with T , the prompt also includes the task’s description C. Finally, we instruct the META-AGENT to select an appropriate set of tools Ts (Ts ⊂ T ) to wait for synthesizing trajectories. The prompt we use for automatic tool selection can be seen in Appx. G.2.
Trajectories Synthesis. Without depending on closed-source models, we enable the METAAGENT to synthesize planning trajectories on its own. Equipped with Ts, we instruct the METAAGENT to synthesize trajectories in a zero-shot manner on the database D adhering to the format of Thought-Action-Observation as defined in Yao et al. (2023). In order to obtain high-quality synthesized trajectories, we filter out all the trajectories with reward < 1 and collect trajectories with exactly correct answers (reward = 1) as the training source for self-differentiation. The prompt for trajectories synthesis can be seen in Appx. G.3.
Self-Differentiation. In order to establish a clear division-of-labor, we leverage synthesized planning trajectories to differentiate the META-AGENT into three sub-agents with distinct functionalities:
• X PLAN-AGENT πplan undertakes question decomposition and determines which tool to invoke in each planning loop (Eq. 2). • { TOOL-AGENT πtool is responsible for how to invoke the tool (Eq. 3) by deciding the parameters for the tool invocation. • ¤ REFLECT-AGENT πreflect engages in reflection by considering all the historical trajectories and providing a reflection result (Eq. 4).
We assume that the planning loop at time t can be denoted as (τt, αt, ot), where τ denotes Thought, α signifies Action, and o represents Observation. α can be further expressed as (αm, αp), where αm is the name of the action, and αp is the parameters required to perform the action. Then the historical trajectory at time t can be signaled as:
Ht = (τ0, α0, o0, τ1, ..., τt−1, αt−1, ot−1). (1)
Eventually, supposing that the prompts of target task information, planning format requirements,
and the question are all combined as S , the responsibilities of each sub-agent can be defined as:
τt, α m t = πplan(S, Ts,Ht), (2) αpt = πtool(S, Ts,Ht, τt, αmt ), (3) τ r, αr = πreflect(S, Ts,H), (4)
where τ r and αr represent the thought and action of the reflection process, and H is the planning history after finishing the answer. The trajectories can be reorganized based on the responsibilities above and fed to the META-AGENT for self-differentiation. Our differentiation is a parameter-efficient finetuning process to achieve resource-efficient learning. We give examples of the training data for each sub-agent in Appx. I. Particularly, for each subagent, we train a specific LoRA (Hu et al., 2022).
Group Planning. At inference time, once the tool name αmt generated by the PLAN-AGENT is triggered at time t, the TOOL-AGENT is roused to decide the parameters αpt transferred to the specific tool. The return result of the tool is treated as the observation ot and handed to the PLANAGENT. After the collaboration between the PLANAGENT and TOOL-AGENT reaches a prediction, the REFLECT-AGENT comes to reflect on the history and provide a reflection result contained in the reflection action αr. If the reflection result indicates that the prediction is correct, the whole planning process ends. Otherwise, the PLANAGENT and TOOL-AGENT will continue the planning based on the reflection information. The specific sequence of the group planning process can be found in the example on the right of Fig. 2.

heading: 3 Experimental Setup
text: Tasks and Metrics. We evaluate AUTOACT on HotpotQA (Yang et al., 2018) and ScienceQA (Lu et al., 2022). HotpotQA is a multi-hop QA task challenging for rich background knowledge, the answer of which is usually a short entity or yes/no. Following Liu et al. (2023), we randomly select 300 dev questions divided into three levels for evaluation, with 100 questions in each level. For HotpotQA, the reward ∈ [0, 1] is defined as the F1 score grading between the prediction and groundtruth answer. ScienceQA is a multi-modal QA task spanning various scientific topics. We also divide the test set into three levels based on the grade, with 120 randomly sampled data in each level. Since ScienceQA is a multi-choice task, the
reward ∈ {0, 1} is exactly the accuracy. Note that due to the limitations of LMs in generating images, for ScienceQA, during the self-instruct stage, we directly generate captions for the images instead.
Baselines. We choose the open-source Llama2 models (Touvron et al., 2023) and Mistral-7B (Jiang et al., 2023) as the backbones of our METAAGENT and sub-agents. The compared baselines include CoT (Wei et al., 2022), REACT, Chameleon (Lu et al., 2023), Reflexion (Shinn et al., 2023), BOLAA (Liu et al., 2023), ReWOO (Xu et al., 2023), FIREACT (Chen et al., 2023a). We detail each baseline in Appx. B. To ensure fairness, we maintain an equal training trajectory volume of 200 for FIREACT and AUTOACT (200 synthesized data). As Reflexion provides answer correctness labels during reflection but other methods including AUTOACT do not, we test all the other methods twice and choose the correct one for evaluation. For all the prompt-based baselines, we uniformly provide two examples in the prompt.
Training Setups. We fine-tune all our models with LoRA (Hu et al., 2022) in the format proposed in Alpaca (Taori et al., 2023). All the training and inference experiments are conducted on 8 V100 GPUs within 16 hours. We detail the hyperparameters for training in Appx. B.

heading: 4 Results
text: Compare to Prompt-based Agent Learning Baselines. As shown in Tab. 1, the Mistral-7B and Llama-{13,70}B models consistently outperform various prompt-based baselines. The Llama70B model even surpasses the agent performance of GPT-3.5-Turbo, achieving a rise of ↑3.77% on HotpotQA and ↑6.39% on ScienceQA. Therefore, whether in a single-agent or multi-agent architecture, prompt-based methods relying on few-shot demonstrations fail to precisely customize the behavior of the agent, which is also supported by the fact that FIREACT widely outperforms REACT and BOLAA in the context of iterative planning.
Compare to Fine-tuning-based Agent Learning Baselines. Further focusing on FIREACT in Tab. 1, despite the aid of GPT-4, FIREACT’s approach of assigning the entire planning task to a single model proves to be burdensome. As a result, its performance on ScienceQA even falls short compared to the prompt-based global planning method, Chameleon. AUTOACT decouples
the planning process and reaches a clear divisionof-labor among sub-agents for group planning, resulting in an improvement than FIREACT, with ↑5.77% on HotpotQA and ↑6.67% on ScienceQA with Llama-70B model. Additionally, AUTOACT achieves self-planning without relying on closedsource models and large-scale labeled datasets, which paves the way for automatic agent learning with open-source models from scratch. In ablation study (§4) and human evaluation (§5), we will further validate that the quality of trajectories synthesized by AUTOACT is not inferior to FIREACT trained on trajectories synthesized using GPT-4.
Single-agent Learning vs. Multi-agent Learning. Under identical settings, multi-agent architectures generally exhibit better performance than single-agent (REACT vs. BOLAA, FIREACT vs. AUTOACT), which aligns with Simon’s theory of bounded rationality. Seemingly contrary to expec-
tations, despite being a single-agent architecture, Chameleon outperforms BOLAA (even FIREACT on ScienceQA). However, we analyze that this can be attributed to the way it leverages tools. In Chameleon, the process of deciding tool parame-
ters is considered a form of tool invocation, and specialized few-shot prompts are designed to guide the model through this process. From this aspect, Chameleon, despite nominally a single-agent architecture, exhibits features resembling a multi-agent one, which does not contradict our initial conclusion. Indeed, we can also explain from the perspective of optimizing objectives. Another well-known principle, Goodhart’s Law (Goodhart, 1984), states that “When a measure becomes a target, it ceases to be a good measure”. This implies that optimizing one objective on the same agent will inevitably harm other optimization objectives to some extent. Therefore, it is not optimal to optimize all objectives on a single agent, and a multi-agent architecture happens to address this issue. However, we analyze in §5 that excessive fine-grained divisionof-labor is not the best approach.
Approach Ablations. Tab. 2 presents the performance of AUTOACT on the Llama-70B model after
removing certain key processes. It can be observed that the least impactful removal is the - reflect. We investigate that in the zero-shot scenario, the model tends to be over-confident in its answers (as also confirmed in Huang et al. (2024a)). It typically only recognizes its errors when there are obvious formatting mistakes or significant repetitions in the planning process. Consistent with previous findings, the removal of the - multi agents leads to a noticeable decrease in performance. A more exciting discovery is that the results of - multi are comparable to those of FIREACT. This indirectly suggests that the trajectory quality generated by the 70B model may be no worse than that of GPT-4. As expected, the performance deteriorates after - fine-tuning, which once again confirms the previous conclusion. To demonstrate the necessity of filtering out planning error data, we specifically remove the filtering process (- filtering) to examine the performance of AUTOACT. The results indicate
that the damage caused by training on unfiltered data is even greater than that of - fine-tuning.

heading: 5 Analysis
text: Larger training data scale does not necessarily mean better results. We evaluate the influence of different training data scales on the performance of self-planning with Llama-{7,13,70}B models on HotpotQA in Fig. 3 (a-c). It can be observed that the overall performance of different models goes to stability with minimal waves once the data scale exceeds 200. We speculate that this may be due to the limited ability of naive self-instruct to boost
internal knowledge of the language model. As the training data increases, the knowledge which can be extracted through self-instruct decreases. Despite our efforts to filter out duplicate data, the mindless increase can inevitably lead to a significant surge in similar data, which undermines the benefits of increasing the data scale and makes it challenging to improve model performance or even leads to over-fitting. To further confirm the role of training data, we decouple the models from the training data and evaluate their training results on trajectories synthesized by stronger models. From Fig. 3 (d-f), we can see consistent conclusions with previous findings. Therefore, maximizing the diversity of the synthesized data in the database may be a key improvement direction for AUTOACT and we leave this for our future work. We can also observe from Fig. 3 (d-e) that the larger the model, the higher the quality of the synthesized data, as the performance of the 7B model shows a gradual increase on self, 13B, and 70B synthesized data.
Moderate division-of-labor benefits group planning performance. To explore the impact of different granularity of self-differentiation, we further subdivide the tool agent, assigning dedicated agents to manipulate each specific tool. We compare the performance of One agent, Three agents (AUTOACT), and the Tool-Specified setting on Hot-
potQA in Fig. 4. It can be observed that excessive differentiation (Tool-Specified) not only fails to achieve better results but can sometimes even be less effective than not differentiating (One) at all. This is consistent with the findings in Qiao et al. (2023a) which indicate that multi-tool joint learning often outperforms single-tool individual learning. Moreover, it appears that the performance loss of tool-specific agents compared to AUTOACT is more significant on harder problems. This is because challenging problems typically require more planning steps and higher levels of collaboration among tools. By unifying tool invocations under one agent, it becomes possible to effectively learn the interconnectedness between tools, thereby compensating for potential information gaps arising from using tool-specific agents. Note the difference from Li et al. (2024), here we are discussing the granularity of division-of-labor among agents with different responsibilities, rather than the voting quantity among mutually equal agents.
Human Evaluation. To get a deeper understanding of the quality of trajectories generated by different methods, we manually compare them from the number of planning rounds, the logical correctness of thoughts, action types, action parameters, and overall coherence. The detailed human evaluation process can be found in Appx. C. The evaluation results are depicted in Fig. 5&6. We can observe a clear advantage for AUTOACT over other methods in the action type and action parameters. This indicates that decoupling the missions of planning and tool invocation can lead to better performance for both, alleviating the overwhelming pressure on a single agent. A more intuitive comparison can be observed in Fig. 5 (a-b). AUTOACT successfully addresses the failure in REACT by employing a more scientific combination of tools and making more accurate tool invocations. Furthermore, AUTOACT tends to consume more planning rounds than other methods (the specific average planning rounds is in Appx. D). This allows AUTOACT to perform better on harder problems. However, this characteristic can be a double-edged sword when it comes to simple problems. A surprising aspect is that AUTOACT can validate its inner answers by continuing more rounds of verification (Fig. 5 (c)). But this can also lead to a longer context, gradually deviating AUTOACT from the original question (Fig. 5 (d)).

heading: 6 Related Work
text: LLM-Powered Agents. The rise of LLMs has positioned them as the most promising key to unlocking the door to Artificial General Intelligence (AGI), providing robust support for the development of LLM-centered AI agents (Wang et al., 2023a; Xi et al., 2023; Wang et al., 2023c,d). Related works focus primarily on agent planning (Yao et al., 2023; Song et al., 2022; Chen et al., 2023a), external tools harnessing (Patil et al., 2023; Qiao et al., 2023a; Qin et al., 2023), collective intelligence among multi-agents (Liang et al., 2023; Liu et al., 2023; Chen et al., 2023c), etc. However, despite their success, existing methods still face two major troubles. Firstly, most agents heavily rely on prompts for customization, which makes it difficult to precisely tailor the behavior of the agent, resulting in unexpected performance at times. Secondly, each agent is compelled to master all skills, making it challenging for the agent to achieve expertise in every domain. In response, our approach leverages a proper division-of-labor strategy and fine-tuning each sub-agent to equip different agents with distinct duties. These agents collaborate to accomplish tasks orderly and effectively.
Agent Fine-Tuning. Despite the vast interest in LLM-powered agents, the construction of agents through fine-tuning has received limited attention. Most early works concentrate on fine-tuning to optimize the model’s reasoning capabilities (Liu et al., 2022; Fu et al., 2023) or tool proficiency (Patil et al., 2023; Qiao et al., 2023a; Qin et al., 2023). Recently, more works have emphasized endowing open-source LLMs with agent capabilities through fine-tuning (Chen et al., 2023a; Zeng et al., 2023; Yin et al., 2023; Shen et al., 2024). However, these works suffer from at least one of the following issues: i) the requirement of one single model to be a generalist, ii) the need for a large amount of annotated data, iii) the need for trajectory annotation of closed-source models. Our approach enables the META-AGENT to synthesize trajectories and achieve a division-of-labor strategy in a zero-shot manner, without relying on closed-source models.

heading: 7 Conclusion and Future Work
text: In this paper, we propose AUTOACT, an automatic agent learning framework for QA that does not rely on large-scale annotated data and synthetic trajectories from closed-source models, while alleviating
the pressure on individual agents by explicitly dividing the workload. Interesting future directions include: i) expanding AUTOACT to more realistic task scenarios (Puig et al., 2018; Zhou et al., 2023a; Xie et al., 2024), ii) boosting more knowledge via self-instruct (as analyzed in §5), iii) iteratively enhancing synthetic trajectories via self-improvement (Huang et al., 2023; Aksitov et al., 2023).
Limitations
In this paper, we focus on constructing an automatic agent learning framework dubbed AUTOACT. Despite our best efforts, this paper may still have some remaining limitations.
Tasks. In this paper, we mainly focus on complex question-answering tasks. However, there are many other more complex interactive scenarios, including web (Yao et al., 2022; Zhou et al., 2023a), household (Puig et al., 2018; Shridhar et al., 2021), traveling (Xie et al., 2024), robotics (Ichter et al., 2022), etc. For example, we have investigated the use of META-AGENT performing random explorations (Xiang et al., 2023; Murty et al., 2024) in virtual environments to replace the process of task and trajectory synthesis through self-instruct and zero-shot planning. We plan to conduct further research on applying AUTOACT to a wider range of tasks based on this in the future.
Boosting Knowledge via Self-Instruct. As analyzed in §5, the planning performance of AUTOACT can be limited by the model’s ability to access internal knowledge through self-instruct. While the current phenomenon allows us to achieve lightweight self-differentiation in terms of parameters and data, it is still necessary to research how to enrich knowledge as much as possible within the constraints of limited data.
Self-Improvement. Recent research has shed light on self-improvement techniques that enhance LLMs by iteratively training them on selfsynthesized data (Zelikman et al., 2022; Huang et al., 2023; Gülçehre et al., 2023; Aksitov et al., 2023). This approach allows the model to continually learn and refine its performance on its own. Our approach also involves training on selfsynthesized data and we believe that further using the iterative thinking of self-improvement will significantly enhance the performance of our method.
Ethics Statement
This research was conducted with the highest ethical standards and best practices in research. All our experiments use publicly available datasets (as detailed in §3), avoiding ethical concerns related to privacy, confidentiality, or misuse of personal biological information. The human evaluation process (as detailed in Appx. C) was carried out strictly with fairness and transparency. Consequently, this research is free from any ethical concerns.

heading: Acknowledgements
text: We would like to express our sincere gratitude to the anonymous reviewers for their thoughtful and constructive feedback. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), Tencent AI Lab Rhino-Bird Focused Research Program (RBFR2024003), and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

heading: A Comparison with Related Works
text: See Tab. 3

heading: B Baselines and Training Setups
text: Baselines. We choose the open-source Llama2 models (Touvron et al., 2023) and Mistral-7B (Jiang et al., 2023) as the backbones of our METAAGENT and sub-agents. The compared baselines are as follows: 1) CoT (Wei et al., 2022), the naive Chain-of-Thought reasoning method. 2) REACT (Yao et al., 2023), a well-known single-agent framework based on few-shot learning that performs planning and action iteratively. 3) Chameleon (Lu et al., 2023), another few-shot single-agent framework that performs planning before action. 4) Reflexion (Shinn et al., 2023), a single-agent framework to reinforce language agents through linguistic feedback. 5) BOLAA (Liu et al., 2023), a multi-agent framework that customizes different agents through prompts. 6) ReWOO (Xu et al., 2023), a multi-agent framework that decouples reasoning from observations. 7) FIREACT (Chen et al., 2023a), a single-agent framework with finetuning on diverse kinds of trajectories generated by GPT-4 (OpenAI, 2023). 8) GPT-3.5-Turbo (OpenAI, 2022). To ensure fairness, we maintain an equal training trajectory volume of 200 for FIREACT and AUTOACT (200 synthesized data). As Reflexion provides answer correctness labels during reflection but other methods including AUTOACT do not, we test all the other methods twice and choose the correct one for evaluation. For all the prompt-based baselines, we uniformly provide two examples in the prompt.
Training Setups. We fine-tune all our models with LoRA (Hu et al., 2022) in the format proposed in Alpaca (Taori et al., 2023). Our fine-tuning framework leverages FastChat (Zheng et al., 2023) using DeepSpeed (Rasley et al., 2020). We detail the hyper-parameters for training in Tab. 4.

heading: C Detailed Process of Human Evaluation
text: To get a deeper understanding of the capability of AUTOACT, we manually compare the quality of trajectories generated by different methods from five aspects. We ask five NLP volunteers to individually select the optimal trajectories generated by all methods in terms of the number of planning rounds, the logical correctness of thoughts, action types, action parameters, and overall coherence. The final results are determined based on major votes. During the evaluation, it is hidden for the evaluators of the correspondence between the trajectories and the methods. We delete the reflection-related parts from the trajectories generated by AUTOACT and randomly shuffle the order of trajectories of each method in each data to minimize the potential bias as much as possible.

heading: D Average Planning Rounds
text: We compare the planning rounds of AUTOACT with various baselines. The win rate of each method is listed in Fig. 6 and comprehensive analysis can be found in §5. Here we present the average planning rounds of various methods on HotpotQA with Llama-2-70B-chat in Tab. 5. Note that to maintain fairness, we exclude the planning steps related to reflection of AUTOACT.

heading: E Task Information
text: Task Name: HotpotQA Task Description: This is a question-answering task that includes high-quality multi-hop questions. It tests language modeling abilities for multi-step reasoning and covers a wide range of topics. Some questions are challenging, while others are easier, requiring multiple steps of reasoning to arrive at the final answer. Task Data Examples: Question: From 1969 to 1979, Arno Schmidt was the executive chef of a hotel located in which neighborhood in New York? Answer: Manhattan
Question: Are both Shangri-La City and
Ma’anshan cities in China? Answer: yes
Task Name: ScienceQA Task Description: This is a multimodal questionanswering task that necessitates a model to utilize tools for transforming image information into textual data. Simultaneously, this task incorporates substantial background knowledge, requiring the language model to acquire external information to
enhance its comprehension of the task. Task Data Examples: Question: Which of these states is the farthest north? Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma Caption: An aerial view of a painting of a forest. Answer: A. West Virginia
Question: Identify the question that Tom and Justin’s experiment can best answer. Context: The passage below describes an experiment. Read the passage and then follow the instructions below. Tom placed a ping pong ball in a catapult, pulled the catapult’s arm back to a 45 angle, and launched the ball. Then, Tom launched another ping pong ball, this time pulling the catapult’s arm back to a 30 angle. With each launch, his friend Justin measured the distance between the catapult and the place where the
ball hit the ground. Tom and Justin repeated the launches with ping pong balls in four more identical catapults. They compared the distances the balls traveled when launched from a 45 angle to the distances the balls traveled when launched from a 30 angle. Figure: a catapult for launching ping pong balls. Options: (A) Do ping pong balls stop rolling along the ground sooner after being launched from a 30-angle or a 45-angle? (B) Do ping pong balls travel farther when launched from a 30-angle compared to a 45-angle? Caption: A wooden board with a wooden head on top of it. Answer: B. Do ping pong balls travel farther when launched from a 30 angle compared to a 45 angle?

heading: F Tool Library
text: To facilitate our agents in automatic task planning, we provide a comprehensive tool library that contains 15 commonly used tools for various complex question-answering tasks. A part of our tools and their corresponding information can be found in Tab. 6. Users can also have the option to expand the tool library according to their specific needs, allowing for more flexible utilization.

heading: G Prompt
text: G.1 Prompt for Self-Instruct See Tab. 7.
G.2 Prompt for Tool Selection See Tab. 8.
G.3 Prompt for Trajectories Synthesis See Tab. 9.

heading: H Database Cases
text: HotpotQA: Question: The deepest part of the ocean, is located in which ocean? Answer: The Pacific Ocean
Question: The famous scientist who discovered gravity, lived in which century? Answer: 17th century
Question: The first successful flight of a power was made by which inventor?
Answer: The Wright brothers
Question: The highest mountain peak in the solar system is located on which planet? Answer: Mars
Question: In the novel "Pride and Prejudice", what is the name of Mr. Darcy’s estate in Derbyshire, England? Answer: Pemberley
ScienceQA: Question: Which of the following is a type of renewable energy? Options: (A) Coal (B) Oil (C) Natural gas (D) Solar power Caption: A picture of a solar cell Answer: D. Solar power
Question: Which of the following is the term for the process by which the Earth’s weather patterns are influenced by the movement of air in the atmosphere? Options: (A) Weathering (B) Erosion (C) Deposition (D) Atmospheric circulation Caption: An image of air currents in the atmosphere Answer: D. Atmospheric circulation
Question: Which of the following is a type of chemical reaction that involves the transfer of electrons between atoms? Options: (A) Combustion (B) Photosynthesis (C) Respiration (D) Electrolysis Caption: An image of a battery Answer: D. Electrolysis
Question: Which of the following is an example of a type of weather phenomenon that occurs when warm air rises and cool air sinks? Options: (A) Thunderstorms (B) Hurricanes (C) Fog (D) Fronts Caption: An image of a front Answer": D. Fronts
Question: Which of the following is the term for the process by which water is purified through the use of microorganisms that consume organic matter? Options: (A) Filtration (B) Sedimentation (C) Biodegradation (D) Disinfection
Caption: An image of a water treatment plant Answer: C. Biodegradation

heading: I Training Data Example
text: Here we give an example of the training data for each sub-agent.
Plan-Agent (generate Thought): Input: (format requirements) (tool usage instructions) Question: The first human-made object to land on the moon, in 1969, was which spacecraft? Thought: I should first search the Moon landing history. Action: BingSearch[moon landing spacecraft] Observation: A Moon landing or lunar landing is the arrival of a spacecraft on the surface of the Moon. The first human-made object to touch the Moon was the Soviet Union’s Luna 2, on 13 September 1959. Apollo 11 (July 16–24, 1969) was the American spaceflight that ...... Thought: Output: Retrieve for more information about Apollo 11
Plan-Agent (generate Action name): Input: (format requirements) (tool usage instructions) Question: The first human-made object to land on the moon, in 1969, was which spacecraft? Thought: I should first search the Moon landing history. Action: BingSearch[moon landing spacecraft] Observation: A Moon landing or lunar landing is the arrival of a spacecraft on the surface of the Moon. The first human-made object to touch the Moon was the Soviet Union’s Luna 2, on 13 September 1959. Apollo 11 (July 16–24, 1969) was the American spaceflight that ...... Thought: Retrieve for more information about Apollo 11. Action: Output: Retrieve
Tool-Agent: Input: (format requirements) (tool usage instructions) Question: The first human-made object to land on the moon, in 1969, was which spacecraft? Thought: I should first search the Moon landing
history. Action: BingSearch[moon landing spacecraft] Observation: A Moon landing or lunar landing is the arrival of a spacecraft on the surface of the Moon. The first human-made object to touch the Moon was the Soviet Union’s Luna 2, on 13 September 1959. Apollo 11 (July 16–24, 1969) was the American spaceflight that ...... Thought: Retrieve for more information about Apollo 11. Action: Retrieve Output: Apollo 11
Reflect-Agent (generate Thought): Input: (format requirements) (tool usage instructions) Question: The first human-made object to land on the moon, in 1969, was which spacecraft? Thought: I should first search the Moon landing history. Action: BingSearch[moon landing spacecraft] Observation: A Moon landing or lunar landing is the arrival of a spacecraft on the surface of the Moon. The first human-made object to touch the Moon was the Soviet Union’s Luna 2, on 13 September 1959. Apollo 11 (July 16–24, 1969) was the American spaceflight that ...... Thought: Retrieve for more information about Apollo 11. Action: Retrieve[Apollo 11] ...... Action: Finish[Apollo 11] Observation: Please reflect your answer based on the history. Thought: Output: The question asks about the first human-made object to land on the moon, so it seems that the Soviet Union’s Luna 2 is more like the answer.
Reflect-Agent (generate Action): Input: (format requirements) (tool usage instructions) Question: The first human-made object to land on the moon, in 1969, was which spacecraft? Thought: I should first search the Moon landing history. Action: BingSearch[moon landing spacecraft] Observation: A Moon landing or lunar landing is the arrival of a spacecraft on the surface of
the Moon. The first human-made object to touch the Moon was the Soviet Union’s Luna 2, on 13 September 1959. Apollo 11 (July 16–24, 1969) was the American spaceflight that ...... Thought: Retrieve for more information about Apollo 11. Action: Retrieve[Apollo 11] ...... Action: Finish[Apollo 11] Observation: Please reflect your answer based on the history. Thought: The question asks about the first human-made object to land on the moon, so it seems that the Soviet Union’s Luna 2 is more like the answer. Action: Output: Reflect[wrong]

heading: Future Work
text: Interesting future directions include: i) expanding AUTOACT to more realistic task scenarios (Puig et al. , 2018; Zhou et al. , 2023a; Xie et al. , 2024), ii) boosting more knowledge via self-instruct (as analyzed in §5), iii) iteratively enhancing synthetic trajectories via self-improvement (Huang et al. , 2023; Aksitov et al. , 2023). Limitations
In this paper, we focus on constructing an automatic agent learning framework dubbed AUTOACT. Despite our best efforts, this paper may still have some remaining limitations. Tasks. In this paper, we mainly focus on complex question-answering tasks. However, there are many other more complex interactive scenarios, including web (Yao et al. , 2022; Zhou et al. , 2023a), household (Puig et al. , 2018; Shridhar et al. , 2021), traveling (Xie et al. , 2024), robotics (Ichter et al. , 2022), etc. For example, we have investigated the use of META-AGENT performing random explorations (Xiang et al. , 2023; Murty et al. , 2024) in virtual environments to replace the process of task and trajectory synthesis through self-instruct and zero-shot planning. We plan to conduct further research on applying AUTOACT to a wider range of tasks based on this in the future. Boosting Knowledge via Self-Instruct. As analyzed in §5, the planning performance of AUTOACT can be limited by the model’s ability to access internal knowledge through self-instruct. While the current phenomenon allows us to achieve lightweight self-differentiation in terms of parameters and data, it is still necessary to research how to enrich knowledge as much as possible within the constraints of limited data. Self-Improvement. Recent research has shed light on self-improvement techniques that enhance LLMs by iteratively training them on selfsynthesized data (Zelikman et al. , 2022; Huang et al. , 2023; Gülçehre et al. , 2023; Aksitov et al. , 2023). This approach allows the model to continually learn and refine its performance on its own. Our approach also involves training on selfsynthesized data and we believe that further using the iterative thinking of self-improvement will significantly enhance the performance of our method