PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion (2024)

Abstract: Recent evaluations of Large Language Models (LLMs) have centered around testing their zeroshot/few-shot capabilities for basic natural language tasks and their ability to translate instructions into tool APIs. However, the evaluation of LLMs utilizing complex tools to finish multiturn, multi-modal instructions in a complex multi-modal environment has not been investigated. To address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark to assess LLMs’ ability to create and edit PPT files based on user instructions. It contains 279 multi-turn sessions covering diverse topics and hundreds of instructions involving multi-modal operations. We also propose the PPTX-Match Evaluation System that evaluates if LLMs finish the instruction based on the prediction file rather than the label API sequence, thus it supports various LLM-generated API sequences. We measure 3 closed LLMs and 6 open-source LLMs. The results show that GPT-4 outperforms other LLMs with 75.1% accuracy in single-turn dialogue testing but faces challenges in completing entire sessions, achieving just 6% session accuracy. We find three main error causes in our benchmark: error accumulation in the multi-turn session, long PPT template processing, and multi-modality perception. These pose great challenges for future LLM and agent systems 1.

heading: 1 Introduction
text: Recent evaluation works for Large Language Models (e.g. ChatGPT and GPT-4 (OpenAI, 2023)) focus on their zero-shot/few-shot abilities on basic natural language tasks (Jiao et al., 2023; Zhong et al., 2023; Wang et al., 2023; Qin et al., 2023a) and their tool-use ability to generate APIs for solving user instructions, such as basic APIs like a calculator in tool transformer (Schick et al., 2023), RapidAPIs in ToolLLM (Qin et al., 2023c), and
1Code: https://github.com/gydpku/PPTC
hugggingface APIs in Gorilla (Patil et al., 2023). However, these tool-use works emphasize the translation of natural language instructions into APIs and ignore the challenge of using APIs in the observation of complex multi-modal environments to finish user instructions. Also, their evaluation approach focuses on comparing the generated APIs with the label API sequence, assuming there’s only one unique solution. This approach becomes impracticable in situations with multiple/unlimited correct solutions. To address these challenges, we introduce Power-Point Task Completion (PPTC), a benchmark that measures LLMs’ performance in creating and editing PPT file tasks based on user instructions. We choose PowerPoint as it includes various elements like textbox, table, and image and supports a wider range of APIs than Word and Excel.
Our benchmark has three distinctive features from other task completion benchmarks: (1) Multiturn dialogue with varying difficulty. Our PPTC benchmark simulates the multi-turn dialogue session between the user and the LLM and contains 279 multi-turn sessions. Each multi-turn session in our benchmark includes 2 to 17 turns. Each turn consists of a user instruction that describes the user’s needs, a feasible solution that provides the correct solution, and the resulting label output file. Some turns can be easily addressed using a single API, while over half of the instructions require multiple APIs for completion. We provide the LLM with a reference API file that contains all feasible APIs for selection. (2) Multi-modality. Finishing the instruction of our benchmark requires understanding the multi-modal PPT file content and using multi-modal API operations (e.g., PPTC has 268 image operation-related instructions). (3) Evaluation based on the final status: We propose the PPTX-Match Evaluation system to evaluate the LLM’s outcome. To identify if the LLM completes the instruction, it checks the PPT file produced by
8682
executing the LLM-generated APIs rather than the LLM-generated APIs, thus all API sequences that lead to the correct final status are acceptable.
To finish the instruction, we use the current instruction, past turns’ instructions (dialogue history), the PPT file content (specific environment information), and the reference API file as the input to prompt the LLM to generate an API sequence as the solution (See Figure 1 (A)). Then we use the API executor to execute the API sequence and return the user the resulting PPT file (See Figure 1 (B)). We name the resulting PPT file as the prediction file. In the evaluation step (See Figure 1 (C)), the PPTX-Match Evaluation system first uses the Python-PPTX library to extract all attributes from the prediction PPT file and the label output file. Then it uses the position relation checker to check if objects’ positions conform to the label relation and the attribute content checker to check if the attribute’s content is matched with the corresponding label attribute’s content. The LLM correctly completes the current turn’s instruction if all attributes
of the file pass these tests.
We measure the performance of three closedsource LLMs (GPT-4, ChatGPT, and Davince-003) and six open-source LLMs (e.g., LLaMa-2) in our benchmark. We further test planning (e.g., CoT (Wei et al., 2022)) and content selection algorithms’ performance based on GPT-4. Evaluation metrics include turn-based accuracy as the ratio of correctly completed turns to the total number of turns and session-based accuracy as the ratio of correctly completed sessions to the overall session count. Experiment results show that GPT-4 is the strongest LLM among all LLMs but still encounters challenges when completing entire multi-turn sessions. For example, although GPT-4 achieves 75.1% turn-based accuracy in the creating new PPT file task, it only achieves 22.7% session-based accuracy as errors made in previous turns. GPT-4 and other LLMs also struggle to process long PPT templates (complex file environment). For example, GPT-4 only achieves 38.1% turn-based accuracy in the editing task. We further find that GPT-4
8683
struggles to finish instructions involving non-text modality operations, especially for position-related operations, such as ’Put object A on the top of the slide’. It only achieves 24% accuracy in these instructions.
In summary, this paper has the following contributions:
(1) We propose the PowerPoint Task Completion benchmark to measure LLM’s task completion performance within the PowerPoint official software. This benchmark contains 279 multi-turn sessions with hundreds of multi-modal instructions in the complex multi-modal environment.
(2) We propose the PPTX-evaluation system to automatically measure LLMs’ performance in our benchmark. We test 3 closed-source LLMs and 7 open-source LLMs and find that GPT-4 is the strongest LLM among all LLMs.
(3) We further analyze LLMs in our benchmarks and find three key error factors: error accumulation in the session, long PPT template processing, and multi-modality perception. These findings pose significant challenges for future LLMs and LLMbased systems.

heading: 2 PPTC Benchmark
text: In this section, we introduce our Power-Point Task Completion (PPTC) benchmark, including the overview of our benchmark, its collection and validation process, and the PPTX-Match Evaluation System for evaluation. We further analyze the statistics information of our benchmark.

heading: 2.1 Benchmark Overview
text: Benchmark components Our benchmark focuses on two basic tasks within PowerPoint: creating the new PPT file and editing the existing long PPT template for measuring long PPT Content understanding. We have gathered 229 multi-turn dialogue sessions for creating the new PPT file and 50 sessions for editing existing templates. Each multi-turn session includes 2 to 17 turns. Each turn comprises three parts: (1) the user instruction (2) the label output file as the ground truth (3) one feasible API sequence for finishing the instruction. Our benchmark also contains an API reference file that includes 49 feasible APIs for various operations and can complete all instructions in our benchmark. For each API, we describe its functionality and arguments and provide usage guidelines. For complex APIs, we also offer example cases. We also
provide a PPT reader function and an API executor for LLMs to process the multi-modal PPT file and execute the API sequence, respectively. The details of all APIs, PPT reader, and API executor are in Appendix A.
Task description To complete the instruction in one turn, in general, the AI assistant must comprehend the user’s current and prior instructions for context. It should also analyze the content of the PPT file to identify relevant objects mentioned in the instruction. Additionally, it needs to select appropriate APIs from a reference API file to achieve the user’s goals. So we use these as the input of the AI assistant and it should output an API sequence as the solution. Then, it executes this API sequence and provides the user with the resulting PPT file as its response (See the whole process in Figure 1).

heading: 2.2 Benchmark Collection
text: Design Principles We follow these principles to design our benchmark: (1) Multi-turn instructions: One session in our benchmark should contain multiturn instructions to finish the user’s complex need. (2) Instructions of varying difficulty: Some instructions can be achieved with a single API, while others necessitate a sequence of APIs for successful completion. (3) Diverse multimodal operations: User instructions should cover a wide range of operations on PPT, such as text-related, image-related, and position-related APIs. (4) Topic Consistency: The dialogue in a session should center around the session topic. Each user instruction in a session aligns closely with the previous instructions (the context), ensuring a coherent and contextually relevant dialogue flow. (5) Practicability First: The session topic and specific instructions should simulate the user’s need in real world
Benchmark Collection and Validation To collect user instructions, we engage 6 skilled crowd workers who craft instructions in accordance with the principles we’ve outlined. To achieve practicability first, we request crowd workers to write instructions based on their actual PowerPoint experience. On each session, the workers are asked to first find and list a practicable session topic. For the editing PPT template task, the topic must based on the template file background and is practicable to the template2. To achieve multi-instructions and topic consistency, the workers write instructions
2We collect 50 PPT templates from the SlidesCarnival website (https://www.slidescarnival.com/).
8684
step by step and make them consistent with the topic. To achieve diverse multi-operations, we ask them not to write session that only involves a single modality operation.
Then we ask the seventh worker to write the feasible API sequence with minimal API usage for each instruction. Next, the workers create the PPT label file by using the provided API sequence. More details and steps for the collection and validation process are in Appendix B.

heading: 2.3 PPTX-Match Evaluation System
text: We design the PPTX-Match Evaluation System to evaluate LLMs’ performance on the PPTC benchmark. It’s based on the PPT file (final status). To judge if the LLM has successfully completed the user instruction in one turn, it compares all nonposition attributes in the LLM’s prediction file with those in the label file and verifies if objects follow the correct position relation. The detailed steps are in Appendix C.

heading: 2.4 Benchmark Statistics Analysis
text: To understand the properties of PPTC, we analyze the instructions and APIs and report statistics about the PPTC benchmark in Figure 2.
The number of turns in a session The session turn number distribution (Figure 2 (a)), measured as the number of turns in a session, shows that almost all sessions have at least 3 turns (between 3 and 13 turns for the 5th to 95th percentile, respectively). The longest session has 17 turns, which is very challenging as the errors made in previous turns can influence the completion of the current instruction.
Diffculty varies in APIs number The number
of APIs in a sequence falls between 1 and 5 for the 5th to 95th percentile (Figure 2 (b)), respectively, shows that our instructions vary from a simple one that can be finished by one API to a complex instruction that requires multiple APIs. The longest API sequence consists of 29 APIs. Generating long API sequences is very challenging as the LLM needs to understand sub-goals in the complex instruction, select appropriate APIs, and generate APIs in a reliable order.
Rich multi-modal instructions Our benchmark has hundreds of instructions that involve multimodalities content (Figure 2 (c)). The "chart" modality has the fewest instructions, with 120, while the "position" modality has the most, with 292 instructions. To finish these instructions, LLMs need to employ related-modal APIs based on the understanding of multi-modal file content.

heading: 3 Algorithms
text: In this section, we introduce the algorithms we considered to enhance the LLM’s performance in our benchmark. These algorithms can be categorized into two approaches: planning algorithms that help the LLM in decomposing the user instruction and solving it step by step and selection algorithms that assist the LLM in choosing important environmental information or APIs.

heading: 3.1 Planning Algorithms
text: Complex user instructions often require multiple intermediate steps to complete. We mainly consider two planning algorithms:
Zero-shot-CoT (Kojima et al., 2022) enables LLMs to autonomously generate intermediate
8685
reasoning processes for complex instruction by prompting LLMs to "Let’s think step by step".
Tree of Thoughts (ToT) (Yao et al., 2023) enables LLMs to follow tree-like reasoning paths, where each tree node represents a thinking state. It leverages LLMs to generate evaluations or votes on different thoughts.

heading: 3.2 Selection Algorithms
text: Combining the whole PPT file and the whole API file into the LLM’s input can result in redundant information and filtering out them would improve the efficiency of the LLM. In this context, we primarily focus on two algorithms for selecting the relevant PPT file content and helpful APIs, respectively:
Content Selection algorithm Firstly, we extract all shapes of the PPT file by Python-PPTX. Next, we prompt the LLM to select the shapes for completing the user’s instruction. We show the prompt in Figure 9 of Appendix D, in which we add three demonstration examples to guide the LLM to do selection. In this algorithm, we replace the whole PPT file with the selected shapes when prompting the LLM to generate the API sequence.
API Selection algorithm The API selection algorithm is based on the embedding similarity to select the most relevant APIs for user instructions. Specifically, we use the text embedding API to get the embeddings of all API descriptions and the current user instruction. Next, we compute the cosine
8686
similarity between the instruction embedding and each API description’s embedding and rank them based on the similarity score. In this algorithm, we replace the whole reference API file with the top k APIs when prompting the LLM to generate the API sequence.

heading: 4 Experiments

heading: 4.1 Large Language Models Selected for Evaluation
text: We assess different cutting-edge large language models using our benchmark. These chosen models showcase a wide array of capabilities and are highly regarded in the field. The evaluated large language models include 3 closed-source LLMs:GPT4 (OpenAI, 2023), ChatGPT, and Text-Davinci003 (Brown et al., 2020). We also consider 7 open-source LLMs: LLaMa-2-Chat (Touvron et al., 2023), Baichuan-Chat, Baichuan-2-Chat (Yang et al., 2023), WizardLM v1.2 (Xu et al., 2023a), Vicuna v1.5 (16k) (Chiang et al., 2023), and CodeLLaMa-instruct (Chiang et al., 2023), and Mistralinstruct (Jiang et al., 2023). The detailed introduction of these LLMs is in Appendix E.

heading: 4.2 Experimental Setup
text: In this section, we provide an overview of the experimental setup utilized to assess the performance of LLMs on our PPTC benchmark.

heading: 4.2.1 Turn-Based and Session-Based Evaluations
text: We consider two performance evaluation approaches in our benchmark: turn-based and sessionbased evaluations. For the turn-based evaluation, we measure the LLM’s ability to finish a single turn. For the session-based evaluation, we measure the LLM’s ability to finish a session containing multiple turns. One core difference is that the turn-based evaluation assumes that the previous turns have been correctly finished but the session-based evaluation does not. We put more details and illustrate the prompts for the two evaluations in Appendix F.
Metrics For turn-based evaluation, we report the turn-based accuracy as the ratio of the number of successfully finished turns to the total number of turns. We also report the average token number of the input of one turn and the average API number for finishing one turn as the cost measurement. For session-based evaluation, we report the session-based accuracy as the ratio of the number
of successfully finished sessions to the total number of sessions. We also report the average value of the token number of all inputs in one session and the average API number required to complete one session as the cost measurement.

heading: 4.3 Implementation Details
text: All closed-LLMs experiments were conducted using the respective language models’ API provided by Azure OpenAI Service. For open-source LLMs, we choose their chat/instructed version with 13 billion parameters model. For the zero-shot CoT method, we add the sentence ’Let’s think step by step’ after the dialogue history of the prompt. Our inference prompts are in Figure 3. We run the four algorithm methods based on the GPT-4 model. More details are in Appendix G

heading: 4.4 Main results
text: We report the results of LLMs in both turn-based and session-based evaluations in Table 1 and 2. From the results, we highlight the following key findings.
(1) Superior Performance of GPT-4: GPT-4 consistently outperforms other closed-source and open-source LLMs in both two tasks. For example, GPT-4 achieves 75.1% turn-based accuracy in the creating new PPT file task, demonstrating its strong capability to finish one turn of the user instruction. GPT-4 also has a lower API cost compared to other closed-source LLMs since its precise API usage. GPT-4 incurs the highest token expense when editing PPT templates. That is because its higher token limit than other LLMs allows us to input more PPT template content.
(2) Code continual pre-training and further instruction finetuning can boost open-source LLMs’ performance.: Current open-source LLMs struggle to match the performance of closed-source LLMs in Table 1. For example, LLaMa-2-chat only achieves 16.2% turn-based accuracy in the creating PPT slide task. Code continual pretraining (Code-LLaMa) and instruction fine-tuning based on LLaMa-2 (WizardLM and Vicuna) can further improve LLaMa-2 potential PPT task completion performance on our benchmark. For example, Code-LLaMa improves LLaMa-2’s turn-based accuracy in the creating new PPT slide task by 20.4 %.
(3) Planning and selection algorithms can improve LLMs’ turn-based performance From Table 2, we observe that the planning algorithms (CoT
8687
and ToT) can further improve the turn-based performance of GPT-4 by 1∼2 percent. However, we find that the more complex ToT algorithm does not outperform the zero-shot CoT algorithm with a 5∼10 times token cost. Content and API selection algorithms can further improve the turn-based performance of GPT-4 by 1∼ 5 percent. That is because they reduce the task difficulty by filtering irrelevant PPT content/APIs in the input prompt. The API selection algorithm also reduces the average token cost by reducing the number of APIs listed in the prompt.

heading: 4.5 Three challenges in our PPTC benchmark
text: From the result Table 1 and Figure 4. we highlight the following three key challenges.
(1) Error accumulation makes LLMs performance poor in finishing the entire multi-turn session.: The performance of all LLMs in handling sessions consisting of multiple turns is notably poor. GPT-4, achieves only a 22.7% session-based accuracy for the "creating new PPT file" task and a mere 6.0% session-based accuracy for the "editing PPT template" task. The session-based evaluation is challenging since errors made in previous turns make the LLM fail to finish the session and also influence the completion of the current turn. Current planning and selection algorithms usually fail to improve session-based accuracy markedly. In some cases, they can even make the performance worse.
(2) LLMs perform badly in processing long PPT template: Current LLMs’ performance in the editing PPT temples task is pretty poor. For example, the strongest GPT-4 only achieves 38.1% turn-based accuracy and 6.0% session-based accuracy in this task. The content selection algorithm can partially solve this challenge by filtering out irrelevant file content, but GPT-4 with it still only achieves 43.1% turn-based accuracy. For opensource LLMs, there’s a risk of information loss due to token limitations (typically 2∼4K tokens limit), which often require truncating lengthy PPT content. When it comes to session-based performance, the accuracy remains nearly zero. That means current LLMs (e.g., GPT-4 and LLaMa-2) still struggle to handle complex and lengthy PPT templates.
(3) Multi-modal instructions increase the LLM’s failure rate significantly. To assess LLMs’ task completion performance for instructions involving multi-modal operations (Table, Chart, Picture, Position, and text), we calculate the average accuracy of GPT-4 for each modality by dividing the number of correctly completed instructions within each modality by the total number of instructions involving that modality’s operation. The results are presented in Figure 4 (a). From the figure, we observe that GPT-4’s performance becomes poorer when processing structured data (Chart and Table) rather than text, with 12.4% and 16.2% lower accuracy. Instructions involving picture-related operations pose a greater challenge
8688
for GPT-4 and GPT-4 exhibits its weakest performance in instructions involving position-related operations, with only 24% accuracy. This underscores GPT-4’s limitations in spatial perception ability.

heading: 5 Analysis
text: In this section, we analyze the reasons for GPT-4’s errors. We further analyze the influence of model size and dialogue history.
5.1 Error Analysis of GPT-4 in our benchmark
To analyze the error made by GPT-4, in our benchmark, we gather 50 wrong samples for each of the two tasks in our benchmark in the turn-based evaluation. We find that these wrong samples fall into four error types and visualize the distribution of these four main error types in Figure 4 (b): (1) Position errors: These occur when GPT-4 struggles with instructions involving position adjustments. For example, when asked to move the shape to the bottom of the slide, GPT-4 wrongly calls the "set_top" API. (2) Calling unavailable APIs: GPT-4 sometimes generates APIs that don’t actually exist in the reference API file, resulting in what we call the "API hallucination problem." (3) Misunderstanding PPT file content: GPT-4’s wrong comprehension of the PPT content leads to incorrect APIs. For example, when instructed to make the font size of the current slide’s title consistent with previous slides, GPT-4 set a font size that is different from what was used in previous slides’ titles. Misunderstanding the PPT content becomes the main error in the editing template task. (4) Unfollowing Powerpoint task rules: LLMs don’t understand the Powerpoint task rules. For instance, GPT-4 may directly insert the new
content into the original title when instructed to rewrite the title. For the session-based evaluation, we also collect 50 wrong examples. And we find that the main reasons for errors are similar. One unique phenomenon in this evaluation is that the LLM would repeat previous errors (e.g., repeatedly employing infeasible APIs) in subsequent turns.
We analyze the error of open-source LLMs and explain the reasons for their performance gap compared to closed-source LLMs in Appendix H.
5.2 Does bigger LLM work better on PPTC?
To investigate how the model size impacts the LLM’s performance in our benchmark, we conduct tests using LLaMa-2-chat LLM with 7, 13, and 70 billion parameters and plot the results in Figure 4 (c). We observe that larger LLM consistently achieve higher turn-based accuracy for both the creating new PPT and editing PPT template tasks. For example, in the creating new PPT file task, we find that the turn-based accuracy increases from 13.2 (7B) to 30.1 (70B). However, we do not observe a clear positive correlation between model size and session-based performance. One possible explanation is that although the 70B LLM can correctly finish more intermediate steps, it still falls short of completing the entire session. A larger LLM may be necessary.
5.3 Does dialogue history help LLMs to generate the API sequence?
To investigate the influence of dialogue history in our prompt (see Figure ??), we make an ablation experiment for the dialogue history component of
8689
our turn-based evaluation prompt3. In this evaluation, the dialogue history contains previous turns along with their feasible API sequences. When we removed the dialogue history from the prompt, we observed a decline in GPT-4’s performance. Specifically, GPT-4 drops its performance from 75.1 % to 73.1 % in the creating new PPT file task and decreases its performance by 6.2 % in the editing template task. This experiment shows the positive effect of the dialogue history, as it helps the LLM to both understand the dialogue background and instruct the LLM to correctly use the APIs, similar to few-shot demonstration examples.

heading: 6 Related Works
text: Large Language Models like ChatGPT, GPT4 (Bubeck et al., 2023; OpenAI, 2023), and Bard have billions of parameters and have been trained on the Internet corpus with trillions of tokens. They can write code (Liu et al., 2023a), prove mathematical theorems (Jiang et al., 2022), pass the professional exam (Zhong et al., 2023; Gilson et al., 2023; Katz et al., 2023), employ other models and APIs (Schick et al., 2023; Liang et al., 2023; Wu et al., 2023; Patil et al., 2023), and also perform well on other basic natural language tasks (Kim et al., 2023; Jiao et al., 2023; Zhong et al., 2023; Wang et al., 2023). That raises the hope of achieving artificial general intelligence (AGI).
To further boost LLM’s performance on the specific task, one approach involves prompting engineerings, such as the chain of thought prompting (Wei et al., 2022; Shi et al., 2022; Yao et al., 2023), self-consistency (Wang et al., 2022) and the least to most prompting (Zhou et al., 2022). Another approach aims to use feedback to improve performance. The self-refine method (Madaan et al., 2023; Shinn et al., 2023) refines the output through iterative feedback and refinement Provided by LLM itself or sparse reward signal. The learning to program method (Guo et al., 2023) learns the task program by inducing the general solutions from the errors (feedback) iteratively and uses the program to guide the test inference.
Task completion benchmarks for measuring large language models. To measure LLM’s task completion performance, Saycan (Brohan et al., 2023) and VirtualHome (Puig et al., 2018) bench-
3The task instruction, current user instruction, API file, PPT content in the prompt are necessary parts for generating the API sequence. So we don’t conduct ablation studies on them.
marks ask LLM to generate the correct action sequence for controlling the robot to finish user instruction. WebShop (Yao et al., 2022) and Android in the wild (Rawles et al., 2023) ask LLM to navigate websites and conduct actions to meet the user requirement. APIBench (Patil et al., 2023) and ToolBench (Xu et al., 2023b; Qin et al., 2023b) involve selecting and using APIs to complete the task instruction. Agentbench (Liu et al., 2023b)assesses LLM as autonomous agents in 8 environments and WebArena (Zhou et al., 2023) considers task completion in web-based interactions. We make a comprehensive comparison for these benchmarks in Appendix I.

heading: 7 Conclusion
text: We introduce the PowerPoint Task Completion benchmark to measure LLMs’ ability to complete multi-turn user instructions within the context of the multi-modal PowerPoint software. We further propose the PPTX-evaluation system to access and compare the performance of 10 LLMs and 2 LVMs. We further analyze the behavior of LLMs and find three main error factors that limit their performance. Our benchmark and findings can help the research community design better AI assistants.

heading: 8 Limitations and potential risks
text: Our benchmark does not consider instructions that involve subjective evaluation. For example, the user may want to make the slide more beautiful. However, it’s hard to automatically evaluate if the generated file (the model output) is more beautiful. Another limitation is that we do not consider the instructions that need non-API operations. For example, the user may want to draw a cat on the slide. That instruction needs the AI-assistant system to draw the cat by dragging the mouse and is still infeasible for LLMs and LLM-based systems. We only consider instructions that can be completed by directly executing the API sequence.
For privacy concerns, we replace all real human names, emails, and addresses with those generated randomly. Completing the virtual PPT task could not hurt humans and we do not see any further potential risk in our benchmark.

heading: A The API Reference File, PPT reader, and API excutor
text: We list all APIs and their descriptions in Figures 7 and 8. We provide 49 feasible APIs.
Compared to the general AI assistant, LLMs still have two limitations for completing the task in our benchmarks: (1) LLMs can not directly process the PPT file. So we provide a PPT reader function that extracts all shapes and their information from the PPT file and transforms them into the text format as the PPT file content. Then LLMs can understand and process the PPT file content. The code of the PPT reader is in the supplementary. We illustrate two examples for how the PPT reader turns the shapes (e.g., title, figure and table) of the PPT file into the text description of them in Figure 5 & 6. (2) LLMs cannot directly use PPT software through a keyboard and mouse. Therefore, we have defined
8692
PPT APIs based on the operational logic within the PPT software. and provide an implementation for these APIs in Python that can swiftly generate PPT files. In future work, it may be possible to explore the use of large multimodal models to understand on-screen content and implement APIs using a keyboard and mouse.

heading: B Details for the dataset collection and validation
text: In the collection process, we employ 6 skilled crowd workers. Our crowd workers comprise professional data science engineers well-versed in PowerPoint. Each worker takes on a specific role in the instructional writing work and is encouraged to write instructions in his/her own words. Workers were asked to spend at least 20 minutes on every session. We delete repeated sessions and short sessions that have no more than 50 tokens. For the editing PPT template task, the 50 PPT templates are collected from the SlidesCarnival website (https: //www.slidescarnival.com/). SlidesCarnival is a free and open-source PPT template website. Each session in the editing task has a unique template.
To ensure the data quality of this benchmark, the principal engineer reviews and refines the instructions and API sequences written by the above 7 workers for initial quality assurance. Then the three authors of this paper further undertake the following validation steps: (1) Assessing Instruction Clarity and Relevance: They examine whether the instructions are clear, contextually related to the session topic, and align with the ongoing conversation. (2) API Sequence Execution: The authors execute the provided API sequences to identify and rectify coding errors. (3) Goal Achievement Check: They verify if the instructions’ intended goals are successfully completed in the label files.
In the event that errors are identified during this validation process, the authors promptly report them to the respective workers for revision. The three authors are computer science senior students and researchers.

heading: C The Details for PPTX-Match Evaluation System
text: Specifically, our PPTX-Match Evaluation System first uses a Python-PPTX Content Reader Module to iterate over all shapes in the prediction PPT file produced with the LLM and the label output file.
A shape in the PPTX library typically refers to an individual object, such as a text box or table. Then our system extracts attributes like text, style, and position of the shapes using the PPTX library. Next, we check all attributes from the prediction PPT file. For non-position attributes (e.g., text content), we first convert it and the corresponding attribute in the label PPT file into two strings, and then we use the Exact Match method to examine if the two strings are the same. If they are different or we do not find the corresponding attribute in the label file, then we find an incorrect match. For the position attribute (e.g., location information), we focus on checking if the objects in the prediction PPT file satisfy the correct position relation <A, B, REL>, where A and B are objects that should satisfy the relation REL. In the benchmark collection process, we ask crowd workers to label the position relation that objects should satisfy to finish the instruction.
If there are no incorrect matches for all nonposition attributes and no rule violations for all position-related attributes, we consider the LLM has successfully completed the user instruction.

heading: D The Prompt for Content Selection Algorithm
text: We put the prompt of content selection algorithm in Figure 9.
8693
8694
8697
E Introductions of 10 LLMs and 2 LVMs
We consider the following 10 LLMs:
• GPT-4 (OpenAI, 2023): The latest LLM in the GPT series. GPT-4 is a cutting-edge, large-scale generative pre-trained transformer model. It offers improved performance and a wider knowledge base compared to its predecessors. It showcases human-level proficiency in several scenarios.
• ChatGPT: ChatGPT is a conversational AI model crafted for dynamic interactions. It’s learned from extensive instruction data and fine-tuned through reinforcement learning with human feedback (RLHF). This empowers it to deliver responses that align with human expectations, maintaining context and coherence in conversations.
• Text-Davinci-003 (Brown et al., 2020): GPT3.5 sits between GPT-3 and GPT-4, enhancing performance via additional instruction tuning. It acts as a link between these models, facilitating comparison. We’ve chosen the TextDavinci-003 variant from the GPT-3.5 series for our evaluation.
• LLaMa-2-Chat (Touvron et al., 2023): LLaMa 2, an auto-regressive open-source language model, employs an optimized transformer design. Chat versions utilize supervised fine-tuning (SFT) and reinforcement
learning with human feedback (RLHF) to match human preferences for helpfulness and safety.
• Baichuan-Chat: It is a transformer model trained on approximately 1.2 trillion tokens. It supports both Chinese and English, with a context window length of 4096.
• Baichuan-2-Chat (Yang et al., 2023): It is a large-scale multilingual language model trained from scratch, on 2.6 trillion tokens. The chat version uses Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align with humans.
• WizardLM v1.2 (Xu et al., 2023a): WizardLM v1.2 is finetuned from LLaMa 2 using supervised instruction fine-tuning, where instructions are created by rewriting the initial instructions step by step.
• Vicuna v1.5 (16k) (Chiang et al., 2023): Vicuna v1.5 (16k) is finetuned from LLaMa 2 using supervised instruction fine-tuning and linear RoPE scaling. It’s trained on about 125K conversations sourced from ShareGPT.com.
• Code-LLaMa-instruct (Chiang et al., 2023): Code-LLaMa is a LLaMa-based LLM designed for general code completion and understanding. Its instruction version further supports the chat function with users. Code LLaMa models feature a multitask training objective consisting of both autoregressive and causal infilling prediction (predicting the missing part of a program given a surrounding context).
• Mistral-7b-instruct (Jiang et al., 2023) Mistral-7b is a 7 billion size LLM that leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length. It outperforms many larger LLMs (e.g., LLaMa-13b) on public evaluation benchmarks. We use its instruction version.

heading: F The details and inference prompts for turn-based and session-based evaluations
text: In the turn-based evaluation, we assume that the previous turns in one session have been correctly
8698
finished. We prompt the LLM to generate the API sequence to finish the current turn’s user instruction. The prompt consists of the task instruction for finishing the current user instruction, the API file containing feasible APIs, the parsed PPT file content from the PPT file, and dialogue history consisting of instructions of previous turns with their feasible API sequences. In the session-based evaluation, we prompt the LLM to finish all turns in a session sequentially. The prompt in this evaluation has two differences: the API solutions for previous turns in dialogue history are the outputs of the LLM instead of the correct API sequences. (2) The PPT content is parsed from the PPT file obtained by executing the previous outputs of the LLM. That means the error made by LLMs in previous turns would influence subsequent turns. We list the inference prompt we used in Figure ??.

heading: G Experimental details
text: Azure OpenAI services4 offer two API types: completion and chat completion. Completion API generates text from prompts, while chat completion API responds based on conversation history and new input. We use the completion API for TextDavinci-003 and the chat completion API for ChatGPT and GPT-4. We set a temperature of zero for deterministic output and a max token limit of 2048. The frequency penalty and top p are kept at their default values of zero and 1, respectively. We use the text-embedding-ada-002 API as the embedding API in the API selection algorithm and set k as 15. If the token number of the input prompt is beyond the token limit, we cut the PPT file content to reduce the token number of the prompt. For the ToT method, we follow the official code to run it5.

heading: H Error Analysis for open-source LLMs
text: To analyze the errors of open-source LLMs and explain why their performance is lower than that of the closed-source LLM (e.g. GPT-4), we collect 50 wrong examples made by WizardLM and CodeLLaMa in turn-based evaluation, respectively. Each example consists of a user instruction and a wrong prediction file compared to the label file.
Drawing from incorrect examples, we find three
4https://azure.microsoft.com/ en-us/products/cognitive-services/ openai-service
5ToT:https://github.com/princeton-nlp/ tree-of-thought-llm
unique error reasons in open-source LLMs to explain their poor performance :
• Unexcutable API prediction sequence We find that open-source LLMs frequently generate API sequences that are either meaningless or improperly formatted. For instance, the WizardLM exhibits a 37% rate of generating incorrect API sequences such as ’API(), API()’ in its erroneous examples. Although CodeLLaMa is less prone to this type of error, it still generates API sequences like ’set_font_size(15, 2, 5)’ that would return errors ’ERROR: set_font_size() takes 1 positional argument but 3 were given’, with a rate of 25%. In contrast, closed-source LLMs like GPT-4 rarely make these mistakes, showcasing excellent code generation abilities and the capability to generate concise and clear API sequences.
• Severe API hallucinations We find that both two open-source LLMs often call APIs that do not present in our API reference file, such as ’delete_table()’, ’choose_slide’ (with the correct API being ’move_to_slide’), and ’choose_slide_by_id’. These mistakes constitute half of the errors made by these LLMs. While GPT-4 sometimes also calls unavailable APIs, it does so at a lower rate.
• Short context size limits the input of the PPT file Open-source LLMs typically have a limited context size of 2k tokens, which proves insufficient for handling lengthy PowerPoint (PPT) template tasks, often comprising multiple slides. Our analysis reveals that the input for open-source LLMs generally includes file information from only the initial 1 to 3 slides. Consequently, these models face challenges in executing instructions related to subsequent slides and cross-slide operations. This limitation leads to the generation of APIs containing incorrect file information or invoking improper APIs. In contrast, closed-source LLMs such as GPT-4 support an extended context size of 32k tokens, enabling them to effectively manage longer templates.

heading: I A Comparison of Task Completion Benchmarks
text: In Table 3, we compare our benchmark with other task completion benchmarks in five aspects. Our
8699
benchmark is the only one that satisfies the five requirements.
8700
8701

heading: Future Work
text: In future work, it may be possible to explore the use of large multimodal models to understand on-screen content and implement APIs using a keyboard and mouse