SlowGuess/NeurIPS_2023_OCR huggingface dataset:
found that GPT-3.5-Turbo often fails to use absolute references when necessary, uses self-reference, and uses wrong criteria for a filter or conditional formatting. Inspecting the wrong range cases, we found that GPT-3.5-Turbo tends to use wrong auto-filling ranges, use wrong chart data sources, and leave out the headers when copy-pasting columns. Another two prominent types of failure are Wrong action and Incomplete solution, which occupy  $33.7\%$  in total. After inspecting the wrong action cases, we found that GPT-3.5-Turbo misunderstands the usage of the atomic actions it incorrectly uses. For example, GPT-3.5-Turbo inserts an unnecessary column to fill in data, ignoring the existing column where the data should be written; GPT-3.5-Turbo also uses a filter to hide columns, which violates the actual usage of a filter. After inspecting the incomplete solutions, we found that GPT-3.5-Turbo fails to fulfill all task requirements in these cases although other steps are correct (e.g. it forgets to set the chart title or to autofill after filling in the first row). In the cases where repeated output occurs, GPT-3.5-Turbo generates the same action repeatedly as it fails to revise its plan according to the given error feedback. GPT-3.5-Turbo occasionally misses setting required arguments or disobeys the task requirements to set incorrect arguments which both make up a small fraction of the failure cases. A small proportion of the failure cases result from the implementation limitations of several atomic actions, which can be fixed by upgrading these implementations.
G Demo
Video demos for our SheetCopilot agent can be found on our project website (https://sheetcopilot.github.io).
Our SheetCopilot possesses the following limitations:
- We have not optimized the number of tokens used to solve spreadsheet tasks. This means that SheetCopilot consumes a large number of tokens and is only capable of solving short-horizon tasks. Future work is needed to focus on designing efficient prompting methods that save tokens or using LLMs with a larger context window.
- State feedback only consists of the basic information of cells, ignoring the state of charts, pivot tables, filters, and frozen panes. SheetCopilot probably creates charts and pivot tables repeatedly as it is unable to observe the outcome of creating these elements.
- Our dataset is manually curated and is not fully fledged yet. More labor work is required to generate more ground truth as numerous correct solutions to each task probably exist. Moreover, the more solutions are collected to conduct evaluation, the more accurate and stable performances we can obtain.
- Our evaluation environment does not support all Excel built-in features, which restricts the task categories our SheetCopilot is able to handle. More effort is needed to continuously upgrade the evaluation system to accommodate more diverse tasks.
Our proposed SheetCopilot possesses the potential to significantly simplify the process of spreadsheet control, which positively impacts productivity and efficiency. By enabling users to interact with spreadsheets using natural language, SheetCopilot reduces the time and effort required to perform a wide range of tasks, such as simple data entry and complex data analysis.
However, it is imperative to consider potential negative impacts. One concern is that the increased ease of spreadsheet control possibly leads to an over-reliance on automation, potentially reducing the need for human expertise in certain areas, which may cause unemployment in society. Additionally, there is a risk that the use of SheetCopilot exacerbates existing inequalities in the workforce as those who lack access to SheetCopilot or who are not proficient in using it may be at a disadvantage.
Another potential risk of using SheetCopilot is that a wrong operation is likely to face undesirable consequences when controlling sensitive data forms, such as tax forms or financial statements. This risk highlights the importance of ensuring the robustness and accuracy of the agent's actions, as well as the need for appropriate safeguards and oversight to minimize the risk of errors.
Additionally, SheetCopilot relies on the capabilities of language models, which are known to potentially inherit biases present in the training data. As a result, SheetCopilot may inadvertently exacerbate existing biases when interacting with spreadsheets and processing data. It is crucial to acknowledge the potential for inherited bias from LLMs in the development and deployment of SheetCopilot and to take proactive steps to mitigate any negative impacts on fairness and equity.