GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
Authors: Tejal Patwardhan
Rachel Dias
Elizabeth Proehl
Grace Kim
Michele Wang
Olivia Watkins
Simon Posada Fishman
Marwan Aljubeh
Phoebe Thacker
Laurance Fauconnet
Natalie S. Kim
Samuel Miserendino
Gildas Chabot
David Li
Patrick Chao
Michael Sharman
Alexandra Barr
Amelia Glaese
Jerry Tworek
Venue: ICLR 2026 Poster (2026)
Keywords: benchmark
real-world tasks
RL environments
model evaluation
reinforcement learning
AI impacts
dataset
evals
benchmarks
multi-modal
computer use
agents
long-horizon tasks
AI
artificial intelligence
ML
machine learning
deep learning
LLMs
language models
Primary area: datasets and benchmarks
TL;DR: We introduce GDPval, a benchmark assessing AI model capabilities on real-world, economically valuable, digital knowledge-work tasks.

Abstract: We introduce GDPval, a benchmark evaluating AI model capabilities on real-world economically valuable knowledge-work tasks. GDPval covers the majority of Department of Labor O*NET Work Activities for 44 occupations across the top 9 sectors contributing to U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with an average of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best frontier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding improves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide a public automated grading service to facilitate future research in understanding real-world model capabilities.

GDPVAL: EVALUATING AI MODEL PERFORMANCE ON REAL-WORLLY ECONOMICALLY VALUABLE TASKS
Tejal Patwardhan"
Rachel Dias
Elizabeth Proehl
Grace Kim
Michele Wang"
Olivia Watkins' Simón Posada Fishman'
Marwan Aljubeh"
Phoebe Thacker'
Laurance Fauconnet
Natalie S. Kim
Patrick Chao
Samuel Miserendino
Gildas Chabot
David Li
Michael Sharman
Alexandra Barr
Amelia Glaese
Jerry Tworek
OpenAl
ABSTRACT
We introduce GDPval, a benchmark evaluating AI model capabilities on real- world economically valuable digital knowledge-work tasks. GDPval covers the majority of US Department of Labor OENET Work Activities for 44 occupations across the top sectors contributing U.S. GDP (Gross Domestic Product). Tasks are constructed from the representative work of industry professionals with av- erage of 14 years of experience. We find that frontier model performance on GDPval is improving roughly linearly over time, and that the current best fron- tier models are approaching industry experts in deliverable quality. We analyze the potential for frontier models, when paired with human oversight, to perform GDPval tasks cheaper and faster than unaided experts. We also demonstrate that increased reasoning effort, increased task context, and increased scaffolding im- proves model performance on GDPval. Finally, we open-source a gold subset of 220 tasks and provide public automated grading service facilitate future research in understanding real-world model capabilities.
INTRODUCTION
There is growing debate about how increasingly capable Al models could affect the labor market whether by automating specific tasks, replacing entire occupations, or creating entirely new kinds of work (Brynjolisson et al.) 2025) Chen al. 2025). Current approaches measure the economic impact of Al focus on indicators such as adoption rates, usage patterns, and GDP growth attributed
to Al (Chatterji et al., 2025 Tamkin et al. 2024 Appel al., 2025) Acemoglu 2025 Bick et al.) 2024 However, historical evidence from technological shifts such electricity, airplanes, and computers-shows that the transition from invention to economy-wide permeation often takes years even decades, requiring regulatory, cultural, and procedural changes (David 1990, Brynjolfsson & Hitt 2000; Brynjolfsson et al.) 2017; Dwivedi et al., [2021] Solow| 1987). Therefore, while infor- mative when available, these methods lagging indicators of Al impacts. We consider an alternate method for understanding the potential economic impacts of AI: directly measuring AI model ca- pabilities. Al capability evaluations provide clearer, more directly attributable evidence about model abilities, allowing us to assess economic relevance ahead of widespread adoption.
Our paper introduces the first version of GDPval, benchmark evaluating AI model performance on real-world economically valuable tasks. GDPval covers the top 9 sectors contributing to U.S. GDP (Gross Domestic Product), with least 30 tasks per occupation the full (and tasks per occupation in the gold subset), across 44 occupations. Each task is constructed based on actual work product created by an expert professional. Given the complexity of automatically grading these tasks, our primary evaluation metric is head-to-head human expert comparison. We also provide an experimental automated grader service for the 220 open-sourced gold subset of tasks. Future GDPval iterations will incorporate greater breadth, realism, interactivity, and contextual nuance.
The initial version of GDPval offers several advantages over existing AI model evaluations:
"qqual contribution.

---

Marufacturing Engineer: Design Financial invectment Analyst: Registered Nurse; Assess of reel isand assemblly
competttor landscape delivery
images create consultaiio report
Film and Video Editor: Greala Highteeegy intro reel video and sudio
Customer Service: Email repporaee discatieteed customer requuent jesting return now
Concierge: Create week-long luxury Bahamas Ennerrry for family of four
Order Clarkc Audit oricing inconsiscanc es purchese orders
Real Estate Agent: Disign sales brochura for new DC property
Recreation worker: Optimize table layout for apring wendor forr
Figure 1: Example GDPval tasks from full set
Realism: Unlike Al benchmarks in the style of an academic test that focus on reasoning difficulty (e.g., Phan et al.] (2025); Hendryykk et al. (2020); |Ren al. (2023): Liu et al.
(2023), tasks are based on actual work product from industry experts, validated through multiple rounds of review, and tied to time and cost required for completion.
Representative breadth: Unlike Al evaluations focused on specific domains like soft- ware engineering (e.g.. Miserendino et al. (025)), the GDPval full set covers 1,320 tasks across 44 occupations, sourced to cover the majority of Work Activities tracked by O*NET for each occupation [U.S. Department of Labor, Employment and Training Administration (2024). This top-down approach allows for representativeness of tasks acToss occupations. We also build on production Al usage analyses (e.g.. Tamkin et al. (2024); Chatterji et al. (2025); Appel et al. (2025)) to cover areas where model adoption is still emerging. Computer use and multi-modality: Tasks require manipulating a variety of formats (c.g., CAD design files, photos, video, audio, social media posts, diagrams, slide decks, spread- sheets, and customer support conversations). Each task also requires parsing through up to 17 reference files in the gold subset, and 38 in the full set.
Subjectivity: In addition to correctness, expert graders often consider subjective factors such as structure, style, format, aesthetics, and relevance. Our dataset also therefore serves as a helpful testbed to assess automated grader performance.
No "upper limit": Unlike metrics that could saturate quickly, our primary metric win rate, which allows for continuous evaluation. Currently, we compare model outputs against a human expert baseline, but we could replace our baseline with increasingly strong models over time and keep evaluating.
Long-horizon difficulty: Tasks require an average of 7 hours work for an expert pro- fessional to complete. On the high end, tasks span up to multiple weeks of work.
2
TASK CREATION
We first identify the sectors that contribute most to U.S. GDP, then source tasks drawn from the highest-earning knowledge work occupations within those sectors.
2.1 PRIORITIZING OCCUPATIONS
GDPval covers tasks from 9 sectors and 44 occupations that collectively earn S3T annually. We detail below the methodology behind our initial version.
To choose the initial occupations, we:
1. Selected sectors that contribute over 5% to US GDP as determined by Q2 2024 Value Added by Industry Percentage of Gross Domestic Product (see Federal Reserve Bank
2

---

of St. Louis 2025 These 9 sectors shown in Table 1. Because current data budget did not allow for collection of tasks across all industries, we start with the highest- contribution industries to GDP in this the first iteration of GDPval, but plan to expand to more industries in future iterations.
2. Selected the 5 occupationsS within each sector that contribute most to total wages and compensation and are predominantly digital. Because we could not collect tasks for every occupation in each industry due to budget constraints, started by prioritizing the top 5 in each industry. We took a task-based approach to determining if an occupation should be classified as "predominantly digital." Specifically, identified all tasks for an occupation from O*NET, a database of occupational data, definitions and tasks from the U.S. Department of Labor. Similar Eloundou et al. (2023), we prompted GPT-4o to classify each task as digital or non-digital, and then classified the overall occupation as digital least 60% of its component tasks were digital. To calculate this percentage, we weighted tasks by the "relevance," "importance," and "frequency" scores for each task reported in NET Task Ratings.
We further validated the representativeness of our digital tasks measure by benchmarking against the AAcemollu & Autor (011)) task content framework. 'hee correlations we observe- digital tasks increasing with non-routine cognitive content and decreasing with routine and manual con- tent demonstaate alignment with established economic measures of work, per appendix A.8.6
For wage and occupation data, we used the Bureau Labor Statistics May 2024 national em- ployment and wage estimates to calculate total wages for 831 occupations (U.S. Bureau of Labor Statistics (2025) and further detailed appendix A.8
Real Estate and Rental and Leasing
Government
Manufacturing
Cansierges
Recreation Worbers
Merhhrical Eigiirers
Compliance Officars
Industraal Engineers
Roal Estate Brokers
First Supervisors of ice and Dctectives
Buyers & Purchasing Agents
Counter
Administrativv Servioes Managers
Receiving, inventory
Property, Estate, Community Association
CNKs, Family, and School Sodial Workers
FirstLiiee Supervisors Product and Operating
Managers
Professional, Scientific, and Technical Services
Health Care and Social Assistance
Finance and Insurance
Software Developers
Registered Nurses
Dustomee Service Representatives
Nurse Pract tloners
Accountants & Auditors
Medical Health Services Managers
Managers
Compuner & information on Systems Managers
First Line Supanisers of Office Administratioo
Par Financall Advisors
Management Specialits
Suapert Wo fiers
ties Commodites Services
Agents
Retail Trade
Wholesale Trade
Information
Managors
Producers Directors
General and Operations Managers
Order Clerks
Film a Video Editors
Detectives Investigarors
Marufacturing
First-Line Supervisors of Retail Sales Workers
Technical Scientfic Products
News Antlysts, Reportens, & Journalitts Techricsins.
Sales Representatives, Wholesale & Narutscturing Technical & Sclantfie Products
Finst-Line Supervicors of Nor-Retal Sales Workers
Figure 2: GDPval includes real-world work from 44 occupations.
2.2 EXPERT RECRUITMENT
We recruited expert industry professionals to create realistic tasks based on their professional work experience. Experts were required to have a minimum of 4 years of professional experience in their occupation and strong resume with demonstrated history of professional recognition, promo- tion, and management responsibilities. The average expert had 14 years of experience. We further
We assigned occupations to sectors by using the 2023 BLS National Employment Matrix from of La- bor Statistics (2025) to map occupations to sectors by identifying the sector with the highest employment for each accupation. For more detail, see appendix A.8
3

---

required experts to pass video interview, background check, training and quiz to partici- pate in the project. Experts were well compensated for their time and experience. Some of the prior employers of industry experts include: Accenture, Actna, Apple, AXA Advisors, Bank of America, Barclays, BBC News, Boeing, Budget Rent a Car, Capital One, Centers for Disease Control and Prevention, Citigroup, Condé Nast, CVS Pharmacy, U.S. Department of Defense, Dis- ney, Douglas Elliman, E*TRADE, Federal Trade Commission, General Electric, Goldman Sachs, Google, Guggenheim Partners, HBO, IBM, JPMorgan Chase, Johnson & Johnson, Kmart, Kirk- land & Ellis LLP, LinkedIn, Lockheed Martin, Macy's s. Massaccusetts General Hospital, Meta, Mi- crosoft, Morgan Stanley, National Park Service, NFL Network, Oracle, Paul, Weiss, Rifkind, Whar- ton & Garrison LLP, Prudential, PwC, Raytheon, Sally Beauty, Samsung. SAP, Scientific American, Sotheby's, s, Telegraph Media Group, Thermo Fisher Scientific, TIME, Twilio, U.S. Department of Justice, United States Air Force, United States Postal Service, Walgreens, Wells Fargo, White & Case LLP, and Whole Foods. More information can found in appendix A.6
2.3 TASK CREATION
Each GDPval task consists of two primary components: a request (often with reference files) and a deliverable (work product). Tasks are based on work performed by experts in their professional experience and then mapped to O*NET occupational tasks for their occupation to ensure broad and representative coverage across tasks (National Center for O*NET Development (025). Experts cre- ated multiple tasks, and multiple experts created tasks for each occupation. More details on task characteristics can be found in appendix A.4 To assess task quality, also asked occupational experts to rate each task on its difficulty, representativeensss, time to complete, and overall quality against real-world standards for their occupation. Each task's dollar value was estimated by mul- tiplying the estimated completion time (self-reported by the expert who wrote the task) by median hourly wages for the corresponding occupation from OEWS data (U.S. Bureau of Labor Statisties, 2025
2.4
TASK QUALITY CONTROL PIPELINE
First stage General Task Guidence
stage reviews Occupation Specific Guidance
Third General Task Guidence
Primary
Not vertatvve
Iteraive
Reeavie
Occaurtonl of speciic koctbac
Oscacaiionl
creand by spaci
vonian genenal and
Jethor
melewar becopio aree
curti prrtes
Nendhicc redeive
Sandbox
hevativ
The the pack an rxport
sandior por vinulee andeeis
Firal Sign Off
Too modiunt
Driginaite taak ridiion
Reviewer ghow
Taee through the restew procass
Sedtack
digne off
Figure 3: Tasks undergo multiple rounds of review ensure realism and quality. More detail on the review pipeline in appendix A.5
All 1,320 tasks in the full GDPval set went through an iterative review pipeline involving both automated model-based screening and multiple stages of human expert review. Each task received an average of five human reviews (with a minimum of three reviews). Across all stages of review, experts provided detailed comments, and tasks were iteratively revised before subsequent reviews to enhance quality and representativennss,, as detailed in appendix A.5
2.5 HUMAN EXPERT GRADING AND AUTOMATED GRADING
To grade the 220 ooun-sourced gold subset, we conducted blinded expert pairwise comparisons, where experts in the relevant occupation were presented with unlabeled work deliverables, including
4

---

Prompt and Referenoe Files
Overall Agreement with Human Expert Graders
100%
Agreement Human Graders (%)
Human Ookd Delierrbble
GPT- Sanppe Resgense
80%
66.7%
40%
20%
Pikk which is better
Iconeddering
0%
Automated Grader
Inter-Humen
(a) Pairwise Grading Setup
(b) Agreement with Humans
Figure 4: GDPval uses pairwise expert comparisons for grading. We also create an experimental automated grader. We find that automated grader agreement is within 5% of human inter-rater agreement on the GDPval gold subset.
those from models and from human experts, and asked rank them based on which better fulfilled the task request given its reference files.
On average, grading each comparison for the gold subset took over an hour. Additional occupational experts were sourced to grade human and model deliverables. Experts provided detailed justifica- tions for their choices and rankings, which enabled us compute our headline win rates for various models compared to the human expert completion.
For the gold subset, we trained an experimental grading model to perform pairwise comparisons in the style of industry professional experts. The automated grader is presented with the request and reference files, two deliverables, and rubric, and asked to pick which completion is better. Although limited, the automated grader is faster and cheaper than expert grading, and achieves 66% agreement with human expert graders, only 5% below human expert inter-rating agreement of 71 1.. Further detail is in appendix A.7
EXPERIMENTS AND RESULTS
3.1 HEADLINE RESULTS
We evaluated GPT-4o, o4-mini, o3, GPT-5, Claude Opus 4.1, Gemini 2.5 Pro, and Grok 4 using blind pairwise comparisons by professional industry experts ? We sampled Claude, Gemini and Grok via the UI to enable the maximum GDPval-relevant performance. For example, for Claude, wanted to evaluate its upgraded file creation and analysis feature (Anthropic 2025). ). For the OpenAI models, enabled the web search tool and the code interpreter tool, with background sampling- We also preinstalled several libraries not available in the base image, see appendix [A.7.5) For plots shown, sampled each model 3 times for each prompt, and then had 3 different human graders grade each sample (yielding 9 comparisons per prompt, per model, across 220 tasks). Claude Opus 4.1 was the best performing model the GDPval gold subset, excelling in particular aesthetics (e.g., document formatting, slide layout), while GPT-5 excelled in particular on instruction following (e.g.. producing the requested deliverable format, including all requested sections) per fig. 8 This distinction is also shown in appendix A.2.4. where GPT-5 performs better pure text, while Claude performs better on file types like pdf,, .xslx, and ppt, demonstrating better visual and aesthetic abilities. We caveat also that the occupations and task types covered by text tend to be different than
We aimed keep comparisons as blind as possible, but model samples may have been identifiable due to stylistic differences. OpenAl outputs often used em dashes, Claude outputs frequently adopted first- person phrasing, and Grok occasionally referred to itself as Grok. Although tilenames were scrubbed of model identifiers, to preserve sample identity, we did not alter style or content, so experts may still have been able to infer model origins
5

---

GDPval: Pairwise Expert Preferences
60%
Wins Ties
Wins Only
Parity
Win rate vs. industry professional (%)
Industry Expert
50%
38.8%
40%
34.1%
27.9%
30%
24.3%
25.5%
20%
12.4%
10%
GPT-40
Grok 4
Gemini 2.5 Pro 04-mini high Model
o3 high
GPT-5 high Claude Opus 4.1
Figure 5: On human pairwise comparisons, models are beginning to approach parity with industry experts on the GDPval gold subset.
GDPval: OpenAl Frontier Model Performance Over Time
Wins + Ties
Wins Only
Win rate vs. industry professional (%)
60%
Parity with
Industry Expert
50%
GPT-5 high
40%
03 high
30%
20%
GPT-40
10%
0%
06/2024
09/2024
12/2024
03/2025
06/2025
09/2025
Release Date
Figure 6: Performance of OpenAl frontier models increased roughly linearly over time on the GDP- val gold subset.
those that involve multi-modal, See table 3 for more information about sampling files. In fig. 5 on the GDPval gold subset, 47.6% of deliverables by Claude Opus 4.1 were graded as better than (wins) or as good (ties) the human deliverable. Model deliverables outperformed or matched expert humans' deliverables in just under half the tasks.
3.2 SPEED AND COST COMPARISON
We analyzed several scenarios to understand the potential speed and cost savings ratio of frontier models on the GDPval gold subset tasks appendix A.2. In the scenarios analyzed, incorporat- ing frontier Al models into expert workflows showed the potential to save time and money relative to
We were not ahle to obtain cost estimates for Claude, Gemini, and Grok. Claude's s new files UI handles files more reliably than the API but does expose time token metadata, preventing comparable cost/time

---

unaided experts. Fig7 summarizes expected savings under "try using the model and if still unsat- isfactory, fix it yourself setup. Here, an expert human samples from a model, reviews outputs, and if unsatisfactory, resamples and repeats. If no satisfactory output is obtained, the human completes the task themselves. Under this setup, as well as other setups (e.g., directly using model outputs, trying the model just once before doing work directly). model assistance can potentially save the expert time and money. We report results using GPT-family models as stand-in for frontier models generally. Our intent in these experiments was study human versus model versus human with model, not to compare cost and time across model families.
GDPval: Speed and Cost Improvements from Al Assistance to Human Experts
GPT-5 high
03 high
1.5x
mi-iini
1.25x
Cost Improvement
unassisted
1X
0.75x
GPT-40
0.5x
0.5x
0.75x
1.25x
1.5x
Speed Improvement
Figure 7: In the scenarios we analyze, models show the potential to save time and money by coupling AT assistance with expert human oversight. Here, we show speed and savings from "try times, and if still unsatisfactory, fix it yourself" approach as detailed in appendix A.2.
3.3 MODEL STRENGTHSS AND WEAKNESSES
We built clustering pipeline to analyze why experts preferred rejected GPT-5 high, Claude Opus 4.1, Gemini 2.5 Pro, and Grok deliverables as shown in fig. g Samples were clustered using the natural language rationales expert human graders wrote for their choices. Grok and Gemini most often lost due to instruction-folloving failures and frequently promised but failed to provide deliverables, ignored reference data, or used the wrong format. Claude and GPT-5 high showed better instruction following, though all models sometimes hallucinated data or miscalculated. A more detailed table of failure modes is available in A.2.6
analysis. Gemini and Grok had similar APT file-handling limitations so sampled completions via their first party platforms in order to solicit the best possible performance.
7

---

Modes: Prevaloncc Label
Foctual
Instrction tollwinng
Multiredis
40%
of Samples
50%
00%
0%
90
gamiin
Srggh
gikk
Figure 8: Across models, experts most often preferred the human deliverable because models failed fully follow instructions on GDPval tasks.
3.4 INCREASING REASONING EFFORT AND SCAFFOLDING
To understand the impact reasoning effort on model performance, we ran GDPval on the o3 and GPT-5 models at low, medium, and high reasoning effort. We found that additional reasoning effort improved performance by up 4.3 percentage points for OpenAl o3 and 6.1 percentage points for GPT-5.
We were also interested in measuring how easily we could improve model capabilities with prompts. For example, many of the observed GPT-5 failure modes due obvious formatting errors. We created a prompt which encouraged GPT-5 to rigorously check deliverables for correctness, check layouts by rendering files images, avoid nonstandard unicode characters, and avoid excess verbosity. The prompt applies generally to multimodal economic tasks and is not overfit to any given question (see appendix A.3 for details). We also improved agent scaffolding by enabling GET requests in the container and performing best-of-N sampling with N=4 and GPT-5 judge
Prompting fully eliminated black -quare artifacts from GPT-5 responses, which previously affected over half of generated PDFs, and reduced egregious formatting errors in PowerPoint files from 86% to 64%. This can partially attributed to a sharp increase in agents using their multi-modal capabilities to inspect deliverables (15% + 97%). Prompting also improved human preference win rates by 5 percentage points in Figure 9b These easy performance gains suggest there are paths to agent improvement on GDPval tasks by training scaffolding them to be more thorough and take full advantage of their multimodal capabilities.
4 OPEN-SOURCING
We open-source the prompts and reference files in our 220-task gold subset Hugging Face (dataset: GDPval) While human expert comparison is still our recommended method of grad- ing, we make an experimental automated grader publicly available as well, which is linked from the Hugging Fave, and also available at wwwissopeai.ccmmm Please note that the tasks in the open sourced set have been scrubbed of information that could be used identify the expert who wrote the task. We also note that, as result of limitations with our automated grader, we don't t provide automated grading results for all tasks in the gold subset. Further disclaimers about the open source gold subset are in appendix
LIMITATIONS
Focus self-contained knowledge work: Tasks in the initial version of GDPval are oriented around knowledge work that can be performed on a computer, particularly around digital deliver- ables. Manual labor and physical tasks are not included in the current version. Moreover, tasks that
'Empriical evidence shows that models often favor their responses. See A.7 for further discussion of limitations with the autograder.
8

---

GDPval: Model Win rates Different Reasoning Efforts
Prompt Tuning Expert Preferences
100%
Cnly
Wins Tils
Wins Only
Win rate industry professional (%)
Win rate vs. industy professionll (%)
80
00%
Partyy with
Party kit
Industy
38 88.8%
31.4%
32.1% 84.1%
05.0%
37.2%
38.6%
40%
Low
03
Medium
03
High 03
Medium High GPT-5
Low:
GPT-S GPT-5
0%
GPT-5 high
GPT-5 high (prompt tuning)
(a) Reasoning effort experiment
Prompt tuning experiment
Figure 9: Model performance improves predictably with increasing reasoning effort. Prompt-tuning and scaffolding improvements also increase GPT-5 performance.
involve extensive tacit knowledge, access to personally identifiable information, use of proprietary software tools, or communication between individuals are out of scope for the current evaluation. We aim build on this future versions of the evaluation.
Tasks are precisely-specifeed and one-shot, not interactive: For GDPval, we provide the full context of the task in the prompt, but in real life it often takes effort to figure out the full context of a task and understand what work on. We are working on improvements GDPval that involve more interactivity and contextual realism. In the meantime, the experiment in the "Unnde--oottetuualizd GDPval" section (appendix A.2.7 demonstrates how model performance degrades with less context.
Dataset size: The GDPval full currently consists of only occupations and 30 total tasks occupation. It is therefore limited, initial cut of knowledge work tasks, not comprehensive evaluation of all possible occupational tasks. We are expanding the dataset size.
Grader performance: Our current automated grader has number of limitations compared to human expert graders. More details about the automated grader are available in the appendix A.7.2
Constructing and running our evaluation is expensive, particularly with industry expert graders. For this reason, make automated grader proxy available, but do not consider it full substitute for industry expert graders.
Cost:
CONCLUSION
In GDPval, contribute the following:
1. Dataset: We create new evaluation dataset (GDPval) measuring real-world, economically valuable knowledge-work tasks.
2. Capability benchmarking:
We analyze quality, speed and cost of deliverables across hu- man industry experts and frontier Al models.
3. Experiments: We test how results shift with differing reasoning effort, prompting, scaf- folding, and context.
4. Open-sourcing: We open-source 220 tasks as part of our gold subset which includes prompts and reference files.

---

5. Automated grader: We release experimental automated grader to improve accessibility of grading.
We hope this work contributes the science of tracking model progress, so that we have better data to assess the social impacts of Al models.
7 ETHICS STATEMENT
Human participants. This work relies tasks and judgments from industry experts (experts, reviewers, and graders). All participants were told the purpose of the project, the types of data being collected, and how the data would be used for research. We don't believe that study meets the definition of human buban-subjects research because we were studying models, not human subjects. Experts served professional contractors producing and evaluating work outputs. This is analogous to standard data-labeling pipelines. Participation was voluntary, with the option to withdraw at any time. Experts were compensated for their time and expertise; compensation was not contingent model outcomes. More detail is available in A.6
Privacy, confidentiality, and data governance. We instructed contributors not submit personally identifiable information (PII), material non-public information (MNPI), and confidential or propri- etary content in any submissions. Reviewers flagged overlooked information, which was subse- quently replaced with realistic, fictional data by contributors. We do not release raw deliverables that could reveal sensitive information. O*NET, OEWS and BLS data were used accordance with their terms.
Potential harms and misuse. GDPval measures model capabilities on economically valuable knowledge work tasks. Results could be misused to make premature claims about job replacement or to target specific roles for automation without context. To mitigate this, we emphasize limitations (capability versus adoption; subjectivity; file and tool constraints) and report results with confidence intervals. These results are research findings and are not intended to inform or justify personnel decisions (e.g., hiring, firing, promotion, compensation). The benchmark excludes dangerous or clearly harmful task types.
ACKNOWLEDGEMENTS
We thank Addea Gupta, AJ Ostrow, Aleksander Madry, Ally Bennett, Alexander Wei, Becky Waite, Ben Gaffney, Brad Lightcap, Casey Chu, Cassandra Duchan Solis, Charlotte Cole, Dane Stuckey, Eric Wallace, Erik Ritter, Evan Mays, Fidji Simo, Gideon Myles, Hannah Wong, Isa Fulford, Jakub Pachocki, James Lennon, Jared Pochtar, Jason Kwon, Jordan Frand, Justin Wang, Julia Steele, Karthik Rangarajan, Kevin Liu, Larry Summers, Leyton Ho, Leo Liu, Leon Maksin, Livvy Pierce, Lindsay McCallum, Manoli Liodakis, Mark Chen, Max Schwarzer. Miles Palley, Miles Wang. Nakul Khanna, Nat McAleese, Nicholas Carlini, Nick Otis, Nick Ryder, Noel Bundick, Paul Radulovic, Phillip Guo, Prashanth R, Rachel Brown, Raoul de Liedekerke, Robert Rotsted, Ronnie Chatterji, Ryan Kaufman, Sam Altman, Sam Bowman, Sherwin Wu, Tom Cunningham, Tom Stasi, Tony Song, Trevor Creech, Wenda Zhou, Wenlei Xie, Wyatt Thompson, and Yara Khakbaz for discus- sion, assistance, and review. We'd especially like to thank the industry experts who contributed their time and expertise to GDPval, without whom this work would not have been possible.
REFERENCES
Daron Acemoglu. The simple macroeconomics of ai. Economic Policy,
2025.
Daron Acemoglu and David Autor. Skills, tasks and technologies: Implications for employment and earnings. In Handbook of labor economics, volume 4, pp. 1043-1171. Elsevier, 2011.
Anthropic. Claude can now create and edit files, 2025. URL cn/news/create--ileee Accessed: 2025-09-25.
Ruth Appel, Peter McCrory, Alex Tamkin, Michael Stern, Miles McCain, and Tyler Neylon. An- thropic economic index report: Uneven geographic and enterprise ai adoption. Anthropic Re-
10

---

search, 2025, URL htt.:///www///aa..ooooo/oo///aaa
Alexander Bick, Adam Blandin, and David Deming. The rapid adoption of generative ai. Technical report, National Bureau of Economic Research, 2024.
Erik Brynjolfsson and Lorin M. Hitt. Beyond computation: Information technology, organizational transformation and business performance. Journal of Economic Perspectives, 14(4):23-48, 2000. doi: 10. 10.1257/jep. 14..e.....4....... 14.4.23. URL https: : /www www.waawwwbago....... www.:/wwwww....trttcaasseeeeeee. 1257
1jep.14.4.22
Erik Brynjolfsson, Daniel Rock, and Chad Syverson. Artificial intelligence and the modern produc- tivity paradox: A clash of expectations and statistics. Working Paper 24001, National Bureau of Economic Research, November 2017.
nber org/ papers/ //220011
Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative ai at work. The Quarterly Journal of Economics, 140(2):889-942, 2025.
Aaron Chatterji, Thomas Cunningham, David Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. How people use chatgpt. Working Paper 34255, National Bureau of Economic Research, September 2025. URL httt:://wwttt nber. org/ wwtpg//pppss//w3222
Wilbur Xinyuan Chen, Suraj Srinivasan, and Saleh Zakerinia. Displacement or complementarity?? The labor market impact of generative ai. Harvard Business School, 2025.
Paul A. David. The dynamo and the computer: An historical perspective on the modern productivity paradox. American Economic Review, 80(2):355-311, 1990. URL /tt.:ttsss//ecooppprrrr. epec.
Yogesh K Dwivedi, Laurie Hughes, Elvira Ismagilova, Gert Aarts, Crispin Coombs, Tom Crick, Yanqing Duan, Rohita Dwivedi, John Edwards, Aled Eirug, et al. Artificial intelligence (ai): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International journal of information management, 57:101994, 2021.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look the labor market impact potential of large language models, 2023. URL
g h/abs/2303 10130
Federal Reserve Bank of St. Louis. Value added by industry a percentage of gross domestic product. FRED Release Tables, 2025. htLhhtps:: http//eee..........eeee.ee www.sssttssse..oooooooooo org.oiseeeddorrreeeaaa.e Accessed: 2025-09-03.
e/tables221d==331eedd2222
Rishi et Handa. Which economic tasks are performed with AI? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761, 2025. URL htt.tttt////..i....//a org/abs/
0503.047611
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020. doi: 10.48550/arXiv 2t.S5///aXxv.200..000 2009003300. URL. /tttts://aaxii..
g //abs/2009.1 03300
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating Ilms as agents. arXiv preprint vrXiv:2308.03688 2023. doi: 10.48550/arXiv.2300 wwt.arxXXXii220006066...
Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier LLMs carn $1 million from real-world freelance software engineering? arXiv preprint
1rXiv:2502./2//5, 2025.
National Center for O*NET Development. Task statements net 28.3 data dictionary. https: 7/www. owee wttp://////.ttoee/////aaa.. h/tp:////////eeeee//e//////a/ stat erantsmets....m .tteentsstmmlll.. 2025. Accessed: 2025-09-03.

---

U.S. Bureau of Labor Statistics. Occupational outlook occupational data, 2025. URL https:
7/ /www.
htm Accessed: 2025-09-03.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their generations, 2024.
Long Phan al. Humanity's 's last exam. arXiv preprint arXiv:2501. 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.1202 2023. doi: 10.48550/aXx...311..2000 URL
https://arxiss ottp:///////////////
abs/ 2anis/22311.0 12022
Robert M. Solow. We'd better watch out. New York Times Book Review, pp. 36, July 1987. URL https://wwzs www.dupecoooomiss....... www.coo//cc/oo/oe http:////...ceeooooooo-rtttttt-p
idf
Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, Michael Stern, Brian Clarke, Landon Goldberg. Theodore R. Sumers, Jared Mueller, William McEachen, Wes Mitchell, Shan Carter, Jack Clark, Jared Kaplan, and Deep Ganguli. Clio: Privacy-preserving insights into real-world doi: 10.48550//rXiv.2412..33
ai use. arXiv preprint arXiv:2412.1366 2024.
URL
htt.ps://erx/ otgg abs/ 13678
William Tomlinson, Pratyusha Kalluri, Yu Jung, and et al. Working with AI: Measuring the ap- plicability of generative AI to occupations. arXiv preprint arXiv:2507.07935, 2025. URL
U.S. Bureau of Labor Statistics. Occupational employment and wage statistics: May 2024 national tables. https: /www. bls. wtt.s/g//////taabeeeee htm 2025. Data reference May 2024; accessed: 2025-09-03.
U.S. Department of Labor, Employment and Training Administration. Work activities onet 28.3 data dictionary. https: www. onetcenter /ttp://cct.ooarr///.//////// 3/xc:////////ecceeeeeeeeee
T_ratings. html 2024. Accessed: 2025-04-20.
A APPENDIX
A.1 DISCLOSURES
A.1.1 AI DISCLOSURE
We used Al models to help with literature review and with tweaking language in the paper. We also used AI coding assistants as part of our regular engineering workflows (e.g., to help find and fix bugs).
A.1.2 SENSITIVE CONTENT AND POLITICAL CONTENT DISCLOSURE
Some tasks in GDPval include NSFW content, including themes such sex, alcohol, vulgar lan- guage, and political content. We chose to keep these tasks as they reflect real themes addressed in various occupations (e.g., film, literature, law, politics). We do not endorse the particular actions views in any of the content.
A.1.3 THIRD-PARTY REFERENCES DISCLOSURE
GDPval contains limited references to third-party brands and trademarks solely for research and evaluation purposes. No affiliation endorsement is intended implied. All trademarks are the property of their respective owners. Some images and videos this dataset feature Al-generated individuals and real people who have provided permission. Names and identifying references to private individuals in GDPval are fictitious. Any resemblance to actual persons or entities is purely coincidental.
12

---

A.2 ADDITIONAL DETAIL ON EXPERIMENTAL RESULTS
A.2.1 SPEED AND COST ANALYSIS, CONTINUED
We use the following definitions:
1.
Human expert professional completion time Hf is the time taken by a human expert profes- sional to complete a task, based on validated self-reported time to complete To calculate human expert professional completion cost Hc, we multiplied the reported task comple- tion hours per occupation by the median hourly wage for each occupation from the U.S..] Bureau of Labor Statistics 22025 On average, our 220 gold subset Hf 404 minutes and Hc $361.
2. Human expert professional review time Ry is an estimate of the time taken to assess a model deliverable by human expert grader. We observe this from our task monitoring software, averaging the time taken to grade for the first time each human expert was asked to grade that question. On average, Ry 109 minutes, and associated human expert professional review cost Rie is on average $86, where Rc is again calculated based on time taken multiplied by median wage data.
3. Model completion time Mr is the time taken for the model to complete a deliverable and Mc is the associated completion cost, based on empirical API speed and cost for the model to complete the deliverable when given a prompt
4. Model win rate w is how often the model deliverable is rated better than the human deliv- erable by the human expert grader.
We then calculate the following ratios:
1. Naive ratio: To measure the ratio of human deliverable versus model deliverable, without accounting for any quality differences or implementation times, we simply divide the aver- age task completion time for human by the average sampling time for a model: Hr /Mr, and analogously for cost: Hc/Mc.
2. Try time, then fix it ratio: To calculate the time with this method, we take the sampling time for the model, add review time Ry for expert assess quality, and then with probability (1 wx) add in the human completion time for any fixes needed for that model for task obtain Ti, and analogously C1
T MT, + + Ry, + W,,
E E Mc. + Re, + +(1 Hc.
(2)
The average time spent is Ti = E[T..,], marginalizing over all tasks i, similarly with C This proxies the setup where human tries using GPT-5 for a task, assesses its quality, and then does the task themselves if the deliverable quality is below their quality bar. Our plug- in estimate of the time savings ratio is: Hr/(Mr + Rr + (1 - w) = Hr/TT, where we use the empirical mean Ti. The analogous cost ratio is Hc/
+ Rc + (1- wcH.).
3. Try times, then fix it ratio: To calculate the time with this method, take the sampling time for the model, add review time Ry for an expert to assess quality, and then add in the human completion time for any fixes needed for that model (based on 1 w;;) 8 We repeat this across n resamples and re-assess steps before the human steps in to fix it:
'During submission, experts reloreep the worldworld time required to complete each task. Multiple oc- cupational reviewers independently validated these times, correcting errors. Because times were relf-reported, it is possible that experts under-estimated or over-estimated time taken
Becaase our experts were recruited specifically for being highly experienced their field, these wage estimates likely underestimate their true market cost.
FFor each task, we collected three API completions per model and averaged the observed response times recorded in the API metadata. We also recorded the average invoiced cost per task.
*We are over-penalizing the model here, because the win rate after each completion likely goes up (because the professional will adjust the prompt to the model to fix the errors) and the review time also goes down as the professional gets more comfortable with the task.
13

---

E[T
((1 w;)) k- (M + RT.,)) + (1 w;) "TT
(3)
k=1
1 (1 (1- w.)" (1 16--1 ww) HT.
(MT,; + RT,;)
(4)
E|Cn.l Σ ((L wi)) (a))_1((M,, + Ro,i)) + (1-w)) " "H,,
(5)
1- (1 wi)" + (1-w;)" Hc.
=(Maa Ro,a)
Wi
This proxies the setup where human tries rounds of using GPT-5 for task, then assesses its quality each time, and then does the task themselves if the model quality is below their quality bar after all attempts. As before, the average time spent is Tn = E[Tn, marginal- izing over all tasks i, similarly with Cn. Therefore, savings are H + RT) / /w) times faster and cost savings are Hc/ ((Me(MM + Rc)/w) times cheaper than human experts.
co, with
0, the time
Table 1: Speed and cost improvements under different review strategies.
Speed improvement
Cost improvement
Model
Win rate
Naive
Try 1x
Try nx
Naive
Try 1x
Try nx
gpt-4o
12.5%
327x
0.87x
0.46x
5172x
0.90x
0.53x
04-mini
29.1%
186x
1.02x
1.06x
1265x
1.06x
1.22x
o3
35.2%
161x
1.08x
1.28x
480x
1.13x
1.47x
gpt-5
39.0%
90x
1.12x
1.39x
474x
1.18x
1.63x
When incorporating time to review and redo work, the payoff from using model shrinks. We do not include consideration of the time taken to review human professional deliverable, although this would commonly occur for tasks in GDPval (either self-review of the professional's 's own work or review by a supervisor the human deliverable is also undesirable. One further limitation of this analysis is that it does not capture the cost of catastrophic mistakes, which can be disproportionately expensive in some domains.
team member's work). We also do not include the possibility that
14

---

A.2.2 WIN RATES BY SECTOR
We provide sector-level breakdown of win rates in fig. 10 Results differ industries. Some sectors have low win rates for all models, while in others (e.g., Government, Retail Trade, and Wholesale Trade), the strongest models approach parity on GDPval tasks.
GDPval: Painwise
Expert Preferences
901-40
perris
grik
eb-ligh
diuck
Manutacturing
Technical
Trade
Figure 10: Win rate by sector
15

---

A.2.3 WIN RATES BY OCCUPATION
We provide a detailed breakdown of win rates by occupation in fig. I1 Results vary: some oceu- pations show consistently low win rates across all models, while others display near parity among multiple models.
GDPval: Painwise Expert Preferences Occupation
Condaeges
Courter and flette Clana
Astidaal Maraaees
Thael Coles Dickans
Theal Canale Salee Agents
Fanslan of Frodecto Opensing
pa 1g Reowving, and lwantty
Duors and Puschaiig
Industiil Engrness
Mechanical
Co Comppoo and interstton Spttess
Fis Line Sumeiioos
Maragers
Wonhers
Complerce Offores
Decreaoon Workers
Admimiatat we Suppon Woneers
Adein strative Austatante
Medcal and Nealh Servicee Maragers
Nurse Prachfoners
Regiraned Numses
Geouraas Covined Boo,
Costame Service Paproaraatioa
Frence Marager
Prance undrwwrree Feadyst
Piescnal Fherdia Adrass
Firm Lao Superfoers Piss
Seep Ruproartaiion Whskess and Pookes
Se Pegresentatios Wideosik Tecinecaion Prodess
Woreers
Vasagers
Aratoss Jeumallb
Audo and Vadco Technrdara
Pllmane Viuc Biicce
Phoducer and Deedoo
Figure 11: Win rate by occupation
16

---

A.2.4 WIN RATES BY DELIVERABLE
We report win rates by deliverable type in fig. 12 Performance varies across formats, with Claude achieving the best results for all deliverables except pure text. GPT-5 high leads for pure text outputs, though overall win rates remain low.
GDPval: Model Win rate by Deliverable File Extension
GPT-40
04-mini high high
GPT-5 high
Claude Opus 4.1
80%
Win Reee (96)
60%
Parity Industry Expert
40:500%
35%
40%
219/
25%
30%
28127%
20%
18%
20%
145%
115
0%
pure text (9.9
pdl (31.3%)
xlex (23.1%) 1%)
pptx (5.8%)
other (30.1%)
File Extension
Figure 12: Win by deliverable file type
A.2.5 WIN RATES BY TIME COMPLETE
We report win rates by task duration in fig. 13 Win rates are highest for shorter tasks (0-2 hours) and decline steadily as completion time increases. This indicates that models perform best faster, less time-intensive tasks.
GDPval: Model Win rate by Time to Complete
Wen nete ve, incuury prolessional PS0
70%
GPT-4o
od-mini high
oB high GPT-5 high
Claude Opus 4.1
58%
Party
45%
48%
35%
33%
35%
87%
25%
88%
20%
25%
30%
19%
20%
8%
10%
Complete
Figure 13: Win rate by time to complete task
A.2.6 ADDITIONAL DETAIL ON MODEL FAILURES ANALYSIS
To generate this table, we analyzed the written justifications provided by expert graders when select- ing preferred completion in the head-to-head comparison. Through manual review, we observed that these explanations largely fell into three high-level categories: instruction following, factual accuracy, and multi-modal formatting. We then used a model to systematically cluster all grader justifications into these categories and further refine them into more specific failure subtypes (e.g., missing deliverables, wrong format, broken formatting). The resulting distribution provides struc- tured breakdown of where models most commonly fail, highlighting that while factual errors relatively rare across all models, a substantial share of performance differences arise from instruc- tion adherence and formatting quality, particularly in multi-modal outputs.
17

---

Claude Opus 4.1
Gemini 2.5 Pro
GPT-5-high
Grok
Factual errors
Produces factual emrors
2.68%
3.43%
2.22%
3.18%
Instruction following errors
Omits requested deliverables
1.77%
7.78%
1.36%
6.01%
Omits required content
3.79%
8.23%
5.40%
8.23%
Omits required visuals
2.42%
3.23%
2.47%
3.43%
Delivers wrong format
0.56%
4.19%
0.81%
4.90%
Multi-modal formatting errors
Provides unusable files Broken formatting
0.96% 2.58%
1.36%
0.81% 5.05%
0.86%
4.65%
8. 8.18%
Detailed model failure mode labels from clustering of human expert preferences percent of all samples
Table 2:
We took the subset of GPT-5 model failures (tasks where the GPT-5 deliverable lost the human expert), and then we asked other expert occupational graders to rate these subset samples as:
The model completion would be catastrophic used real life because is harmful or dangerously wrong (e.g., insulting a customer, giving the wrong diagnosis, recommending fraud, or suggesting actions that will cause physical harm).
Catastrophic:
2. Bad: The completion was bad and not fit for use, but not offensive or dangerous (e.g., rambling nonsense, completely irrelevant, or incoherent answers).
3. Acceptable but subpar: The completion was acceptable (and could be used) but the hu- man produced a stronger response (e.g., model response lacked helpful detail compared to the human).
4. N/A: Disagree with original expert grader; the model completion was better than the human completion.
GDPval: GPT-5 high Failure Analysis
60%
Percentage Tasks
47.7%
40%
26.7%
22.9%
20%
2.%%
Acceptable but subpar
Catastrophic
Figure 14: Experts rated GPT-5 model failures by categorized by severity of failure.
The most common categorization GPT-5 model failure was "acceptable but subpar." Another roughly 29% of ratings for bad catastrophic (with roughly 3% of failures marked catas- trophic). The 23% of ratings for "model better" roughly corresponds to the level of inter-rater agreement observed in fig. 4b
18

---

A.2.7 UNDER-COR CONTEXTUALIZED GDPVAL
To assess how models handle task ambiguity, created modified version of GDPval with de- liberately lower-context prompts. These shorter prompts omitted additional context such as where locate specific data within reference files, how to approach the problem, or detailed formatting expectations for the final deliverable; the models had to "figure it out." On average, these revised prompts were 42% the length (by token count) of the original prompts.
This setting helped measure aspect of professional knowledge work previously unaddressed in our evaluation; navigating ambiguity by figuring out what work on and where to get the necessary inputs. We collected and graded GPT-5 completions with expert human graders and found the model's performance was worse on under-specified prompts. In particular, the models struggled to figure out context.
As note: this experiment was run on an earlier version of the GDPval gold subset, and therefore the observed win rates do not match those in the main text of the paper.
GDPval: Model Win
Concise GDPval Prompts
udts
20,
42%
Ni Rate
10%
GFTher
Model
Figure 15: On the underspecified version of GDPval, GPT-5 performed worse as it struggled to figure out requisite context.
19

---

A.3 ADDITIONAL DETAIL ON PROMPT-TUNING
Here is the prompt we give the agent to elicit capabilities (lightly edited to remove some specific details of our scaffolding setup).
Prompt
Special characters Never the character (U+2011). since it will render poorly some people's computers. Instead, always use (U+002D) instead. Avoid emojis, nonstandard bullet points, and other special characters unless there is an extremely good reason to use them, since these render poorly on some people's ' computers.
Graphics embedded within PDFs/slides Make sure that any diagrams plots are large enough to be legible (though not so large that they are ugly or cut off). In most cases they should be at least half the page width. Plots and charts to visualize data are good. Simple graphies (like a flowchart with arrows) are good. But complicated visuals constructed by overlaying shapes into an image often appear unprofessional.
PDFs Always LibreOffice to create the PDF (it must be LibreOffice! If LibreOffice not in- stalled, you can install yourself). Other libraries sometimes show weird artifacts on some computers. Fonts Always use fonts which are available across platforms. We recommend Noto Sans Noto Serif unless there is an extremely good reason to use something else. If you must use another font, embed the font in the pptx/word/etc doc.
Deliverable text- Do not link to submitted files in the deliverable text (links not supported the interface where these will viewed). Ideal deliverable text concise and to point, without any unnecessary fluff. 4 sentences max. Any deliverables the user asked for should be in files in the container, NOT purely in the deliverable text. If portion of the task was unsolvable (for instance, because internet not available), mention this in the deliverable text. Your submission should be complete and self- coff-contained. Even if you are unable fully complete the task due to limitations in the environment, produce as close to a complete solution as possible.
Verbosity Always be clear and comprehensive, but avoid extra verbosity when possible. Filetypes If the prompt does not request specific filetype, use "standard" filetypes like PDF, PPTX, DOCX, XLSX, MP4, ZIP, etc.
Video files (mp4, mov) Extract a string images from the video files and check images see whether the visual elements are corrupted.
Mandatory formatting checks Before you submit your deliverable, you MUST perform the following mandatory formatting checks. Take your time, do these thoroughly, they extremely important! PNGs using LihreOffice. This includes pptx, docx, pdf, xlsx, etc. Convert it so that each page or slide a separate PNG. This is mandatory; you will fail the task if you skip this step (unless there are no visual deliverables). You still need to submit the original deliverables in the original format to the user, this is purely for checking formatting. STEP 2: Display the PNGs. You trying distorted, blank, hard to read (dark text on dark background or light text on light background), or otherwise poorly formatted. Look at each image thoroughly, zoom in if you need to see more closely. Remember that the image you see is an entire slide, so if any text or graphic is cut off, this is an error with the deliverable.
STEP Convert all visual deliverables
if the text graphics
cut off. overlapping,
STEP 3: Programmatic formatting checks. For highly visual submissions (e.g. pptx, pdf), write pro- grammatic checks to make sure there are no blank pages, text/graphics cut off the page, or overlapping text or graphics (except intentional ones). Also check that if there is a page or slide limit, it is respected. STEP 4: Summarize the prompt's deliverable instructions, and match that to the portion of the deliv- erable that addresses it.
STEP Right before submitting, check that the deliverables you have produced exactly what you want to submit: deliverables should contain exactly the files you want to submit, with no extra files. Check that these deliverables are not corrupted in any way by opening cach to make sure it is well-formatted.
any of these checks reveal formatting issue, fix them and go through steps 1-5 again. Take your time, thorough, remember you can zoom in on details.
This is IMPORTANT and MANDATORY, go through each step onc-by-one meticulously! Every formatting сrror is a MAJOR ISSUE THAT YOU NEED TO FIX! There is no time limit, be thorough, go slide by slide page by page.
Finally on the last line of your output text, add CONFIDENCEIXX,, where XX is an integer between and 100, inclusive, indicating your confidence that the submission is correct, follows instructions, and is well-formatted.
We performed best-of-N sampling by prompting a GPT-5 grader with the prompt, reference files, and deliverable files for four different submissions, then asking it pick the best.
20

---

Table 3: Capabilities, file handling, and UI vs. API differences across models
Capability
Claude Opus
Gemini
Grok-4
o4-mini
GPT:S
4.1
Pro
Input modalities
Text, image
Text image. audio, vidco
Text, image
Text, image
Text, image
Text, image
Output modalities
(audio
TTS/ ive)
Context window
200k
IM
256k
200k
200k
400k
Max output tokens
32k
64k
Not published
100k
100k
128k
Supported imputs
pdr. tut json: images: ipg- gif,
-dd,, docs, xllx, cov, ixt;
(obs.) .dl,
pdr, docs, xlsx,
Same as o4-mini
pdr, .ooc, xllxx,
docx; images:
-ug. irg
csv, txt;
con, txt;
weep
images:
images:
images: gin.. JPg
jig: audio: mmp3, wav, video: mp4
jPg .ipp
Unsupported inputs
(obs.)
-gloc (export required)
-gloc (export required)
mm33, direct autio files)
mp4 (no direct audiotvidzo)
dependent)
rejected; .xip unreliable (docs: official list)
Supported outputs
IxI; via tools: JoV, pag. påf
AsI; via TIS/Live:
.xxt
-oxt; via tools: JCSN, -pn- -pdf
.ixt via tools: .eSN, -ung -pff
.xxt
way
Max file size (typical)
NIP (obs.)
Apps: 100MB
Not published
512M//llc (webltools)
512MW/ile (web/tools)
S12MB/ile (web/tools)
30MB/ilee
(non-videa),
20 fleeschat
2GB (video);
Files API:
2GB/ille
Notes. "(obs.)" behavior observed in GDPval tests. Grok-4, vendor does confirm modaalitiessteet- and confext=256k. Claude processes PDFs >> 100 pages as text-only.
A.4 ADDITIONAL TASK CHARACTERISTICS
Table 4: Summary statistics for GDPval gold subset tasks
Mean
Std
Min
25%
50%
75%
Max
Overall quality (1-5)
4.47
0.32
3.18
4.30
4.50
4.70
5.00
Difficulty (1-5)
3.32
0.95
1.00
3,00
3.00
4.00
5.00
Representativeness (1-5)
4.50
0.74
2.00
4.00
5.00
5.00
5.00
Avg time to complete (hrs)
9.49
13.75
0.50
2.38
5.00
10.00
100.00
Dollar value of task
$398.46
$599.45
$12.59
$93.72
$174.81
$386.03
$4,114.20
Table 5: Summary statistics for GDPval full set tasks
Mean
Std
Min
25 %
50 %
75%
Max
Overall quality (1-5)
4.55
0.43
2.00
4.33
4.56
5.00
5.00
Difficulty (1-5)
3.20
0.92
1.00
3.00
3.00
4.00
5.00
Representativeness (1-5)
4.43
0.76
1.00
4.00
5.00
5.00
5,00
Avg time to complete (hrs)
8.63
24.70
0.25
2.00
4.00
8.00
605.00
Dollar value of task
$391.44
$1,296.67
$8.53
$70.70
$147.31
$354.12
$32,028.70
A.4.1 FILES AND ATTACHMENTS
Many traditional evaluations rely on text-in/text-out task formats. GDPval tasks incorporate broad range of real-world file types (such spreadsheets, documents, presentations, images, audio, video, and specialized formats like CAD). 67.7% of tasks required interaction with at least one reference file.
21

---

Table 6: File counts for GDPval gold set tasks
Mean
Std
Min
25%
50%
75%
Max
Reference files
1.92
3,47
0.00
0.00
1.00
2.00
38.00
Deliverable files
1.54
2.64
0.00
1.00
1.00
1.00
36.00
A.4.2 N TASKS, SKILLS, AND WORK ACTIVITIES
To ensure broad occupational representativeness, analyzed the O*NET tasks skills and general work activities represented by GDPval tasks. The dataset covered 208 unique O*NET tasks, 25 occupational skills, and 26 work activities.
Most GDPval tasks involve multiple OENET tasks, skills, and work activities.
Table 7: O*NET Tasks, Skills, and Work Activities coverage in gold set
Total unique in OCNET
Total in gold subset
Coverage (%)
O"NET Skills
35
25
71.4%
OENET Work Activities
41
26
63.4%
NET Tasks
1,470
208
14.15%
A.4.3 TASK SPECIFICATION
Occupational experts conducting human grading rated the specificity of instructinnss provided in each prompt. 89.07% % of tasks were rated well-specified, indicating the instructions closely matched real-world expectations of clarity and detail.
Table 8: Task specification scores
Label
% gold set %, full set
Underspecified
8.28%
8.41% 4
Well-specified
89.07%
89.34%
Overspecified
2.66%
2.26%
A.4.4 TASK REPRESENTATTIVENSS
Professional Services Qualification: Technology and intellectual property attorney with partner roles at multiple AmLaw 100 firms in New York and California, and 15+ years of experience advising clients on emerging technologies, advertising, antitrust, and cross-border disputes and transactions.
Quote: Legal tasks included details that felt true to practice, like ambiguous fact patterns, disclo- sure of relevant legal considerations along with non-legal business goals, and realistic reference documents.
Healthcare Qualification: Nursing professional with 18+ years of expertise in emergency medicine, renal management, care coordination, and healthcare operations. Skilled in quality assurance, case management, and professional education.
Quote: These tasks captured the complexity of the role, requiring not only keen for the physician's 's words, but also careful attention to clinical accuracy and professional formatting.
Retail Trade Qualification: Strategic retail executive with 15 years of experience growing prestige and niche beauty brands through national account leadership. $1B+ P&L ownership, and data- driven omnichannel strategies.
Quote: These tasks mirrored the work performed regularly, including developing revenue fore- casts, conducting competitive analysis, building executive-level presentations, and driving strate- gic initiatives for key retail partners within global organization.
Finance Qualification: Fintech and Wall Street leader with 20+ years experience in wealth man- agement, asset management, and capital markets across global institutions and startups.
22

---

They reflected real-world scenarios that nuanced and individualized, situations that only someone with years of experience in the field would fully comprehend. The language and details used in the tasks were directly drawn from actual industry practice, making them authentic and grounded in real-world application.
Quote:
Wholesale Trade Qualification: National Accounts Sales Manager for US, China, and Sweden based brands/factories with over 25 years of experience selling to US based retailers. Quote: All the tasks were in fact based upon real world tasks with back-up reference files and real-world data.
Manufacturing Qualification: Lead Industrial Engineer with S+ years of experience managing large-scale projects and leading teams of 10+ engineers in industrial operations.
Quote: The redesign tasks stood out as especially true to real-worke practice because they in- cluded specific design components and blocks, along with detailed drawings that incorporated precise measurements. They emphasized practical considerations such as visibility and optimiz- ing walking distances to improve overall productivity, exactly the kind of detail-oriented focus that reflects actual enginecring and operational priorities.
Government Qualification: Executive leader with 15+ years working at strategic and operational levels in government and non-profit sectors in housing, human service and labor market pro- grams.
Quote: Many of the tasks demand the integration of multiple sources of information, nuanced decision-making, and tailored the work to varied audiences we serve in the workplace.
Real Estate and Leasing Qnalification: Seasoned commercial real estate broker with 10 years of experience in investment sales, leasing, and managing real estate offices and agents. Quote: The tasks capture the dynamics and expertise unique to specific sectors and settings.
Information Qualification: An experienced senior journalist and content leader with over 20 years top-tier media, global corporations, and high-growth startups.
Quote: Most importantly, the tasks are anchored in real-world challenges and workplace goals. They push past obstacles, achieve workplace goals, and deliver real-world solutions and products.
Additional Detail about Expert Qualifications Less than 10% of applicants were selected to con- tribute tasks our full set. The industry experts also brought occupational diversity, representing different company sizes, locations, and sub-specialties. Each occupation had a minimum of 5 quali- fied professionals.
Experts for each occupation were required to have previous experience in that specific occupation and sector based on the O*NET occupation definitions (of Labor Statistics 2025).
A.5 FURTHER DETAIL ON TASK QUALITY CONTROL
A.5.1 MODEL-IN-THE-LOON TASK REVIEW
We used OpenAl models to automatically screen each task submission across a variety of criteria and flag possible
omissions including: ensuring the task is relevant to the selected O*NET occupation, verifying the request involved tasks performed primarily on a computer, flagging if the task complexity was simple (e.g., if the task seemed like 5 minutes of work instead of longer- term piece of work), and indicating if there were no deliverable and reference files attached.
Because models can make mistakes, experts were instructed take model feedback a suggestion rather than a direction. Experts retained final responsibility for task accuracy and completeness; the model did not autonomously alter tasks.
A.5.2 HUMAN EXPERT REVIEWERS
Human reviewers conducted multiple rounds of review on each task. Reviewers were primarily sourced from the original expert pool based on demonstrated excellence in task creation. Initially, our researchers manually reviewed all tasks to identify experts who produced consistently high- quality tasks; these individuals were trained and promoted reviewers. The most skilled reviewers were further trained to become lead reviewers, responsible for identifying, mentoring. and promoting additional qualified reviewers from within the expert pool. Throughout the review process, the
23

---

research team regularly performed quality conttoll checks tasks signed off by reviewers, ensuring ongoing alignment and quality standards.
A.5.3 ITERATIVE REVIEW PROCESS
The iterative review process included least the following 3 stages:
1. Generalist initial review: A generalist reviewer confirmed the task adhered to project requirements.
2. Occupation-specific expert review:
An occupation-specific reviewer assessed the repre- sentativeness of the task for the occupation, and confirmed that the task was possible for another member of the occupation to complete with the provided context.
3. Final iterative reviewer feedback loop: A third expert reviewer provided iterative feed- back and worked with experts until the task met our rigorous quality standards.
Prior the main review pipeline, all task writers completed "sandbox" task. After initial onboarding survey confirming that they understood campaign requirements, writers created first task in the "Sandbox" and received iterative feedback from an experienced reviewer. A writer could begin contributing to the live campaign only once their sandbox task was signed off as meeting our quality and representativeness standards.
A.6 HUMAN BASELINER RECRUITMENT, TRAINING, AND QUALITY CONTROL
This section provides additional detail the recruitment, compensation, training, and evaluation procedures for human baseliners and occupational experts, following recommended reporting prac- tices (Handa al., 2025).
We recruited expert industry professionals to create realistic tasks based on their professional work experience. Experts were required have minimum of 4 years of professional experience in their occupation and strong resume with a demonstrated history of professional recognition, promotion, and management responsibilities. The average expert had 14 years of experience. We further quired experts to pass a video interview, a background check, training and a quiz to participate in the project. Experts were well compensated for their time and experience.
A.6.1 RECRUITMENT AND POPULATION CRITERIA
Human baseliners and occupational experts were recruited for each occupation through postings on platforms like LinkedIn and targeted outreach. Applicants were screened using their resumes and Linkedln, Al-assisted interview, and, for graders, an occupation- speciifc questionnaire. Eligibil- ity required at least 4 years of full-time experience in the target occupation and fluency in English (all tasks were written and graded in English). Approximately 10% of applicants met these criteria and were selected; 90% were excluded based on insufficient experience. Experts were predominantly based in the United States, United Kingdom, and Canada.
Demographic information on baseliners was not collected beyond region and prior work experience. Therefore, full demographic reporting is not available.
No author served as a baseliner or grader.
A.6.2 COMPENSATION
All human contributors were compensated at professionally competitive hourly rates for their occu- pation, experience level, and geographic region. Rates varied by occupation and resume strength, but all contributors were paid well above U.S. minimum professional contracting wages, with a min- imum hourly rate of $40/hr Task writers and graders were compensated at the same levels during training, and performance- based bonuses were occasionally offered to increase throughput subject quality checks.
24

---

A.6.3 TRAINING PROCEDURES
All experts who created tasks participated in live training calls covering grading standards, exemplar lasks, common errors, and expectations around deliverable quality. Contributors also had access to office hours and dedicated Slack channels for ongoing clarification and support throughout task writing and grading. Writers and graders were compensated for training time.
A.6.4 EXECUTION AND QUALITY CONTROL
Experts determined the number of tasks they completed per session (instrument length was not fixed). Items were not randomized.
Quality control was performed throughout data collection. Our staff monitored Average Handling Time (AHT), and reviewers flagged low-effort low-quality submissions. Tasks failing quality thresholds were excluded from the gold set. For more on the review pipeline, see appendix A.5
Humans and models did whereas models interacted through APIs or first-party UIs depending on file-handling reliability.
equivalent Uls: humans worked through graphical interface,
A.7 AUTOMATED GRADER DETAILS
A.7.1 AUTOMATED GRADER CONSENSUS METRICS
To measure automated grader performance, we measured the agreement rate between scores given by the automated grader human expert graders for the same sample. We also compared grading agreement between human experts who had graded the same sample.
Human-automated grader Agreement. For given sample 8, let the human score H and au- tomated grader score A take values in {0, 0.5, 1}. where indicates preference for the model de- liverable, indicates preference for the human deliverable, and 0.5 indicates a tie. The agreement between human and automated grader is defined as
AHA E Al]
The model-level human-automated grader agreement the mean of AA over all samples for that model.
Human Inter-Rater Agreement. For a given sample 8, let the human scores H1 and H2 take values in p {0, 0.5, 1}. We measure human inter-rater agreement the following expectation over two randomly sampled human ratings
A = Hell
For given sample, we estimate this quantity by the empirical mean over all pairs ratings for that sample. The final human inter-rater agreement for model is the mean of these sample-level scores over all samples with at least two human graders. Existing grader inter-reliability statistics such as Cohen's 's kappa, Fleiss' kappa, and Krippendorff's 's alpha are less directly applicable here, since our graders output ordinal scores {0, 0.5, 1).
A.7.2 AUTOMATED GRADER CORRELATION RESULTS
Over three automated grader sweeps on our datase average human-automated grader agreement was 65.7% and human inter-rater agreement was 70.8%. Plots below show 95% confidence intervals obtained by bootstrapping (resampling with replacement the available automated grader scores or human grades for each sample, computing the mean per sample, and averaging across all samples or for the specified model).
Metrics were calculated over all samples where the automated grader did not encounter systems errors and returned a valid score. We also excluded 12 tasks (out of the 220 in our open-sourced eval set) that the automated grader frequently could not grade was less likely to grade reliably due to its limitations, described later.
25

---

Our automated grader, based on GPT-5-high. shows lower correlation with human expert graders when assessing outputs from capable OpenAl models. This aligns with empirical evidence that models often favor their own responses Pannikssser et al. (2024). Both agreement metrics highest for less capable models, since their outputs are easier to distinguish from human deliverables and less likely to be preferred.
Overall Agreement with Human Expert Graders by Model
Automeated Grader
Inter-Humen
100%
Agreamentt Human Graders
84.5%484.24
80%
73.5%62..0%
8622%71.5%
71.4%
68.0%
66.2%
55.3%
06-mini-gg
Figure 16: Average human-automated grader agreement is most closely aligned with human inter- rater agreement for con-OpenAl models. Both agreement metrics are highest for less capable mod- els, as they can be more frequently distinguished from human deliverables and are less likely to be chosen.
A.7.3 AUTOMATED GRADER RUBRICS
Rubrics used by the autograder are first written by experts trained rubric writing, then reviewed by another human expert from the same occupation the original task writer improve quality. We iteratively improved these rubrics using a model prompted for rubric refinement, and every revision was checked again by expert rubric writer. This process helped ensure that the rubries were clear, consistent, and aligned with domain expectations.
A.7.4 AUTOMATED GRADER LIMITATIONS
In the spen-source set we mark 12 out of 220 tasks as ungradable due to limitations of the automated grader.
1. Internet Access: Tasks which strictly require internet (e.g., tasks that ask agents to find music online and download it) are not possible to grade because the grader does have internet access.
2. Python: The automated grader operates in a container that only allows for running Python. Because of this, we excluded 3 Software Developers tasks that require running other lan- guages and downloading external dependencies to properly test.
3. Font Packages Although the automated grader has most aetrically-identicn fonts (c.g.. Liberation Sans instead of Arial), some font packages used in human deliverables still causes certain deliverables to be rendered differently than they would appear on a computer that has these fonts installed.
4. Speech-to-text transcription: The automated grader has limited speech to text function- ality inside the container, and struggles with non-voice sounds.
A.7.5 AUTOMATED GRADER PACKAGES
To ensure the model can process a wide variety of file types in GDPval, the following packages pre-installed in the base production Docker image. These were also made available to the agent during sampling of OpenAl models.
jupyterlab=
jupyter-core=s,.0 5.1 jupyter--erver=2.11....
jupyterlaa-ppyment==0000 jupyterlab-server==2 2.2
26

---

aiohttp==3.9.5
torchvision==0.200 20.1
nypercorn==0.14.3
PyMuPDF==1.21.2
notebook==6.5.1
paf2image==1.66.
nbclassic==004455
pyth3==00.7
1y1. 10.2
h5py==3.8.0
0..
tableg=a3.8.0
websocketss=10.33
rarfile==4.0
:qdm==4.64.0
odfpy==1.4.1
natplot11b==36633
pymc==4.0%
natplottiib-veen==0011..1
jax==0.2.28
uumpy==122.0
pyx1sb==1.0.8
numpy-fimancial=11.00
keras==2.6.0
scipy=m1.14.1
xgb0ost==1.4.2
p
1oguzu-=0.5.3
$.atsmmodell==0113..5
plotly==5.3.0
aympy==1.13.1
graphviz==0.17
0aaborn==011..2
fuzzymazzy=00.18.0
1.
pydot=1.4422
n1tk==3.9.1
gensim==4.3.1
plotnine==0,10.1
pypandoc==1.6.3
shapely==1.7.1
einops==0.3.2
fiona==1.9.2
reportlab==3, 6..1.....
3eopandas==0.11m 10.2
gradic==2.2.15
0.1
mutagen==1.45.7
pydub==0.25.1
librosa==0.8.1
moviepy==1.0.3
svglib==1.1.0
opencv-Pyytion=44555....
gtts==2.2.3
Pillow==9.1.0
textblob=0015..3 15.3
rasterio==1.3.5
python-pptx=0066.1
rdf1ib==6.0.0
openpyx1=33.0.10
rdkit==2024.9.6
xm1-python-0044.3
biopython=
geopy==2.2.0
cairosvg==2.5.2
sclkit-image==020..
markdownffy==0.9.3
folium==0.12.1
anytree==2.8.0
wordcloud==1.9.2
pdfpllumeer==0.6.2
faker==8.13.2
trimesh==3.
3
svgwrite=m1.4.1
soundfile==0, 10.2
pdfzw==0.4
kerykeion==2, 1.16
pyzbar==0.11 1..
pdfkit==0.6.1
dlib==19.24.. 24.2
pycountry==00.7.3
mtcnn==0.1.1
countryinfo==0.1.22
imgkit==1.2.2
tabulate==0.9.0
chardet==3.0.4
2ha0==0.39.0
bokeh==2.4.0
pylog==1.1
tabula==1.0.5
Pyprover==0.5.6 5.
1.melott-py==0.00.1
Pytesseract=
exchange__cllenddrs=33..
@rcode==7.3
weasyprint==53.3
basemap==1.3.9
pronouncing==0.2.00
pygraphviz==1.7
cryptography==3.4.88
networkx==2.8..
spacy==3.4.4
Pyttsx3==2.90
requests==2.31.0
nashpy==0.0.35 0.33
mne==0.23.4
6ocx2txt=#0.8
Pyopenss1=21.0000
typping-exttensiios==41.1.100000
torch==2.5.1
attabrickk-sgl--ooeettt--......
torchaudio=s2.5.1
ddtrace"=2.8.1
torchtext==0.18.0
datadog"=0.433
27

---

pytest"=8.2.0
catboost
pytest-coy"=5.000
1ightgim"=4455.0
oytest-jsorrteportti..... coverage==7.5.1 0
imblearn
imba imbal = 12.3
We also installed the following additional packages, and we tell the model in the prompt it has access to these additional packages:
libreoffice
a
av==11.0.0
cadquery==2.4.0
7..0
pedalbooard=
pyloudhorm==0.1.1
srt==3.5.3
x1rd==2.0.1
28

---

A.8 FURTHER METHODOLOGICAL DETAILS ON SELECTING OCCUPATIONS
A.8.1 ASSIGNING OCCUPATIONS SECTORS.
We assigned occupations to sectors by using the 2023 BLS National Employment Matrix from of La- bor Statistics 2025) to identify the sector with the highest employment for each occupation. This involved filtering to "Line Item" occupations, taking the first two digits of NAICS codes, dropping "total employment" rows, summing 2023 employment, and assigning each occupation to the sector with the largest share of employment.
29

---

Sector
% GDP
Top Occupations and Total Compensation (in Billions USD)
Real Estate and Rental and Leasing
13.8%
Propety/REi/Commnniy Association Managers $24.54B
Counter and Rental Clerks $$17.22B
Real Estate Sales Agents $13.53B
Real Estate Brokers $4.55B
Concierges $1.80B
Manufacturing
10.0%
First-Line Supervisors of Production and Operating Workers $51.07B
Buyers and Purchasing Agents - $39979B Shipping, Receiving, and Inventory Clerks $38.50B
Industrial Engineers - $37.79B
Mechanical Engineers $31.57B
Professional, Scientifie, and Technical Services
8.1%
Software Developers $239.18B
Lawyers $136.66B
Accountants and Auditors - $135.44B
Computer and Information Systems Managers $121.44B
Project Management Specialists $108.77B
Government
11.3%
Compliance Officers $33.80B
Administrative Services Managers $32.03B
Child, Family, School Social Workers $24.10B
Finst-Line Supervisors of Police and Detectives $17.00B
Recreation Workers - $11.51B
Health Care and Social 7.6%
Registered Nurses $323.05B
Assistance
First-Linc Supervisors of Office/Admin Support $107.02B
Medical Hcalth Services Managers $77.93B
Nurse Practitioners $40.58B
Medical Secretaries & Admin Assistants - $37.87B
Finance and Insurance
7.4%
Financial Managers $147.74B
Customer Service Representatives - $123.70B
Securities, Commodities, and Financial Services Sales Agents
$52.14B
Personal Financial Advisors - $43.33B
Financial and Investment Analysts - $$39.7BB
Retail Trade
6.3%
General Operations Managers $477.16B
lst-Line Supervisors of Retail Sales Workers - $58.27B
Pharmacists $45.12B
Private Detectives & Investigators S2.39B
Wholesale Trade
5.8%
Sales Reps, Wholesale & Mfg (Except Tech/Scientific) $103.21B
Sales Managers $97.16B
Sales Reps, Wholesale Mfg (Tech/Scientifie) $33.66B
lst-Line Supervisors of Non-Retail Sales Workers $22.43B
Order Clerks $3.86B
Information
5.4%
Producers & Directors $16.60B
Editors - $8.18B
News Analysts, Reporters. and Journalists $4.41B
Audio Video Technicians $4.30B
Film Vidco Editors $2.41B
Table 9: Sectors, their value added percentage of U.S. GDP (Q2 2024), with representative top occupations and total compensation billions (USD).
A.8.2 DETAIL ABOUT O*NET DATA SOURCE
Occupations in GDPval. We arrived at 831 occupations by filtering to "Detailed" occupations from the May 2024 OEWS national employment and wage statistics Bureau of Labor Statistics 2025) exclude any aggregate employment categories. We dropped "All Other" occupations,
30

---

which are catch-all categories within broader group that bundle together occupations that don't fit into any of the detailed occupations in that group. Dropping "All Other" occupations left us with 761 occupations.
A.8.3 CALCULATION OF TOTAL WAGES EARNED BY OCCUPATION.
Estimated total wages earned is calculated as total employment * mean annual salary for jobs with annual salaries, and total employment * hourly salary typical work year of 2080 hours for jobs with only hourly salaries. The determination of which jobs had annual vs. hourly salaries was included in O*NET data. 2080 hours is cited as a "typical work year" by the Bureau of Labor Statistics (BLS), assuming someone works 40 hours per week. This is an imperfect estimate (eg., the BLS acknowledges actors "generally do not work 40 hours per week, year round") but is the most precise estimate provided by the BLS.
A.8.4 COMPUTING DIGITAL SCORES FOR OCCUPATIONS.
To classify occupations as predominantly digital, we use a task- based approach. For many occupa- tions, the O*NET database contains task statements and ratings that list all the tasks performed by a worker in an occupation TO The OENET data is provided on the 6-digit SOC occupational code level (SOC-6). We map the OENET SOC-6 occupations and the corresponding tasks to occupations in the OEWS dataset which reports wages at the 4-digit SOC level ("SOC-4"). For each SOC-4 occupation, we classify its tasks either digital or non-digital using prompted GPT-4o model that receives both the occupation and task and the prompt, "For each Task, Occupation, and Sector, clas- sify the task as primarily manual/physical (Return 0) or primarily digital/knowledge work (Return 1). Reply with ONLY 0 or 1. 0 = manual; = digital." We then calculate the weighted share of digital tasks for each occupation. Occupations are classified as digital if their digital share exceeds
a threshold of 0.60.
To calculate the weights for our weighted task share, we use task ratings data from OE surveys, which includes the relevance, frequency, and importance of each task of the occupation We first calculate an Adjusted Task Score for each combination of 6-digit SOC occupation and task. This score is defined as the simple average of the three normalized task ratings: task frequency, task importance, and task relevance. Each rating is normalized relative to the maximum observed rating (e.g. the importance ratings out of 5) If one of these ratings is missing for task, we impute the value with the mean of that rating across all tasks within the same occupation. For example, a task lacks a frequency rating, we assign it the average normalized frequency rating of all tasks in the occupation.
[]]
We then aggregate these 6-digit Adjusted Task Scores into 4-digit Adjusted Task Scores (for each set of 4-digit SOC occupations and tasks). We do this by summing the SOC-6 Adjusted Task Scores of SOC-6 occupations within a SOC-4 occupation for each task For example, the SOC-4 oc- cupation Computer Occupations, All Other combines two 6-digit SOC occupations (Information Security Engineers and Penetration Testers) which have one task in common: "Identify security sys- tem weaknesses, using penetration tests." This task has two SOC-6 Adjusted Task Scores which are added together to create the SOC-4 Adjusted Task Score.
Next, we calculate the Weighted Task Share for each combination of 4-digit SOC occupation and task. The Weighted Task Share is the Adjusted Task Score of the occupation-task pair divided by the sum of Adjusted Task Scores of that occupation. For each occupation, the sum of Weighted Task
Note that while OENET distinguishes between Core and Supplemental tasks in its task data, we treat these two task types equally in our calculation of task share.
Forr the two occupations without OENET 28.3 task ratings ("Facilities Managers" and "Medical Dosimetrists"),
used task ratings from O*NET 29.0.
The maximum importance value is 5, and the maximum relevance value is 100. The maximum frequency value is 7, and the values correspond to categories (e.g. 1="yearly or less." 7="hourly more"). We treat these frequency ratings as proxy for relative task importance. While the scale is categorical rather than interval, rating of 7 unambiguously indicates that a task performed more often and is typically more central day-to-day work than a task rated 1. If a SOC-4 accupation is mapped SOC-6 occupation, the SOC-6 and SOC-4 Adjusted Task Scores the same.
31

---

Share across all its tasks is equal to one. The Weighted Task Share gives us measure of the relative significance of each task for a given occupation. These Weighted Task Shares are the weights used to calculate the weighted share of digital tasks for each occupation.
Table 10: Tasks and Digital Score for Counter and Rental Clerks (SOC 41-2021)
SOC Cirds
Occupation Vill
Took
Weizhted Tink Share
Digital Clasefication
Occupation Digital Scvee
41-2021
Counte and Restal Clerks
Conpute chages for merchardise or services and receire paseecats
6.77
6.97%
41-2021
Count Restal
odes services restal, repais, dry
6.07%
6.67%
clexsing, and shorage.
41-2021
Coune and Restzl Clecks
Esylan recial lees, polices, and procelxess
6.538
6.55%
41-3001
Couner Resull
Provide infoenction
ovillshi-
0.50
6.50%
descrigtios. Advis cus loteers
41-2021
Counter and Rental Cierlo
and car merchandis
6.52%
6,526
41-2021
Courier and Restal Cierla
Civet castener and dacse fax opec quility and quat-
0.89%
1000%
anmecrusdie sngaht venua.
41-2021
6.629
6629
dare.
41-2021
Courcer and Restal Clerla
Inspect and sijest reatal iema to seet needs of contener.
6.38%
0L00%
41-3021
Counee Restal
Propure contd cltaiiig cootonat signatare and
0.3%
6.15%
41-3001
Curter and Rezial Clerlo.
Reni lomn, smmnn, and acsepe triam.
for
of criiece
6.50 0.
0.00%
41-3021
Currrr and Restal Clarlo.
Kerp reconde momactirr and number oie
6.375
41-2021
Curttr and Renial Clarlo.
Reaave, cxaminse and amdos allened clanall
6.2%%
1L00%S
strrci, or requendd
41-3021
Currer and Rental Clarlo
Kearee namee reguanoo time and keep roamts
5.99
5.89%
41-2001
Coune Restal Clecks
Prepere nechordise
rental.
5608
0.00%
41-2031
Couner and Resual Clerks
Recornass) and provide abrice on wide valery
5.79
5,67%
gredhcts and serkees.
41-3021
Couner
eqiipeer gartkipoats spactng everss
increaticall
100.015
63.86%
64%
Table 11: Occupation Digital Scores
Sector
Occupation
Digital Score
Finance and Insurance
Financial Managess
0.984
Finanoe and Insurance
Custemer Service Representatives
1.000
Finance Insurance
Securities, Commodirics, and Financial Services Sales Agents
Pinance Insurance
Personal Financial Advisors
1.000
Finance Insurance
Finuncial and Investment Analysts
0.962
Government
Compliance Officess
0.849
Government
Admmistrative Services Managers
0.93
Government
Child, Family, and School Social Woekers
0.92
Supervisors of Police Detecrives
0.76
Government
Recreation Workers
0.625
Health Care and Sucial Assistance
Regidiered Nurses
0.810
Health Care and Social Assistance
First-l inc Supervisors of Office and Administraiive Support Workers Medical and Health Services Managers
0.966
Health Cure and Social Assistance
.95
Health Care and Social Assistance
Nurse Pracitionars
0.926
Health Care and Social Assistance
Medical Secretaties and Administrative Assistants
0.93
Information
Producers and Directioss
0.912
Information
Editors
1.000
Information
News Analysts, Reposters, and Journalists
0.967
Information
Audio and Video Technicians
0,793
Information
Film and Video Editoos
L.000
Manufacturing
Firt-LLine Supervisors of Production and Operating Workers Buyers Purchasing Agents
0.79
Manufacturing
0,850
Manufacturing
Shipping. Receiving, and Inventory Clecks Industial Engineers
0.73
Manufacturin
0:93
Manufacturing
Mochanical Engineers
0.949
Professionall Scieniific, and Technical Services
Software Developers Lawyers
1.000
Professional, Sciemtifie, and Technical Services
1.000
Professional, Scientific, and Techntical Services
Accountants Auditoes
1.000
Professional, Scieniific, and Technical Survices
Computer and Information Systems Managers
0.95
Professional, Sciemific, and Technical Services
Project Management Specialists
1.000
Real Estate and Kental and Leasing
Property, Real Estate, and Community Association Managess
0.852
Real Estate and Rental and Leasing
Counter und Rental Clerks
0.638
Real Estte Rental
Leasing
Real
Agents
0.879
Esatee Rental Leasing
Real Estate
Real Estate and Rental and Leasing
Conciergos
0.71
Retail Tride
General and Operations Mansgers
0.93
Retail Trade
Finst-Line Supervisurs of Retaal Sales Workets
0.75
Retail Tride
Phurmacists
0.905
Retail Trade
Detectives and Inwasigatoes
Wholesale
Sales Representatives, Wholesale Manufacturing, Except Technical Scientific Products
Wholesale Trade
Sales Managers
0.95
Wholesale Trade
Sales Represenatives, Wholesale and Manufacturing, Technical and Sciencific Products
0.857
Wholesale Trade
Finst-Line Supervisors of Non Retail Sules Workers
0.77
Wholesale Trade
Order Clerks
0.94
A.8.5 HANDLING MISSING DATA.
1. Missing Task Statements. Some occupations in OEWS lacked associated task statements ratings. Forty-seven of these were broad "All Other" categories without component tasks
32

---

twelve others were split into finer sub sub-occupations in O*NET 29.0 (as of August 2025). For the latter, we incorporated the full set of component tasks from their sub-occupations in O*NET 29.0. The exact reconciliation of how we mapped these 12 occupations is be- low. We distinguish between SOC-6 and SOC-4 because these levels meaningfully differ in granularity. SOC-6 (O*NET-SOC) occupations have task-level ratings, whereas many SOC-4 (BLS SOC) aggregates do not have their own task lists and must be constructed by combining constituent SOC-6 occupations. We used the Bureau of Labor Statistics- provided SOC-6 to SOC-4 crosswalk to perform all aggregations.
(a) Tour and Travel Guides: This SOC Code is broken out into two occupations: Tour Guides and Escorts and "Travel Guides") We added the tasks from both occupations.
(b) Miscellaneous Construction and Related Workers: This SOC Code is broken out into three occupations: '"Segmental Pavers"| "Weatherization Installers and Techni- cians" and "Construction and Related Workers, All Other"] We added all of the tasks from "Segmental Pavers" and "Weatherization Installers." " "Construction and Related Workers, All Other" is a general occupation category without component tasks.
(c) Teaching Assistants: This SOC Code is broken out into three occupations: Teach- ing Assistants, Preschool, Elementary, Middle, and Secondary School, Except Spe- cial Education, Teaching Assistants, Special Education and Teaching Assistants, All Other We added the tasks from Teaching Assistants, Preschool, Elementary, Middle, and Secondary School, Except Special Education, and Teaching Assistants, Special Education. Teaching Assistants, All Other is a general occupation category without component tasks.
(d) Buyers and Purchasing Agents: This SOC Code is broken out into three occupa- tions: |Buyers and Purchasing Agents. Farm Products, Wholesale and Retail Buyers, Except Farm Products and Purchasing Agents, Except Wholesale, Retail, and Farm Products We added the tasks from the three occupations.
(e) Substance Abuse, Behavioral Disorder, and Mental Health Counselors: This SOC Code is broken out into two occupations: Substance Abuse and Behavioral Dis- order Counselors and Mental Health Counselors We added the tasks from both occu- pations.
(f) Clinical Laboratory Technologists and Technicians: This SOC Code is broken out into six occupations: Medical and Clinical Laboratory Technologists, Cytogenetic Technologists, Cytotecchollogist/] |Histotcchnollgistt Medical and Clinical Labora- tory Technicians, and Histology Technicians) We added the tasks from all these occu- pations.
(g) Special Education Teachers, Kindergarten and Elementary School: This SOC Code is broken out into two occupations: (Special Education Teachers, Kindergarten and SSeciall Education Teachers, Elementary School We added the tasks from both occupations.
#These occupations were: Entertainers and Performers, Sports and Related Workers, All Other; Postsec- ondary Teachers, All Other; Production Workers, All Other; Office and Administrative Support Workers, All Other; Teachers and Instructors, All Other; Surgeons, All Other; Information and Record Clerks, All Other; Community and Social Service Specialists, All Other; Educational Instruction and Library Workers, All Other; Sales and Related Workers, All Other, Education Administrators, All Other: Social Workers, All Other: Legal Support Workers, All Other: Food Preparation and Serving Related Workers, All Other; Personal Care and Service Workers, All Other; Food Processing Workers, All Other, Motor Vchicle Operators, All Other; Finan- cial Clerks, All Other; Media and Communication Workers, All Other; Counselors, All Other; Social Sciences Teachers, Postsecondary, All Other; First-Line Supervisors of Protective Service Workers, All Other; Dentists, All Other Specialists; Material Moving Workers, All Other; Helpers, Construction Trades, All Other; Drafters, All Other; Media and Communication Equipment Workers, All Other; Metal Workers and Plastic Workers, All Other; Cooks, All Other, Designers, All Other; Life Scientists, All Other; Building Cleaning Workers, All Other; Precision Instrument and Equipment Repairers, All Other; Grounds Maintenance Workers, All Other; Religious Workers, All Other; Artists and Related Workers, All Other; Textile, Apparel. and Furnishings Work- ens, All Other: Gambling Service Workers, All Other: Transportation Workers, All Other; Extraction Workers, All Other: Entertainment Attendants and Related Workers. All Other; Woodworkers, All Other; Underground Mining Machine Operators, All Other; Agricultural Workers, All Other; Logging Workers, All Other; Rail Transportation Workers, All Other; Communications Equipment Operators, All Other.
33

---

(h) Home Health and Personal Care Aides: This SOC Code is broken out into two
occupations: Home Health Aides and Personal Care Aides We added the tasks from both occupations.
(i Property Appraisers and Assessors: This SOC Code is broken out into two oc- cupations: AApraisers and Assessors of Real Estate and Appraisers of Personal and Business Property] We added the tasks from both occupations.
() Miscellaneous Assemblers and Fabricators: This SOC Code is broken out into two occupations: Assemblers and Fabricators, All Other and Team Assemblers) We match to Team Assemblers since Assemblers and Fabricators, All Other general occupation category without component tasks.
(k) Electrical, Electronic, and Electromechanical Assemblers, Except Coil Winders, Tapers, and Finishers: This SOC Code is broken out into two occupations: Ellecti- cal and Electronic Equipment Assemblers and Electromechanical Equipment Assem- blers We added the tasks from both occupations.
()) First-Line Supervisors of Transportation and Material Moving Workers, Except Aireraft Cargo Handling Supervisors: This SOC Code broken out into four oc- cupations:
First-Line Supervisors of Helpers, Laborers, and Material Movers, Hand, First-Line Supervisors of Material-Moving Machine and Vehicle Operators/First-LLiin First-Line Supervisors of Passenger Attendants and First-LLine Supervisors of Transportation Workers, All Other We added the tasks from First-Line Supervisors of Helpers, Laborers, and Material Movers, Hand, First-Line Supervisors of Material-Moving Machine and Vehicle Operators, and First-Line Supervisors of Passenger Attendants. First-Line Supervisors of Transportation Workers, All Other is general occupation category without component tasks.
2. Missing Task Ratings. There are SOC-6 occupations which do not have any task rating in O*NET 28.3 or 29.0. These correspond to 34 SOC-4 occupations [55 Among these, SOC-4 occupations (Data Scientists and Web and Digital Interface Designers) have task ratings for some of the component SOC-6 occupations which allow us to compute the Ad- justed and Weighted Task Share measures. For the rest of the 32 SOC-4 occupations that have no O*NET task ratings, we cannot compute the Adjusted or Weighted Task Share measure. Instead, we proxy the Weighted Task Share as follows: for each combination of 4-digit SOC occupation and task, calculate the number of times the task appears (i.e., task frequency) for the occupation and divide by the sum of task frequency of all tasks of that occupation. For example, the 4-digit SOC occupation "Special Education Teachers, Kindergarten and Elementary School" combines two 6-digit SOC occupations (Special Ed- ucation Teachers, Elementary School and Special Education Teachers, Kindergarten) has 43 unique tasks. Among these 17 tasks appear twice. Thus, the sum of task frequency across 43 tasks is 60. For each task that appears once, the proxy Weighted Task Share is 1/60 0.0017, 17, and for each task that appears twice, the proxy Weighted Task Share is 2/60 = -..0033.
A.8.6 VALIDATING THE DIGITAL TASKS MEASURE
We benchmark our "knowledge work" classification method against the task-content framework of Acemoglu & Autor 2011).
Theeee SOC-4 occupations are: Aircraft Service Attendants, Bus Drivers, School, Calibration Technologists and Technicians, Cardiologists, Crematory Operators, Data Scientists, Disc Jockeys, Except Radio, Emergency Medical Technicians, Emergency Medicine Physicians, Entertainment and Recreation Managers, Except Gam- bling, Financial and Investment Analysts, Financial Risk Specialists, First-Line Supervisors of Entertainment and Recreation Workers, Except Gambling Services, First-Line Supervisors of Security Workers, Fundrais- ing Managers, Health Information Technologists and Medical Registrars, Hydrologic Technicians, Legislators, Lighting Technicians, Medical Records Specialists, Orthopedic Surgeons, Except Pediatric, Paramedics, Pe- diatric Surgeons, Project Management Specialists, Public Relations Managers, Sales Representatives Ser- vices, Except Advertising. Insurance, Financial Services, and Travel, School Bus Monitors, Shuttle Drivers and Chauffeurs. Software Developers, Special Education Teachers, Kindergarten and Elementary School, Substi- tute Teachers, Short-Term, Taxi Drivers, Teaching Assistants, Except Postsecondary, Web and Digital Interface Designers.
34

---

We further contextualize this validation against recent empirical work examining how Al systems are already being applied to economic tasks in practice. Tomlinson et al. (2025) measure the appli- cability of generative AI to occupations by mapping usage patterns onto NET task descriptions, while Handa 2025) analyze millions of real-world Claude interactions to understand which oceu- pational tasks
delegate Al systems. While these studies focus observed Al usage rather than model capability, their findings are consistent with the patterns we observe when benchmarking our digital-task measure against the Acemoglu and Autor framework that AI concentrated in cognitively intensive information- information-processing tasks.
The framework Acemoglu Autor (2011) I based on the U.S. Department of Labor's O*NET survey, which collects data on the activities, work "content", and abilities required for each occupa- tion. The framework aggregates these measures into five scores:
1. Non-routine cognitive: Analytical.
2. Non-routine cognitive: Interpersonal.
3. Routine cognitive.
4. Routine manual.
5. Non-routine manual physical.
Each score is computed composite measure of select O*NET "Importance" scales. For example, the "Non-routine cognitive: Analytical" score for each occupation is computed by summing the (normalized) values of the "Analyzing data/information" work activity, the "Thinking creatively" work activity, and the "Interpreting information for others" work activity. A high numerical value for an occupation for a given score indicates that the occupation relies heavily on that type of work.
We compute the Acemoglu & Autor (2011) scores for each occupation and then compare them with our measures of knowledge work (that is, the share of digital tasks and binary "knowledge work" indicator for each occupation).
In our first set of results, we compare each Acemoglu & Autor 2011) task-content score with the share of digital tasks in an occupation. The patterns are clear: occupations with higher digital-task shares score systematically higher on the non-routine cognitive dimensions and lower on the manual dimensions. In other words, the more occupation relies on digital tasks, the more it resembles cognitive, non-routine work.
0/5
600
580
58 Dip Avik
M/
Figure 17: Distribution of occupations and task contents
In our second set of results, we look at the relationship between the Acemoglu & Autor (01 scores and our binary measure of "knowledge work." In the following figure, we plot each occupation' value for each score, and color occupations by the paper's knowledge-work classification: blue for occupations identified as knowledge work and red for others. The pattern again clear- knowledge- work occupations cluster at the top of the non-routine cognitive distributions and the bottom of the routine and manual distributions. Taken together, these results suggest that our digital- task classification is closely aligned with the economic literature cognitive/manual work.
35

---

Figure 18: Scatterplot of digital tasks and task contents
36


Reviews:
[{"review_id": "fSP3pZm0nC", "reviewer": "Reviewer_uLMP", "summary": "The authors introduce an evaluation dataset for \"real-world economically valuable tasks\" (from occupations with predominantly digital tasks) and show how current frontier models perform on these tasks compared to experts.", "questions": "Questions:\n- Can you explain the notion of “significantly limited” in 213?\n- I don’t understand how the last sentence in line 290 squares with the rest of the paragraph. The previous sentence seems to say that only 47.6% of model deliverables were as good or better than the human deliverable so I don’t understand where the “just over half the tasks” statement comes from. Could you help me understand that, please?\n- Why couldn’t cost estimates for Claude, Gemini, and Grok be obtained? API cost for estimates should be available for all three models, no?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors aim to provide a more granular evaluation to capture the economic impact of AI, an area that is currently lacking data\n- The experts that the authors recruited had extensive experience and were very carefully vetted \n- The authors provide a smaller open-source evaluation dataset as well as an initial version of an automated grader for these tasks\n- The writing is overall clear.\n- The iterative task review process with multiple expert review rounds (and an initial sandbox round with additional feedback) was very well thought-through and designed.", "weaknesses": "Note: I am willing to update my score but currently, there is a lot of methodological information missing in the paper that prevents me from fully assessing the work. Happy to revise my score once more details are provided.\n\n- Line 091: Where do these time estimates come from?\n- Line 106: How was the 5% threshold chosen?\n- Given that in the current version only jobs with predominantly digital tasks were included, the premise of GDPval seems overstated. The abstract states that GDPVal is “an evaluation assessing AI model capabilities on real-world economically valuable tasks” but doesn't specify that it is limited to occupations with predominantly digital tasks only.\n- The paper only states the set of 44 occupations but not the tasks/task descriptions for each of these. I strongly encourage the authors to add, to the appendix, a list of high-level summaries of the tasks and to which occupation they belong to allow for a more meaningful assessment of the paper. For example, I find it puzzling that \"registered nurse\" is in the 44 occupations considered and would have needed a summary of tasks added for this occupation to verify whether this classification made sense. I understand the authors’ concerns that they do not want to make all task specifics/prompts/files/etc. available due to potentially identifying information of expert, but they can make a high-level description of the task and the occupation it belongs to available.\n- Figure 7(a) and (b): Why were OpenAI models singled out here? How does this analysis look like for the Claude or Gemini families? Overall, most analyses in the paper single out OpenAI models and don’t allow for an accurate understanding of frontier model capabilities. Given that the full tasks aren’t open-sourced and the analysis hence cannot be completed for other models by third parties, I kindly ask the authors to provide the more detailed analysis for non-OpenAI models, too, to allow for better understanding and comparability of frontier model performance. E.g., how would Figure 7(b) look like for Claude models (which was overall the best-performing model according to Fig. 6? Similarly, in Section 3.3, this analysis was only done for improving formatting. How much of a performance increase would models like Claude or Gemini have gotten if the model was prompted to strictly follow instructions? \n- The reported details for the human baseliners and recruited experts are insufficient. For example, there is largely no information about the recruitment process given. Also, the paper only states that experts were “well compensated” but given the impact of compensation of performance, this should be made transparent. See this paper for the level of detail required for human baseline reporting: https://arxiv.org/pdf/2506.13776. Relatedly, how were occupational experts (line 206) sourced? \n- It’s unclear to me what “their requests” mean in line 172. Who assigned them “their requests”? Were multiple experts from the same occupation tasked with creating tasks? If this wasn't the case, it’s unclear to me how content validity (i.e., coverage) was ensured for each occupation.\n- It feels like the claim in line 287/288 that GPT-5 excelled in particular on accuracy is overstated. From Fig 8, the % of total examples of failure modes seems too close for accuracy to say that a model “particularly excelled” (within +/- 1% for Claude and GPT-5) and in addition, Grok 4 showed the same % as GPT-5 so I don’t understand why GPT-5 is being singled out here. Are these significant differences?\n- The human uplift study design has not been fully explained. How were scenarios picked? Why were only OpenAI models tested if Claude Opus 4.1 was better overall? \n- It’s unclear if the authors had an IRB or applied for an IRB exemption, which would be necessary for the experimental setup with significant human involvement. Please clarify.\n- Can you provide quantitative data for the clustering pipeline? I.e., a table where there is quantitative information per model on how often the rationale fell into one of the buckets listed in line 311 ff., how often the rationale was unclear, etc. Otherwise it’s hard to judge the true differences across models. It’s also unclear if these differences were statistically significant, please add this information.\n- Line 335: Note that there is a bias if you use the same model to judge the same model (e.g., GPT-5 judge to judge GPT-5 output); it’s best practice to at least compare the judgement to other judge models, see this paper: https://arxiv.org/abs/2404.13076. I'd suggest to at least add this to the limitation section.\n\nNit picks (only affected presentation score):\n- The authors talk at multiple points about the \"gold set\" but it's unclear how the tasks for the gold set were selected.\n- There seems to be a reference error in line 075.\n- Lines 155ff.: It’s unclear to me how the list of prior employers is relevant for describing the experts.\n- The formatting in lines 216/217 seems off, shouldn’t this be a full line?\n- The format looks again off in lines 260/261, these should be full lines. Did you change anything about the underlying template?\n- Figure 8 requires a lot of zooming in to be readable.\n- Line 327/328: this statement is too qualitative. How much did performance improve? I'd suggest to least add a reference to the corresponding figure here.\n- The paper lacks a proper related work section. While the authors very briefly mention previous work in one line in the introduction, they don’t sufficiently explain how their work is novel/different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aopskd1AzJ", "reviewer": "Reviewer_ky9x", "summary": "This paper introduces GDPval, a new benchmark to evaluate the performance of large AI models on real-world economically valuable tasks.", "questions": "* The paper mentions “We fine-tune GPT-5 on GDPval data and measure clear improvements in human win-rate” in the conclusion part. Given that GPT is a close-looped LLM, can the authors provide further explanation about the meaning of fine-tuning here?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This work introduces a benchmark (GDPval) that directly connects model performance to real-world economic value. The framing around professional tasks across multiple industries gives the research strong societal and economic relevance, and it has clear potential to become a foundational evaluation backbone for assessing AI models’ productivity capabilities.\n\n* The benchmark covers 9 major sectors and 44 occupations, representing a large portion of GDP-contributing work activities. The inclusion of tasks curated from professionals with extensive industry experience makes the evaluation dataset both realistic and diverse.\n\n* The paper reports quantitative progress of recent LLMs on GDPval but also analyzes key performance drivers such as reasoning effort, context size, and scaffolding, offering actionable insights into how models achieve expert-level deliverables.", "weaknesses": "* While the authors claim that they will open source their benchmark, I do not see any anonymous link or supplementary that contains the related benchmark file. Given that this benchmark is the major contribution of this paper, the authors should make sure the benchmark will be open-source.\n\n* Since the benchmark focuses mainly on U.S. GDP-related activities and professional writing tasks, it may underrepresent other economically relevant domains (e.g., manufacturing, logistics, physical or multimodal tasks). Broader coverage would improve generalizability.\n\n* The grading pipeline and automatic scoring system are promising but may introduce bias or subjectivity, particularly when comparing AI outputs to human experts. More discussion on inter-rater consistency and rubric validation would strengthen the reliability of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMwbwXggsb", "reviewer": "Reviewer_JEiu", "summary": "The paper describes GDPval, a benchmark of 1320 occupational tasks written and reviewed by professionals in the relevant occupation. Each task involves the creation of a deliverable file (document, slide deck, video, etc). 7 LLMs were tested on GDPval and their outputs were scored by relevant professionals compared to professionally-created deliverables. The best-performing model achieved a win rate of about 45% against human experts. In supplementary analyses, the authors analyze the types of errors performed by different models, how reasoning effort contributes to performance, and how much improvement can be gained by prompt tuning. They also develop an automated grader that nearly achieves the same agreement on GDPval outputs with human graders as humans.", "questions": "1. The tuned prompt in Appendix A.3 implies the GPT-5 agent has significant computer-use ability, but the footnote on page 2 says other LLMs were interacted with through the UI; why was GPT-5 evaluated in a different context?\n2. Relatedly, are all of the models listed in Figure 6 capable of outputting videos, slide decks, CAD files, PDFs etc?\n3. Figure 3 is never referenced or described. What is the sandbox? Does an expert contribute multiple tasks, only the first of which goes through the iterative sandbox? Who are the \"occupational experts\" mentioned in the second stage?\n\n### Comments\n- The paper is missing prompts for the task-level \"digital\" classifier and the occupation-level \"knowledge work\" classifier; without carefully reading the appendix, the fact that there were two classifiers was confusing (the paper mentions both digital tasks and knowledge work, without making the distinction clear)\n- O*NET isn't developed by the BLS, so calling them \"U.S. Bureau of Labor Statistics Work Activities\" in the abstract isn't accurate. (To be pedantic, it's developed by the North Carolina Department of Commerce under sponsorship from the Employment and Training Administration, part of US DoL)\n- For percentage of digital tasks, the main text should reference the appendix section where the weighting method is described (page 21). (this weighting method is also a bit strange, since the 1-7 frequency codes have categorial meanings: 1=\"yearly or less\", ..., 7=\"hourly or more\")\n- the note about fig 12 being on a different version of the test set should be in the figure caption, as this was confusing\n- the more standard names for \"SOC-4\" and \"SOC-6\" codes are \"SOC codes\" and \"O*NET-SOC codes,\" as one is not simply a truncation of the other. Was the BLS-provided crosswalk used to do the mapping between the two taxonomies? (https://www.bls.gov/emp/documentation/crosswalks.htm)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 4, "strengths": "1. This is an impressive and ambitious research project.\n2.The question is timely and of very high importance.\n3. Gathering realistic tasks from professionals is a much better way of benchmarking economic potential of AI than existing approaches.\n4. The amount of time, effort, and money needed to develop, review, and evaluate the tasks is very high.", "weaknesses": "1. The quality of the paper does not live up to the scale and ambition of the methods, with typesetting issues (see lines 075, 262, 288, 294), many missing details, and figures with no explanation or references.\n2. The paper is sparse on details of how the models were run (how do these LLMs edit videos or produce CAD files?)\n3. The prompts (both for GPDval tasks and the other classifiers run), model outputs, and human reviews do not appear to be available\n4. The discussion of related work is very brief, only a few sentences; also, missing a few of the most important related papers:\n\t- Eloundou et al. \"GPTs are GPTs: Labor market impact potential of LLMs.\" Science 384.6702 (2024): 1306-1308.\n\t- Tomlinson et al. \"Working with AI: Measuring the applicability of generative AI to occupations.\" arXiv preprint arXiv:2507.07935 (2025).\n\t- Handa et al. \"Which economic tasks are performed with AI? evidence from millions of Claude conversations.\" arXiv preprint arXiv:2503.04761 (2025).\n\n\n### Overall evaluation\nThis research should obviously be published--it's very important and impactful, and the authors deserve credit and recognition for this work. However, the effort put into the paper falls short of the effort put into the research. The paper really should have some more time and care put into the writing: missing prompts and details should be added, all figures should be referenced and described, and related works should be fleshed out. More information should be provided as to the execution and outputs of the models. It's also not clear to me that ICLR is the best venue for this research. Despite that, I have to give the paper a high score simply due to its importance and impact. However, the authors should really improve the paper before any camera ready so it lives up to the impact of the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)