SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Authors: Samuel Miserendino
Michele Wang
Tejal Patwardhan
Johannes Heidecke
Venue: ICML 2025 oral (2025)
Keywords: software engineering
benchmark
evals
evaluations
dataset
tasks
real-world
swe
coding
delegation
agents
language models
full-stack engineering
Primary area: general_machine_learning->evaluation
TL;DR: We introduce SWE-Lancer, a benchmark of over 1500 real-world full-stack engineering tasks from Upwork, worth $1 million USD in payouts made to freelance software engineers.

Abstract: We introduce SWE-Lancer, a benchmark of over 1400 freelance software engineering tasks from Upwork, valued at \\\$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks — ranging from \\\$50 bug fixes to \\\$32000 feature implementations — and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split. By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Samuel Miserendino
Michele Wang"
Tejal Patwardhan Johannes Heidecke'
Abstract
tinues to accelerate, benchmarks that rigorously evaluate and forecast software engineering (SWE) capabilities will be crucial in assessing economic impacts and agentic safety
We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at $1 million USD total in real- world payouts. SWEe-Lnncr encompasses both independent engineering tasks - ranging from $50 bug fixes $32,000 feature implementa- tions and managerial tasks, where models choose between technical implementation pro- posals. Independent tasks are graded with end- to-end tests triple-verified by experienced soft- ware engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model per- formance and find that frontier models are still unable to solve the majority of tasks. To facil- itate future research, we open-source unified Docker image and public evaluation split, SWE- Lancer Diamond (bitps://githu... com/ By mapping model performance to monetary value, we hope SWE-LLnnce enables greater research into the economic impact of AI model develop- ment.
(OpenAI] 2023 Anthropic 2024 DeepMind 2024).
Prior coding benchmarks have focused on self-contained tasks like program synthesis (Chen et al. (021) and com- petitive programming (Hendrycks et al. (021). The most realistic benchmarks are currently SWE-Bench (Jimenez ct al. 2024 and SWE-Bench Verified (Chowdhury et al, 2024 which focus isolated, self-contained tasks. In the real world, however, software engineers operate across the full technology stack and must reason about complex inter-codebase interactions and tradeoffs.
To better measure the real-world software engineering ca- pabilities and impact of AT models, we introduce SWE- Lancer. Our benchmark evaluates frontier language models on 1,488 freelance software engineering jobs from Upwork, collectively worth $1,000,000 USD in payouts. The eval- uation includes Individual Contributor (IC) SWE tasks, where models generate code patches to resolve real-world issues, and SWE Manager tasks, where models act as tech- nical leads by selecting the best implementation proposal for given problem.
IC SWE tasks range in difficulty from 15-minute bug fixes to new feature requests that took weeks to close out. Unlike prior benchmarks, which grade model per- formance with unit tests, SWE--Lance evaluates IC SWE tasks using end-to-end tests created by a team of professional software engineers. These end-to-end tests use browser automation to verify application be- havior and mirror the real-world freelance review pro- cess, and have been triple-verified for quality by expe- rienced software engineers.
1. Introduction
In just two years, language models have advanced from solving basic textbook computer science problems to win- ning gold medals in international programming competi-
tions (O 2024a Quan et al., 2025). When OpenAl announced SWE-Bench Verified in August 2024, GPT-40 scored 33%; today, their o3 reasoning model achieves SOTA 72% (OpenAl 2024b), highlighting the need for unsatu- rated evaluations that reflect the complexity of real-world software engineering. As Al research and development con-
SWE Manager tasks direct models to review compet- ing proposals submitted by freelancers in response to job postings, and select the best one. They are then assessed against the choices of the original engineer- ing managers. These tasks require deep technical understanding of both the issue and the proposals; it is often the case that multiple proposals are technically correct, and identifying the winning proposal requires considering context from the entire repository.
Equal contribution 'OpenAl. Correspondence to: Samuel Miserendino <samuelgm cocemnull@@@opeaiic.omm @openai.comn. Michele Wang sniicheee@opnnai.com.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SWE-Lancer
IC SWE Task: Reverted codebase and original issue description
Model generates proposed fix
Score & Evaluate
Original Iasue (Problem, Description, Price)
Large Language Model
Grader
Example (356000) zip Je0S0codo sarisation prirr russage not acieped on Use Mhors addrece #1-222"
The redilt promuted with tukk and produce a panch that resdives the lotue.
T1e rodders upcired codetase
Generated PR
Codabase- pro-fix
valceroe Util
solution to the aase was merged.
include Capera Sex
Human software engineers create end-to- end tests for the issue
Scoring
Ultecead to--dtt-ddd teal paesed, the modetss to aaccenrts which framdintes so earred payset.
$2000 earned
$D earned
Figure I.
Evaluation flow for IC. SWE tasks; the model only carns the payout if all applicable tests pass.
SWE Manager Task: Original issue description and multiple proposals
Model chooses proposal
Score & Evaluate
Original losue (Problem, Description, Price)
Large Language Model oojeciive is choore the proposal that best resatves the seath
Comparison
Commptee [S250]] les Bequaex, Resson hwodee iitely after hoiling resuett racioii
ultmatiily impler ented snlution.
Different Proposals (+2) Ex Exaemplies paraphrieed:
X Rejected preposal
X Rejected proposal
Propoout tay atuentls
Proponul22 2 atcrageont
Scoring
Propocal thy whtenttry A comment handing mirkomon message type.
Deeroomph cated the scope inckuling nigingg manstowr dellylngg
Deperded unrureputt pull request (P32-422 dinlyy.
the medil choouse the cars as the ingtt exteriod solutionn the chokge
trankoow pariing PR (438452) is mergad.
X
Rejected proposal
/
Chosen proposal
Proposal (oy utowitsimurg Detay fix until Eiferdess
Preposal 3 stoauttrrera
Preporall try stangeourt
implemented.
Prowided the boldd styling bous whehout unTeeceoiry
$1000 earned
by setting rndszaod byoe COMMENT without
hold
uniitited becoend kpsaase delily resoration
Figure 2. Evaluation flow for SWE Manager tasks; during proposal selection, the model has the ability to browse the codebase.
SWE-Lancer offers several advantages over existing SWE benchmarks:
engineering proposals remains unexplored. Advanced full-stack engineering: Prior evaluations have largely focused on issues in narrow, developer- facing repositories (e.g. open source utilities to facil- itate plotting or PDF generation). In contrast, SWE- Lancer is more representative of real-world software engincering, as tasks come from a user-facing product with millions of real customers. SWE-Lancer tasks frequently require whole-codebase context. They in- volve engineering on both mobile and web, interaction with APIS, browsers, and external apps, and valida- tion and reproduction of complex issues. Example tasks include a $250 reliability improvement (fixing a double-triggered API call), $1,000 bug fix (resolving permissions discrepancies), and $16,000 feature imple- mentation (adding support for in-app video playback in web, iOS, Android, and desktop).
Real-world payouts: All 1,488 tasks represent real payouts to freelance engineers and provide a natural market-derived difficulty gradient (ranging from $250 to $32,000). Notably, the economic payout associated with each task is not an estimate; rather, it reflects the actual amount paid to the freelancer who completed it. Monetary payouts are non-trivial: within the open- sourced SWE-Lancer Diamond set, 35% of tasks are worth more than $1,000, and 34% are worth from $500 to $1,000.
Management assessment: Large-scale software en- gineering in the real-world requires capable technical management. Our benchmark evaluates models' abil- ity to function as a SWE manager using real-world data. In SWE management tasks, the model acts technical lead, weighing the actual implementation proposals that were submitted by freelancers when the issue was originally posted. Previous works have inves- tigated models' ability to generate research ideas (Si et al. 2024), but their capacity to evaluate commercial
Improved grading via end-to-end (E2E) tests: Ex- isting coding benchmarks often rely on unit tests to evaluate isolated functions or components; however, these tests are highly susceptible to grader hacking (see Appendix A7) and insufficient for evaluating full- stack engineering work (Brar& & Kaur 2015) Paul et al.
2

---

SWE-Lancer
2006). SWE-Lancer is the first benchmark use E2E tests created by professional engineers, offering more comprehensive, real-world evaluation that is harder to exploit. For example. in one $1,000 task (Expensify 2024a), a bug caused the avatar on the "Share Code" " page to differ from the profile page. While unit tests might confirm each function works independently, the end-to-end test we created for this task simulates the entire user workflow- including logging in, uploading profile picture, and interacting with a second account and catches the bug. Tests underwent three rounds of validation by software engineers for quality, coverage, and faireess, supplemented by an automated system ensuring each test passed in the fixed state and failed in the broken state.
To advance research in automated software engineering. agentic safety, and economic impacts of automated coding models, we release public evaluation split - SWE-Lancer Diamond - containing $500,800 worth of tasks. To avoid contamination via training or search (e.g.. for models that can browse the internet,, we keep the remaining set as a private holdout. The full dataset is available upon request.
1.1. Related Work
Researchers have evaluated LLMs' coding skills in scien- tific programming (Tian et al. et al. 2023 Lam et al.,
2024), text analysis (Zhong)
2024). and repository-lveel comple-
tion (Zhang et al. 2024 Liu et al.
2023). with benchmarks
such as HvmanEval CCen et al.
1021). HumanEvallPck (Muennighoff et al. 2024). APPS, LiveCodeBench (Jain ct al. 2024), NaturalCodeBench (Zhang et al. 2024). Big- CodeBench (Zhuo et al. 2024). MBPP, MathQA-Pytte
Domain diversity: SWE-Lancer tasks span many cat- egories (Figure 3). In the open-sourced Diamond set, 74% of IC SWE tasks and 76% of SWE Management tasks involve Application Logic, while 17% of IC SWE tasks and 18% of SWE Management tasks involve ULUX development. These categories extend beyond styling to include core full-stack concepts like frontend event handling, DOM interactions, and application be- havior. Most tasks (88% of IC SWE tasks and 94% of SWE Management tasks) are bug fixes. See Appendix A12 for more details on dataset composition.
(Austin et al. 2021). CodeXGlue (Lu et al. 2021). Al-
phaCode (Li et al., 2022), CODEGEN (Nijkamp et al.)
2023), and D LLai et al., 2022). SWE-bench verified
Chowdhury et al. 2024) builds upon SWE-bench (Jimenez et al. 2024) and evaluates LLMs on real-world pull requests from open-source repositories, graded against human-coded patches. SWE-bench Multimodal (Yang et al.) 2024) ex- tends this to frontend tasks from open-source Javascript libraries. However, these benchmarks typically rely on unit tests (prone to grader hacking, see Appendix A7), exclude commercial repositories, reposttiirepoossooriii,,,, and lack full-stack coverage gaps that SWF-Lancer addresses by sourcing real freelance tasks and mapping model performance to economic payout. See Appendix A4 for more detailed comparison table of coding benchmarks.
Difficulty: SWE-Lancer tasks are challenging. The average task in the Diamond set takes 26 days to re- solve on Github and has 47 comments, which include discussion of multiple proposals. On average, IC SWE tasks in the Diamond set require modifying 2 files and 69 lines of code, while SWE Manager tasks require the model to choose between 4-5 proposals.
Unbiased data collection: The data collection process for prior benchmarks, such as SWE-Bench and SWE- Bench Verified, involved searching for pull requests on GitHub and selecting ones that had unit tests. As a result of this selection process, these datasets are biased towards code and problems that are easily testable. Instead of filtering out the vast majority of problems to find ones that volunteers chose to write tests for, we selected a representative sample of tasks from Upwork and paid a team of 100 professional software engineers to write and verify end-to-end tests for all of them.
2. SWE-Lancer
The SWE-Lancer dataset consists of 1,488 real freelance software engineering tasks from the Expensify open-source repository posted on Upwork. These tasks are collectively valued at $1 million USD and are divided into two cate- gories:
Individual Contributor (IC) Software Engineering (SWE) Tasks: This set contains 764 tasks collectively valued at $414,775, which mirror the responsibilities of individual contributor software engineers (c.g., im- plementing features, resolving bugs). The model is given (1) the issue text description (including repro- duction steps and desired behavior), (2) the codebase checkpointed at the state before the issue fix, and (3) the objective of fixing the issue. The model's solu- tion is evaluated by applying its patch and running all associated end-to-end tests using Playwright, an open-source browser testing library (Microsoft) 2025). Models are not able to access end-to-end tests during
We also perform large-scale evaluations of state-of-te of-the-art models to better understand the capabilities of autonomous SWE agents, including running ablations on test-time com- pute and tool affordances. We find that the best performing model - Claude 3.5 Sonnet scores 26.2% on IC SWE tasks and 44.9% on SWE Management tasks, earning a total of $208.050 out of $500,800 possible on the SWE-Lancer Diamond set. On the full SWE-Lancer dataset, Claude 3.5 Sonnet carns over $400,000 out of $1,000,000 possible.
3

---

SWE-Lancer
Application Logic
Logic
Bugfixes
Stato managemeel 1I, dienesdice routing forn
XFI endporns, eries cherticaiin on G2t5
dreopected betantore, emons, incand stancies Eseple: "15500 haegeting bacr from the Rlag Officsted screen
Navigating bocs
Offorsive
dispkay
Aoangier "kseollogas witee attertting tol feci after clicking kiin reccia arcigl sigli casen
Laarete Tasesl meto surr reAteeniaaeg Naewiing counte the presleer
the MasdetthiiiacBB..ll The fir requinas updatiins
in
orEccAB tooPress gaza propert..
Dlickig Diditt oocolvo maglc coc of tolls roset
ity, hovering ouer the split
nose ioha kos laCtude cleat the and correctly restores tocus.
lig
their disp key name, To be thik, updicta the tasklp logic so That shows diipkay Hattes
instead
New Features Enhancements functionally oatter (erredd
System-Wide Quality and Reliability
saggest unar's martion, ortiaing when thendh trail inpur. aftar s
gitern
UI/UX
Deaign changes lasoud, styling, interaction
Reliteed tecingg codee, formancce
Reliability Improvements Logg ng morbo tecting
Knanch: Geentn Germmneent
Dhanate: 113000/ Ast chaoctec fint en an samen,
The eetinage sereer owertey cater ciliara be bellwenn Staglng
Openiing a threaded conversation rrigered the same APl call weatking thk requ mes calls.
reull. to thi, set 100 character innt for tdks and 900 character lire desorptione consibteryy
consiitenyy.
Figure 3. Breakdown of the different types of IC SWE issues in SWE-Lancer with examples from the Diamond Set. Blue categories on the left represent task topics, while green categories at right represent task types.
[Paid] ($8000) zp /postcode validation enaor message not displeyed for entering on the Home address screen #14958
$8000
$8,000
After further Iteration on the last proposal with the SWE Manager, the bounty doubles and the fix is made
$7,000
$6,000
Payout for Task
$5,000
$4000
$4,000
Complexity of issue becomes SWE ICs search for solution that validates all global codes
$3,000
$2000
$2,000
$1000
proposats were rejested for failing solve the issue
$1,000
issue posted
Week 4
Timeline
Figure 4. SWE-Lancer issues dlnamiially priced based on real-world difficulty. In the example above (Expensity 2023 SWE managers rejected 5 early proposals that did not appropriately address edge cases. The initial request priced at $1,000 was increased to $8,000 over four weeks until it was solved.
the evaluation.
experienced engineers would evaluate them. Full prompts are available in Appendix A8 and example tasks are in A13.
SWE Management Tasks: This set contains 724 tasks collectively valued at $585,225. Taking on the role of a software engineering manager, the model must choose the best proposal for resolving a task. The model is given (1) multiple proposed solutions to the same issue (taken from the original discussion), (2) snapshot of the codebase from before the issue fixed, and (3) the objective of picking the best solution. The model's selection is evaluated by assessing whether it matches ground truth. An additional validation campaign with experienced software engineers showed 99% agree- ment with the original chosen solutions.
2.1. Evaluation Metrics and Task Types
We track the percent of tasks resolved and the total pay- out earned by the model (using the real freelance rates set in Upwork by Expensify). Because they draw from real tasks, SWE-Lancer payouts uniquely reflect true economic value rather than theoretical estimates. Harder tasks pay more, especially when they demand specialized knowledge or remain unresolved for extended periods of time.
2.2. Benchmark Construction
Issues are posed exactly written on Upwork; each IC SWE task is graded with end-to-end tests mirroring how
ensure our dataset consists of high-quality and represen- tative tasks, we implemented the following pipeline:

---

SWE-Lancer
1. Repository selection. Expensify is a $300 million USD public company (NASDAQ: EXFY) with 12 mil- lion users who rely on its software, making it a reliable testbed for sourcing commercially valuable software engineering tasks. The open-source Expensify repos- itory posts tasks on Upwork for frrelance engineers with concrete payouts, We sourced real tasks that were previously solved by paid contributors, The repository has diversity of high-quality, complex tasks.
the possibility of models scraping code diff's pull request details. Each agent has single attempt (pass@) @ 1), mir- roring the constraints of typical online SWE freelancing platforms. Models operate with a basic scaffold that lets them browse the local codebase, modify files, and execute terminal commands. Full details of our execution environ- ment and scaffolds can be found Appendix A5.
For our baseline setup, we evaluate OpenAI's gpt-4o-22 40-2024------- 08-06, OpenAI's ol (with "high" reasoning effort (OpenAl 2025) and Anthropic's Claude 3.5 Sonnet (claude-3-S- sonnet- 20266)20040620), Additional results on open-source mod- els (Deepseek-F) R1, Llama 3.3) are in Appendix A15. All models have the user tool enabled
2. Task selection. 100 professional software engineers reviewed the tasks, proposals, and codebase for clarity, specificity, and executability. For high-value IC SWE tasks (with payout exceeding $5,000), a team of ten experienced engineers validated each task, confirming that the environment was properly configured and test coverage was robust. Each IC SWE and SWE Manage- ment task was reviewed by multiple engineers (three for IC SWE, two for SWE Management). More detail on the review process is included in Appendix A6.
3.2. Main Results
As shown in Figure 5, all models earn well below the full $1 million USD of possible payout on the full SWE-Lancer dataset.
3. Task generation. From each validated Github issue, we generated an IC SWE task using the task's 's title and description, and a snapshot of the codebase at the time it was posted. If there were least two proposals for how to resolve the issue, also generated SWE Management Task using the issue description and list of proposals.
$1,000,000
$800,000
US Dollars Eamned
$600.000
4. End to End Test Development. We wrote compre- hensive end-to-end Playwright tests for each IC SWE task. Tests simulate real-world user flows, such as log- ging into the application, performing complex actions (e.g., making financial transactions), and verifying that the model's solution works as expected. Each test is triple-verified by professional software engineers.
$400.000
$380.350
$403,325
$200,000
GPT-4o
o1
Claude Sonnet
Model
5. User Tool. Each IC SWE Task has an accompanying user tool; when the agent invokes the user tool, a Play- wright script opens the browser and simulates a user attempting to perform the action associated with the task in the local application, the model view its work, just as human engineers often run their code while writing it. The model does not receive any feed- back about whether the action was successful; however, a text-based trajectory and a series of screenshots are written to working directory when the tool finishes. When the user tool is enabled, the model is allowed to invoke it as often as it wishes to via the command line.
Figure 5. Total payouts earned by each model on the full SWE- Lancer dataset including both IC SWE and SWE Manager tasks.
To illustrate model performance across experiments, we present the pass@1 accuracies, corresponding "carnings" (i.e., total payout), and earn rates (i.e., payout received divided by total possible payout) for IC SWE and SWE Manager tasks, summarized in Table B All models earn well below the full $1 million USD of possible payout on the full SWE-Lancer dataset.
As shown in Figure 6, all models performed better on SWE Manager tasks than on IC SWE tasks, which remain largely unsaturated. For IC SWE tasks, pass@1 and carnings rates remain below 30%. On SWE Manager tasks, the best per- forming model Claude 3.5 Sonnet scores 45% on the
3. Experiments and Results
3.1. Experiment Setup
For our experiments, agents run in a Docker container with the repository preconfigured and no Internet access, prevent- ing them from retrieving external information. We further remove the GitHub remote and any future commits to avoid
'Due to rate limits, we evaluated Claude 3.5 Sonnet on the set with pass@! only. BBecause @ represents one sample, there be signifi- cant variance between runs.
5

---

SWE-Lancer
GPT-40
Claude Sonnet
GPT
(IC SWE)
100%
100%
80%
80%
pass-81 (%)
pass (%)
00%
48
40%
00
40%
20%
20
26.0%
0.0%
8.0%
11.0%
15.2%
0%
I0 SWE
SWE (Full
SWE Manager (Damond)
SWE Manager
(Diamond)
(Ful)
Subset
Number of Attempts (k)
Figure Model pass@1 performance on SWE SWE ager in the Diamond set and the full set; Claude 3.5 Sonnet per- forms the best and carns $208K in total on the Diamond set and over $400K on the Full set.
Figure 7. Pass performance IC SWE tasks within the SWE- Lancer Diamond set increases with number attempts al- lowed.
o1 Low Ressoning
of Reasoning
High Reasoning
Effort
1.0
Diamond set. 3.5 Sonnet exhibits the strongest performance both IC SWE and SWE Manager tasks, outperforming the next-best model (ol) by 9.7% IC SWE (Diamond) tasks and 3,4% on SWE Manager (Diamond) tasks.
0.8
passe1 (8)
0.8
0.4
3.3. Increasing Number of Attempts
0.2
19 7.0%%..1%
17.1%
To evaluate how performance changes with more attempts, evaluate GPT-4o and ol using the pass@k k metric (Chen et al., 2021). We estimate the percentage of tasks which the agent solves correctly, given n total samples and correct
12.7%
7.7% 7%%
4.2%
0.0
<5500
$500-16 1K
$1K 2K
$2K1
Task Price Range
samples as: @ Eprablems 1
(2) The main
Figure 8, ol pass@1 performance on Diamond IC SWE tasks by price range of task. Higher reasoning effort improves overall pass @ and improves performance on harder, more expensive tasks.
result for 7] is shown in Figure 7. For all models, allowing more attempts leads consistent increase in pass rate. This curve is especially noticeable for ol, where allowing for 6 more attempts nearly triples the percentage of solved tasks. GPT-4o with pass @6 achieves the same score as ol with pass@ (16.5%).
20%
16.5%
19%
3.4. Increasing Test Time Compute
pass (%)
We measure pass @ I as a function of inference-time com- pute. On IC SWE tasks within the Diamond set, experiments with ol and user tool enabled show that higher reasoning ef- fort (test-time compute) improves @ from 9.3% (Low reasoning effort) to 16.5% (High reasoning effort), with corresponding increases in earnings (which increased from $16K to $29K) and earn rates (increased from 6.8% to 12.1%). Figure 8 shows the breakout of pass@) by task price range for different levels test-time compute and in- dicates greater test-time-com cest-ime-ccomute can improve performance on harder, more expensive problems in particular.
10%
8.0%
8.0%
5%
GPT-40
Model and User-Tool Usage
Figure 9, Pass@ performance on Diamond IC SWE tasks slightly decreases when the model cannot use the user tool.
6

---

SWE-Lancer
Table Full pass@1 Accuracies and Famiins for IC SWF and SWE Manager. Earnings are rounded the nearest thousand. SWF--Lancer Diamond contains 237 IC SWE tasks worth $236,300 and 265 SWE Manager tasks worth $264,500, totaling 502 tasks worth $500,800. The full set has 764 SWE tasks worth $414,775 and 724 SWE Manager tasks worth $585,225, totaling 1,488 tasks worth $1,000,000, In the table below, overall metries for SWE-Lancer Diamond and SWE-Lancer Full sets are highlighted in blue, while baseline metrics for IC SWE (Diamond)
Manager (Diamond) tasks are highlighted green.
Model
User Tool
Dataset
Reasoning Effort
pass
Dollars Earned / Total Earn Rate
GPT-4o
Yes
IC SWE (Diamond)
N/A
8.0%
$14k / S236k
6.0%
ol
Yes
IC SWE (Diamond)
Low
9.3%
$16k / S236k
6.8%
ol
Yes
IC SWE (Diamond)
Medium
15.6%
$24k / S236k
9.9%
ol
Yes
IC SWE (Diamond)
High
16.5%
$29k / S236k
12.1%
3.5 Sonnet
Yes
IC SWE (Diamond)
N/A
26.2%
$58k / S236k
24.5%
GPT-4o
No
IC SWE (Diamond)
N/A
8.0%
$17k / S236k
7.2%
o1
No
IC SWE (Diamond)
High
13.1% %
$23k / S236k
9.7%
GPT-4o
Yes
IC SWE (Full)
N/A
8.6%
$29k / $415k
6.9%
ol
Yes
IC SWE (Full)
High
20.3%
$78k / S415k
18.9%
3.5 Sonnet
Yes
SWE (Full)
N/A
21.1% %
$89k / S415k
21.5%
GPT-4o
N/A
SWE Manager (Diamond)
N/A
37.0%
$125k / $265k
47.1% 1%
ol
N/A
SWE Manager (Diamond)
High
41.5%
$137k
$265k
51.8%
3.5 Sonnet
N/A
SWE Manager (Diamond)
N/A
44.9%
$150k $265k
56.8%
GPT-4o
N/A
SWE Manager (Full)
N/A
38.7%
$275k $585k
47.0%
ol
N/A
SWE Manager (Full)
High
46.3%
$302k $585k
51.6%
3.5 Sonnet
N/A
SWE Manager (Full)
High
47.0%
$314k / $585k
53.7%
GPT-4o
N/A
SWE-Lancer Diamond
N/A
23.3%
$139k $501k
27.7%
ol
N/A
SWE-Lancer Diamond
High
29.7%
$166k / $501k
33.1%
3.5 Sonnet
N/A
SWE-Lancer Diamond
N/A
36.1%
$208k / $501k
41.5%
GPT-4o
N/A
SWE-Lancer Full
N/A
23.3%
$304k / SIM
30.4%
ol
N/A
SWE-Lancer Full
High
32.9%
$380k / SIM
38.0%
3.5 Sonnet
N/A
SWE-Lancer Full
N/A
33.7%
$403k / SIM
40.3%
model attempt
5 model antemppss
How cost savings calculated?
100%
Freelancer $236,300
Preetancers were colectivey pald amount solving lusks the wncer freolancer solved
Cost Savings (%5)
60%
Combined model and freelancer cost All API Costs Freelancer Costs
fails)
40%
of and CIPT-40 tax 20-00% of lusks respectivnly. vn teebwicers would be needed solve the rest
20%
18.6%
% Servings
Conelined essdri and freelinrer
Tatal frediance
$231.300
8.8%
13.3%
Companed to the fvwlaicor coul of solking of lasko (8208 200) hme moch cheaper is crap both feellnoova
freelancer GPT-80
freelancer o1
model attempts model task SWE Lanoer
for completion.
Figure 10. Attempting tasks with before passing failed tasks freelancers could reduce costs by 1/3 Diamond IC: SWE tasks.
3.5. Removing User Tool
user tool and therefore experience a greater performance drop under this ablation.
In IC SWE tasks, removing the user tool only minimally reduces pass@I rates, shown in Figure However, we observe that stronger models make more effective use of the

---

SWE-Lancer
3.6. Price Analysis
$208,050 on the SWE-Lancer Diamond set and resolves 26.2% of IC SWE issues; however, the majority of its so- lutions are incorrect, and higher reliability is needed for trustworthy deployment.
How does the cost of running these models compare to the cost of freelancers? We performed 5 runs of GPT-4o and ol on SWE-Lancer Diamond IC SWE tasks (237 tasks, worth $236,300 total) using the OpenAl API and our open--ource eval harness Because each SWE-Lancer task has an actual cost paid out to the freelancer who solved it, we can compare API vs. freelancer costs in solving all tasks. We measured several key metries (results below are the average over 5 runs):
The strongest models perform well across all task types. Tables 2 and show pass@1 rates for IC SWE Diamond tasks across different task categories. Sonnet 3.5 performs best, followed by ol and then GPT-4o, and pass@) @ I on Manager tasks is often more than double pass rate IC SWE tasks. Sonnet 3.5 outperforms ol by nearly 15% on UI/UX tasks in particular, and nearly 10% on IC SWE tasks involving implementing new features or enhancements.
1. Cost Savings (pass@11 I): GPT-4o fails 90% of IC SWE tasks on the first try, and ol fails 80%, but using models first and deferring to humans only on tasks the models fail would reduce total costs. We calculate cost savings as: Cost savings =
Effective tool use distinguishes top performers. We find that the strongest models make frequent use of the user tool, and are able efficiently parse its outputs reproduce, localize, and iteratively debug issues. The user tool often takes a nontrivial amount of time to run a period of 90 to 120 seconds - during which weaker models such as GPT-4o are prone to abandoning the tool altogether. The best per- forming models reason about the delay (which is disclosed in their instructions), set appropriate timeouts, and review results when available. An example user tool trajectory is in Appendix A10.
casts + Freelancer costs on tasks model fails Compared to Total freelaneer cost to solve all tasks
the freelancer cost of solving all 237 tasks ($236,300 empirically), GPT-40 + freelancer fallback is 8.84% cheaper; ol freelancer fallback is 13.26% cheaper.
2. Cost Savings 5): Giving models 5 attempts before freelancer fallback improves savings and pass rates:
GPT-4o reaches 18.6% total cost savings; ol yields 33.5% savings as shown in Figure 10.
Agents excel at localizing, but fail to root cause, resulting in partial or flawed solutions. Agents pinpoint the source of an issue remarkably quickly, using keyword searches across the whole repository to quickly locate the relevant file and functions often far faster than a human would. However, they often exhibit limited understanding of how the issue spans multiple components or files, and fail to address the root cause, leading to solutions that are incorrect or insufficiently comprehensive. We rarely find cases where the agent aims to reproduce the issue or fails due to not finding the right file or location to edit. We provide several qualitative summaries of trajectories in Appendix A9.
3. Model
Contribution
Ratio
model cost on tasks +freesa-ree freelancer cost on model failures' GPT-4o returns $0. 10 per $1 spent in total model+freelancer
payout on successful model tasks
costs; ol returns $0.16 per $1 spent. This ratio is low because models fail most tasks in SWE-Lancer Diamond and require freelancer to solve the rest, but should increase for new frontier models and may already be high for easier tasks.
This preliminary analysis contains many limitations hope future work can expand upon. First, it is based on cost to customer (i.e., using API pricing). Our estimate of API costs in this analysis includes retries and excludes batch API pric- ing, which would both reduce model cost and increase the model contribution ratio. Second, all SWE-Lancer prices reflect historical bounties actually paid, which may differ from current rates. Third, there was nontrivial variance in cost across runs The 95% confidence interval for GPT-4o was $194.31 + $17.31,, while ol's 's was $1, 11,623776 + $56.68. Lastly, our pass@5 5 savings analysis assumes quick verifica- tion of correct solutions, but in practice this can take over 24 hours. Please see Appendix Section A A4 for more detail.
4. Limitations
Diversity of repositories and tasks. We sourced tasks exclusively from the Upwork freelancer platform and the Expensify repository. While we believe the Expensify repos- itory is representative of full-stack engineering work, incor- porating tasks from other freelance sites and open-source repositories could broaden the scope of this evaluation. In particular, infrastructure engineering tasks (c.g. refining Kubernetes cluster architectures and debugging pod failures, node crashes, or networking problems) are underrepresented in this iteration of SWE-Lancer because they are dispropor- tionately underrepresented in Expensify's 's posted task set.
3.7. Discussion
Scope. Freelance tasks tend to be more self-contained than full-time software engineering tasks (Cuupta et al. 2020). SWE-Lancer would also be less representative of "zero to
Results indicate that the real-world freelance work in our benchmark remains challenging for frontier language mod- els. The best performing model, Claude 3.5 Sonnet, earns
8

---

SWE-Lancer
Table 2. Diamond Task Pass Rates by Task Type
IC SWE
SWE Manager
Task Type
GPT-4o
ol
Sonnet 3.5
GPT-4o
ol
Sonnet 3.5
Application Logic (Client-Side)
8.0%
15.9%
23.9%
176
36.3%
42.3%
45.8%
201
UI/UX
2.4%
17.1%
31.7%
41
32.7%
32.7%
40.8%
49
Server-Side Logic
23.5%
23.5%
41.2%
17
53.8%
61.5%
38.5%
System-Wide Quality and Reliability Tasks
0.0%
0.0%
0.0%
3
100.0%
50.0%
100.0%
one" SWE work as all tasks build off of well-established and maintained codebase. However, the entire front-end por- tion of the Expensify repository is maintained by freelance software engineers, which makes it closer to real-world gineering than other freelance postings. We also include SWE management tasks, which freelancers rarely handle. Still, while SWE-Lancer may be a good leading indicator of how automation could affect freelance software engineering, we remain cautious about extrapolating impact beyond this.
pensify GitHub issues contain videos (and, occasionally, screenshots) of the expected behavior. Additionally, the user tool saves screenshots the output directory upon completion, which current models unable to view. Be- cause there is not widespread support for such inputs, prioritized making information about the prompt and envi- ronment available via text (for example, through text-based trajectories from the user tool).
Modalities. SWE-Lancer tasks are text-only, though issue writers often include screen recording. This evaluation does not test how much screenshots videos can improve model performance.
6. Conclusion
We introduce SWE-Lancer, a benchmark for evaluating model performance real-world, freelance software engi- neering work. This software engineering benchmark maps model performance to real monetary value, and improves realism model benchmarks by evaluating more complex full-stack software engineering and management skills. We hope SWE-Lancer enables greater research into the eco- nomic impact of Al model development.
Environments. Unlike real engineers, the models cannot ask clarifying or follow-up questions.
Contamination. Contamination is common limitation of coding benchmarks that rely on open-source repositories. SWE-Lancer Diamond tasks originate from public GitHub issues between 2023 and 2024. Depending on model cutoff date, contamination in training data is possible. Moreover, because tasks are posted online, a model with browsing capabilities or an internet search tool can look up solutions. Therefore, when running this eval, is important either disable browsing or limit sites accessed to prevent direct- lookups, and to ideally also post-hoc filter for any instances of cheating. This will help ensure evaluation results measure real capability rather than contamination. Note that Table 4 in Appendix A2 shows no clear performance improve- ment for tasks predating the models' knowledge cutoffs, suggesting limited impact of contamination for those tasks.
5. Future Work
Economic analysis. Model performance SWE-Lancer maps to real-world economic value; each task in the bench- mark has an associated payout that was sent to a freelancer via Upwork. We hope this benchmark enables research into the broader societal impacts of autonomous agents, including impacts labor markets, productivity, and the acceleration of AI R&D.
Multimodality. Our current evaluation harness does not support multimodal inputs; however, the majority of Ex-

---

SWE-Lancer
Impact Statement
Al models with strong real-world software engineering ca- pabilities could enhance productivity, expand access to high- quality engineering capabilities, and reduce barriers to tech- nological progress. However, they could also shift labor demand-especcally in the short term for entry-level and freelance software engineers-and have broader long-term implications for the software industry. Improving Al soft- ware engineering is not without risk. Advanced systems could carry model autonomy risk self-improvement and potential exfiltration, while automatically generated code may contain security flaws, disrupt existing features, or stray from industry best practices, a consideration that is impor- tant if the world increasingly relies on model-generated code. SWE-Lancer provides a concrete framework to start tying model capabilities to potential real- world software automa- tion potential, therefore helping better measure economic and social implications. By quantifying Al progress in soft- ware engineering, we aim to help inform the world about the potential economic impacts of AI model development, while underscoring the need for careful and responsible de- ployment. To further support responsible Al progress, we open-source a public eval set and report results on frontier Al models to help level-set the implications of Al progress. Future work should explore the societal and economic impli- cations of Al-driven development, ensuring these systems are integrated safely and effectively.
Acknowledgements
This evaluation would not have been possible without the help of those OpenAl who work on foundational sys- tems used by teams across the company. In particular, we would like to thank Evan Mays and Kevin Liu for underly- ing evaluation infrastructure development, Nat McAleese for coming up with the name SWE-Lancer and for valuable feedback and collaboration on the price analysis, Rachel Dias for human data vendor sourcing and setup, Jack Ed- wards for open-source codebase cleanup and price analysis support, Grace Kim for task validation support, Oliver Jaffe, Jack Edwards, Giulio Starace and James Aung for support with rumning open-source models, Patrick Chao for plot- ting library development, Olivia Watkins and Philip Guo for valuable feedback on the price analysis, Phoebe Thacker and Mia Glaese for human data team leadership, Olivia Watkins, Patrick Chao, Miles Wang. Leyton Ho, Andy Ap- plebaum, Leon Maksin, Joel Parish, Kevin Liu, Aleksander Madry, and Alex Beutel for feedback earlier drafts of the paper, the many software engineer annotators for quality- checking tasks, the OpenAl Preparedness, Safety Systems, and Research organizations for sponsoring this work, and the Upwork and Expensify developers and contributors for open-sourcing their original process.
10

---

SWE-Lancer
References
Expensify. Issue #41239; Add support for copy/pasting im- ages ios. ptttps://gitthe .tm//cccmmmcmmmmm Expens fy/ 2024b.
Anthropic. Responsible scaling policy.
Tech-
App/
nical report, Anthropic, October 2024. URL
ht
Gupta, V., Fernández-Crehhut, J. M., and Hanne, T. Free-
m/
lancers in the software development process: A sys- tefhadee mapping study. Processes, 8(10): 2020. doi: 0..3390/p:81012155 BRL httth:://///tt mdpi com/ foon/2227-97178///
Enthropic-Respono Sewelle--Saainn---e Powpic-lllllee
pdf
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language mod- cls, 2021. URL https: pestps://arxiie http//axivv... www.iiiw.ggg..oooooooooooo halg///b///// a210 07732
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E.,, Burns, C., Puranik, S., He, H., Song. D., and Steinhardt, J. Measuring coding challenge competence with apps, 2021. /tt.httsss//aaxvvv.ooo 2105.09938 09938
Brar, H. K. and Kaur, J. Differentiating integration testing and unit testing. In 2015 2nd International Conference on Computing for Sustainable Global Development (IN- DIACom), pp. 796-798, 2015.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T. Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodehench: Holistic and contamination free eval- uation of large language models for code, 2024. URL
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y.. Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M.. Khlaaf, H.. Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, I.., Bavar- ian, M. Winter, C., Tillet, P. Such, F. P., Cummings, D., Plappert, M. Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N. Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J.. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Me- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained URL https: http://axxvvvv..aaa///
Jimenez, C. E., Yang, J., Wettig, A., Yao, Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
/ 000.00////b//1000000000 06770
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong. R., Zettlemoyer, I.., tau Yih, W., Fried, D., Wang, S., and Yu, T. Ds- 1000: natural and reliable benchmark for data science code generation, 2022. URL https://arxiv..
abs 22122
Lam, M. S., Teoh, J., Landay, J. A., Heer, J., and Bernstein, M. S. Concept induction: Analyzing unstructured text with high-level concepts using lloo.. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 24, pp. 1-28. ACM, 2024. doi: 10.1145/ 3613904 URL http: //dp::::://.../// doi org/10 1 34453613904444 3904 3642830
on code, 2021.
2107. 03374
Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sher- burn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang, J., Liu, K., and Madry, A. Introducing swe-bench verified. arXiv preprint arXiv:2407.01489) 2024.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d' Autume,
DeepMind, G. safety framework, 2024.
Introducing the frontier
URL https:
C. Babuschkin, I., Chen, X., Huang, P.-.., Welbi, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition-
the
framework
Expensify.
zip/postcode validation error message not displayed for entering '," on the home ad- dress screen. http:::/ttts/ttttt thub. cwm/ coom/Epensiify.
Issue #14958:
level code generation with alphacode. Science, 378 (6624): 1092-1097, December 2022. ISSN 1095-9203. doi: 10. 10.11126/sciennc.aag11555 URL hhttp://dx... abq1158
happissuee/14995 2023.
org
Expensify. Issue #25889: Dev: Share code avatar dif- fers from profile avatar. https: //tpt/gg.ttt//////// com/
Liu, T., Xu, C., and McAuley, Repobench: Benchmark- ing repository-level code auto-completion systems, 2023. /
PrppensifyAApp//ii httppp//sssusse/555 2024a.
03091

---

SWE-Lancer
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C.,, Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N.. Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning bench- mark dataset for code understanding and generation, 2021. URL http:s///aaxii..orr/bbb/////////
Tian, M., Gao, L.., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong, H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y. Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E.,, and Peng, H. Scicode: A research coding benchmark curated by scientists, 2024. URL
2407.13168
Mierosoft. Playwright: Fast and reliable end-to--nd test- ing for modern web apps. dev/ 2025.
Yang, J., Jimenez, C. E., Zhang, A. L.,, Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S. I., and Press, O. Swe-bench multimodal: Do ai systems generalize visual software domains?, 2024. URL https://
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q. Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models, 2024. httph::t////xiiiiie
arxiv. org/ abtg//ab//// 2410. 03859
abs/ 2bs/2308.0 07124
Zhang, S., Zhao, H., Liu, X., Zheng, Q. Qi, Z., Gu, X., Zhang. X., Dong, Y., and Tang, J. Naturalcodebench: Ex- amining coding performance mismatch humaneval and natural user prompts, 2024. URL
Nijkamp, E., Pang, B., Hayashi, H., Tu, L.., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi turn program synthe- sis, 2023, URL httpp:://arxiv.oom org/ /220
org/ abs /00/a/2405.000 04520
13474
Zhong. R., Zhang. P., Li, S., Ahn, J., Klein, D., and Stein- hardt, J. Goal driven discovery of distributional differ- ences via language descriptions, 2023. URL https
OPenAL.
preparedness framework, openai pwwpareddeessee...
Openai
2023.
URL
//arxiv. http//bb//232// .00.3022.3333
pdf
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R. Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brun- ner, S., Gong, C., Hoang. T., Zebaze, A. R., Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, 7., Ya- dav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q., Wang, Z., Lo, D., Hui, B., Muennighoff, N., Fried, D., Du, X., de Vries, H., and Werra, L. V. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL https
OpenAL. Learning to reason with llms, 2024a.
OpenAI. Openai o3 and o3-mini- days of openai: Day 12, 2024b.
OpenAL. OpenAl Authentication API Documentation, 2025. URL httts:://pllt.fomm.ooo.. wpw.aaaaaooooo.ooooo.ooooo c docs / /api
Paul, R., Tsai, W. T., Chen, Y., Fan, C., Cao, 7 and Huang, H. End-to-end (e2e) testing and evalu- ation of high-assurance systems. In Pham, H. (ed.), Springer Handbook of Engineering Statistics, Springer Handbooks, pp. 439-467, Springer, London, London, 2006. ISBN 978-1-85233-806-0. 978- 18811-84628-288-1 24. URL https:/ /ttp://doi.oooo /ogi.org/
www.xi..rg/abs.... 15877
doi: 10.1007/
10. 10.1007/773 1 84628 288 1 24
Quan, S., Yang, J., Yu, B., Zheng, B., Liu, D., Yang, A., Ren, X., Gao, B., Miao, Y., Feng, Y., Wang, Z., Yang, J., Cui,
Z., Fan, Y., Zhang, Y., Hui, B., Lin, J., and Qwen Team,
A. G. Codeelo: Benchmarking competition-level code generation of Ilms with human-comparable elo ratings.
arXiv preprint 20X112250.00015 2025. URL https: 7/arxiv. org/ abs 2000 01257
Si, C., Yang, D., and Hashimoto, T. Can Ilms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL buttps://arx: org/abs/2409. 04109
12

---

SWE-Lancer
A. Appendix
A.1. Pass@1 Rate Analysis by Category
Table 3 shows pass rates for SWE-Lancer Diamond categorized by the tasks' nature of work.
Table 3. Diamond Task Pass Rates by Nature of Work
IC SWE
SWE Manager
Task Type
GPT-4o
ol
Sonnet 3.5
GPT-4o
ol
Sonnet 3.5
Bug Fixes
9.6%
19.2%
28.4%
208
34.7%
39.9%
44.0%
New Features Enhancements
0.0%
4.8%
14.3%
21
69.2%
69.2%
61.5%
13
Maintenance, QA, Testing, or Reliability
0.0%
12.5%
0.0%
8
75.0%
50.0%
50.0%
4
To determine each task's category (as shown in Tables 2 and 3), we had ol review each entire Github issue thread, including all descriptions and comments, three separate times for each IC SWE and SWE Management task. If these three categorizations were not unanimous, we manually assigned the final categories.
A.2. Contamination Analysis
Table 4. Diamond Pass @1 Rate Analysis Based on Task Creation Date.
IC SWE
Manager
Model
Cutoff Date
Pass@ 1 (Before)
Pass@1 1 (After)
Pass@ (Before)
Pass@1 (After)
GPT-4o
2023-10-01
0.038 (n=79)
0.101 (n=158)
0.373 (n=142)
0.244 (n=123)
o1
2023-10-01
0.139 (n=79)
0.177 (n=158)
0.387 (n=142)
0.407 121)
Claude 3.5 Sonnet
2024-04-01
0.178 (n=118)
0.345 (n=119)
0.452 (n=210)
0,436 (n=55)
A.3. Further exploration SWE Management Tasks
Within SWE Management tasks, the winning proposal is often selected for nuanced reasons, such as addressing edge cases or reducing external dependencies. For example, one task (Expensify 2024b) involves selecting proposal for implementing image-pasting iOS. The winning proposal handled multiple clipboard formats (e.g., base64, blob). minimized permission prompts, and aligned with native iOS behavior by enabling direct pasting into the compose bar. Other proposals were deprioritized due to reliance on third-party libraries, complex modifications, or inconsistent cross-platfore support.
13

---

SWE-Lancer
Example: [$20000] Add support for copy/pasting images on iOS #41239
Description (paraphresed): Users want the ability to copy an Image to their clipboard and paste It directly Into chat in the Expensify iOS app. Currently, users must save the image and manually upload it, which is inefficient.
IC SWES submit proposals
original bounty: $250 (April 30)
bounty increase $500 (May 16) $1K (May 23)
final bounty: $2K (July 9)
Proposal by @ShridharGoll
Proposal by (suunaox
...Poooosall by @s77rtt
Proposed Changes& & Optional Images Add option Paste image' in the only there's image copied oipboard.
Proposed Changes & Optional Images Use case Handling bese64 image data
Proposed Changes Optional Images
RCTUITextVier
Patch canPactezntccion: mathod
force show the Paste option the clibbaad has an image
Clicking option open the attachment modal with the image.
2. Patch pastes method to call poxt-IpptDDiPaass Implement texcIngu:DicPaste this method will just serd event to component viaw via onPaste prop Android: ReactEditText
method
Fatch onTaxtCantaxttaan
method
have displ Lyt lenieeaallll we'll add Eetptayfteeeeeeee Uisingitt which have uri of the copied mage as input. New option in attachment picker:
call tcsteMatcher
2. Implemant PastoMktcher, this class will just send event to the component view via onPaste prop
App: Composer
Use case 2: Handling images copled as
Add onPaste prop and call Clipboardl.c LinggenN / Clipb
bloos Ishortened to (R)
mand..geiinage
Update: rrassd a PA that covers changes... leaving minor
proposal adds Image" images,
proposai detects alppbard content baseed strings blobs, converts objects, enables users to intuitively througn long-press gestures or 'Paste image" button.
This propesal components comprehersiveyy image paste support Compaser both (OS Android.
copied clipboard opens images In the attachment modal for sharing:
menu, format, stores
SWE Managers provide feedback proposals and ultimately accept reject them
Menager Feedbback: 'Pesting image from the the clipboerd seams workaround, because when we copied the Image, we click "capy image" not "copy Image addresss so wa may need to handle the image base84 string.
Manager Feedbeck (after some and forth): think your mothod detect onPasteFile and make the paste opoion appear after copying an image from the gallery is unveliable If Clipboard.get gew.rgggggggggghhi ) Android, and Clipboard. Tengge IInage() hancle anPasteF11 nativaly in Mar karkdownTaxtinn and use Cliphoard to get the image content. What do you think?
SWE Manager Feedback: "Can you share test branch? Also are we able to use the same Android, Clipboard Android"
us fieOtjessl
Clipboard. serooorrddgggttinngeeeo getgng
Also, your proposal doasn't have how we will detect tolle in the normal by long clicking the input then pasta"'
IC SWE: Yes, I've updated the proposal.
X Proposal Rejected
Proposal Rejected
Proposal Accepted Solution comprehensive and native, handling all cases. it handled gallery and web images as raw image content, and supporting iOS and Android.
Solution was not comprehensive. only worked for Image URLs, which uncommon, and falled to support raw image data.
Solution was not comprehensive. it handied images copied from the gallery but failed to support web Images.
Figure 11. Real-world illustration of the iteration process for submitting and choosing solutions to Github issues. This task was incorporated into our SWE-Lancer dataset.
14

---

SWE-Lancer
A.4. Coding and Software Engineering Evaluations Comparison Table
Table Comparison of SWE-Lancer existing coding and monetary metrics, incorporates full-stack tasks and management tasks, and evaluates using execution based end-to-ent by engineers. In contrast, other benchmarks rely pre rre-xxiiting tests within repositories or tests generated by LLMs.
benchmarks: SWE-Lancer uniquely maps model performance col-to-ent end tests aand-crafted
Evaluation
Langnage
Dataset Source
Data Type
Capabilities Tested
Monetary
Metric?
Evaluation
Method
Evaluation
Source
Method
Open-source and commercial
Professional
Hand written by professional software engineers
SWE- anser
Javascripe Typescript
Expensify repo of issucs posted on Upwork
software engineering
Software engineering
Fnd tolttt--eni end test
problem statements
Gub issues from
SWE. Bench Verified
Python
Open GifHub repox
all developer frameworks
maintenance
Unit lest
original repository
Software maintenance tasks using multimodal
GidHab issues focused on developer tools and utilities
Jnvascripe Typescript/ HIML/ CSS
SWE-Bench Multimodal
Open-source GitHub repos
*
Unit test
Soureed from original repository
data
GitHub issues from Python repositories
that meet costain criteria (posi Jan '23, fork, 100+ stars, 808++ Python,
Unit test, similarity metrics
Open-source Python
Code completion
Sourced original repusitory
RepoCoder
Python
tests)
Github--C Sauuuub-oode Dataset and newly crawled GitHab data (post-Feb 2023)
Python and Java
Single and multi code snippets
Code retrieval and completion
Similarity metrics
RepoBench
*
N/A
Coding problems designed for competilive programming competitions
LeesCode. AtCoder, and CodeForces
Code generation
Generated using L.I.ss
LiveCodeBBnnch
Python
Unit test
Problem statements
User queries from online coding services
Generated using GPT-4 refined by humans
NaturalCodeeeench
Python and Java
derived from real-world user queries
Code generation
Autemated
unit resting
Natural language
Generated using
BigCodeBench
Python
Human expert
curation
tasks composed
maltiple substeps
completion
Code
Unit
GPT-4 refined
by bumans
HumanEval
Python
Hand-written by humnan experts
Python l'nnctions
Code
*
Unit tests
Hand-wiittnn by experts
generation
Python/ / Java JSS/ PHIP/ Ruby Go C/CH)
CodeXGLUE
Gutllu repos &
Open-source
14 Lask-specific codeltext datasets
undersianding
Code
*
similarity
Unii
Benchmark authors (Ia et al., 2021)
newly sets
generation
metrics
Crowd soureed, are expert verified
Short eatry-level Python l'ncctioss
Code
МBPP
Python
*
Unit test
Crowd sourced
completion
Solving mathematical ward
Python
Converted from MathQA
Sourced from MathQA
MatiQA. Python
Python
prohlems using
program
Corretnnsss
syathesis
Python
AlphaCode
C++/
Scraped from Codeforees
Competition coding problems
Code
*
Unit test
Scrapod from Codeforces
Pytho//
generation
Multittrn coding problems and prompts
Human expert
Code completion
Human expert
МТТB
Python
Unit teat
written
written
Collected from StackOverflow
Data science problems
Callected from StackOverflow
DS-1000
Python
Fixing code
Unit test
Comparing Learnings Against Prior Benchmarks: For SWE-Lancer's IC SWE tasks, one comparable benchmark is SWE-Bench Multimodal, where GPT-4o solved 12.2% of visual programming tasks when given the best scaffold and multimodal capabilities. This is comparable to GPT-4o scoring 8.0% on our IC SWE Diamond tasks when the model can ingest image screenshots via the user tool. On SWE-Bench Verified, the industry standard benchmark for software engineering, GPT-4o currently scores 38.8% using the best available scaffold, but only 8% on SWE-Lancer Diamond IC SWE tasks likely because SWE- FrE-Bench Verified focuses on bugfixes, and SWE-Lancer includes open open-ended problems (adding new features, reliability) and full-stack development. This highlights the need for more challenging, unsaturated
15

---

SWE-Lancer
SWE benchmarks.
A.5. Additional Scaffold Details
Each agent is executed within Microsoft Azure Standard_ virtual machine with vCPUs and 8GB RAM. We execute agents within Docker containers using a pre-built SWE-Lancer image, which we open-source. The container has 64 GB of shared memory allocated and 192GB of maximum memory available. All the libraries the agent can access (i.e., the ones relevant for the Expensify repository) are installed in the container.
The agent does not have multimodal image or video inputs. For each rollout, the maximum number of tool calls is 100 and the maximum time allowed 3 hours. All rollouts are conducted with temperature 1.0.
Except for our pass@k @k experiments, each is just a single rollout.
A.6. Task Curation Criteria
Project maintainers post Github Upwork
Users in-depth proposals
Problem, Description, Price
Root
problem
winning proposal submitrrr demonstration thei solution, discuss dependenciea, etc.
Winning propesal is implemented
Example: [52000] Add support for capy! pasting images on 108 #41289
Need add now feature
Codebase
Proposed solution
Papanelyylgn
Add new option "astt image" in menu chats, using the code below
Abnored
/besitop
Aos
Used for IC SWE Tasks
Asriits
Aveuba
Manegement Tasks
more...
Figure 12. Lifecycle of an Upwork task. First, project maintainers post a software engineering task to Upwork with an associated payout. Then, users submit in dn-tepth proposals for fixing the issuc. Finally, assigned reviewer selects the best proposal (used to create SWE Manager tasks). The winning proposal is implemented (used to create IC SWE tasks).
We manually filtered tasks according to the following criteria. Each task in our final set was screened by at least three experienced human software engineers.
1. The issue is sourced from an open-source repository.
2. The issue has associated, concrete payout.
3. The issue was resolved in the past.
4. The issue is well-specified and clear models must be able to actually understand the task and implement fix, without being able to ask for clarification.
5. The PR that resolved the issue actually fixes this fix is confirmed by executing locally.
There are additional criteria for SWE Management Tasks:
1. There
least well-defined proposals for the issue.
2. The proposals considered contain sufficient information about the correct fix.
Further detail is provided below on how we assess specific criteria and labeled other attributes for each given task.
To confirm that the issue was well-specified and clear, we asked software engineers to rate each issue from 0-3 (below) and only accepted issues where the majority human engineers rated them as
Read the issue description again here, and imagine that you experienced software engineer with 2-3 years of relevant experience and familiarity with the codehase. You have been instructed full access to the codebase and can use the title, description, reproduction steps, and any attached videos/images from the linked issue when forming your solution.
that successfully resolves it. You have
Please note, however, that you are not able to ask for clarification and would need to work exclusively from the above information. Please do not read the comments below the issue when making your decision-yoo must rely solely the content the issue itself and codebase/does.
16

---

SWE-Lancer
Please assign rating from using the rubric below,
0- The issue is well- sell--peeified. and it is clear what is required for successful solution.
1- There some blanks to fill in about the issue, but there is sensible interpretation of what is required for successful solution.
The issue is vague, and there room ambiguity. It unclear what successful solution would look like.
is almost impossible to understand what you are being asked to do without further information.
Issue Well-Specified Score: [0 to 3] Please explain your choice above. Include references to specific filenames, function/- class names, lines of code where relevant. [Text box]
To confirm that the tests were valid for all reasonable, correct solutions, we asked engineers to rate the tests according rubric and only accepted issues where the majority human engineers rated them as
Please re-review pull request that addressed the issue. Does contain valid, well-scoped tests so that reasonable solutions pass?
Please assign rating from to using the rubric below.
0- The tests perfectly cover all possible solutions.
1- The tests cover the majority of correct solutions; however, some unusual solutions may be missed.
The tests work, but perfectly reasonable solutions may be missed by the tests. tests are too narrow/broad they look something different than what issue about.
A.7. SWE-Lancer Robustness to Cheating
Existing SWE benchmarks in the literature often rely unit tests that are narrow and prone to cheating, An example of cheating by ol SWE--Bench Verified shown below:
An excerpt from the problem description is included below:
Queryset raises NotSupportedEEror when RHS has filterable-Fals, attribute.
In response, ol adds an underscore before filterable so that the branch never executes:
2: ASSISTASTT ALL
django/db http:/d/mmeeeeee//eeeeeeeeeeeeee
<<<<<< SEARCH
if not getattr (expression, 'filterable', True):
raise NotsupportedError
expression._ class _aab____lss____________ name + "_is_disallle fie filterr 'clause.'
if not getattr (expresiion, '_filterable', True): raise NotSSupporteEEroo
expressiion__class:: name_ + '' dis_disal lowed_in tetttheffltett clause.''
>>>>>> REPLACE
As language models grow more capable, is increasingly important to publish challenging, un-hackable software engineering benchmarks. SWE-Lancer's numan-validated end-to-end tests emulate the actual workflow of human engineer, rather than relying on a handful of selectively chosen test cases. As a result, SWE-Lancer is inherently more resistant to cheating and can more accurately reflect a model's 's capacity to produce genuine, economically valuable solutions for real-world engineering challenges.
17

---

SWE-Lancer
A.8. Prompting
Models are prompted with the following general template for IC SWE Tasks. We did not find variance in performance when including vs. excluding the prices from the promtp.
18

---

SWE-Lancer
SWE Task Instructions
"""You are expert software engineer maintaining the Expensify fy repository. It has already been cloned, and is present in /app/expensify (your CWD). You have been tasked with fixing the following issue:
(issue title)
(full HTML documentation of issue, including how to reproduce the bug, the actual expected behavior, and which platforms are affected)
The reward for the correct fix is $(price).
You do not need to push your changes to the repository, we will deal with this for you.
not attempt run the app; simply make your changes and then end your turn.
Do not end your turn until you have submitted your code changes.
You absolutely must not leave action items TODOs.
You will be provided with user tool, which you can call by running "bash -i -o uuser=tool"" in the command line, which has user open the app with your local changes and perform the actions specified in the bug report. It could take up to a minute for the user to finish using the app. When you call the user tool, when the user finishes hhtp/expensi.fyuusrr_oooo
will be populated with folders called output_browserl browsers as were used by the user during their test.
n for as many Inside each folder, you will find a file called trace.trace, which contains the trajectory the user tock. Here are few things to note:
You will need to parse this file programmatically in order to open it. then the "shal" property
If a row has "type":"screen""" "screencttcen will correspond to a ipeg file in the resources folder, which is a screenshot of the browser at that time.
If you want view the page as HTML, rather than a JPEG, you can look for rows that are called "frree":rrrae"ssapshoo""" you can print them out to see the HTML at this point in time.
You should review the rows with type "type" set to "log", "before", or "after" as they are important and show what actions are being taken.
All of the rows are in order, so e.g. you can find an interesting log row and then look at the next row that is frame-snapshot to see the HTML.
should this to help you iteratively debug and fix the issue. You can call it as often you need to.
When you call the user tool, you wait 90, 000 milliseconds (e.g. by setting the timeout argument to 90000) mun
Models are prompted with the following general template for SWE Management Tasks.
19

---

SWE-Lancer
SWE Management Task Instructions
E"" are an expert software engineering manager working on the Expensify repository. You have tasked your team with addressing the following issue:
(issue title]
(full HTML documentation of issue, including how to reproduce the bug, the actual vs. expected behavior, and which platforms are affected)
You will be paid 'price') if your team fixes the issue. Since you are manager, your task is to select the best proposal regarding how to fix the issue from the below options:
(list of all proposals}
For your convenience, the Expensify repository has already been cloned, and is present in /app/expensify (your CWD) You should not attempt to run the app or implement the entire fix, but you can review the codebase to understand the context of the issue.
When you are ready to submit your answer, please create a JSON file in happ:/rreess.tt/mmmaggee..eddd
json with the following format:
int,
You must write your output to this path exactly: htapp//xpensstyymmmaeee.. json'. You must not submit to anywhere /mnt/ just write to htap://eeensstt/maaaneeeee.ee json' exactly!
A.9. Qualitative Analysis of Rollouts
We conduct qualitative analysis of task instances from SWE-Lancer, focusing model-generatee solutions and debugging strategies. The analysis is based on agent trajectories under the default configuration of o1, 4o, and Claude 3.5 Sonnet.
Issue Summary. The Expensify app fails to display a validation error message when users enter a comma (.) in the zip/postcode field on the Home Address screen, allowing invalid input. This issue occurs across multiple platforms and may stem from missing input sanitization or incomplete validation logic. The bug was observed staging and production, potentially introduced during the N7 Account Settings update (Issue #1 #14958).
Note end-to-end tests. The end-to-end tests developed for each issue robust and reflect the key considerations from the original Github discussion. While this issue initially concerned the commas in zip-code validation, much of the discussion in the Github thread focused on improving validation overall, including country-specife regex dictionaries and handling edge cases where the user hasn't selected a country or a country lacks a postal system. Thus, our end-to-end tests measure not just whether the model's code causes the validation emror to appear when entering a comma, but also whether can comprehensively handle the full range of global zip-code formats.
20

---

SWE-Lancer
ol Interaction Summary
1. Locate the Target File
The model begins by scanning the directory structure to find relevant files:
!1s -1 /app/ expensify
searches for zip-related validation logic:
!grep -irn "zip" /app/expensify
To find the isValiddippaoo function, it narrows the search:
!grep -irn "ValidationUtills /app/expensify//rre I grep -i zip
!grep -irn "isWalidZipCode" /app/expensity/sro
The model identifies the right file to modify and sees that we use regex to validate zip code:
grepp - "function isValidRipCode"
The model finds a reference to AddressForm. ssx aims to understand how the zipPostCode field being processed and how zip code input values are sanitized before being validated or used
grrep -C 40 " "zipPostCode?. " trim" /app/ http//exepnssf///sseeeeesseeeese
components AddressFormm.
2. Search for the Issue
The model looks for where ZIPCODE appears in the codebase. Through this, it finds the regex used to validate US zip codes.
ggeep -irn "ZIP_CODE:
/app/expensi fy/sr
The model looks more to see how textttAddressFomm is used in the overall codebase, and specifically how zipcode is used.
Igrep - 50 50
import AddressForm" happ:/expessitt/ssr//pppappe AddressPage.tsx
3. Diagnose the Validation Logic
The model reasons through the root cause of why a validation error does not display when a user enters comma in the "zip code" field in the "Home Address Screen", and concludes that the likely issue is that the InputWrapper for the zip code field bypasses the form validation logic by using onVa lueChange. The error is being set the validator but not displayed due to either: 1) missing triggers for validation during typing (enChange) focus loss (onBlur), or 2) Mismatched field IDs causing the validation result not propagate correctly.
It hypothesizes that the fix involves removing the manual state handler for the zipcode field, the "Home Address" form now relies solely on the form's built-in validation logic (via the va idator prop in the FormProvider), and entering comma in the ZIP/postcode field will immediately show the "incorrect zip format" error.
(a) Validation is not triggered dynamically (e.g., on typing or focus loss).
(b) Mismatched field IDs (zipPostCode vs. Sitbress2ipCooe may prevent the error from display- ing-
4. Modify the Code
21

---

SWE-Lancer
The model makes patch, removing the custom onVa ueChange so that the form's natural validator invoked and the zip/postcode error now displays expected when the user enters comma.
diff --git a/sstt/conppnnnttt/Adtssssssssss
index 71928c1 ce425c 100644
a/src/ htaa://cmmmpeeett/dddeeaaaoo.mm
+++ b/stc/ccmponete htstt//ccmmoneett/dddesssammmaa
@e 7 +182, @0 function AddressForm (1
/>
< </View> <View
/>
(/* Remove onValueChange so this field uses the
form's s standard validation x/}
<InputWrapper
Input
LnDutID=-INNFTI_DSS. ZIP 2tt___DDS.2I_PPPS_CC____1
00 7 +189, 00 function AddressForm (1
orValueechangeessseelpoogg defaulttaale===(ippcoee)
5. Clean Up and Submit
The model submits the patch, believing the issue is resolved.
The model's fix addresses the basic issue of the
showing up when the types comma, but it does not fully address other issues which were captured in the real PR.
Evaluation. ol successfully identified the relevant files, including Val Salidatiin innut ils. t and AddressF o orm. t sax, and correctly recognized that by removing onVa lueChange could fall back to the form's normal validator, which would solve this particular issue. However, ol's solution was not fully comprehensive and did not pass our end-to-end test. didn't 't address other issues which were all addressed in the pull request and Github discussion:
Missing Validation Triggers: ol's patch does not enable validation on key events like onBlur or onChange, which is necessary for providing immediate feedback to users. The real pull request added logic validate dynamically when the user interacts with the field.
No Sountry-Specifie Validation: ol's patch does not handle country-specifis specific zip code formats or provide tailored feed- back. The pull request implemented dynamic validation based on sountry-specifie regexes and included z pF rmat hints to guide users.
Does Not Address Non-Zip Countries: ol patch overlooks cases where some countries do not require a zip code. The pull request included logic to handle these scenarios, such as disabling or hiding the zip field when applicable.
Error Message Improvements: ol's patch does not enhance error messages for clarity or localization. The pull request ensured errors were dynamically adjusted and localized based on the user's country and language settings.
4o Interaction Summary
1. Locate the Target File
The model scans the directory structure to find relevant files:
os listdir (" h/aapp/xxeeessfy//
22

---

SWE-Lancer
navigates to the source code:
sre_path = "/app/eexeesiff//rre"" os bi.iistiir (src_path)
After multiple iterations, it finds and opens AddressPage.
addreas_page_pptth
os. as.path. (pages_pataee "AdrressPgeetssx (pepth.. with open (addreess_page__atteee 'r") as f:
f.read
print patpssss_spppe__otteeeeeeeee [:2300]]]
The model searches for AddressForm, which likely contains input validation, directing it to Val i calidatioon dedaaaintilssttee........ where the zip/postcode validation logic is located.
2. Search for the Issue
The model searches for postcode-related validation within Va iidationUe inntt ls. ts a by writing a Python function.
finds the iwtalisdiiciiiico (zipCode: string) function and its associated regex for validating
zip codes.
The model then examines CONST. ts to locate the regex:
US:
regex: / [0-9] (5) (?: (- (0-9)
Despite correctly identifying the regex that flags commas invalid, the model notices that the validation message is not shown, indicating the issue lies in how emrrr are handled within AddressForme sxx The model searches for the error message related to zip/postcode validation and hypothesizes it is triggered by:
errors zipPostCode translate
I'privatePersonaaeaaalll.oo iicoorreet22pPormat,, zipFormat: countryZipFormat }
3. Modify the Code
The model decides modify the code ensure that validation errors display for the zip code field.
Instead of fixing the validation logic, it incorrectly modifies the regex itself.
The model identifies and modifies AddressForm. tsx to allow multiple zip code formats, but this change does not fix the validation issue:
88 -3615,7 +3615,7 7 e0 const CONST { GA: (), GB: {
regex:
(0-9000-)0000000000000-------- 9] [A-2-CIKMOV] / regex: / ( [A-2] [R] [0-9A-2]?
\s ([0-9) [ABD-AJLNP-OR-21] ()))
samples: 'LA102UX, BL228FX, BD159LU, WR4G 6LH',
},
GD: (),
0e -4134,7 +4134,7 7 00 const CONST =
UG: (),
UM: (},
US: {
regex: /=[0-9]]5) 1101)]0(:21110999-111111 (4)-)) regex: /=6_1[0-20-20-991111100111
samples: '12345, 12345-1234, 12345 1234',
23

---

SWE-Lancer
UY: O,
4. Attempt to Use the User Tool
The model attempts to test the fix using the user tool but struggles with execution. does not verify if an error message appears in the UI, failing to confirm whether the issue is actually resolved.
5. Clean Up and Submit
The model submits the modified regex, believing the issue is resolved.
However, the fix does not surface error message when a enters comma, meaning the original problem persists.
Evaluation. GPT-4o followed a similar approach ol, correctly identifying the necessary files and analyzing the regex pattern. However, it also overlooked the UI mechanisms responsible for displaying validation errors, failing recognize that the issue stemmed from how validation logic was triggered. Instead of modifying the validation logic, GPT-4o attempted to adjust the regex, which did not address the core problem. The model conducted minimal testing and did not verify whether error messages appeared as expected in the UI. It also did not attempt full form submission to validate its fix, limiting the effectiveness of its approach.
Claude 3.5 Somnet Interaction Summary
1. Locate the Target File
The model scans the Expensify repository:
1s -la /app/ /eep//expsiiff
then searches for files related to addresses and validation, writing function to so:
import os
def search files (directory, keyword)
matching_files for root, dirs, files in os wa.walk (directory)
for file in files:
if keyxord.lower/ in file. lower ()
matching_fille atp.a..peeeeeeedeeeeee.. (os path, join file))
return natching_files
address_files fi http:///we..aa/gooooooo////////a 'address')
print ("Adrese-relate__flle hotpleach....ooooottttttccooooor for file in address_files: print (file)
print ("
validation files search files /tpp://ppnsiii///c///oooooo cttp:///sstcccmmoonstmssssss for file in validation. files: print (file)
'validat')
2. Identify the Validation Logic
The model identifies AddressForm, tss as where validation logic is implemented. It reads in the content to find where zip code postcode validation is implemented, and then uses regex to find the specific validator function.
import re
with open (' http://pppppiaia/a////////////// content file. read ()
(x') file:
24

---

SWE-Lancer
locate the validator function
val walliddtor_aaccee compile (r' lusetorr__sseaalabaae 11backk ((.+?) \s*=>\s {((++)))
re.000ALL))
validator_maten match validator_paatt pattern search (content)
if walidator__ateeet match:
print Divallddtoor,funciiii "
else:
3. Modify the Code and Submit
The model modifies the validator function to enforce the no comma rule, generating the following diff. This modifies AddressF orm. tax to ensure that entering a comma the Zip Postcode field triggers a validation error message, by adding a new validation rule that specifically checks for a comma.
Although this fixed the specified bug, the zip code validation logic is too restrictive, and doesn't 't support international zip codes which are checked for in our end-to-end tests.
diff --git a/sre///mmmmmeess//tttttsssfo index 4470481d2be hcw.dd20 100644
htwptt/ttttttcccccccocccco. +++ b/src//omppone http://coaapootttte/eeeeeeeteeem /wt.raarrsso.............
00 -118,7 +118,10 e0 function AddressForm({
arrotttil..dddrrroossssgggggoooo 'firstName', translate (?
cantAccont.errror.fftttttaaa
if (couttrySperiicciippeeex
(valuess.ziPosttodde &4 values zipPostCode includes ()) I
errors. - translate t (pprirateeerroolllloaallll.....
.noCommaAllowed)) ;
else if (countryspeclficclpeeeex
(! [ecouttryySeccc:ii./.ppeeeex test (values. zipPostCode? trim()
tcUpperrCaee
if waliorrtoottiie-BBeutttttuuuuuud (values.
alsPost(oode..trimm))
ezrors. translate('
privatePPrssona Detaiin ssror.r. I incorrect2ipPormat!
(ziphormat: countryziprorman
00 -126,8 8 +129,11 08 function AddressForm({
zipPostCode translate (" common error.
fieldRequired')
else if (:CONST.GENERIC, GENERIC CEATIST.GEEERRCCPP..OLC 21P_CODE_REGEE test (values?.
zipPostCcde?. () ? tolpperrCas ?? ())
errors errors.zipPostcom zipPostCod = translate ('priiaateeerronlllltalllll.ese
LicerreettiiFormatt
return errors;
Evaluation. Claude 3.5 Sonnet was extremely quick to identify validation logic as the root cause of the issue and fixed the specified bug. but was not fully comprehensive and did not pass our end-to-end test. Despite fixing the logic so commas the zip code field now get flagged, and identifying that validation was failing for valid international zip codes, Claude did
25

---

SWE-Lancer
address other issues which addressed in the pull request:
Missing Country-Specifie Zip Code Validation: Claude's patch used a generic validation approach and did not ZIP code formats. The pull request introduced regex-based validation system that dynamically applies the correct format based on the selected country.
incorporate
No Dynamic Format Guidance: Claude's patch did not provide users with real-time hints on the expected ZIP code format for their country. The pull request added a feature that dynamically updates the hint text to show the correct format when a country is selected
Failure to Handle Countries Without ZIP Codes: Claude's solution assumed that all countries require a ZIP code, leading to false validation errors places where postal codes are not used. The pull request correctly allows empty ZIP fields when a country does not have postal system.
Lack of Centralized Validation Logic: Claude's patch scattered ZIP code validation across multiple locations in the codebase. The pull request centralized validation logic into a utility function (ValidationUtils), making it easier maintain and extend.
Inconsistent Error Messaging: Claude's patch did enhance error messages, making them generic and unhelpful. The pull request ensured that error messages now specify the correct ZIP format when validation fails, guiding users to enter the correct value.
Validation Not Integrated with User Actions: Claude's ' solution did not validate ZIP codes dynamically based on user input. The pull request implemented onBlur and onChange validation triggers, ensuring that errors appear in real-time as the user types.
A. Model Usage User Tool
Here we provide an example ol using the user tool to help solve an issue in which an error page appears when new user navigates to a concierge link after finishing the onboarding flow, when instead they should be able navigate the concierge chat.
Example of ol using the user tool
1. Extracting Logs and Snapshots: The model parses t race. . trace to extract logs and frame snapshots from the user tool, to understand key UI transitions and find error messages or timeouts related to the "join" button or concierge navigation.
prace_path = hrtp:
logs []
with open (trace (trace_path, "r", excoding= as E: for line in f:
data ison. loads (line)
if data. get ("type") in ["log", "before", "after"]; apgena (data)
for entry in logs: print (entry)
2. Searching for Missing Elements Snapshots: The model then searches frame snapshots for the "Join" button so it can see if the UI is rendering it correctly. If "join" was missing, could mean rendering issue missing translation. If "join" was present but disabled, it suggested logic issue in the button's properties.
trace_path /aapp/xppens frame Saame_snapshote
with open (trace path, "r", encoding= as F: for line in f: data json. loads (1ine)
26

---

SWE-Lancer
if data. ( ("type") == "frame-snapshot"; frane_snapshots. (data)
snapshots__earr_allll
start_caa call 15 25250. 356
end c5 = 55271.52 523
snap aaame_snapshotee time chap.get ("timestamp")
if time is not None and start_caa start_call_15 <= time <= end_call__s 15:
peaeehot__eer_cll_ss.ppppp (snap)
for 1, snap enumerate her.ppoists.eeerrr........ 15)
html snap, enap.get ( ("snapshot", (1) get "")
if "Join" in htmi:
print (f"Snapshot_ hahaa__ Join' _in_it_)
if "join" in html: print (f"Snapshot_ (1 haas_ _inunninn_tt""
3. Checking for the Join Button's Disabling Logic: The model then searches user tool traces for isDi sabled conditions affecting the button by running
subprocess. run ( (["grep", "-Rni", "Join", "sccc/pages/signnn""
It finds the following logic, which supported its suspicions that the button incorectly disabled due to network status account errors.
<Button
Lboisableee-tttttookk..o.f..... II !account? .message)
success
large
isloadinnf(accoutt
onPress=(1) => Session.signuieenn signUUsser ()
pressOnEnter
style= ([tt([[(syyssbb))) mb2]]
4. Fixing the "Join" Button: The model removed isDi sabled from the code to that the Join button could always be clicked.
5. Fixing the Missing Translation for "Join": The model checked if wel comeS gnupFo rm. join was missing anbd added a transition. Without defined translation, "Join" would not appear on the button.
diff --git s/src//anguaggs//eeee setaaaagaseeeeeeeeee ts s/src/Languages//nn index 1000000..111111 2/srcc/aaguaggs//er 100644
03/src//anguages//eee
88 6 +1,9 eC export default welcomeSignUpForm: join: 'oin',
},
common: cancel: 'Cancel', yes: Yes',
27

---

SWE-Lancer
6. Fixing Concierge Navigation Issues:
The model then examined Conci erge age. tsx and found the app tried to go back when there was no history. It added a check to prevent navigation errors. This helped resolve the root cause of the issue: it prevented the app from breaking when without prior history wanted to navigate to the Concierge page.
diff --git index 1234567. .baddef 100644
b/s b/ssc//pages/conciirrggsssss
--- a/s +++ b//sxc/pagee//onccerggeeeee
88 -35,7 +35, @e function ConciergePage (Isession
ConciergaPageProps]
( sessssonn II !session. seession.uuthooenn {
Session. grgeloon..igotttaacddderrrrtoooo
return;
else if Inavigation.canGoeel
navigation. goBack ; // Only ravigate back if history exists.
7. Verifying the Fixes with the User Tool: After applying all fixes, the model reran the user tool to confirm it worked appropriately.
In summary, o1 used the tool to iteratively debug the issue by extracting logs, analyzing UI snapshots, and identifying key failures. It first ran the tool to collect trace. trace logs, filtering for log, before, and after entries to detect errors and timeouts. It then inspected it was disabled. Upon discovering issues with button accessibility and missing translations, ol modified the UI logic and added missing translations, then reran the user tool to validate fixes. Finally, it used the tool to confirm that new users could successfully sign up and access Concierge without encountering timeouts or navigation errors.
snapshot data to verify if the "Join" button was rendered correctly and whether
A.11. Holdout Set
We maintain a $499,200 private holdout set to mitigate risks from contamination and model browsing. This internal-only set contains $178,475 worth SWE tasks that were augmented by professional software engineers: to construct this split, we held the task description and cases for issues constant, and instructed software engineers familiar with the Expensify repository reintroduce the bug issue in distinct but equivalent way. This approach mirrors real-world software development cycles where recurring issues (such downtime unexpected behavior) have diverse and persistent root causes. As with all tasks, these issues were triple-eerified by professional software engineers for clarity, faithfulness, and feasibility.
A.12. Additional WWE-Lancer Dataset Details
Here describe more details around dataset composition for both the SWE-Lancer Diamond set and full set.
Table 6. Number of tasks by task type within SWE-Lancer Diamond and SWE-Lancer Full datasets.
Diamond
Full
IC SWE
SWE Manager
IC SWE
SWE Manager
Task Type
%
%
n
n
%
Application Logic (Client-Side)
74.26%
176
75.85%
201
75.92%
580
70.99%
514
UI/UX
17.30%
41
18.49%
49
16.23%
124
22.38%
162
Server-Side Logic
7.17%
17
4.91%
13
7.33%
56
5.39%
39
System- Miite Quality and Reliability Tasks
1.27%
0.75%
2
0.52%
4
0.83%
Infrastructure/Deeooss
0.41% 0
28

---

SWE-Lancer
Table 7. Number of tasks by task within SWE-I Diamond and SWE--Lance Full datasets.
Diamond
Full
IC SWE
SWE Manager
IC SWE
SWE Manager
Nature of Work
%
%
%
%
Bug Fixes
87.76%
208
93.58%
248
87.04%
665
92.13%
667
New Features or Enhancements
8.86%
21
4.91%
13
8.90%
68
5.80%
42
Maintenance, QA, Testing, Reliability improvements
3.38%
1.51%
4
4.06%
31
2.07%
Table Resolution time comments count within SWE-Lancer Diamond and SWE-Lancer Full datasets.
Diamond
Full
IC SWE
SWE Manager
Both
IC
SWE Manager
Both
Metric
Resolution Time
25.00
27.99 51.55
26.15
20.33
25.00 47.39
21.47
Comments
41.94
47.01
36.39
41.74
Table 9. Proportion of Tasks in each Price Range for SWE-Lancer Diamond and SWE-Lancer Full
Price Range
Diamond
Full
Under $500
31.5%
44.2%
$500 $1000
33.9%
28.1%
$1000 $2000
25.3%
23.9%
$2000 and up
9.4%
3.8%
Table 10. IC SWE statistics for SWE-Lancer Diamond and SWE-Lancer Full datasets
Metric
Diamond
Full
Mean
Max
Mean
Max
Lines of Code
69
1812
55
1812
Number of Files
2
21
39
Issue Word Count
271
466
270
2842
A.13. Example IC SWE and SWE Management Task Prompts
Below, we share an example, filled-in IC SWE task prompt, exactly as it is given to the model.
Example IC SWE Task
You are an expert software engineer maintaining the Expensify repository. It has already been cloned, and is present in /app/expensify (your CWD) You have been tasked with fixing the following issue:
[HOLD payment 2023-10-09] [$500] new tab opens when dewnloading offline
## Action Performed:
From user Send an attachment to user
2. Be offline for user before opening the attachment
3. Click on the attachment sent from User A
4. Click on the download button at the top
#1 Expected Result:
the inactive download button shown.
29

---

SWE-Lancer
## Actual Result: A new tab is opened
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
## Platforms:
<!---
Chock off any platforms that are affected by this issuc
--->
Which of officially supported platforms is this issue eccurring on?
-[] Android native
-[] Androic Chrome
10s / native
i0s / Safari
[x] MacOs / Chrome / Safari
[]] MacOs Desktop
*Veessoon Number:** 1.3.58-0
Reproducible in staging?:44
**Reproducible in production?:e<
**If this was caught during regression testing, add the test name, and link from TestRail:**
EEmail or phone of affected tester (nc customers):::
*ogg::+* https///rrakkovrrooo wwtp////apppasllly//sssttooooeee
silbttes//hotco///ideeaa: Any additicnal supporting documentation
http://wwww..aaooooo/./////sss/a f
bttps::/gittub.comm/BB f03052651a
#Bxpemssify/Expeesiff Issue JRL:..
AAIssue reported by:** @daveSeife
**Slack conversation:** httpss:/exxensiify. wttp:/i..ww.laa....oooooooooo httpa///..aaaaacc///cc/cceeeee/a p1692409444967919
[View all open jobs GitHub] 1a93.sseeelLaaassssaeeeeaaaaaaaa
<estiile<<umyy>tppuurk Automation Not Edit</sunmary>
<ul>
<li>Upwork Job URL: https:
<li>Upwork Job ID: 18991369888112748///7
<li>Last Price Increase: 2023-09-19</11>
<li>Aut offers: </li>
<ul>
<li>fedirjh
Reviewer 26848952</1i>
<li>daveSeife Reporter | 26848953</1i>
</ul></1i>
</ul>
</details>
The reward for the correst fix is $500.0. 0. You do not need to push your changes repository, will deal. with this for you.
Do not attempt to run the app; simply make your changes and then end your turn. Do not end your turn until you have submitted your code changes. You absolutely must not leave action items TODOs.
30

---

SWE-Lancer
Below, we share example, filled-in SWE Manager task prompt, exactly it given the model.
Example SWE Manager Task
You are an expert software engineering manager working on the Expensity repository.
You have tasked your team with addressing the following issue:
[Paid] [98000] zip /postcode validation error message not displayed for entering ' on the Home address screen
Action Performed:
1. to Settings > profile> personal details >Hoe address
2. type the zip /postcode
## expected Result:
i: should show validation error message
## Actual Result: does not show validation message
# Workaround:
unknown
## Platforms:
<!---
Check off platforms that are affected by this issue
--->
which our officially supported platforms is this issue occurring on?
-[] ] Android native
-[] Android / Chrome
-I ios / native
ios / Safari
-[] MacOS Chrome / Safari
[ Macos / Desktop
Version Number: 1.2.67-7
Reproducible in staging?:**
#*Reproducible production?:** new feature
**If this was caught during regression testing, add the test name, ID and link from TestRail:**
*+maiil phone affected tester (nc customers):::
http:///aaaooolloo/////////////s +Nlttes//hoooos//dddss
http:://ecccooggggoc.cccccccccer -5525fe66e651.mmot
htt.:///w....a.oaoo....oo.oooooa -087aaf5490fc.mg
+ *#Epensi fy/Expessify Issue JRI:.*
**issu reported by:** @gadhiyamanan
**Slack conversation:*x httpss//eeeeetff....aa.accocccce p16/5888992/973319
(View all open jobs on GitHubl wtwtthub.com/eeee httpa//......oooooaaaaa////////e AAppes 3Aopen
31

---

SWE-Lancer
<detail=<<summrr>pppooo Automation - Do Not Edit</summary
<v1>
2 <li>Last Price Increase: 2023-03-02</1i>
<1 <1i>upwork Job URl:: https: https://www.cr wwtp://wwwwpppwwoo.//// www.soos//////t..cctoottoootsstt
<11>Upwork Job ID: 16377233//32330/11
</ul>
</details>
You will be paid $8000.0 your team fixes the issuc. since you are manager, your task is to select the best proposal regaording how to fix the issue from the below options:
Proposal:
Proposal
## Restating the problem
Some unwanted characters allowed zip code
# Root Cause
'Regex is not being validated correctly for zip code
Solution
Hore are some conditions for the post code:
Every postal code system uses only A-Z and/or 0-9 and sometines space /dash
postal codes (ex. Ireland outside of Dublin), but we' ignore that here.
every country
The shortest postal code format is Sierra Leone with NN
The longest is American Samoa with MNNNN-MMNNNN
You shoula allow
space dash.
Should not begin or end with space or dash
[a-2009] (a-2099)- [00, 10) [a-20-9]s*
This regex should cover all those conditions
source:
httpp//cww...caaa/////////332323 2/19944322/15373344
So I'll just this validation to the 'Personal information Page as well as the 'Home Address Screen'. Not posting code diff due to changed proposal guidelines
* Note:**
As mentioned by @Heamanator in https://expensifgs sttpeeesssfyyyl.lllllll... wocc.ccc//arcchis/////////iii 116754452481151119 15 18583887788 775338827.973311 973319sci ##с0049Ннм// we also have allow letters like in Canadian zip codes, therefore allowing the alphabets
Proposal: 1:
ORIGINAL PROPOSAL (comment #1424743785)
## Proposal
### Please re-state the problem that we are trying to solve in this issue. The validation the 'AddressPage" component is insufficient, leading to many need to add proper validation for all the fields the component.
problems.
### What is the root cause of that problem? The root cause behind this issue is that none of the values the home address are validated properly, the only thing checked is if the fields are empty not in
32

---

SWE-Lancer
'addressPage'.
https://github.c htttss///gtiub..cc//xxsssss/////
htt.://100000.1.aaaaaaaaaaaaaa.. 6dfa adfa/arc/ http:///ooollll/eesssooaallllll
Adt./adddrsssseeee ja 1 13-1.122
### What changes do you think we should make order to solve the problem? The validation in 'AddressPage" is outdated and does not validate all fields
properly. The best case would be to check the validations performed in CompanyPage and then code, but all fields are correctly validated.
the Walidationutils to ensure that not just the zip
Adding the required checks for each field:
For each required field, can it through Waliidtionntiils wtw.llls..sssiiiie.. ise. () first.
++Zip code*:: Zip codes across the world can vary differently, they need not be of digits only. suggest just do basic sanity check and not do anything else
**Solution 1**
Create map of regexes with respective country specific fic zip code matching and then validate according to user input.
**Solution 2 (Better IMO)**
It docsn't t really matter whatover the zip code the user enters. we can just perform a basic sanity check and then 1et the user enter whatever they want. The regex for the same is ([Ww] [Ww\= 100, 10) [Aw]s'.
ADDITIONAL NOTE (coment #1424971869).
"My pproposal| (attps://github wwtp:aa...o////ppeeeeeeeee
1424743785 will resolve http:///ww....oo.oooo///////staa as well |"
FURTHER DISCUSSION (comment #1427798502):
"Thanks for the feedback and context! For the zip code in Personal Information page, since we want to support zip code from any region the world, isn't really possible to create single super regex to perform validation according to the country.
We presented with choices right **Solution 1**
Create map of regexes with respective country specific zip code matching and then validate according to user input.
**Solution (Better IMO)
It doesn't really matter whatever the zip code the user enters. can just perform a basic sanity check and then let the user enter whatever they want. The regex for the same is [[[w]] [ww-- [ 10) [Aw]s
As for the other fields, no need for the detailed validaticn and we can perform just the basic sanity check."
ADDITIONAL CLARIFICATION (comment #1431621152):
"My [proposal] hnttps://gihhb htteesr//gthuu..cc/////eesss/// http:///wcc.caaa//////////////aa
-1424743785) will resolve http:://ptttbb..cmm/xxoott https//////p/sssss//s5/3333 woll
33

---

SWE-Lancer
potential issue, you please shed light on this: Cur most connon use case involves the user searching for an address in the address line and choosing from the autocomplste. However, sometimes the zip code from that API is incomplete or doesn't match the stricter letting them correct it FURTHER UPDATES (selected excerpts from conversation by same auther)
(comment #1427798502 also mentioned):
"Thanks for the feedback and context! For the zip code in Personal Information page, since we want to support zip code from any region in the world, it isn't really possible to create single super regex to perform validation according to the country. We can just do basic sanity check, country code zip code."
can optionally do map of
(comment #1442138427):
"If we proceed with adding the map to SCONST. js', stcre each country's name, iso code, and the regex. Then if the user picks country that does exist the map, we fallback to a simpler approach. That might be the best scenario."
(comment #1450806934):
"Yes please let's a keep it all the E/App repo for now. we proceed with adding this in E/A, we should create a new file for storing the regex map. Or we could place it in 'CONST.js- map to 'CONST.Js', (2) create new file for storing the regex map, or (3) create new npm package for this."
Ha, sure. We could either (1) procsed with adding the
(comment #1459820688):
"While investigating more,
regex. those cases, maybe should help the for them fix it."
Me'11 show
(End of unified proposal from "Prince-Mendirattaa..
Proposal: 2:
## Proposal
## Please re-state the problem that trying to solve in this issue. Validation errors are not shown in the address page with zip code
### What is the reat cause of that problem?
We don't t valicate zip codes. In fact the particular page don't t see any validation except empty check validation. hctps://github. ht/p://hhbbccm///ppess/////////p http:///i///.epepee//ttt/tta//// perrsonallbttaaill////eeessssss
### What changes do you think we should make order to solve the problen?
<!-- DO NOT POST CODE DIFFS -->
Wo add proper validation for all fields in the particular page including the
34

---

SWE-Lancer
*/ concerned zip code.
we al reeadyy have validations for these in other flows like cemрany step. We can reuse the same functional connlityy ity here as well.
For ex: We already have validation util for zipcode here battps://githuae /xpossst///gg//at////aaaa//s//a/
Proposal:
## Proposal
## Re-state the problem
z zip/postcooe validation error message not displayed for entering on the Home address screen
### What is the root of that problem?
The root cause of this problem is the nen-oxistent validation function for zip/ postcode field.
### What changes you think should make order sclve the problem?
<!-- DO NOT POST CODE DIFFS --> Solution is very simple!!! Refore clicking save button, just val salidate the zip/ postcode field by using this function and show the validation error message i the entered wrong.
..
/**
@param (String) postcode @returns (Boclean)
Sunction i (possccc)))) { return (2: 25/ test (postcode);
...
### what alternative ive solutions did you explore? (Optional
Optionally, we can use postal code API for postal code validation from https:// developer.bring com/ httm/aapppostallcce
Proposal: 4:
## Proposal
### Please re-state the problem that we are trying to solve in this issue. Some unwanted characters are allowed in zip code
### What is the root cause of that problem?
'Regex is not being validated correctly for zip code
## What changes do you think should make in order to sclve the problem?
35

---

SWE-Lancer
<!-- DO NOT POST CODE DIFFS
'Regex' is not valida idated for zip code. I'd propose adding an existing validationutils function for zip code new function if it's ready there. Then hecking that to all forms that have zip code.
**oot We can also handle unexpected punctuation like by either removing it showing error.
Proposal:
ORIGINAL PROPOSAL (comnent #1427806164):
# Proposal
### Please re-state the problem that are trying to solve in this issue.
', is allowed in zip / postcode input validation
space-only )) is allowed all fields but not actually saved
## What is the root cause of that problen? There's validation check for all fields Home address screen, except 'required' validation.
htcps::/ggituub.ccm//pppg/
main root http://rrfff////errroooataaaaala
Then why are they saved and then reset to previous values when re-visit this page? ( #15003)
passes current validation and "updateHomeAddresss" api is called. The issue is that we pass trimmed values as api params so actually empty values are sent to server which causes api to fail. reverted back to criginal.
That's why values
### What changes do you think we should make order to sclve the problen?
**For zipcode:**
checked context:
be aware that the current validation on that page only allows numbers (and an optional -) but in the new Home Address form we have to also allow letters since some countries have letters in their zip codes
So think just allewing letters in current validation of zip code which was used in bank account step should be simple solutior. https://github.coom/a http://///.p......oo.cocaaaaaaaa CONST. L0870
New zipcode regex for Home address would be: 1/aa-A-20C 20- (5) [?: ] [a (=-2A-20-9] (41)
We can add new flag param called 'shculdAl www.w/lwwwweersr in 'isValidRipCode" function and this in both Home address page and Fersonal information page. bttps://github.com
**For all fields:**
Validate trimmed value instead raw value by using "istequireeFullFFillele
function defined in WalidationUtils" (optional) For other strict validatiens, we can use existing logic in 'CompanyStep
36

---

SWE-Lancer
*+ddonn***
noticed no-trim issue happens "CompanyStep' well, which has the same root cause as #15003 is correct in frontend val validation but incorrect backend validation because it's trinmed on api call. It would good to fix them too in this issue.
https://user-image
2978-b3al6a2eell3,
UPDATED PROPOSAL (comment #1436038776):
"## Proposal
[Updated] based on scope change https: //tp::/gtttttt/ wtt. Expenst fy/ http///pp/aaass///aaaa ssseccmment
- http:tthbb.cmmmpeeeeetee httpp///sssusss///9995/////// / ssuecommentt 42270c00 64)
1131443974
(No major changes shown, just referencing the original propesal and stating it has updated.)
For your convenience, the Expensify repository has already been cloned, and is present in /app/expensinnneeeeee fy (your CWD). You should d not attempt to run the app or implement the entire fix, but you can review the codebase to understand the context of the issue.
when you are ready to submit your answer, please create a JSON file in /app/ expensi fy wtw.a//geeeaeeeeiieieeieoe deciiiioeddeiioissos.. json with the following format:
Locpttedppropssla____ id: int,
You must write your output to this path exactly: /aapp//eeenniit//mameee dew.aaggerrdddddissss... decisions. json'. You must not submit anywhere
just write to /app/
expensi Ty/manager_deer _ecci cionn, j exactly!
A.14. Price Analysis Additional Details
To estimate API costs, we created new OpenAl API keys and measured the amount spent after each run. We used 2p1-40-2024-08-66 and OpenAl's ol with high reasoning effort.
We also calculated these additional metrics (results below the average 5 runs):
1. Cost Ratio, not conditioned on success: Ignoring success and simply calculating total model API cost total freelancer cost across all tasks: GPT-4o costs 0.08% of freelancer cost and ol costs 0.7%.
2. Model Earnings on correctly solved tasks Model API Cost over all tasks: On solved tasks: GPT-4o earns $109 for every $1 of API cost, and ol earns $20 for every $1 of API cost.
A.1 Evaluation of Open-Source Models
In addition to the main models evaluated Section 3, we also ran preliminary evaluations on two open-source models: Deepseek-BR RI and Meta Llama 3.3 70B Instruct; however, performance here may be underelicited due to some limitations described below.
37

---

SWE-Lancer
Table 11. Preliminary pass@1 Accuracies and Farnings for IC SWE and SWE Manager open-source models. Farnings rounded
the nearest thousand. SWE-Lance, Diamond contains 237 IC SWE tasks worth $236,300 and 265 SWE Manager tasks worth $264,500, totaling tasks worth $500,800. The full set has 764 IC SWE tasks worth $414,775 and 724 SWE Manager tasks worth $585,225, lotaling 1,488 tasks worth $1 $1,000,000.
Model
User Tool
Dataset
pass@1
Dollars Earned
Total
Earn Rate
Deepseek-RI Llama 3.3 70B
Yes
IC SWE (Diamond)
3.8%
$4k $236k
1.6%
Yes
IC SWE (Diamond)
4.6%
$5k $236k
1.9%
Deepseek- R1
Yes
IC SWE (Full)
5.2%
$17k S415k
4.1%
Llama 3.3 70B
Yes
IC SWE (Full)
5.9%
$17k S415k
4.0%
Deepseek-RR R1
N/A
SWE Manager (Diamond)
40.4%
$121k / $265k
45.6%
Llama 3.3 70B
N/A
SWE Manager (Diamond)
40.4%
$120k / $265k
45.2%
Deepseek-RR RR Llama 3.3 70B
N/A
SWE Manager (Full)
46.7%
$276k / $585k
47.2%
N/A
SWE Manager (Full)
44.1%
$260k / $585k
44.5%
Deepseek-BR R1
N/A
SWE-Lancer Diamond
23.1%
$124k / $501k
24.8%
Llama 3.3 70B
N/A
SWE-Lancer Diamond
23.5%
$124k / $501k
24.8%
Deepseek-RI
N/A
SWE-Lancer Full
25.4%
$293k / SIM
29.3%
Llama 3.3 70B
N/A
SWE-Lancer Full
24.5%
$277k S1M
27.7%
The preliminary results in Table reflect one seed on our default scaffold. We suspect a more customized scaffold could result in significantly higher performance on IC SWE tasks (which are more dependent on scaffolding) from these open-source models, especially Deepseek-R1. We used the same scaffold used run the OpenAI and Anthropic models; customizing this scaffold to better serve open-source models would likely elicit better performance. Below, summarize the common failure modes we observed from Deepseek-RI and Llama 3.3 70B under the current scaffold setup:
Poor instruction following led to models struggling to navigate the codebase and localize the issue. Our scaffold provided guidance for how the model could familiarize itself with the repository, essentially instructing the model to begin by running is with Python block, waiting for reply, and always proceeding step by step. While models like Claude 3.5 Sonnet, GPT-4o, and ol followed these instructions to understand the codebase structure, Deepseck-RR in particular would not follow these instructions and would rarely inspect the directory tree, instead either hallucinating the repository structure or running superficial context-blin search, with something like the below:
import os
target_ file "UserProfLicPage found None
for root, dirs, files in wa.walk wa. ("/norkkpace/prrjec"""" if target_file in files:
found = path. foin (root, target file)
break
print ("Found:", found if found else "Not, found")
The above code from RI incorrectly assumes the repo lives under http///wwwpaeeppojeeeetttt relies on name match (which would miss if the file actually called UserProfilePagee tsx or userProfilePage js, and returns binary outcome (found vs. not found) instead of listing related files.
2. Despite instructions to proceed step by step, models would try to solve the whole task at once. Our scaffold instructed models to not try to solve the problem all at once, but both Deepseck-RI and Llama 3.3 would try to write a full patch right away instead fully understanding how the problem arises. Claude, GPT-4o, and ol all frequently grepped the codebase with various keywords and read the contents key files to understand the issue - we did not observe this with the open-source models.
38

---

SWE-Lancer
3. Models would give up early. Particularly with Llama 3.3 70B, that it would frequently end the task before making significant progress finishing it, despite eval harness containing prompts give up until the patch is completed.
4. Models would repeatedly retry the same failed strategies. Especially with Deepseek-RI, the model would often try similar code blocks at each turn, despite seeing output that those attempts unsuccessful.
5. Open source models were substantially less likely to use the user-tool. If they did invoke it, they frequently failed to correctly parse its output when was invoked.
39


Reviews:
[{"review_id": "tZeYDI3r87", "reviewer": "Reviewer_FybT", "summary": "The paper introduces a benchmark SWE-Lancer using 1488 freelancing tasks from Upwork.\nThe benchmark offers many advantages as compared to the existing SWE benchmark: connects solving an SWE task directly to economic benefits, more challenging problems, a diverse data set including UI/UX tasks.\nThere are two types of tasks:\ni) Individual Contributor (IC) tasks, where models generate code patches to resolve issues\nii) Software Engineering (SWE) Manager tasks, where models select the best implementation proposal\nApart from proposing the benchmark, they tests vLLMs on these tasks and found that the best model, Claude 3.5 Sonnet also resolves only  40.9% of the issues.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this thoughtful review! We appreciate your feedback and address your points below.\n> Multiple runs and CIs\n\nExcellent point. In response to your comment, we performed 3 runs of GPT-4o and o1 on the IC SWE and SWE Manager Diamond subsets to provide confidence intervals in the camera-ready version if accepted. We are still running Claude (which takes longer due to rate limits). We calculate 95% CIs using the standard bootstrap procedure that resamples model attempts per problem to approximate the metric’s distribution. Our results are below:\n- GPT-4o (IC SWE Diamond): avg pass@1 = 8.0% (95% CI [4.2%, 11.4%])\n- o1 (IC SWE Diamond): avg pass@1 = 17.5% (95% CI [13.1%, 21.9%])\n- GPT-4o (SWE Manager Diamond): avg pass@1 = 35.1% (95% CI [30.7%, 40.0%])\n- o1 (SWE Manager Diamond): avg pass@1 = 40.8% (95% CI [35.3%, 46.0%])\n\n> Partial progress metrics\n\nGreat idea! In response to your feedback, we updated the grading and reporting modules of our open source scaffold to report both full and partial credit. We hope this will allow external researchers to experiment with this idea more easily! \n\n> Single repository\n\nWe appreciate this concern and recognized it as a key limitation in our paper. Still, we believe our contribution provides a strong foundation for expanding to more repositories in the future. We initially attempted a multi-repository approach but realized we needed to de-risk our process on a single sufficiently diverse repository first. We chose Expensify for its range of apps (e.g., web, iOS, Android, desktop), tasks (UI/UX, application logic, server-side logic, system-wide reliability improvements, managerial tasks), and maturity (Expensify serves millions of customers). Building our benchmark helped us create a scalable task-creation process to tie LLM performance to economically valuable work. By open-sourcing our methodology and code (with modular components that are applicable across repos), we hope community contributors can now expand to additional codebases.\n\n> Multimodal data\n\nGreat point! Our environment already supports image screenshots via the User Tool. In response to your comment, we added a utility to the open-source SWE-Lancer codebase to automatically download each task’s video – along with README instructions to enable easier video task creation in the future. Let us know if this addresses your concerns! \n\n> Motivation for managerial tasks\n\nWhile many believe LLMs will soon automate IC coding tasks, others argue managerial tasks (like oversight of proposed code changes) will still require human supervision (see e.g. https://www.wsj.com/articles/how-ai-tools-are-reshaping-the-coding-workforce-6ad24c86). In the Expensify repository, managers—who review and select the best technical proposals—earn the same as the freelance engineers implementing the solutions, and we wanted to reflect the full range of repository tasks. Results on management tasks are also unique in helping evaluate the “management is protected from LLMs” hypothesis. For those interested exclusively in the IC SWE tasks, we report these metrics separately in our open source code. We will clarify this in the camera-ready version if accepted.\n\n>Dynamic pricing\n\nGreat question! You’re right that a sufficiently powerful model might solve certain tasks right away – effectively preventing price hikes altogether. But that’s exactly the disruption our paper sets out to measure. Our benchmark reflects how tasks are priced in today’s freelancer markets, where unresolved jobs see their rates climb over time. By quantifying this status quo and then applying LLMs to it, SWE-Lancer demonstrates the extent of the economic impact these models would have.\n\n> Diamond Set\n\nThank you. We will clarify earlier that the “Diamond” subset is the publicly released subset worth $500,800 in payouts; the rest is a private holdout. \n\n> Bias \n\nWe mitigate test-based bias with human-written E2E tests that mirror realistic user flows, rather than unit tests from the original codebase (biased towards code people chose to write tests for). Still, all benchmarks have biases, so we encourage using multiple real-world benchmarks to capture a broader range of capabilities.\n\n> Language split\n\nAll of the coding tasks are in JavaScript, TypeScript, and React or React Native.\n\n> Data contamination\n\nBecause these tasks are from a public repository, we checked for data contamination by comparing pass@1 on tasks published before vs. after model knowledge cutoff dates (Table A.2) and found no evidence of memorization.\n\n> Open-source models\n\nWe haven’t tested open-source models yet but plan to run Llama and DeepSeek for the camera-ready version. We’ve also open-sourced our evaluation harness and the Diamond Set to allow others to test models.\n\nThank you again for your valuable insights. We hope we have addressed your concerns and respectfully invite you to consider raising your score if you feel the benchmark has improved as a result of these clarifications.", "claims_and_evidence": "Yes, the claims are generally supported by the evidence. \nIt would be good if authors could do multiple runs and provide confidence intervals. The LLMs are quite stochastic and concluding from a single number is misleading.", "methods_and_evaluation_criteria": "The evaluation criteria used are pass @1, pass @k and real-world economic benefit.\nYeah, these make sense for the concerned applications. \nHowever it would be better if authors could also propose metrics that can evaluated based on the partially correct trajectory as well, i.e., the issue is not resolved but somewhat have done something reasonable. This will be beneficial for understanding where do current model stands.", "theoretical_claims": "Experimental paper -- No theoretical claims", "experimental_designs_or_analyses": "Yes, I checked them namely, increasing the number of attempts, test time computation and removing tool usage.", "relation_to_broader_scientific_literature": "The key contribution of proposing a benchmark of more complex and realistic problems is beneficial for the community.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and easy to parse\n- I believe the contribution of adding more realistic problems is quite strong\n- They employed professionals (human annotated data) for writing and verifying end to end tests \n- Experiments are conclusive\n\nWeakness:\n- tasks are only from the Expensify repository. This limitation it quite strong IMO.\n- Would be good to include multimodal data\n- Motivation of introducing managerial task is rather weak. I think a benchmark just with IC would also be valuable for community\n- The associated costs are dynamically priced, however, in actual deployment LLM's would solve it immediately (for the lowest price) ---- i believe it would not wait for weeks to get price hiked up?", "other_comments_or_suggestions": "line 86: explain diamond set when it occurs for the first time.\n\nthe paper mention about bias in SWEbench due to having unit tests in them; however i believe there is also strong bias is swe-lancer due to only having test based data, that too from expensity. I think getting rid of bias is difficult and my point is to not as a benefit or a disadvantage (line 134:144).\n\nAlso, I think on page 1-3, the advantages of swelancer is a bit verbose with some repeats and perhaps can be shortened.\n\n### Update after rebuttal\n\nAll my questions have been addressed. \nI also agree with the motivation to include managerial tasks. \nFor authors: Please include the promised changes in the final version. \nI have changed the score to accept.", "questions_for_authors": "Do you know the spit of tasks based on coding language as well? That would be helpful as well.\n\nDo you have hints or trends that you may have observed in execution that shows if existing LLM is trained on similar data?\n\nDid you also try some open-source models on these tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9sEsYWHh72", "reviewer": "Reviewer_E3n6", "summary": "This paper introduces SWE-Lancer, a benchmark of 1488 real-world freelance software engineering tasks from Upwork valued at $1million USD in actual payouts. The benchmark includes both Individual Contributor (IC) tasks where models generate code patches to fix issues, and Software Engineering Manager tasks where models select the best implementation proposal. Tasks are evaluated using end-to-end tests with experiments showcasing that even the best performer (Claude 3.5 Sonnet) solves only 26.2% of IC tasks and 44.9% of Manager tasks on the public Diamond set, indicating these real-world tasks remain challenging despite recent progress in coding capabilities.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your generous review and strong endorsement! Below we address your points.\n\n> Unbiased data collection vs. platform bias\n\nGreat point! SWE-Lancer  focuses on a single repository and freelance tasks, so it isn’t free from platform bias. By ‘unbiased data collection,’ we mean we didn’t selectively pick tasks or rely on pre-written tests; we took tasks as posted on Upwork and had professional engineers write E2E tests. We will clarify that ‘unbiased data collection’ refers to minimal artificial filtering, not the absence of any underlying biases.\n\n> Most tasks are bugfixes \n\nThis reflects the distribution of the real-life Expensify repo, as it is a mature production system where most tasks involve debugging and incremental improvements. With SWE-Lancer, we set out to ‘containerize’ a company in an unopinionated manner, so we didn’t filter for task type or ease of testing. We acknowledge that one drawback is that the distribution may not be ideal or interesting; however, it ensures that our dataset is realistic. \n\n> Benchmark is expensive to run\n\nWe recognize that running SWE-Lancer is resource-intensive, as there are 1488 tasks which each run in their own VM. We open-sourced our evaluation harness so researchers can select smaller subsets to manage cost (e.g., some researchers run 5–10 SWE-Lancer tasks locally). In response to your feedback, we have also updated our open-sourced repository with a SWELancer-Lite split with 174 tasks (vs. the 502 originally in the Diamond set).\n\n> SWE-Lancer task distribution\n\nGood question! We note in the limitations section that freelancer work tends to be more self-contained and well-scoped than full-time software engineering work. Furthermore, SWE-Lancer would be less representative of SWE work that is going “zero to one,” as even the tasks involving adding a new feature are building off of a well-established and maintained codebase. We will be sure to elaborate on this in the camera-ready version. \n\n> Characteristics of challenging tasks\n\nWe analyzed pass rates by domain and task types (Tables 2 and 3 of paper) as well as individual rollouts. Overall, models performed much better on SWE Manager tasks than IC SWE tasks, likely because qualitative analysis is easier than execution. Models tend to perform better on IC SWE bugfix tasks as those are easier to root-cause vs. more open-ended tasks (adding new features) or reliability changes that span more of the codebase. All models were worse at IC SWE tasks involving application logic, and GPT-4o and o1 in particular performed worse than Claude 3.5 Sonnet on UI/UX tasks. \n\n> Artificial variation of price in prompt\n\nGreat question. We found that omitting price information did not materially impact performance, so we decided not to pursue this direction further; however, in response to your feedback, we will directly evaluate sensitivity to the price offered in prompt as a new experiment for the camera-ready paper, if accepted. \n\n> Multiple valid solutions to the same problem\n\nThis is a great question, and a challenge for SWE benchmarks. To address this, we employ end-to-end tests, which are more flexible than unit tests because they are agnostic to the specific implementation methodology and pass as long as the end behavior is correct. Of course, the model’s implementation could still deviate from the tests in a way that they fail to account for. To mitigate this, we allow the model to invoke a User Tool, which opens the browser and performs the action associated with the task (e.g. uploading a profile picture). You can read more about the User Tool in section 2.2.5 of our paper. \n\n> Cases where models pass E2E tests but are still not fully correct\n\nWe have manually reviewed rollouts and haven’t observed this; however, the possibility of cheating and/or grading errors is a concern for every benchmark, so we cannot rule this out. We find that end-to-end tests are significantly harder to cheat on compared to unit tests, as it is significantly harder to fake correct behavior when the entire system is involved. We include examples of unit test hacking occurring in SWE-Bench in the Appendix to illustrate some of the advantages of end-to-end testing.  \n\n> Extending beyond correctness\n\nWe agree this is an exciting future direction, and will mention this explicitly under Future Work for the camera ready version of our paper. We did not want to make the evaluation too complex to run, nor the results too difficult to interpret, and imbuing notions of code quality and maintainability into the benchmark would have contributed significantly to its complexity. We believe that determining the best way to test for these is an open research question, and would encourage others to leverage our dataset and open-source codebase to experiment with more complex evaluation setups. \n\n> Frequency of manual assignment\n\nManual assignment was performed in 15% of cases.  \n\nThank you again for your encouraging feedback!", "claims_and_evidence": "The key claims are well-supported by the comprehensive methodology and results! However, I would appreciate if the authors could ammend:\n\n1. In Line 134 to 144 the authors claim, that previous benchmarks were biased (eg, SWE-Bench involved selecting PRs with unit tests), while SWE-Lancer follows \"unbiased data collection.\" However, this doesn't accounts for already biased distribution of tasks present in Expensify and UpWork in general (line 406-411). Therefore, the same should be clarified upfront while describing the SWE-Lancer unbiased.", "methods_and_evaluation_criteria": "The evaluation approach is sound:\n\n1. The use of end-to-end tests rather than unit tests is particularly valuable, as it replicates real-world behavior of human reviewers.\n2. Mapping model performance to real monetary value provides a meaningful economic context that goes is more meaningful than accuracy metrics.\n3. Including management tasks addresses an important aspect of software engineering that previous benchmarks have neglected.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is rigorous with comprehsnive evaluation on multiple frontier models. A minor limitation however is that the benchmark draws exclusively from one repository (Expensify) and one platform (Upwork), which may limit generalizability across different domains of software engineering.", "relation_to_broader_scientific_literature": "The paper thoroughly positions SWE-Lancer relative to existing benchmarks articulating clearly how SWE-Lancer differs from previous benchmarks like SWE-Bench by focusing on real-world, full-stack commercial engineering rather than isolated programming tasks. The comprehensive comparison table in Appendix A4 also clearly highlights the unique contributions of SWE-Lancer.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The benchmark's economic valuation framework provides a novel and intuitive way to assess the practical impact of model capabilities.\n2. The inclusion of both implementation and management tasks offers a more generalizing view and evaluation of software engineering abilities.\n3. The rigor in experiments and analysis is commendable.\n4. The authors open-source the benchmark and environment, which is great for future research.\n\nWeaknesses:\n\nApart from points mentioned above,\n1. The end-to-end tests may still have limitations, such as them being static, with limited guarantees on coverage.\n2. Similar to prior benchmarks, the benchmark still primarily contains questions related to bug fixing (~90% of tasks), compared to new features, maintenance, etc. (Table 3)\n3. The benchmark is very expensive to run in terms of computational cost and time, limiting immediate adaptation by the community. Any specific efforts to address the issue would have made the impact of SWE-Lancer stronger.", "other_comments_or_suggestions": "The paper is well-written and is a valuable contribution to improve SWE agents further. However, it would have been great, if authors had made specific efforts in increasing the accessibility of the dataset, given the high cost required to evaluate on SWE-Lancer.", "questions_for_authors": "1. How might the distribution of tasks in SWE-Lancer differ from those encountered by full-time software engineers rather than freelancers? Does this affect the benchmark's representativeness of general software engineering work? \n2. What analysis have the authors done on the characteristics of tasks that models find particularly challenging versus those they solve more easily? Are there patterns in task types, or importantly domains that predict model performance?\n3. In prompts, authors provide \"The reward for the correct fix is ${price}.\" Given prior research has shown artificially incentivizing LLMs with monetary rewards can improve their performance, have authors evaluated artifically varying price to see change in model performance? For instance, could performance on easier tasks, been decreased simply because of lower amounts provided in the prompt?\n4. How does the benchmark account for multiple valid solutions to the same problem? Especially for more complex tasks, there may be various approaches with different trade-offs that could all satisfy the end-to-end tests?\n5. Have authors seen cases, where the models pass the end-to-end tests, but are still not fully correct?\n6. Have you considered extending the benchmark to include aspects of software engineering beyond correctness, such as code quality, maintainability, or adherence to best practices? If not, this should be part of limitation, so the community does not make incomplete conclusions from the paper.\n7. In Lines 620-621, authors mention: \"If these three categorizations were not unanimous, we manually assigned the final categories.\" it would be interesting to see how often the manual verification was done?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6ylCuCmtb", "reviewer": "Reviewer_ue7i", "summary": "This paper introduces SWE-Lancer, a benchmark for evaluating language models' capabilities in real-world software engineering tasks. The benchmark comprises 1,488 freelance software engineering tasks from Upwork, collectively valued at $1 million USD in actual payouts. SWE-Lancer includes two distinct task categories: (1) Individual Contributor (IC) SWE tasks where models generate code patches to resolve real-world issues, and (2) SWE Manager tasks where models select the optimal implementation proposal from multiple options. Unlike previous benchmarks, SWE-Lancer evaluates performance across the full technology stack with end-to-end tests created by professional software engineers.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful and thoughtful review! We’ll address your points below.\n\n> Limited description of the agent framework: Although the details of the agent framework are mentioned in the appendix of the paper, it is still difficult to fully understand its implementation. The paper would benefit from a more detailed discussion of the agent framework in the main text rather than relegating it to the appendix.\n\nThank you for flagging! We included information about our agent harness in the appendix to satisfy the ICML page limit requirements. We’ve also open sourced eval-harness code so anyone can plug in an API key and run our eval. We are happy to update the main paper with more details about the agent framework in the methodology section in the camera-ready version if helpful. Are there specific aspects of the agent framework you’d like us to share details on? We are happy to clarify them here if helpful! \n\n> Given that SWE Manager tasks showed higher success rates than IC SWE tasks, do you believe this reflects a genuine difference in capability, or could it be related to the evaluation methodology (selection from options vs. generation)?\n\nGreat question! Our manager tasks involve selecting from predefined options, and the base rate of selecting a correct answer by chance in a multiple-choice setting is higher than that for an open-ended question. Therefore, the “lower bound” for what a model could achieve is by definition higher in our management tasks than our IC tasks that involve writing code. Still, the management success rate on our evaluation is much higher than can be attributed to chance alone, and the difference in success rates is meaningful (management success rates are more than double that of IC SWE tasks), suggesting there is also a capability difference at play. Qualitative analysis of trajectories also indicates that models we tested are stronger in conceptual thinking required for management tasks versus in full hands-on coding, where they are more likely to make a mistake. A potential future improvement to our evaluation is to create a more open-ended management test to ensure measured differences are due to capability; however, given that the actual tasks for managers in real life in our dataset required managers to select proposals from predefined options, we kept this grading schema to better match reality. It’s also worth noting that manager tasks in SWE-Lancer do not capture all management skills, which also include defining project timelines, deciding resource allocation, and making strategic/organizational plans.  \n\n> How might the benchmark be extended to evaluate human-AI collaboration more directly, beyond the current user tool experiments? This seems like an important direction given that collaborative scenarios may be more realistic in practice than fully autonomous AI software engineers.\n\nExcellent question - we agree this is an important direction for future work! We have open-sourced our evaluation harness and the diamond split of our dataset, and we encourage the community to build upon this to further explore human-in-the-loop approaches. We regret that we had limited time and resources while building the evaluation, and were not able to explore this exciting direction. \n\nThank you again for your valuable insights. We hope we have addressed your concerns and respectfully invite you to consider raising your score if you feel the benchmark has improved as a result of these clarifications.", "claims_and_evidence": "The claims made in the submission are well-supported by evidence:\n\n- The claim that SWE-Lancer provides a more realistic evaluation of SE capabilities is supported by the detailed description of task selection, construction, and verification processes. The use of real freelance tasks with actual monetary values and end-to-end testing provides stronger evidence than previous benchmarks.\n\n- The performance claims for different models are backed by experimental results presented in tables and figures, showing pass rates and earnings across different task types, categories, and conditions.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand:\n\n- The division into IC SWE and SWE Manager tasks allows for evaluation of different aspects of software engineering capabilities.\n\n- Using real-world freelance tasks with actual monetary values provides a economic significance to the evaluation.\n\n- The end-to-end testing approach is a significant improvement over unit tests used in previous benchmarks.\n\n- The evaluation of models under different conditions (reasoning effort, number of attempts, with/without user tool) provides insights into factors affecting performance.", "theoretical_claims": "The paper does not make theoretical claims that require formal proofs.", "experimental_designs_or_analyses": "- The comparison across multiple frontier models (GPT-4o, o1, Claude 3.5 Sonnet) provides a good overview of current capabilities.\n\n- The ablation studies on reasoning effort, number of attempts, and user tool access offer insights into factors affecting performance.\n\n- The reporting of both pass rates and earnings gives a more complete picture of model performance.", "relation_to_broader_scientific_literature": "It connects to broader research on automated software engineering, agentic systems, and the economic impact of AI.", "essential_references_not_discussed": "The paper appears to cover the relevant prior work in software engineering benchmarks and LLM evaluation.", "other_strengths_and_weaknesses": "Strengths:\n\n**Real-world relevance**: Using actual freelance tasks with real monetary values provides a realistic evaluation of LLM capabilities.\n\n**End-to-end testing**: The use of comprehensive end-to-end tests rather than unit tests is a improvement over previous benchmarks.\n\n**Economic dimension**: Mapping performance to monetary value adds an important perspective on the practical impact of AI capabilities.\n\n**Comprehensive evaluation**: The paper evaluates multiple models under various conditions, providing a rich understanding of current capabilities and limitations.\n\nWeaknesses:\n\n**Limited description of the agent framework**: Although the details of the agent framework are mentioned in the appendix of the paper, it is still difficult to fully understand its implementation.", "other_comments_or_suggestions": "The paper would benefit from a more detailed discussion of the agent framework in the main text rather than relegating it to the appendix.", "questions_for_authors": "- Have you analyzed whether there are systematic patterns in the types of tasks that models fail to solve correctly? This could provide valuable insights for targeted improvements in model capabilities.\n\n- Given that SWE Manager tasks showed higher success rates than IC SWE tasks, do you believe this reflects a genuine difference in capability, or could it be related to the evaluation methodology (selection from options vs. generation)?\n\n- How might the benchmark be extended to evaluate human-AI collaboration more directly, beyond the current user tool experiments? This seems like an important direction given that collaborative scenarios may be more realistic in practice than fully autonomous AI software engineers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SNf4gsgDbD", "reviewer": "Reviewer_h9KZ", "summary": "The paper introduces a benchmark of 1488 software engineering freelance tasks consisting of managerial and coding tasks to evaluate LLM performance against real world tasks. The benchmark helps get a clearer picture about the potential social and economic impacts of AI. For individual contributor/coding tasks, comprehensive e2e tests are developed and a user tool is provided to mimic developer's actions associated with the task in a browser. \nPerformance of various LLMs is measured as Pass@1, earning, and earning rate. Reported results show a significant gap between SOTA LLMs' performance and what human can achieve. This gap is more spelled out for individual contributor tasks where pass@1 remains below 30% versus managerial tasks for which this value gets up to 45%.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful and thoughtful review! We’ll address your points below.\n\n> There are certain relevant and seminal works that are not cited in the paper. For example, Evaluating Large Language Models Trained on Code by Chen et al. and Program Synthesis with Large Language Models by Austin et al. are among such cases.\n\nThank you for this flag. We do include Chen et al.’s paper, HumanEval, in the Related Work section, but we will cite directly and add to Table A.4. We sincerely apologize for not citing Austin et al. and will include it in the camera-ready version if accepted to the conference (as per ICML rules we cannot update the submitted PDF in response to rebuttals). \nTo be more comprehensive, we also plan to add:\n- Lu et al. (2021) – CodeXGLUE\n- Li et al. (2022) –  AlphaCode\n- Nijkamp et al. (2022) – CODEGEN\n- Lai et al. (2023) – DS-1000\n\nPlease let us know if there are others we should include. We sincerely apologize for our oversight and appreciate you pointing this out.\n\n> More importantly, they could have compared their learning from reported experiments in their work against some of the claims or experimental results reported in other existing works.\n\nGreat point! In terms of comparing learnings:\n- For IC SWE tasks, one comparable benchmark is SWE-Bench Multimodal, where GPT-4o solved 12.2% of visual programming tasks when given the best scaffold and multimodal capabilities. This is comparable to our GPT-4o scoring 8.0% on our IC SWE Diamond tasks when the model can ingest image screenshots via the user tool.\n- On SWE-Bench Verified, the industry standard benchmark for software engineering, GPT-4o currently scores 38.8% using the best available scaffold, but only 8% on SWE-Lancer Diamond IC SWE – likely because SWE-Bench Verified focuses on bugfixes, and SWE-Lancer includes open-ended problems (adding new features, reliability) and full-stack development. This highlights the need for more challenging, unsaturated SWE benchmarks. \n- SWE-Lancer is unique in evaluating engineering management skills – we did not identify a direct analogue in prior works; we welcome any suggestions for related work. \nWe will include these comparisons in the camera-ready version of our paper if accepted to ICML.  \n\n> After reading the paper I'm still not clear how the user tool is helping. The paper says \"The model does not receive any feedback about whether the action was successful\" it also says \"a text-based trajectory and a series of screenshots are written to its working directory when the tool finishes.\". Should I assume by \"action was successful\" authors mean whether e2e test passes successfully?\n\nWe’re happy to clarify! When the model calls the user tool, it automatically performs actions relevant to the task on the website (specifically, by using the Playwright library to simulate user actions), and sends the model a report (screenshots + a text-based log of browser states). The user tool does NOT provide feedback to the model on whether it passed or failed the end-to-end tests, but it does allow for iterative debugging and verification of hypotheses. We created the user tool to mirror how humans iteratively check their code by running it and inspecting the output.\n\n> Based on Figure 9, it looks like user tool is not helping GPT-4o much, can authors explain why that's the case for GPT-40 but not O1.\n\nDefinitely! Smarter models are able to “get more out of”  the user tool. GPT-4o often fails to properly use the user tool. For instance, as we outline in section A.9. of the appendix, the model fails to review the user tool logs that indicate the actual behavior. \n\n> One thing that I could not follow is whether managerial tasks are also collected from Upwork, the idea of having a few proposals and letting LLM decide which one is more appropriate is not aligned with my understanding of the tasks on a platform like Upwork. \n\nHappy to clarify! The tasks in this benchmark were posted on both Upwork and GitHub. When a task is posted (e.g., “solve X bug”), interested freelance software engineers are required to first submit proposals for solving the task on the corresponding Github issue (see the Expensify/App GitHub repository for examples). A freelance engineering manager then reviews all proposals and chooses the best one. The engineer whose proposal is chosen then gets to actually implement the solution. For each task with multiple proposals, we created a “managerial task” by collecting all the freelancer proposals, having models choose the best proposal, and comparing model choices to the real manager’s. You are right that not all Upwork software engineering tasks entail a proposal selection process – this just reflects Expensify’s public vetting process.\n\n Thank you again for your valuable insights. We hope we have addressed your concerns and respectfully invite you to consider raising your score if you feel the benchmark has improved as a result of these clarifications.", "claims_and_evidence": "The main claim of the paper is raising more awareness about LLMs potential in automating real-world software engineering tasks to get a clearer picture of their social and economic impacts. The claim is clear and is backed up by the reported experimental results.", "methods_and_evaluation_criteria": "The prepared benchmark based on Upwork and the evaluation criteria to gauge the LLM performance makes sense.", "theoretical_claims": "There is not much theoretical claim in the paper so this box is less applicable.", "experimental_designs_or_analyses": "The experimental design looks sound, e.g. authors are mindful of the possible contamination issues and have some mitigation plans. The analysis accompanied the experimental results including the ablation studies looks sound as well.", "relation_to_broader_scientific_literature": "Studying AI/LLM performance against real-world tasks that are tied with monetary reward is a critical and relevant topic to the broader scientific literature since they can adjust expectations about what can versus cannot be expected from AI and ground everyone's perception of AI's potential.", "essential_references_not_discussed": "There are certain relevant and seminal works that are not cited in the paper. For example, Evaluating Large Language Models Trained on Code by Chen et al. and Program Synthesis with Large Language Models by Austin et al. are among such cases.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The comprehensive benchmark of real-world software engineering tasks with curated e2e tests that are tied with monetary rewards is a valuable asset for the research community. On top of that, the experimental results and providing a realistic picture of AI's capability in this domain is extremely valuable and will trigger valuable discussions in the research community.\n\n**Weaknesses**\n- The weakest aspect of the paper is the literature review section. Authors could have done a more comprehensive study of the existing literature. More importantly, they could have compared their learning from reported experiments in their work against some of the claims or experimental results reported in other existing works.", "other_comments_or_suggestions": "No other comments/suggestions.", "questions_for_authors": "- After reading the paper I'm still not clear how the user tool is helping. The paper says \"The model does not receive any feedback about whether the action was successful\" it also says \"a text-based trajectory and a series of screenshots are written to its working directory when the tool finishes.\". Should I assume by \"action was successful\" authors mean whether e2e test passes successfully? \n\n- Still related to user tool, based on Figure 9, it looks like user tool is not helping GPT-4o much, can authors explain why that's the case for GPT-40 but not O1.\n\n- One thing that I could not follow is whether managerial tasks are also collected from Upwork, the idea of having a few proposals and letting LLM decide which one is more appropriate is not aligned with my understanding of the tasks on a platform like Upwork. Maybe my knowledge is limited but can authors clarify?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SWE-Lancer, a novel benchmark for evaluating large language models on software engineering tasks derived from real-world freelance job postings on Upwork, specifically from the Expensify repository. The benchmark includes approximately 1500 tasks, encompassing both individual coding contributions and managerial decision-making (proposal selection), collectively valued at $1 million based on actual payouts. A key contribution is the use of end-to-end tests, verified by experienced engineers, to assess task completion, alongside mapping performance to potential economic earnings. The authors evaluate several frontier LLMs, demonstrating that these models currently struggle with the complexity and demands of these real-world tasks.

Strengths highlighted by reviewers include the benchmark's novelty and real-world relevance, using actual freelance tasks tied to economic value (FybT, E3n6, ue7i, h9KZ). The use of end-to-end tests, verified by experienced engineers, was seen as a significant improvement over prior benchmarks relying solely on unit tests (FybT, E3n6, ue7i). The inclusion of managerial tasks (E3n6, ue7i) and the mapping of performance to monetary value (FybT, E3n6, ue7i, h9KZ) were also praised. Furthermore, the paper was generally considered clearly written (FybT) and the open-sourcing of the benchmark subset and evaluation harness was commended (E3n6).

Several potential shortcomings were raised. A primary concern, noted by FybT and E3n6, is the benchmark's reliance on a single software repository (Expensify) and platform (Upwork), potentially limiting the generalizability of the findings. Reviewers FybT and E3n6 also questioned the claim of "unbiased data collection," pointing out the inherent biases within the source platform and repository itself. The high computational cost and time required to run the benchmark were flagged by E3n6 as a barrier to broader adoption. E3n6 also noted the benchmark's focus on bug-fixing tasks (~90%) and the limitations of static end-to-end tests regarding coverage. The initial lack of confidence intervals (FybT), metrics for partial progress (FybT), evaluation beyond correctness (e.g., code quality, E3n6), and a comprehensive literature review (h9KZ) were also mentioned. Some reviewers sought further clarification on the agent framework (ue7i) and the utility of the 'User Tool' (h9KZ).

In deliberation, the authors provided substantive responses and committed to several revisions. Regarding the single-repository limitation, the authors acknowledged this constraint but justified it as a necessary step for developing a scalable process, positioning the open-sourced framework as a foundation for community expansion. They clarified that "unbiased data collection" referred to their methodology of not selectively filtering tasks based on testability, rather than implying the absence of source bias, and promised to refine this language. To address accessibility concerns due to cost, they created a smaller 'SWE-Lancer-Lite' subset. The predominance of bug-fix tasks was explained as reflective of the mature source repository chosen for its realism. While acknowledging the limitations of E2E tests and the lack of code quality evaluation, the authors argued for the current focus on correctness as a pragmatic starting point and flagged quality assessment as important future work. Crucially, the authors provided confidence intervals for key results, updated their tooling to support partial credit metrics, committed to significantly expanding the literature review and comparison with prior work, and offered clarifications on the User Tool and managerial task origins. They also planned evaluations on open-source models for the camera-ready version.

Considering the strengths and the authors' constructive engagement with reviewer feedback, the core contribution – a large-scale, economically grounded benchmark using realistic tasks and robust E2E testing – remains significant. While the limitation to a single repository is notable, the depth and realism offered represent a valuable advancement over existing benchmarks. The issues raised concerning methodology, reporting, and literature context appear to have been adequately addressed or committed to be addressed by the authors. Therefore, the substantial merits of providing a challenging, relevant, and economically interpretable benchmark for advanced software engineering AI agents seem to outweigh the remaining limitations, meeting the standards for publication at ICML.
Decision: Accept (oral)