Can AI Freelancers Compete? Benchmarking Earnings, Reliability, and Task
  Success at Scale
Authors: name: David Noever; name: Forrest McKee
Published: 2025-05-16 22:42:04

Abstract: This study explores Large Language Models (LLMs) as autonomous agents for
real-world tasks, including freelance software development. This work presents
a new benchmark that evaluates LLMs on freelance programming and data analysis
tasks derived from economic data. We construct the benchmark using synthetic
tasks created from a Kaggle Freelancer dataset of job postings, with all job
prices standardized to USD (median fixed-project price around 250, and an
average of 306). Each task is accompanied by structured input-output test
cases and an estimated price tag, enabling automated correctness checking and a
monetary performance valuation. This approach is inspired by OpenAI's recent
SWE-Lancer benchmark (1,400 real Upwork tasks worth 1M total). Still, our
framework simplifies evaluation using programmatically testable tasks and
predicted price values, making it highly scalable and repeatable. On this
benchmark, we evaluate four modern LLMs - Claude 3.5 Haiku, GPT-4o-mini, Qwen
2.5, and Mistral. We report each model's accuracy (task success rate and
test-case pass rate) and the total "freelance earnings" it achieves (sum of
prices of solved tasks). Our results show that Claude 3.5 Haiku performs best,
earning approximately 1.52 million USD, followed closely by GPT-4o-mini at
1.49 million, then Qwen 2.5 (1.33M) and Mistral ($0.70M). We analyze the
distribution of errors per task and observe that the strongest models solve the
most tasks and rarely fail completely on any project. We discuss the
implications of these results for the feasibility of AI as a freelance
developer, the advantages and limitations of our automated benchmark approach,
and the gap between performance on structured tasks versus the true complexity
of real-world freelance jobs.