The RealHumanEval: Evaluating Large Language Models’ Abilities to Support Programmers
Authors: Hussein Mozannar
Valerie Chen
Mohammed Alsobay
Subhro Das
Sebastian Zhao
Dennis Wei
Manish Nagireddy
Prasanna Sattigeri
Ameet Talwalkar
David Sontag
Venue: Accepted by TMLR ()

Abstract: Evaluation of large language models for code has primarily relied on static benchmarks, including HumanEval (Chen et al., 2021), or more recently using human preferences of LLM responses.  As LLMs are increasingly used as programmer assistants, we study whether gains on existing benchmarks or more preferred LLM responses translate to programmer productivity when coding with LLMs, including time spent coding. We introduce RealHumanEval, a web interface to measure the ability of LLMs to assist programmers, through either autocomplete or chat support. We conducted a user study (N=243) using RealHumanEval in which users interacted with seven LLMs of varying base model performance. Despite static benchmarks not incorporating humans-in-the-loop, we find that improvements in benchmark performance lead to increased programmer productivity; however gaps in benchmark versus human performance are not proportional---a trend that holds across both forms of LLM support. In contrast, we find that programmer preferences do not correlate with their actual performance, motivating the need for better proxy signals. We open-source RealHumanEval to enable human-centric evaluation of new models and the study data to facilitate efforts to improve code models.

Sebastian Zhao Published in Transactions on Machine Learning Research (01/2025)
The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
Hussein Mozannar*
bmoozamorrrmmccrroofftttto
Microsoft Research
Valerie Chen Carnegic Mellon University
vchen228amdrew.com eduu
Mohammed Alsobay
mosobay@mit. edu
Massachusetts Institute of Technology
Subhro Das
subhro.Da@@ibm.com
MIT-IBM Watson
IBM Rescarch
sebbyshoo@Werkeeeyy..dkk
University of California, Berkeley
Dennis Wei
dweithus.ibm.
MIT-IBM Walson AI Lab
IBM Research
Manish Nagireddy
mawist..aggrrdddyy@.mmmmm
MIT-IBM Watson Al Lab
IBM Research
Prasanna Sattigeri
peattig@us.ibm.com ibm.
MIT-IBM Watson Al Lab
IBM Reseorch
Ameet Talwalkar
atalwalkar @ggmail com
Carnegic Mellon University
David Sontag Mussachusellss Institute of Technology
daontag Acsail mit edu
*: Equal contribution.
Reviewed on OpenReview: https: openreview. net/ forum? id= I90
Abstract
Evaluation of large language models for code has primarily relied on static benchmarks, cluding HumanEval (Chen al. 2021), or more recently using human preferences of LLM responses. As LLMs increasingly used programmer assistants, study whether gains existing benchmarks more preferred LLM responses translate to programmer produc- tivity when coding with LLMs, including time spent coding. We introduce RealHumanEval,, a web interface to measure the ability of LLMs to assist programmers, through either au- tocomplete or chat support. We conducted a user study (N=243) using RealHumanEval in which users interacted with seven LLMs of varying base model performance. Despite static benchmarks not incorporating humans-in-the-leee we find that improvements in benchmark performance lead to increased programmer productivity; however gaps in benchmark

---

Published in Transactions on Machine Learning Research (01/2025)
sus human performance are not proportional-a trend that holds across both forms of LLM support. In contrast, we find that programmer preferences do not correlate with their actual performance, motivating the need for better proxy signals. We open-source RealHumanEval to enable human-centric evaluation of new models and the study data to facilitate efforts to improve code models.
1 Introduction
Coding benchmarks such as HumanFval (Chen et al. evaluating the capabilities of large language models LLMs) as programming becomes a valuable application
2021 and MBPP (Austin et al. 2021 play a key role in
through products such as GitHub Copilot (Github 2022) and ChatGPT (OpenAT 2022). These benchmarks quantify LLM abilities by measuring how well model can complete entire coding tasks. As LLMs are in- creasingly adopted as programmer cesistants-providing chat responses or autocomplete suggestions, rather than full code generalions prior works have argued for bringing humans-in-lhe-looo to evaluate LLMs (Lee et al. 2023 Chiang et al., 2024 A predominant human-centric approach collects human preference judg- ments of intermediate LLM outputs, whether between pairs of LLM responses (e.g., Chatbot Arena (Chiang et al. 2024)) or, for coding in particular, using programmer acceptance rates of LLM suggestions (e.g., products such as Github Copilot (Bird al. 2022 ) However, such evaluation may not capture the LLM's
downstream impact on programmer productivity.
Evaluating the utility of LLMs on downstream productivity requires conducting user studies where program- mers code with LLM assistance. While a set of small-scale user studies have been conducted to primarily build a qualitative understanding of how programmers use LLM assistance, they are typically restricted to evaluations on one model and one form of LLM support, primarily relying on commercial tools like Github Copilot or ChatGPT (Barke et al. 2023; Mozannar et al. 2024 Vaithilingam et al.) 2022 Ross et al. 2023 Liang et al. 2023 Peng et al., 2023). To enable evaluations of broader set of L.Mss and lower the barrier to conducting these studies, platform consists of a code editor where programmers can solve coding tasks with two common forms of LLM assistance: programmers can either ask questions to the LLM through a chat window or receive code completion suggestions through an autocomplete system inside the editor. The interface also supports ex- ecuting and testing code and logging telemetry which can be used lo compute productivity metrics, which we operationalize as the time to complete a task or number of tasks completed, and preference metrics, including average acceptance rates of suggestions and the likelihood of copying code from chat responses.
introduce online evaluation platform, Real BealHumanEvalP (Figure D The
Using RealHumanEval, we conduct user study with 243 participants to understand the effect of model's benchmark performance and the form of LLM assistance on time to complete a task and number of tasks completed. Each participant was assigned to one of seven conditions: a control condition with no LLM support, three conditions with autocomplete support from either CodeLlama-7b (Rozière et al. 2023 CodeLlama- (Rozière et al.] 2023), or
et al.] 2020). and finally three conditions where the editor is equipped with a chat window powered by the chat variants of the previous models in addition to GPT-4o {O 2022). We deliberately select model families with increasingly higher benchmark performance and consider model pairs within each family with similar benchmark performance to understand the effect of autocomplete versus chat assistance. Through the study, we collected a dataset of interactions on 888 coding total tasks, where 5204 autocomplete suggestions were shown and 1055 chat messages were sent.
Overall, we find that improving a model's base performance on existing coding benchmarks leads to improve- ments in the time spent completing tasks. These trends were present across both chat and autocomplete interactions, validating the potential "generalizability" of benchmarks to more realistic contexts. However, we observe that gaps in benchmark versus human performance are not necessarily proportional, suggesting that further gains in benchmark performance do not necessarily translate into equivalent gains in human task completion. We also investigated whether human preference metrics, such as the average acceptance rate
The choice of naming our platform RealHumanEval is to imply that the evaluation is done with the help of real humans. It does not imply that it a real evalnation while other benchmarks are real, nor does imply that this the real version of the Human Eval benchmark.
2

---

Published in Transactions on Machine Learning Research (01/2025)
RealHumanEval
Researcher
Interface
Logging
Metrics
Productivity metrics include...
Set-up
Code Telemetry:
Autocomplete suggestions
Chat resporises
Time to complete task
tasks completed
Define Tasks:
Autocomplate interactions:
Preference metrics include...
Chat interactions:
participants:
Distribute into experimental conditions
suggestions accepted
% chats copied
Our study using RealHumanEvvl
Benchmark performance
Human preferences
Human productivity
GPT-3.5
GPT-4o
GPT-3.5 GPT-40
GPT-3.5
GPT-4o
34B
7B
34B
7B
34B
7B
RealHumanEval
No human involvement
In-depth human involvement
Figure We introduce RealHumanEval, an end-to-end online evaluation platform of L.LM-assisted coding through autocomplete suggestions and chat support. The goal of RealHumanEval is to facilitate human- centric evaluation of codo-generating LLMs, simplifying the workflow for rescarchers to conduct user stud- ies to measure the effect of LLM assistance downstream human productivity and preferences. We se- lected 4 families of LLMs of varying sizes (CPT-4o, GPT-3.5, CodeLlama- 34b, CodeLlama-7b) for use with RealHumanEval to study the alignment between static benchmark performance, subjective programmer pref- erence judgments, and programmer productivity.
of suggestions and the likelihood of copying code from chat responses, are aligned with human performance metrics. While these preference metrics are readily available in real deployments of LLM systems compared task completion time and thus can be attractive proxy metrics (Ziegler al. 2022) we find that they are only correlated with programmer perceptions of LLM helpfulness but necessarily with actual program- performance. The dissimilar findings between benchmarking and human preference metrics highlight the importance of careful evaluation to disentangle which metrics are indicative of downstream performance.
In summary, our contributions are as follows:
An open-source platform RealHumanEval to encourage more human-centric evaluations of code LLMs
An evaluation of 7 code LLMs of varying performance using RealHumanEval to provide insights into the alignment and discrepancies between benchmark performance and human preferences with down- stream human performance. Our findings emphasize the importance of studying how programmers interact with code LLMs through user studies to identify muances in programmer- interactions.
We release the dataset of interactions collected from this study guide the development of better coding assistants.
3

---

Published in Transactions on Machine Learning Research (01/2025)
2
Related Work
Table 1: A comparison of our study against prior studics understanding programmer-LILY interactions in terms the number of participants, models, lypes of LLM interaction, and lasks. Note that Cui et al. (004h) was a field experiment and thus not a controlled user study with a fixed number of tasks.
Study
# participants
# models
Autocomplete?
Chat?
# tasks
Vaithilingam et al. Vaithilingam ct al.
2022
24
X
3
Peng al. (Peng et
2123
95
X
Barke et: al.
Barke et al.
2023
20
X
Prather ct al. Prather cf al.
2023
19
1
X
1
Mozannar
Mozannar
2024
21
X
Vasconcelos et al. (asconccloss cl al.
2023
30
X
Cui ct al. (Cui et al.
2024
1974
X
Ross et al. Ross ot al.
2023
42
X
4
Chopra et al. CChopra et al.
2023
14
AX
1
Gu al. Gu al.
2024
22
10
Kazemitabant ct al.
Kazemitabaar et al.
2023
69
45
Nam et al. (Nam al.
2024
32
2
Ours
243
7
17
Coding Benchmarks. Benchmarks are essential for tracking the progress of LLMs, and coding benchmarks are key piece (Achiam et al.] 2023 Laskar et al. 2023 Zan al. 2023 Hou et al.) 2023). Morcover, the coding ability of an LLM can be informative of its reasoning abilities (Madaan
2022 thus, performance on coding benchmark is of broader interest. While HumanFval (Chen et al. 2021 | and MBPP (Austin et al. 2021) are the most commonly used coding benchmarks, many extensions and further benchmarks have been
proposed [Lu et al. 2021; Nijkamp et al. 2023 Zhu et al. 2022 Liu et al. 2023 Jimenez et al.) 2023:
Khan et al. 2023 Yang et al. 2024; Yan et al. 2023 we highlight a few: Eval Plus exlends Human Eval
test cases (Liu et al. 2023), MultiPI-E (Cassano et al. 2023) to other languages, ReCode with robustness checks (Wang et al. 2022), HUMANEVALPACO (Muennighofi et al. 2023) with code repair and explanation tasks, and buggy- Eval (Dinh et al. 2023) with bugs in the reference code. Relatedly, the DS-1000 (Lai et al.] 2023) benchmark evaluates models' abilities on data science problems that require using external librarics. More involved evaluations include the multi-turn program evaluation benchmark (Nijkamp et al, 2023) and SWE-bench (Jimenez et al. 2023). which requires the LLM to resolve GitHub issues. While existing benchmarks evaluate diverse set of LLM behaviors across models, these benchmarks do not, however, include a programmor-in-the-loop as would be the case in the inercasingly common use case of programming with Al assistance. Our evaluation complements this existing line of work by conducting a user study, where programmers put the utility of these models to the test as assistants, rather than independent code generators.
Preference Metrics. Instead of relying solely coding benchmarks' pass@k metrics, which consider only the functional correctness of I.T.--geerated code, recent work has advocated for incorporating human preferences, which may better reflect how LLM code could be useful to programmer without necessarily being functionally correct (Dibia et al.] 2023). Preferences are generally collected after a single turn (c.g.. after a single L.M response or suggestion) and thus can be collected at scale (Bird al. 2022 Chiang al., 2024) or even simulated with LLMs (Dubois et al.) 2023 Zheng et al.] 2023). Given that preferences are only form of intermediate feedback, this study, we evaluate whether human preferences provide signal for downstream productivity gains when coding with LLMs.
Programmer-LLit Interaction. Prior work conducting user studies where programmers code with LLM assistance has primarily focused on two forms of LLM support, autocomplete suggestions (Vaithilingam
let al.
2022
Peng et al. 2023 Barke et al.] 2023 Prather et al.) 2023 Mozannar et al.) 2024 Vasconcelos
ct
2023
Cui al.
2024 and chat dialogue (Ross al. 2023 Chopra al. 2023
Kazemitabaar
et al.
2023
Gu al.] 2024 Nam al.) (2024). While these studies have made progress in understanding
programmer-L.A. L.MM interactions, all studies only consider one I.IM-often Copilot ChatGPT-ann
4

---

Published in Transactions on Machine Learning Research (01/2023)
Code
Change Thome
Show Inst
Time left: 15:02
A/ Looka complood
class Tokeniiznn:
(b)
(c)
Init _s nax_vocab,, sirab_stze=200)
User
baidd vocalaslay nethed
self.max, vocab size vax_vooab__aa size
How do count occurrence sach ward in lixt of strings
provided Tokenaer clm.
self word_too self.id word {
{
A rokenvier is on object that coneerts words numercal IDs. The dbjcctive the
def tokenize text):
eturn text, Tower() (p
Chabot
haid. vocaubbry method follow:
vocabularysself, corpus):
function the niti library count occureenee fist strings example:
Fou
Wond_ [frq Counter()
The primary goal
For sentence in carpusi
Autocamplete suggestion
il
foeed_freeuuee updaae senteence)
word.
Copy
sell weldred shculd
each werd your
corpus unique rumerical
Suport nitk
Mdenillrr (ID).
texts teiis enaple sentence" "Aoother excmple sentence")
sdfiid to word is the revens mapping, where each unique ID commepoods to word
methos srould only couider froquent
# print(nn::); [g(a]]
(eint(eg)))) ngD3 count())
corpes, specifio
and appear
Output:
Run
Submit
Figure 2: We introduce Heaa,EaEE,,,,, an online evaluation platform for LLM-assisted coding, The plat- form consists of (a) customizable task description, (b) the code editor which shows autocomplete suggestions in grey, and (c) the chat assistant. Above the editor, users can check their task progress and the amount of time left, reset the editor, change the editor theme, and view study instructions. Below the editor, they can run and submit their code.
form of LLM support- either autocomplete or chat, making difficult to compare outcomes and metrics across models and across forms of support. In Table compare the aspects of our study with prior works that have conducted user studies where programmers code with LLM support. To our knowledge, ours is the first study to consider models of varying performance capabilities and multiple forms of support. Additionally, we note that the majority of studies have similar participant profiles as ours (i.e., students with some programming experience and industry professions), though a few focus exclusively on novice programmers (Kazemilabaar et al., 2023 Prather et al. 2023 Finally, multiple studies have limited scope in terms of the number and types of coding tasks that are considered (e.g., focusing on one minesweeper game (Prather et al. 2023) or simple plotting tasks (Ross et al. 2023)), which differ from the breadth of tasks that have been evaluated in benchmarks and are We contribute web platform RealHumanEval to enable case of human-centric evaluation of more models and forms of support. Beyond applications of coding assistance. our study contributes to the broader literature studying human interactions with LLMs (Lee et al. 2023
Collins et al.) 2023 Lee et al. 2022 Dang et al.) 2022 Jakesch et al. 2023 Köpf et al. 2023 Jo ct al.
2023
Brynjolfsson et al. 2023).
3 Eeal
We introduce RealHumanEval, web-based platform to conduct human-centric evaluation of LLMs for programming through the workflow shown in Figure [ We created RealHumanEval to facilitate largo-scale studies of programmers coding with LLMs, eliminating the need for participants to perform any additional installation of a bespoke IDE or study-specific extension or to have access to special hardware to serve study-specific models.
Interface. As shown in Figure 2 RealHumanEval incorporates many basic features of common code editors and the functionality of programming interview sites such as Coet Given coding task that consists of
5

---

Published in Transactions on Machine Learning Research (01/2025)
natural language description, partial code (e.g., function signature), and unit tests that evaluate the task, HeaanvalEEval allows the programmer to write code with assistance from an LLM to complete the task. The platform has a panel that displays the natural language description of a task, as shown in Figure 2(a), alongside partial code to solve the task. Participants then write their code for the task in the code editor and can lest their code with a button that checks the code against lest cases and runs their code directly. The editor displays any errors, available, and whether the code passes the unit test. Once the programmer completes the task, new task can be loaded into the interface. For our user study, we only use a single code editor file, however, RealHumanEval can support multiple-file projects.
Forms of LLM Assistance. RealHumanEval supports two forms of LLM assistance: autocomplete- based and chat-based. Examples of autocomplete and chat assistants include GitHub's 's Copilot (Github 2022) Replit's Ghostwriter (replit) 2023), Amazon CodeWhisperer (Amazon 2022 and ChatGPT (OpenAí 2022
In autocomplete-bused assistance, the programmer writes code in an editor, and the LLM displays code suggestion inline, which is greyed out as shown in Figure bb). The LLM is assumed to be able to fill in code given a suflix and prefix. A suggestion, based on the current code body in the editor, appears whenever the programmer pauses typing for more than two seconds when the programmer requests suggestion by pressing a hotkey. The programmer can accept the suggestion by pressing the tab key or reject it by pressing escape or continuing to type.
In that-based assistance, the programmer writes code in an editor and has access to a side chat window where the programmer can ask questions and get responses from the LLM, as illustrated in Figure C The LLM is assumed to be chat model. The programmer can copy and paste code from the LLM's responses into the editor. Currently, the interface supports any LLM invoked via an online API. Further information the implementation of both forms of assistance is in Appendix A and Appendix C
Telemetry logging. HeaaHuvaE logs all user behavior, including interactions with LLM support. For each autocomplete suggestion, we log the following tuple S.), Rs, Ar] where S,) is the prefix and sulfix of the code based on cursor position at the time of suggestion i, Rg is the LLM suggestion, and A, is a binary variable indicating whether the suggestion was accepted. All the logs are stored in a dataset Dac- For chat-assistance, we log for each user message the following tuple (X,, M,, R2, J= where X, is the code at the time of mossage i, M, is the user message (including prior chat history), R, is the response from the LLM for the message, and C is the number of times code was copied from the LLM's response. All the logs are stored in a dataset Dehat. Moreover, every 15 seconds, the interface saves the entire code the user has written.
Metrics. From the telemetry logs, RealHumanEval provides multiple metries to analyze programmer be- haviors: the number of tasks completed (completion is measured by whether the submitted code passes a set of private test cases), time to lask success (measured in seconds), acceptance rale (fraction of suggestions shown that are accepted, for autocomplete), and number of chat code copies (counting when user copies code from LLM response, for chat) among other metrics.
4 Study Design
Using RealHumanEval, we conducted a user study to evaluate (1) the impact of LLM assistance on pro- grammer performance as a function of the LLM's performance on static benchmarks and (2) whether luman preference metrics correlate with programmer productivity metries.
Overview. For the entire duration of the study, participants are randomly assigned either to a control group, where they experienced the no LLM condition, to the LLM-assisted group, where they experienced the autocomplete or chat support condition. For autocomplete- based support, the window in Figure (c) is hidden. For chat-based support, no autocomplete suggestions are shown in Figure [b). Participants are only assigned to one condition to minimize context switching, given the relatively short duration of the study, The study was conducted asynchronously using the RealHumanEval platform; participants were told not to use any outside resources (e.g., Google), and cannot paste any text originating outside the app into the oditor. Specific instructions are in Appendix A The first problem was a simple task (i.c., compute the sum and product of list.) for participants to familiarize themselves with the interface. Participants are given 35
6

---

Published in Transactions on Machine Learning Research (01/2025)
Name
Task Type
Description
is ___ored
Algorithmic
Given a string of words, write function to count the number of boredoms, where boredom a sentence that starts with the word 'I.
event scheduler
Algorithnic
Given a list of events represented as a tuple (start timee end time, scure), write a function that schedules the events to maximize total importance score.
order by points
Algorithmic
Write function which sorts the given list of integers in ascending order according to the sum of their digits.
encude message
Algorithmic
Write a function that eucodes message by swapping the case of all letters and replaces all vowels in the message with the letter that appears 2 places ahraad of that vowel the English alphabet.
triple_siue sum_ sero
Algorithnic
Given a list. of integers as an input, write a function that determines if there are three distinct elements in the list that sum to zero.
even odd count
Algorithmic
Given number, write function that computes both the number of even and odd digits.
product
Algorithmic
Given a list of numbers, write a function that returns tuple consisting of and product of all the integers.
is multiply prime
Algorithmic
Given a number, write a function that determines if the number is the multipli- cation of 3 prime numbers.
count nums
Algorithnic
Given an array of integers, write a function that returns the number of elements which has sum of digits 0.
table transform named
Data manipulation
Given a dataframe with features (age, color, dates, height), write function to transform it exactly to the following output dataframe.
table_ transform _nnnamed
Data manipulation
Given a dataframe with unnamed features (coll, col2, col3, col4), write function to transform it exactly to the following output dataframe.
(2)
Lest
Data manipulation
Given two arrays of numbers, write function that compares the means of two populations.
retriever
Edit & Augment
Fill out this class which retrieves similar vectors from a collection of vectors, which supports calculating distances from query vector, retrieving the most similar vectors to query, and generating similarity matrix for multiple queries.
Ingin_ hnttiiiittoo
Edit Augnent
Fill out this class which manages hashing, adding users, removing users, and changing passwords with verification.
authentication, which supports password
caleulator
Edit & Augment
Fill out this class which is a special ralculator that keeps track of the previous operations performed and fix a set bugs with the existing implementation..
tokenizer
Edit Augnent
Implement the build vocabulary method, which creates dictionaries mapping words unique IDs and vice versa and considers only the most frequent words, in the provided Tokenizer class.
Table 2: Task descriptions for 17 tasks evaluated in study which comprise algoirthmic problems, data manipulation problems, and problems that require editing and augmenting existing code. See supplementary material for full task specification and starter codc.
minutes complete as many tasks as possible. If 10 minutes pass and the participant has not completed the task, a button appears to provide the option to skip the task.
Tasks. We designed 17 coding tasks for the flat that can be categorized into three categories: (a) algorithmic problems from HumanEval (e.g., solve interview-style coding), (b) data manipulation problems (e.g., wrangle input dataframe into desired output), and (c) editing and augmenting code lasks (e.g., fill in provided code scaffold to achieve desired behavior). While the set of tasks does not evaluate all types of coding problems exhaustively, they do capture tasks of varying difficulty and solutions of varying length, as well the use of different programming skills, leading to varying opportunities benefit from LLM support. A short description of each task can be found in Table 2 We chose 17 tasks to build diversity across tasks while being able to collect enough samples per task. We ensured that no LLM model considered in the study, in addition to GPT-4o, could solve all tasks perfectly, so that programmers would not simply accept all LLM suggestions and that each (ask could be solved in under 20 minutes by an experienced programmer (validated through pilots with the authors and volunteer participants). to ensure that these were reasonable questions to consider for a user study. These 17 tasks are distributed into five sets, where each sot consists of a different mix of task types in varying orders but shares the first two tasks. Each participant is randomly assigned to one of these sets. The LLMs are not aware of the task descriptions unless the programmer types
7

---

Published in Transactions on Machine Learning Research (01/2025)
them in the editor chat window; this is to simulate the real world where the task description represents the programmer's 's hidden true intent. We provide examples of the coding tasks in Appendix B and in full in the supplementary materials.
Conditions. For the autocomplete conditions, we chose base LLM models that naturally generate next-word predictions, whereas the "chatty" variants of the base models are employed for the chat con- ditions. To evaluate the effect of LLM capabilities, we selected three types of models that demon- strate clear gaps in performance lected 7 LLMs for our study: 4 from the Code Llama family (Rozière et al. 2023 (CodeLlama-7b, CodeLlama-7br brbbinstruct, CodeLlama- 34b, GPT series (Brown et al. 2020) (GPT-3. 5-turbo, GPT-3. etturbo-instrune and GPT-4o). To avoid confu- sion, we refer to the autocomplete conditions by the base name of the model: CodeLlama-7bh CodeLlama-344 and GPT-3.5 (refers to with chat: CodeLlama-7b (chat) (refers to ModeLlama-7bl instruct), CodeLlama-34b (chat) (refers to CodeLlama-345 instruct), GPT-3.5 (chat) (refers to choices of parameters, system prompts, and other considerations are provided in Appendix
existing benchmarks (as shown in Figure [1] In total, we se-
along with three models from the
t--uuroo-nnstuut)) and the chat conditions by the base name of the model
5-turbo) and GPT-4o (chat). Specific
Participants. We recruited 263 total participants from university mailing lists and social media to capture a range of coding experiences. We verified that participants were above 18 years of age, resided in the United States, and correctly completed a simple Python screening question. Out of the 263 participants, we filtered out those who did not complete any task or did not write code for a period of 15 minutes during the study to arrive at 243 final participants. Of the 243 participants, 35% identify as Female. In terms of occupation, 80% are Undergraduate or Graduate Students studying computer science, 11% work in Software Development and 7% work in AI. While majority of our participants were students, only 35% of participants had less than 2 years of professional programming experience. We ensured that participants were roughly equally distributed across experimental conditions based on programming experience. 11% had never used any form of AI for coding while 66% of participants use AI at least once a woek for coding. Participants were provided with a $15 Amazon gift card as compensation. This study was approved by institutional IRB review.
User study metrics. To quantify the benefits of LLM assistance on the number of tasks completed and time to task success, we report the gap between each condition where some form of LLM assistance was provided and the control no LLM condition, which we denoled as A. For example, for lime to lask success, A < 0 for LLM support indicates that participants took less time to complete tasks with the LLM. In addition to the quantitative metrics, we also ask post-study questions to obtain participants subjective measures of their interactions with the LLM: we ask participants to rate the helpfulness of the LLM on a scale of [1, 10] and to describe how the LLM support provided (if any) was helpful and how it could be improved. We also measure two preference metries, suggestion acceptance rate and percentage of chat code copies.
5 Results
We report results for data collected from 243 participants split across the seven conditions; since condition assignment is randon each condition has around 25 to 35 participants (except for No LLM, which has 39 participants). Participants completed total of 888 coding tasks (mean of 3.6 tasks per person) on average in 358 seconds (std=153 seconds), were shown 5204 autocomplete suggestions ((Pael), l), with an average 11.3% acceptance rate, and received 1055 messages from the chat. I.LMs ((Dehat)) [), with 35.8% of messages having at least one copy event. In the following analyses, we conduct ordinary least squares regressions with Benjamini- Hochberg correction and use a significance level of 0.05. A more in-depth analysis of both datasets and results is in Appendix D
Providing LLM assistance reduces the amount of time spent coding. To measure the productiv- ity gains of LLM assistance to programmers, we look at two metries: the amount of time spent coding (in seconds) and the number of tasks completed. We first distill our observations for each metric by comparing performance for cach model type (i.c., combining autocomplete and chat models) against the No LLM condi-
Thh assignment is random across all conditions except the GPT-4o condition, as that was performed individually a later date once GPT-4o was released.
8

---

Published in Transactions on Machine Learning Research (01/2025)
GPT-3.5
CodeLlama-34p
CodeLlama-7b
GPT-4
No LLM
100 in Avg Task Duration better)
100
-50
0
50
-2
in Num Tasks Completed better)
(a) Difference in task completion time (in seconds) com- paring LLMs to the No LLM condition.
(b) Difference in number tasks completed compared to the No LLM condition.
500
12%
5
No LLM
75%
450
CodeLlama7b
Task Duration (s)
400
14%
Tasks Completed
-0%
-10%
CodeLlama7b (chat)
19%
21%
CodeLlama34b
350
CodeLlama34b (chat)
300
GPT-3.5
GPT-3.5 (chat)
GPT-4o (chat)
100
Condition
Condition
(c) Average lask completion time (in seconds) by condition.
(d) Average number of tasks completed
by condition.
We measure the effect of LLM support user study performance mean task duration in seconds (a.c) and number of tasks completed across model type (b,d). In (a) and (b), compute A, the difference between each model aype-aggrggating conditions corresponding to the same model type, c.g., Codellama7b and Codellama7b down the same metrics for each of the seven conditions and mark the percentage improvement over the No LLM condition. We observe that better LLM support can improve task completion time, but not necessarily increase the number tasks completed. Error bars denote standard errors- the standard deviation divided by the square root of the sample size (i.e., across participants), where each participant contributes single data point.
Figure 3:
and the No LLM condition for each metric. In (c) and (d), we break
tion As shown in Figure a), we find that compared to the No LLM setting where participants spent an average of 400 seconds per task, GPT-3.5, CodeLlama-344 and GPT-4o models reduce the amount of time spent per task by an average of 78, 64, and 60 seconds respectively (p = 0.04, = 0.12, and p 0.10). In contrast, CodeLlama-7b models slightly increase the average time spent on a task by 10 seconds. However, we do not observe statistical differences across any of the conditions the number of tasks completed, as shown in Figure 3(b,, meaning no form of LLM support allowed programmers solve more problems than they otherwise would have their own. We hypothesize that benefits in task completion were not observed because of the short duration of the user study (35 minutes) and the amount of time it takes to complete each task, though we do observe an increase in the number of tasks attempted.
We now consider how our observations using HuaanEalll implicate the broader code LLM evaluation landscape, specifically the use of static benchmarks and human preference metrics.
Are LLM performance on static benchmarks informative of user productivity with LLM as- sistance? We find that improvements in model-specific evaluations benchmarks tends to also improve human performance both productivity metrics in the user study (i.e., CodeLlama-7b models led to the
In Appendix we repeated the analyses controlling for task difficulty and observed same trends.

---

Published in Transactions on Machine Learning Research (01/2025)
GPT-3.5
CodeLlama-
Autocomplete
Chat
CodeLlama-7b
GPT-4o
0.0
0.1
0.2
0.3
0.4
0.5
0.20 Autocomplete: % Suggestion Accepted
0.00
0.05
0.10
0.15
2
8
Chat: % Code Copied
Al Helpfulness( f)
(a) Percentage of chat messages copied for chat conditions.
(b)
Percentage of autocom- (c) Rating of LLM helpfulness across both plete suggestions accepled. autocomplete and chat conditions.
Figure 4: Measuring participant preferences of different models by the amount of interaction with chat (a) autocomplete systems (b), with standard error. We find that preference judgments align with the reported helpfulness of the LLM assistant post-study (c); however, these preferences do not necessarily align with their actual lask performance.
500
completion time (s)
Mean task
450
Codelliema7
Codellama7h
400
CodeLiama7o
Codeliama7o
350
CodeL mm244 (chat)
GFT4 (chag
Codalamm34b (chas)
GPT- (chat)
Cookelama34b
GPT 3.5
Codellama34b
G.5
300
GPT3 (chat)
40
90
40
50
60
70
HumanEval score
MBPP score
Figure 5: Average task completion time (in seconds) plotted against LLM performance on static benchmarks (HumanEval and MBPP) for each of the LLMs evaluated in RealHumanEval. Error bars denote 95% confidence intervals.
least number of tasks completed, while GPT-3.5 models led to the most). Interestingly, this trend holds even when considering motrics with chat and autocomplete separately, in Figure ((c-)). However, significant gaps in benchmark performance result in relatively indistinguishable differences in terms of human performance. In Figure we plot task completion time against static benchmark score. While we do not necessarily expect performance gaps to be consistent, we find that, after certain point, additional gains static benchmarks may not translate to practical utility. The Pearson correlation between RealHumanEval average completion time (in seconds) and HumanFval benchmark score is -0.60 (p=0.15) and for MBPP it is -0.65 (p=0.11), which suggests a non-significant correlation, however, looking at the raw data in Figure 5 reveals the nature of the correlation. For instance, CodeLlama-344 (chat) is 19% better over CodeLlama-1 7b (chat) models on HumanFval, and participants are 22.8% (95% CT [2.8, 38.7)) faster on average complete a task with 34b 7b. Yet, GPT-3.5 (chat) model outperforms CodeLlama- yet participants equipped with GPT-3.5 (chat) models are only 8.3% (95% CI [-11.2, 24.6)) faster than those with CodeLlama-34b (chat). Surprisingly, also find no statistically significant difference between GPT-4o (chat) and CPT-3.5 (chat) in terms of task completion time.
(chat) by 85% on HumanFval, and
Do human preferences align with productivity? We also consider programmer preferences for the LLM assistant's suggestions on autocomplete and chat: the average suggestion acceptance rate and the average copics-per-response respectively. While GPT-4o, GPT-3.5, and CodeLl mma-34b models reduced the amount of time spent coding over CodeLlama-78 we do not find the same trends reflected in human preferences. As shown in Figure 4(a,, find that suggestions from Codel.laaa344 are less likely to be accepted at 5%
10

---

Published in Transactions on Machine Learning Research (01/2025)
LLM-assisted
Algorithmic Problems
False
True
Manipulation
Edit/Augment Code
300
400
500
Avg task duration (s)
Figure 6: Average task duration with and without LLM assistance with standard error by task category.
compared to 15% and 9% for GPT-3.5 5 and CodeLlama-7b (p 0.001 and p = 0.19). The same ordering occurs for the percentage of chat messages copied (27% versus 35% and 29%, though not significant) in Figure 4 b). 4b).
By analyzing the participants' qualitative responses, discussed in Section E we identify potential factors that may have contributed to these preferences, including perceived lack of context in CodeLlama-34b inggestions and slight increase in latency in CodeLlama-341 (chat) responses. These results suggest. that various external factors that might be difficult anticipate priori can easily affect human preferences even if they do not impact downstream productivity.
5.1 Additional User Study Observations
Findings on the effect of the form of LLM support and task type further illustrate the importance of evaluation with humans in the loop.
Chat support is perceived to be more helpful than autocomplete support. Even though autocom- plete and chat variants obtained similar performance on static benchmarks and participant performance in both conditions conditioned participants in the post-study questions significantly more helpful than autocomplete models (p 0.001), shown in Figure [[c. Again, observe that CodeLlama-34B models tend to be rated less helpful (3.3 out of 10), than the other two models (4.19 and 5.09 out of 10 for CodeLlama-70 and GPT-3.5). To no
model type relatively similar, we observe that chat models rated by
surprise, GPT-4o (chat) is rated as the most helpful with 6.9 out of 10 followed closcly by GPT-3.5 (chat) with 6.3 out 10.
The benefits of LLM assistance can vary by task type. We also analyze the time spent on each task category, comparing when participants have access LLM assistance versus the control condition. As shown in Figure 6 we find suggestive evidence that LLM assistance was particularly effective in reducing the time programmers needed to solve data manipulation tasks, by 26.3%, and slightly less so for problems that required editing and augmenting existing code, by 17.1%. In contrast, we found that LLMs were unhelpful on algorithmic problems, increasing the amount time spent by 11.4%. A breakdown by individual task is in Appendix D
Alternative Measures of Autocomplete Suggestion Quality. The fraction of suggestions accepted by programmers in the autocomplete conditions is myopic measure of the interaction between the program- mer and the LLM. Programmers often accept suggestions to them with code styling and then delete them promptly or verify them at later time [Mozannar et al.) 2024). On the other hand, programmers may reject suggestions inadvertently. A measure that is less myopic than a fraction of accepted sugges- tions, is the persistence of an accepted suggestion in a programmer's code. In Figure 7 we track each accepted suggestion over time see if it remains the programmer's code. We find that suggestions from GPT-3.5 and CodeLlama34b persist more frequently compared to suggestions from CodeLlama7 confirming our productivity assessments.
11

---

Published in Transactions on Machine Learning Research (01/2025)
0.9
Models
GPT-3.5
CodeLlama34b
CodeLlama7b
0.7
Stil in Code Fraction
0.6
0.5
0.4
0.2
0.1
20
30
40
50
60
Time after Acceptance (s)
Figure 7: For each accepted suggestion across the three models for autocomplete conditions, we track after 15 seconds, 30s, and 60s whether the accepted suggestion was still found exactly in the user's code. We track the fraction of the accepted suggestions that persisted across the three time points.
6 Discussion
In this work, we introduce Heaanval,, human-centric evaluation platform for code LLMs, and conduct user study using the platform to measure programmer productivity, measured by time to complete a task and number of tasks completed, assisted by different LLMs. We now discuss how the platform can be easily used for future studies to evaluate new models and interactions as well as how the data collected from our study can be used to improve coding assistants.
An open platform to support interactive model evaluation. We believe RealHumanEval can be adopted evaluate newly released LLM models in a more meaningful way and become standard for eval- uation. As demonstrated across several studics of AT-assisted programming, the effects of coding assistance show considerable heterogeneity by task, model, language, and user population, among other factors. Re- alHumanEval was built with this heterogeneity in mind, as a toolkit to enable systematic exploration of these factors within the research community. Specifically, researchers can add their own models by providing an API endpoint, add new tasks as JSON files, and configure their own values of LLM parameters (c.g., temperature, time to trigger, elc.) through a cent ralized configuration file. As an open-source project, the platform is also designed to be extensible, allowing researchers to also add their own custom components the interface with minimal friction. We will also release tutorials with the open-source platform to make it easy possible for others to get started.
Improving coding assistants and leveraging study data. We summarize participant suggestions on how coding assistants could be improved (more detail in Appendix E.. Participants overwhelmingly felt that LLMs struggled to infer the appropriate context to provide the most useful support from the information available, highlighting the need for benchmarks that capture settings where LLMs need to infer intent from partial or fuzzy instructions. There are also opportunities to improve autocomplete and chat assistants to be better programming partners (Wu et al. 2023 For example, autocomplete systems might benefit from personalization of when participants would benefit from suggestions and dynamically adjusting the length, while chat-based systems could be improved to have better, more tailored dialogue experience and better integration with the editor. Toward these goals, we release the datasets of user interactions that can be leveraged as signals of user preferences and behavior patterns. For example, the data collected in our study presents an opportunity to build and evaluate simulation environments that mimic how programmers write code with an LLM. The data can also be used to fine-tune the models; the dataset of interactions with autocomplete models Dnc reveals which suggestions programmers accept and which they reject, which
12

---

Published in Transactions on Machine Learning Research (01/2025)
be used to update the LLM and generate suggestions that maximize the probability of being accepted by the programmer.
Limitations. Firstly, we acknowledge that a set of 17 coding tasks does not span the entire set of tasks a professional programmer might encounter in their work and may limit the generalizability of our evaluations of the 7 models. We encourage future work to leverage RealHumanEval to conduct further studies with a more extensive set of tasks and with more models. We included the seven models chosen to be representative of different scales of LLMs, notably we did not evaluate models from the Claude series Anthropic (2024) but expect GPT-4o to have relatively similar performance. Our goal is not to benchmark all available LLMs, but to look at trends between human productivity and LLM performance. Furthermore, our work includes more studied models than prior work with human (Table []. Second, the coding tasks we used are of short duration, while real-world programming tasks can take hours to months. This presents a trade-off in study design: short tasks allow us to evaluate with more participants and models in shorter period but give us a less clear signal compared to longer-term tasks. Third, HeaaHuna does not fully replicate all functionality existing products such as GitHub Copilot may have the study may underestimate exact productivity benefits. Such products complex systems comprising more than single LLM, where many details are hidden and thus not casily replicable. We release RealHumanEng to enable others to build more functionality in an open-source manner.
Societal implications. While our evaluations focused on productivity metrics, there are additional metrics of interest that may be important to measure when studying programmner interactions with LLM support. On the programmer side, further evaluations are needed to understand whether programmers appropriately rely LLM support (Spiess al. 2024 and whether LLM support leads to potential de-skilling (Bommasani et al. 2021). Further, our metrics do not consider potential safety concerns, where LLMs may generate harmful or insecure code (Pearce et al. 2022 Perry et al. 2023
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXit:2303.0877, 2023.
Amazon. Ml-powered coding companion amazon codewhisperer, 2022. URL httpps://aae www.on.aazz..ooooom com/ codewhisperer/
Anthropic. Introducing the next generation of claude, 2024. URL http:://rw...attrrp.....c//////
3laude-
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXic:2108.071 07732, 2021.
Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(00PSLA1):85 111, 2023.
Christian Bird, Denae Ford, Thomas Zimmermann, Nicole Forsgren, Eirini Kalliamvakou, Travis Lowder- milk, and Idan Gazit. Taking flight with copilot: Early insights and opportunities of ai ai-powered pair- programming tools. Queue, 20(6):35-57, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiu preprint arXin:2108.. 07888.7788.00 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond. Generative ai at work. Technical report, National Bureau of Economic Research, 2023.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering. 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (tte:2177.3374, 2021.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating lims by human preference. arXiv preprint AKKi::2403.043322 2024.
Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z Henley. Conversational challenges in ai-powered data science: Obstacles, nceds, and design opportunities. arXiv preprinl ar Xie:23300 16164, 2023.
Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.011 01694, 2023.
Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The productivity effects of generative ai: Evidence from a field experiment with github copilot. 2024.
Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. Beyond text generation: Supporting
writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. pp. 1-13, 2022.
Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi-Sangdch, Han Liu, and Salcema Amershi. Aligning offiine metrics and human judgments of value for code generation models. In Findings of the 2023.
Association for Computational Linguisties: ACT 2023, pp.
Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. Advances in Neural Information Processing Systems, 36, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2023.
Github. Github copilot your ai pair programmer, 2022. URL https:://githii httpps//ggtuuu.co///eaeeeeeee//e htt.hh/b.cccceeeauuuee//pooooott
Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M Drucker. How do analysts under- stand and verify ai-assisted data analyses? In Proceedings of the CHI Conference on Human Factors in Compuling Systems, pp. 1 22, 2024.
Sandra G Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the human faclors and ergonomics society annual meeting, volume 50, pp. 904-9009 004-008. Sage publications Sage CA: Los Angeles, CA, 2006.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. arXiv preprint Xrx:::2308. 10620, 2023.
Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. Co-writing with opinion- ated language models affects users' views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2023.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth In- ternational Conference on Learning Representations, 2023.
Eunkyung Jo, Danicl A Epstein, Hyunhoon Jung, and Young-Ho Kim. Understanding the benefits and challenges of deploying conversational ai leveraging large language models for public health intervention. Tn Proceedings of the 2023 CHT Conference on Human Factors in Computing Systems, pp. 1 1-16, 2023.
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara Ericson, David Weintrop, and Tovi Gross- Studying the effect of ai code generators supporting novice learners in introductory programming. In Proceedings of the 2023 CHI Conference on Human Factors in Compuling Systems, pp. 1 1-2 2023.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retricval. arXiv preprint ar arXx:22303.0000 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. International Conference on Machine Learning, pp.
PMLR, 2023.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. A systematic study and comprehensive evaluation of chatgpt on benchmark datasets. In Findings of the Association for Computational Linguisties: ACL 2023, pp. 431 381-469, 2023.
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference Human Factors in Computing Systems, pp. 1-19, 2022.
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard- Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. Evaluating human-language model interaction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
URL https:
//openreview http:/reww....eetttttmtaaaaauuut
Jenny T Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann. Can gpt-4 replicate empirical software enginecring rescarch? arXio preprint arXiv:2310. 1.X.:221000112.1 01727, 2023.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2023.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Thirty- fifth Conference on Neural Information Processing Systems Datasels and Benchmarks Track (Round 1), 2021. URL https://openrevviet ttte/foorumiid-11AAAa
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384-1403, 2022.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Hussein Mozannar, Valerie Chen, Dennis Wei, Prasanna Sattigeri, Manish Nagireddy, Subhro Das, Ameet Talwalkar, and David Sontag. Simulating iterative human-ai interaction in programming with 1lms. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in al-assisted programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 16, 2024.
Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Oclopack: Instruction Luning code large language models. In The Twelfth International Conference on Learning Representations, 2023.
Daye Nam, Andrew Maevean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using Ilm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1 13, 2024.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Represenlations, 2023. URL https:://ppenreriew.ntt
torum//d=iaYc_KpP225
OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https:://opeaai.com// chatgpt/
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan- Gavitt, and Ramesh Karri. Asleep the keyboard? assessing the security of github copilot's 's code contributions. In 2022 IEEE Symposium Security and Privacy (SP), pp. 754 768. IEEE, 2022.
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity; Evidence from github copilot. Xiviv preprint arXim:2302.0653 2023.
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Bonch. Do users write more insocure code with ai assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Securily, pp. 2785-2799, 2023.
James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luiton-Reilly, Garrett Powell, James Finnie-Ansley and Eddie Antonio Santos. "it's weird that it knows what i want". "; Usability and interactions with copilot for novice programmers. ACM Trans. Comput.-Hum, Interact., 31 (1), nov 2023. ISSN 1073-0516. doi: 10.1145/361 33667. URL http:://doo..or///00.466//6//
replit. Meet ghostwriter, your partner in code., 2023. URL
Steven Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. The programmer's assistant: Conversational interaction with large language model for software development. In Proceedings of the 28th International Conference Intelligent User Interfaces, pp. 491-514, 2023.
Baptiste Rozière, Jonas Gchring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et. al. Code llama: Open foundation models for code. arXiv preprint XrXi::2308. 12950, 2023.
Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint ar arXiv:2301. 206:0030000653, 2023.
Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Susmit Jha Prem Devanbu, and Toufique Ahmed. Quality and trust in llm-generated code. ar preprint arXiv:2402.0004 2024.
Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Fxtended Abstracts, pp. 1-7, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Vera Liao, and Jennifer Wortman Vaughan. Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in ai-powered code completions, arXiv preprint arXiv:2302.072.58 2023.
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code generation modcls. In Proceedings of the 61sl Annual Meeling of the Associalion for Compulational Linguistics (Volume 1: Long Papers), pp. 13818 138488,333, 2023.
Tongshuang Wu, Kenneth Koedinger, et al. Is ai the better programming partner? human-human pair programming vs. human-ai pair programming. arXiv preprint arXi::206600511 2023.
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao,
Li Zhu, Shuiguang Deng, et al. Codescope: An exccution-based multilingual multitask multidimensionaal benchmark for evaluating Ilms on code understanding and generation. orXiv preprinl orXi::2311.08588 2023.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet n12code: survey. In Proceadings of the 61st Anmual Meeting of the Association for Compulational Lánguistics (Volume 1: Long Papers), pp. 7443 7464444444- 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhnang, Zhanghao Wu, Yonghao Zhnang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a- judge with MT-bench and chatbot arena. In Thirty-seuenth Conference on Neural Information Processing Systems Datusels and Benclemarks Track, 2023. URL htttp:: http:///ww.......e..oomootttccaa
Ming Zhu, Ancesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. Xlcost: A benchmark dataset for cross
cross-lingual lingual code intelligence, 2022. URL https://arxiv.orr /bs//
2206.08474
Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sit- tampalam, and Edward Aftandilian. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 21-29, 2022.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Appendix
A User study details
A.1 RealHumanEval interface screenshots
We show examples of the RealHumanEval web interface used in the study; autocomplete conditions (Figure and Figure 9 and chat conditions (Figure 10 Note that the interface is the same as that of the autocomplete conditions for the no LLM condition, except there no LLM to provide any inline code suggestions.
Pows
31.01
wilst Emplee sum ahout be equel to and emply product stould be equal to
lonGrumerrs)
(0.1) sun_podudj1.22
inbtun (9.1)
12
product
Quuinee product
17
FRu
deSubnn for Tooking
Figure 8: Screenshot of the autocomplete LLM assistance interface in our user study.
CoaeeCook
Change Theme
Shee Chernds Butlontre
Shaw Inatuations
34:36
intggrr in Enjty sum should equal to and emery product should equal prnnuag)
p00 sum.sooouut 2,3.
[1h, 24)
Run
Submi for Testing
Oubut RunSubiit:
Dstges:
Figure 9: Another screenshot of the autocomplete LLM-assistance interface in our user study.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Hoand Coobs
Change Tiere
Mew the Tach Iutrettd
34:33
and a prodic ot of all the integees in lit Emery shoule equal is and emery procud ahoull repue produck(])
Claa
1)
sum pwo.uant11112 2,3,4)
Chatbot
You can use the "sunt] finction get the cum of the elersents in Here un sooemtll
Dapy
(onttal proitt
jype your easeag....
Suomt Testing
Output RuneBubent:
Outgar: #i aboan horo when Rin dilatt prosand.
Figure 10: Screenshot of the chat LLM-assistance interface in our user study,
A.2 User Study Instructions
Before participants enter the main interface, they are provided with the following text:
After you fill out the information here, click the Start Experiment button to proceed.
Please DO NOT refresh press back you may lose fraction of your progress, if needed you can refresh while coding but you will lose your code.
Your name and email will NOT be shared with anyone used in the study.
Note that there chance the interface may not have A1, that is bug. By performing this task, you consent to share your study data.
In all conditions, a pop-up is displayed with the following instruction: Welcome to the user study! You will first complete a tutorial task to make you familiar with the study.
You will be writing code in Python only and use only standard python libraries and only numpy and pandas.
After the tutorial task, you will have 35 minutes total where you will try to solve as many coding tasks as possible one at time.
It is NOT allowed to use any outside resources to solve the coding questions (e.g., Google, Stack- Overflow, ChatGPT), your compensation is tied to effort only.
A.2.1 Autocomplete Condition
You will write code in the interface above: a code editor equipped with an AI assistant that provides suggestions inline.
The AI automatically provides suggestion whenever you stop typing for more than seconds.
19

---

Published in Transactions on Machine Learning Research (01/2025)
You accept suggestion by pressing the key [TAB] reject suggestion by pressing [ESC].
You can also request suggestion at any time by pressing [CTRL+ENTER] (Windows) or [CMD+ENTER] (Mac).
You can run your code by pressing the run button and the output will be in the output box at the bottom in grey.
Press the submit button evaluate your code for correctness. You can submit your code many times as you wish until the code is correct.
If you cannot solve one of the tasks in 10 minutes, a button "Skip Task", only press this button if you absolutely cannot solve the task.
Note: please be aware that the AI assistant is not perfoct and may provide incorrect suggestions. Morcover, the Al may generale polentially offensive suggestions especially if prompted with language that is offensive.
A.2.2 Chat Condition
You will write code in the interface above: code editor equipped with an AT assistant chatbot in the right pancl.
The AT chatbot will respond to messages you send and incorporate previous messages response. The Al does not know what the task or the code in the editor.
When the AT generates code in its response, there is COPY button that will show up above the code segment to allow you to copy.
You can test your code by pressing the run button and the output will be in the output box at the bottom in grey.
Press the submit button to evaluate your code for correctness. You can submit your code as many times as you wish until the code correct.
If you cannot solve one of the tasks minutes, a button "Skip Task", only press this button if you absolutely cannot solve the task.
Note: please be aware that the AI assistant is not perfect and may provide incorrect suggestions. Morcover, the Al may generate polentially offensive suggestions especially if prompted with language that is offensive.
A.2.3 No LLM Condition
You will write code in the interface above: code editor.
You can run your code by pressing the run button and the output will be in the output box at the bottom in grey.
Press the submit button to evaluate your code for correctness. You can submit your code as many times as you wish until the code correct.
If you cannot solve one of the tasks in 10 minutes, a button "Skip Task", only press this button if you absolutely cannot solve the task.
20

---

Published in Transactions on Machine Learning Research (01/2025)
A.2.4 Post Dost-Study Questionnaire
Thinking of your experience using Al tools outside of today's session, do you think that your session today reflects your typical usage of AI tools?
How mentally demanding was the study? (1-20)
How physically demanding was the study? (1-20)
How hurried or rushed was the pacc of the study? (1-20)
How successful were you in accomplishing what you were asked to do? (1-20)
How hard did you have to work to accomplish your level of performance? (1-20)
How insecure, discouraged, irritated, stressed, and annoyed were you? (1-20)
Overall, how useful/helpful was the Al assistant? (1-10)
In which ways was the AI assistant helpful? What did it allow you to accomplish? (freo-text)
How could the Al suggestions be improved? (free-text)
Additional comments (Optional): anything went wrong? any foodback? (froc-text)
To ensure consistency in responses to scale-based questions, we labeled 1 with "low" and either 10 20 with
"high" depending the question.
A.3 Data release considerations
We took the following measures mitigate potential ethical concerns regarding the release of the study. First, the study protocol was approved by institutional IRB review. Second, before participating in the actual study, all participants were provided with a consent form outlining the study and the data that would be collected part of the study (including interaction data with LLMs) and provided with the option to opt out of the study if they so choose. Finally, after data collection and prior to public data relcase, the authors carefully checked all participant interactions with LLMs, particularly chat dialogue, to ensure that no personally idenbifiable information was revealed.
B
Task Design
B.1
Task categories
Algorithmic coding problems: Many coding tasks require programmers to implement algorithmic think- ing and reasoning and are widely used to evaluale programmers in coding interviews. To identify algorith- mic coding problems, we sample representative problems from the HumanEval dataset CCen et al. (2021). Given gpt-3. 5-turbo''s high performance on this type of problem, we ensure that we also include problems where fails to solve the problem on its own. We evaluated each question using test cases from the Hu- EanEval dataset. We included the following problem ids from HumanFval: is bored 91, is _s_ multiply primme 75, encode. message 93, count nums 108, order pointssntss 145, even odd count 155, sum product 8, triple sum to_ zero 40. In addition, we created a custom problem called wvent_schedulee All tasks with unit tests will be released.
Editing and augmenting existing code: When working with existing repositories, programmers will often need to edit and build on code that may have been written by others (Sobania et al. 2023). We designed questions where participants are either provided with either code scaffold fill or with code body that they are asked to modify the functionality of. When designing such questions, we take care avoid common implementations (e.g., a traditional stack and queue) that would have appeared in LLM training data. We
21

---

Published in Transactions on Machine Learning Research (01/2025)
also constructed set of test cases for each question. The four problem names are calculator, tokenizer, login authenticator and retriever.
For example, here is the login authenticator problem description:
Your goal to implement the LoginAuthenticattor class, which will be used authenticate users of system. The class will include the following methods:
hash password (Private Method): Creates hash of given password. Accepts password (string) and returns the hashed password using any hashing technique.
add user Method: Adds new user to the system with username and a password. It checks if the username already exists, hashes the password if it does not, and stores the credentials. Returns True succossful.
remove_usee user Method:
Removes a user from the system by deleting their username entry from self. auser_creeentiieee if it exists. Returns True if successful.
change password Method: Changes & user's password after authenticating the user with their old password. If authenticated, it hashes the new password and updates self. user credentials. Returns True if successful.
The programmer is given the following initial code:
class LoginAuthhenicattrr
def -_intt__ (self):
# DO NOT CHANGE
self user user_credentiall O dictionary for bashed_ passeo_
def _wahaahh__a__e passsordstr password):
WRITE HERE return
def add add_user (self, usernane, password);
# WRITE CODE HERE return
def authenticate user (echettiatte_user(eell,, (self, usernane, password):
# DO NOT CHANGE
#Checks if the given username and password are valid if username not in self userruuurrrrrrereeeee user__ceeeettalle return False return self.user: credeccceeeti (username) == self passwwshsssoodd (password)
def remove_user (self, username);
WRITE CODE HERE return
def change_password (self, usernane, old_password, mex_passvord):
WRITE return
Data science tasks: Given the increased usage of data many engineering disciplines, programmers are often involved in data science problems. We design data science problems inspired by the DS-1000 dataset Lai et al. (2023), where participants need to perform multiple data manipulation and wrangling operations and return resulting Pandas dataframe. To ensure that an LLM cannot achieve perfect accu- racy on its own, we only show an example of the input and target dataframes without providing specific instructions on each operation. The code will be evaluated based on the norsectness of the dataframe in an wlement-wise fashion. The four problem names are table trrnssorrn. named, table transform unnamedl, table transform 2rannform_uummmmd2 and test.
Here for example the problem table transform unnamed1:
22

---

Published in Transactions on Machine Learning Research (01/2025)
Given the imput pandas Frame:
coll
col2
col3
col4
col5
0
6
1
5.38817
3
2
1
9
2
4.19195
5
8
2
10
8
6.8522
8
3
6
7
2.04452
8
1
10
8.78117
10
10
Transform this DataFrame to match the following output structure, rocognizing the rela- lionship between the input and output DataFrames:
coll
col2
co13
0
6
8.38817
15
9.19195
2
25
14.8522
31
10.0445
4
32
11
18.7812
0
0
0
0
0
0
Implement function named transform lfttoomm_mfffooo that takes the input DataFrame and returns the transformed DataFrame, discovering and applying the patterns between them.
The programmer is given the following initial code:
import pandas as pd from io inport StringI0
# Original dataset data 1.> coll, co12, col4, co15
9G1188.38818733033355855 33,2
19949914999414329446 8,8
10.8.6. 8.8.668822195503865555888 88
5.772.004622977155775 ,8,7
6,10.8.78117336399454,888 10,
Read the dataset into a DataFrame (Striing (data))
def transforn_ _f (df):
# Your code here
print (transform_df (df))
B.2 Task organization
We created five task sets where we fixed the first task (in addition to the tutorial sum product task) and varied the remaining tasks randomly ensuring split across the categories. The five sets are:
Task Set 1: even count, triple_sum sum to zero, table transform named, tokenizer, en- code message, test, event. scheduler.
23

---

Published in Transactions on Machine Learning Research (01/2025)
2. Task Set 2: even even_odd _en__dd__ouut, is bored, login authenticator, is amiltppl_pprime count nums, table transform transform_named,, calculator.
3. Task Set 3: even odd count, count nums, calculator, table transform unnamed2, lo- gin authenticator, encode message, is bored.
Task Set even odd count, order by points, retriever, triple sum zero, lokenizer, event scheduler, encode message.
5. Task Set 5: even aven_odd count, is multiply_ prime, table transform unnamedl, test, is bored, order by points, triple_sum sum to zero.
C LLM Details
90
80-
70
Po558
HumanEval
MВPP
CodeLlama77
CodeUama34b (chat)
GPT-3.5 (chat)
CodeLlama7b (chat)
Cada4
Figure 11: Pass@1 of LLM models and their chat variants on two canonical benchmarks, HumanEval and MBPP (results from (Rozière et al.] 2023 Liu et al. 2023) ) showing that CodeLlama-78 models perform worse than CodeLlama-344 models, which are less performant than GPT-3.5 models.
We select three models of varying benchmark performance as shown in Figure 11 Here we provide links to model weights (where applicable) and any additional details.
CodeLlama (7b, 34b) and CodeLlama Instruct (7b, 34b). Accessed from https: together xyz/ Note that the base model variants are no longer available from this source. The license for the CodeLlama models is at LICENSE
GPT-335-turbo. Specific model version is 11..33..-turb-0661 -013. Accessed through the OpenAI API. This closed model and does not have an associated license.
GPT-s.--turbo-nstruuct Accessed through the OpenAl API. This is closed model and does not have an associaled license.
GPT-4o. Accessod through the OpenAI API. This a closed model and does not have an associatod license.
LLM parameters. For LLMs, we used a lemperature setting of to ensure varied responses. For autocomplete LLMs, we needed a way set the the length of the suggestions with a fixed number since base LLMs are not trained with EOS token and thus do not know when to stop generating code. We first looked at how current open-source Copilot systems determine the suggestion length for the autocomplete suggestions, We found that all open open-source systems (such as Fauxpilot and HuggingFace's personal Copilot m use fixed-length suggestion, and each with different length parameter. We experimented in initial study pilots with different choices and found that token length 64 made the suggestions more likely to be correct
httpe://huuggingfccc..oo// http:///wccccccco////ooooooaa--. copplltt
24

---

Published in Transactions on Machine Learning Research (01/2025)
while not being too short. However, to allow future systems to more smartly pick the suggestion length, decided to make the suggestion length random (truncated Gaussian) on the interval [10,120] with mean 64 so that we can learn from this data in an unbiased manner. If we were to use model confidence, we would have to use an arbitrary threshold to know when to stop generation, which may be problematic in unforeseen ways. Equipped with our study data and interface, future work can pick this confidence threshold in a more sound manner by trying maximize acceptance rate. Since the design of RealHumanEEval was modular, it should be easy to plug in future mechanisms. For the chat LLMs, we set the max token parameter 512 tokens constant.
Why we did not select other model candidates. Of the CodeLlama models available to use at the time of the study, we omitted 1odeLlama-13b. We did not select CodeLlama-131 as its performance on HumanEval very similar to the 7b variant. Additionally, Codellama--70b and Claude-3.5-sonnel had not been released when we conducted the study,
C.1 Prompts used
We used the following system prompt for all chat-based LLMs:
You are an expert Python programmer, be helpful to the user and return code only Python.
For autocomplete-based based LLMs, the first line of the prompt is always the following:
# file is main.py. ONLY CODE IN PYTHON IN THIS FILE
These prompts help to ensure that LLM responds Python.
D Additional Results
D.1 Pre-registration
We pre-registered our study design prior to data collection but not the analysis plan htpps::://spprrditteet org/blind. K3P KiJ, We deviated from the initial plan by adding a condition with GPT-4o as post- hoc exploration when GPT-4o was released. Due to the limit on the number of participants who completed the task within the timeframe the study, we thus ended up with fewer participants in the final dataset than we originally anticipated being able collect (i.c., per condition instead of 50 per condition). As result, we opted pool together data from the same model class to study both hypotheses, All other additional analyses in this work exploratory and were not pre-registered.
D.2 Dataset Analysis
We post-processed both datasets to ensure they did not reveal any identifying information about participants contain harmful language.
Autocomplete dataset. Recall that users had the option to request suggestions via hotkey or were provided the suggestion after some time. As shown in Figure 12 participants are much more likely to accept suggestions if they request them. Interestingly, CodeLlama 34b suggestions seemed be more preferred than Code when requested.
Chat datasct. We analyze the 1055 chat messages participants sent across the three conditions, as shown in Figure 13 On average 2.7 messages were sent per task with a length of 104.8 characters. We note that there is a particularly long tail in terms of words appearing in chat messages because many questions contained implementationnspocifee variables. In accordance with our findings that LLMs were most useful for data manipulation tasks, we also find that participants engaged with LLM support the most for those tasks.
25

---

Published in Transactions on Machine Learning Research (01/2025)
GPT-3.5
CodeLlama-344
CodeLlama-7b
0.0 0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Non-requested: % Suggestion Accepted
Requested: % Suggestion Accepted
Figure 12: Comparing the acceptance rate for when participants requested suggestions with when they were automatically provided with suggestions by the autocomplete system.
1000
200
Frequency
150
Frequency
750
500
100
50
250
0
0
10
1000
2000
3000
Number of messages per task
Length of Chat Message
Algorithmic Problems
600
Frequency
Data Manipulation
400
200
Edit/Augmmet Code
2.5
3.0
3.5
4.0
0
Number of messages
Words appearing in chat msgs
Figure 13: Analysis of the number of messages per task (top left), the length of chat messages (top right), the number of messages sent per task category (lower left), and the frequency of words appearing in chat messages (lower right).
D.3 Accounting for task difficulty
To facilitate comparisons between different sets of tasks, which may have varying difficulty, the value of each metric is z-scored within the task set:
M J.M.t
M
26

---

Published in Transactions on Machine Learning Research (01/2025)
where Mi, is the value of metric M achieved by participant i, z-scored within task set AM, and M,t are the mean and standard deviation of metric M for task set 1, across all participants. We rerun our analysis for performance metrics and present results in Figure 14
GPT-3.5- I-Turbo-lnsstuct
GPT-3.5 5-Turbo-Instruct
3odeLlama-34b
CodeLlama-34p
CodeLlama-7b
CodeLlama-7b
GPT-3.5-Turbo
GPT-3.5-Turbo
GPT-4o
GPT-4o
CodeLlama-334-inssruucc
Codellama-34b-Insttuct
CodeLlama-77-Instruuct
CodeLlama 7b Instructt
No LLM
No LLM
00.5
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
in Avg Task Duration (z-score)
in Num Tasks Completed (z-score)
Figure 14: Performance results across models, z-scored to account for potential variation in task difficulty across sets.
D.4 Task completion time
In Figure 3 we find the most significant. differences between models in terms of task completion time. We further analyze task completion time across multiple axes.
By task type. When comparing when participants have access LLM assistance versus the control condition, as shown in Figure D we find suggestive evidence that LLM assistance particularly effective in reducing the time programmers needed to solve data manipulation tasks and problems that required editing and augmenting existing code, but not for algorithmic problems. We also analyze whether participants benefited from LLM assistance on an individual task level in Figure We observe that trends for individual tasks within a category are similar, indicating the importance of understanding how programmers interact with LLMs for different types of lasks.
Verifying outlier behavior. We plot histogram of task completion times in Figure 16 to verify that across participants, there were not significant number of outliers. We also performed a similar check by plotting across conditions in Figure 18 to ensure that there was not, differing behavior across participants (e.g., no bimodal behavior within given condition).
D.5 Code Quality Metrics
Code Comments. Code written with the assistance of the LLM will inherit some of the characteristies of the writing style of the LLM. One instance of that is comments the code written. We investigate the number of comments written by participants for the different types of interaction with the LLM: autocomplete, chat, or no LLM. We count how many additional comments are the code participants write compared to the number of comments in the provided code participants complete. Participants in the autocomplete conditions wrote 0.85 + 0.1 additional comments, in the chat condition wrote 0.59 + 0.08 comments and those in the No LLM condition wrote 0.41 + 0.13 comments. Participants writing code with autocomplete LLM write twice many comments as those without an LLM (p 3e (e- 6). There are two possible explanations for this increase: first, programmers usually prompt the LLM with inline comments to get suggestion they desire, and second, we often observe that code generated by LLMs is often heavily commented. This indicates that we can potentially diffcrentiate code written by programmers with LLM assistance by the number of comments in the code.
D.6 TLX Results
We measure cognitive load via a series of questions from the NASA Task Load Index (TLX) Hart (0066 summarized in Table 8
27

---

Published in Transactions on Machine Learning Research (01/2025)
sum jum_product
LLM-assisted
test
False
True
table_tran transform named
table transform_ unnamed1
table transform unnamed2
tokenizer
login_ authenticator
retriever
even odd count
triple_ semeessm_uuu zero
encode encode_message
bored
is pl_mulliplyyprrmg prime
caunt_
order becirby___iinntt
200
400
600
800
Avg task duration (s)
Figure 15: Time to task completion with and without LLM assistance, reported by task and grouped by task category, with standard error.
Model
Frustration Performance
Temporal Demand
Physical Demand
Effort
Mental Demand
GPT-4
9.83
8.40
13.00
3.60
11.33
12.00
GPT3.5
8.11
9.11
12.74
4.71
11.80
11.37
CodeLlama-34b
13.54
7.96
11.18
5.18
10.86
10.93
CodeLlamm-7b
11.88
6.50
13.88
4.88
10.65
14.50
GPT3.5 (chat)
10.09
9.28
12.19
4.94
10.88
12.09
CodeLlams-34b (chat)
11.04
8.00
13.44
5.16
11.40
12.88
CodeLlama-7b (chat)
9.54
7.43
12.57
6.75
11.93
11.82
No LLM
9.62
7.56
13.51
5.95
11.79
12.10
Table 3: TLX scores across conditions.
28

---

Published in Transactions on Machine Learning Research (01/2025)
Mean Median
120
100
Frequency
80
60
40
20
0
O
250
500
750
1000
1250 1500
1750
Time to Complete a Task (seconds)
Figure 16: Histogram depicting the distribution of task completion times across all participants and condi- tions. The histogram overlaid with dashed lines representing key statistical measures: the mcan (red) and the median (green).
GPT-3.5
CodeLlama34b
CodeLlama7b
GPT-3.5 (chat)
GPT-4o (chat)
CodeLlama34b (chat)
CodeLlama7b (chat)
No LLM
-40
-200
400
800
A in Avg Task Duration
better)
Figure 17: Violin plot of the difference average task duration times (in seconds) between the No-LLM condition and all other conditions.
29

---

Published in Transactions on Machine Learning Research (01/2025)
GPT-3.5
CodeLlama34b
700
CodeLlama7b
Time to Complete Task (s)
GPT-3.5 (chat)
GPT-4o (chat)
600
CodeLlama34b (chat)
CodeLlama7b (chat)
No LLM
400
300
200
tutorial
1
2
3
4
5
Task Index Solved
Figure 18: For each of the seven conditions, we plot the average time for participants to complete the tutorial task, the first task they solved, the second task they solved, and so on.
5
No LLM
Codellama7b
Avg Num Submits per Task
4-
Codellama7b (chat)
Codellama34b
Codellama34b (chat)
3
GPT-3.5
GPT-3.5 (chat)
2
GPT-4o (chat)
1
Condition
Figure 19: Average number of submissions by participant per task across condition. We do not observe
difference in the number of attempted runs, indicating that participants did not try to brute force solutions.
30

---

Published in Transactions on Machine Learning Research (01/2025)
Examples of helpful interactions
Actionable response to question
Brainstorm test cases
how would geet the day from this 2019-03-06 where want the number
defeven,
Chatbot
You can use The 'day' attribute of the datetime.da date class the day of the month. example
even_sountt>+
num numm/ts
import datatime
do datertmm 3,
day date day printddey printed
euen mmmnt count pont sour (21aa)) mnaieow soum]tab5/8 adatleenn add count IMN
Examples of unhelpful interactions
Verbose explanation. code
Incomplete. illogical generation
User
Writ python cade detarmine whether
number the product prime numbers
www.muplyyyymmc.cm primee
Chatbot
number product prime numbers we can follow the steps below:
desermine
1. Create function -s.product_nt. nuum' wtw.cc_ttt__pimsssuuuuu that
primesal
accepts
Initialize variable 'count" to keep track the number of prime factors.
3 Create helper function is, primm(()) that accapts integer checks whether is number.
Figure 20: Examples from of helpful and unhelpful chat and autocomplete interactions from the user study, While these examples showcase how LLM assistance can improve programmer productivity (e.g., by providing actionable responses and generating test cases), they also highlight how programmer- interactions can improved. We discuss design opportunities collected from post-task participant responses in Section and provide more examples in Appendix F
E Design Opportunities
To understand the design opportunities around improving the coding assistance provided through RealHumanEval, we analyzed a post-study question on how coding assistants could be improved. Answers to the question were collected in free response format and were optional, though it was answered by the majority of participants. We summariie participant suggestions into general comments that could apply both types of interactions and identify autocomplete- and chat-specific suggestions!
Both autocomplete and chat models need improved context. A theme that spanned both types of interactions and model lypes was the perceived lack of context that the LLM had about the general task when providing either suggestions or chat responses (example shown in Figure 20 While one might expect that a more performant model might mitigate these concerns, we do not observe a significant decrease in mentions regarding this issue for GPT-3. models compared to both CodeLlama-78 and CodeLlama-33 models. general, may not. be obvious how to concisely specify the full "context" " recall that we intentionally considered set-up where the LLM is unaware of task T' to mimic realistic constraints but the development of new interfaces to facilitate context specification and mechanisms to prompt for additional task-specific information could improve LLM generations. Additionally, further baseline checks can be implemented to minimize concerns mentioned by participants (c.g., ensuring that the LLM responses are provided in the correct programming language, beyond prompting-basee approaches implemented in our study). We note
We omit. obvious, blanket suggestion for replacing the assistant with better LLM, model-only performance one of the independent variables in our experiment and more performant model would undoubtedly improve the assistance provided.
31

---

Published in Transactions on Machine Learning Research (01/2025)
that issues surrounding context control have also been highlighted in prior work (Chopra et al.
2023
Barke
et al. 2023
Autocomplete-speeffc suggestions. We highlight the three most commonly mentioned avenues of im- provement across all three model types. (1) Minimize suggestion frequency: Participants noted that the frequency of suggestions appearing the code editor could disrupt their train of thought. To address this issue, may be preferable to allow participants to turn off the LLM model when they are brainstorming the next steps or to modify the LLM to detect when participants may not need as frequent suggestions based on their current coding behavior. Moreover, we observe quantitatively that participants are between 3 - 10x more likely to accept an assistant's 'ss suggestion if they requested it, as shown in Figure 12 (2) Dynamic suggestion length: A common issue with autocomplete interactions noted by participants was the presence of "incomplete variable definitions or function implementations" " and "fragmented code" (e.g., Figure 21 (left)). As this behavior is a product of the fixed length of LLM generations, autocomplete assistants can be im- proved by ensuring the suggestion is complete before terminating generation. (3) More concise suggestions: Finally, participants also noted that code completions could be more concise, as "it was overwhelming" and "large chunks of code... start deviating from the task question" (e.g., Figure 21 (right)). It is an open question to determine the appropriate length for how much code to generate in a context-aware manner.
Chat-specific suggestions. There were three common suggestions shared across models. (1) Responses should focus on code, rather than explanation: It is well known that chat LLMs tend to generate verbose responses, which could be detrimental when used as programming assistants. An example of a lengthy re- sponse is in Figure 23 In particular, participants noted the additional time required to read large blocks of texts and suggested to "get rid of all explanations and stick to code only, unless the user specifies they want explanations." Additionally, when focusing code, participants suggested that the chat assistant could anticipate alternative implementations (2) Improved dialogue experience: First, instead of making assump- tions about potentially ambiguous points in a programmer's 's question (e.g., as in Figure 22), participant suggested that the LLM "could ask clarifying questions or provide multiple suggestions." " Additionally, in particular for CodeLlama-7b, participants asked for better consistency across multiple chat messages (e.g., "It wasn't 't able to refer back to previous messages that I had sent when answering a question."). (3) Better integration with code editor: Currently, the burden is on the programmer to appropriately prompt the chat assistant with questions and then to integrate chat suggestions into the code body in the editor. This onus can be reduced by more readily incorporating "the code and the most recent error, if any, as well as the test case that generated it in the context for the assistant" and "autocorrect code" based on its suggestions.
Why was ModeLlama-33 34b less preferred by users? Based on participants' survey responses, we identify two potential reasons that might qualitatively explain why CodeLlama was less preferred for both au- tocomplete and chat. For autocomplete, the lack of context was a particularly prevalent issue in responses for CodeLlama-344 mentioned by 54% of responses, compared to 32% and 28% of CodeLlama-78 and G responses respectively. In particular, participants noted that the generated suggestions were often irrelevant to the prior code and in the wrong programming language. We show examples of rejected sugges- tions that illustrate a lack of context from participants who interacted with the CodeLlama- model in Figure 24 For chat, while there were no exceptional concerns about lack of context, CodeLlama-344 had the most mentions of latency as a point of improvement (6 mentions as compared to only and mentions for CodeLlama-7b and GPT-3.5 respectively). For example, one participant noted that "the responses are slow so sometimes it was faster to go off of my memory even if I wasn't sure if it would work." " Indeed, we found that CodeLlama-344 response time (about 10 seconds) was on average twice slow as either CodeLlama-7b or GPT-3.5 5 (about 5 seconds). We note that this slight delay did not significantly impact any participant's 's performance metrics.
E.1 Opportunities to use data
Simulating programmer-LLM interaction. The data collected in our study presents an opportunily to build and evaluate simulation environments that mimic how programmers write code with an LLM, Essentially, the simulator could be used to more efficiently replicate the results of RealHuma and evaluate wider set of models. However, despite initial work on simulating programmer- interaction Mozannnr et al. 2023 building a useful simulator requires significant training and validation. Our dataset
32

---

Published in Transactions on Machine Learning Research (01/2025)
provides training data for both chat and autocomplete interactions: The dataset of interactions with the chat models Dehat allows us to simulate the queries programmers make to the chat assistant given the code they have currently written. The dataset of interactions with the autocomplete models Dac can allow us to simulate finer-grain interactions with LLM suggestions such as verifying and oditing suggestions, among other activities outlined in (Mozannar et al, 2023). To validate a proposed simulator, one should test whether it faithfully replicates the trends observed in Reaa before it can used as an evaluation benchmark.
Optimizing suggestions from human feedback. In addition to using the human feedback data to simulate the interaction, one can use it to fine-tune the models. For instance, the datasct of interactions with autocomplete models Dac reveals which suggestions programmers accept and which they reject, which can be used to update the LLM and generate suggestions that maximize the probability of being accepted by the programmer. Moreover, the dataset also captures how accepted suggestions were edited over time, which can be used to generate suggestions that are more likely to persist in the programmer's code. Finally, LLM that is not instrructionttuee usually requires specifying a maximum generation length parameter to intentionally randomized
stop the generation of code suggestion. In autocomplete implementation, the maximum suggestion length of the generated suggestion to be between the range [10, 120] with mcan token length of 64. This design decision can provide yet another signal about when the LLM should stop generating code.
F
Example user interactions
eefeevee_ode jounttiuum
O
Driginal
odd
data
colll col col8, coll
IRr k In mumi
5.1.5 388 H88867544033364/
mml 7% 2 ewe
9.2.4 1915965140172988844 5.
10,8,6 35219990339555566 8.1
5.2.2 6,7,044572492 044452222251577444
datasot OataFrame
of pd.rend pdd.eed__cw(Stnnnne Cldoal
for Telle wnmumetti (140)
der transform, di(all
In Add col8 col4 and cols
col-elf e61 LoW] dfii)
(oll)) (rouneills 17200
Figure 21: Examples of problematic autocomplete suggestions: incomplete suggestion (left) and starting new irrelevant function (right).
33

---

Published in Transactions on Machine Learning Research (01/2025)
you see any errors in this function?
order hy_points/umm:
ans =[]
for number innume:
palit=fintell str(number) ifxis
FFnumber 0;
split0]= spllitt(]]
ans.sorti
return (+[1] forxinans]
Chatbot
Yes, there is an error in the provided function. When sorting the "ans' list, it uses the default sorting order, which ascending
sum of split digits. However, the desired behavior is to sort in descending order based on the sum.
To fix this, you can pass lambda function as the key" argument to the .sari(]" method, which specifies
should sorted based the negated
User
The goals to sorted ascending order based on the sum the split digits
Chatbot
Apologies for the confusion. the goal is to sort the list in ascending order based on the sum of the split digits, the original code any additional arguments, sorts the list in ascending order.
The ssort)) method, without
Figure 22: Example of a chat interaction where the chat assistant could have proactively asked more clarifying questions. Note that the chat agent response is shortened (by excluding code snippet) for brevity.
Write python code to determine whather a number isthe product of prime numbers
Chatbot
To determine whethera number the product of prime numbers, can follow the steps below:
Create function is_product__de Btt.uutt.o.....rmmeeemee merrssmmmm))) that accepts integer numm as parameter.
Initialize variable "count" keep track number prime factors.
3. Create a helper function it_prime(n)) that accepts integer 'n' and checks whether is prime number. This function will True prime, otherwise.
square each number a prime factor of num"
(inclusive) using for loop. Checkit
If the
number
factor, increment the "oount"
If the count' exceeds 3, break
loop as there can't be more than prime factors.
After the loop, check the countt is equal to
If it is equal to 3, return True as "numm the product 3 prime numbers.
If it is not equal to 3, return False.
Here the implementation the aboov steps:
defis_product_d Btt____________eee__e [_prtmetnum): count=0
return intimattssgrtlll
nangel)
retum
return
for In ranget2, intumattsssnnmnmmm
Frum senaddn_nmmelll
countt>1.
break
return Dount am
function is_product_ad pprimes/num) check if number num" is the product of prime numbers. The function will return True is, and False otherwise.
cau
Figure 23: Example an overly lengthy response from a chat interaction.
34

---

Published in Transactions on Machine Learning Research (01/2025)
Import pandas as pd
class Tokenizer:
from import StringiO
Init [rsll_ssofttt 200abb self. max, vocab sze-max vvccab size ward to id=if)
Texcludingge ingeode define detefrrmae
del transform, df
Your code here
del tokenizelseif, texti:
printtdr "hhighh""" int" K/PRE>
do not change
it Split
import pandas is import Stringl@
return
1d wordget[wword M
<OST
Figure 24: Examples of rejected suggestions from CodeLlama-344 -44b, which failed to consider the context. of existing code: (left) the suggested code tried import the same packages that are already present and (right) the suggested code trails off into irrelevant, non-Python text.
35


Reviews:
[{"review_id": "QqRAJ6qh1l", "reviewer": "Reviewer_tH1K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* This kind of human-centric study is very important to understand the actual impact of LLMs.\n* The RealHumanEval platform that the authors will release will be potentially useful for future studies of this kind.\n* The paper is very well written and the study was rigorously executed. I can't point to any important issues in the methodology.\n* The study was pre-registered (though with differences in the analysis and findings, but still better to have some pre-registration in any case!)\n\n# Weaknesses\n\n* I missed having a more concrete sense of the actual 17 tasks. I saw the problem IDs in the Appendix, but it'd have helped to have at least a short description of the actual tasks to be completed.\n* I was also expecting an analysis correlating each LLMs performance on HumanEval (or other benchmark), vs its impact on productivity. That is discussed in the text in finding 1 (\"(1) Are LLM performance on static benchmarks informative of user productivity with LLM assistance?\"), but pointing at specific data points.\n* One bit about the results seem puzzling: LLM assistance reduces time in each task, but doesn't generally increase the number of tasks completed. I suggest the authors clarify why this is the case. I'd suspect that the time saved in the duration of the study is generally not enough for a completely new task, due to duration of each task and the overall study. If so, this seems just a scaling issue (i.e., a longer study would see corresponding impact in both metrics), and perhaps can just clarify this (unless I missed it).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reports on a human evaluation (N=243 participants) of code LLMs, where human programmers receive tasks that to complete. Across 7 condition (one with no LLM assistance, 3 with instruction-tuned LLMs under an autocomplete feature, and 3 with chat LLMs, from CodeLlama variants to GPT-4o), participants were given time to complete as many tasks as possible, from a pool of 17 task. Generally, LLM assistance seems to help participants finish tasks faster, thus gaining in productivity. However, the productivity gains do not fully reflect differences in the static evaluation of HumanEval (large HumanEval gaps don't translate do large productivity gaps). The authors will release the RealHumanEval platform to support future human-centric evaluations of code LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please expand more in the main text about the actual tasks. Since the authors have space left, I'd suggest a table with 1-2 sentence descriptions of each task.\n* I'd also suggest a simple correlation analysis between HumanEval performance and the productivity metrics in RealHumanEval, to accompany the corresponding finding in Section 5\n* The numbering of the findings is a bit confusing, since \"Providing LLM assistance reduces the amount of time spent coding.\" is not numbered, but is the first result; then, the next one is labeled (1). In 5.1, the numbers are dropped. Perhaps the authors can standardize these headings, as they find fit.\n* [Minor] There are some occurrences of \"GPT-4\" in the text (as opposed to 4o): e.g., in Section 4 paragraph \"Tasks\", Section 5 paragraph \"Providing LLM assistance ...\". \n* [Minor] Fig 6 caption missing a space before \"seconds\"", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MDM5vUhgMk", "reviewer": "Reviewer_M2x2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis paper is an ambitious effort to confront an open and important challenge, that of understanding how static benchmarking results are related to the actual usefulness of code models for human programmers.  I found it very useful to be able to make comparisons both among different models, as well as across use modes (i.e., autocomplete vs. chat).  Table 1 does a nice job of encapsulating the merits of this study.  Beyond the experimental findings, the web interface for gathering more data, as well as the resulting dataset, can stand as significant contributions as well.\n\n**Weaknesses**\n\nWhile I think that this paper does make some solid contributions, I find that it overstates the conclusions that can be drawn from their experiments, and under emphasizes the potential contribution of their web interface and dataset.\n\n - The use of the word “productivity” here seems like a stretch.  Productivity is something much more complex than the ability to finish 3 small coding tasks during a 35 minute sprint window.  I think most people think of productivity on a much longer timescale than this, and a study that meaningfully addressed productivity would want to consider other important questions like the effect of the tool use on programmer's skills over time.  This language should be refined or clarified, at least in some instances early in the paper, to something along the lines of “helping programmers quickly complete a series of small tasks.”  While this paper is not meant to be an in-depth discussion of the complexities of productivity, its scope should be specified in more detail, as I feel that the current broad claims around productivity lack supporting evidence. \n\n - I am having trouble understanding the name of the paper and dataset: RealHumanEval.  As noted in the paper, this is not the first effort to have real humans evaluate LLM code output.  Also, this work is seemingly not an extension of HumanEval (e.g., ImageNet->ImageNet-v2).\n\n - I do not think it is true to say that code LLMs will always be used with a programmer in the loop (e.g., top of page 4), especially on such simple tasks.  Instead, people aim to develop coding agents that can work without direct human interaction on low-level tasks.  There are many modal settings for LLM code models, and this work explores one (important) setting.\n\n - I think there are some issues with experiment design (e.g., participants only being assigned to one condition; short time window; small number of tasks, some of which may be overly simple and not require LLM help), but nothing disqualifying, as long as the writing correctly reflects the conclusions that can be supported.\n\n - I think the creation of the web interface is a strong contribution, but after reading the paper I am left wondering how the authors envision it being used in the future.  Is this an open source software tool, that someone can adopt and deploy on their own for data collection?  Or does it live on the web somewhere?  What does it take to make changes, e.g., to the set of underlying models or tasks? I think it should be clearly stated how exactly one may or may not be able to use this tool in the future.\n\n - I also think the dataset could represent a significant portion of the value created by this paper.  However, while it mentions the data from each interaction that is saved into the AC and Chat datasets, the paper does not explain what these datasets might be useful for.  Some indication of how these datasets may be used to produce better code models, or better evaluations for code models, would be very useful.  As it stands now, these datasets are treated more as artifacts logging your experiments than a valuable resource for future community development.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses one of the most important challenges that exists is bringing LLMs to mass-use production across many fields: evaluation.  With a focus on coding, this paper aims to bridge the gap between the understanding of performance provided by existing static benchmarks and the actual impact that a coding assistant will have on human output.  The authors produce a web interface tool that enables the measurement of how helpful a set of LLMs are for completing small coding tasks.  The interface also allows for a comparison between these models when used in the form of a chat assistant, or as an autocomplete tool.  Human study results show that stronger models according to static benchmarks are in fact more useful for helping humans complete coding tasks.  Their collected dataset is released for future public use.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarification of the scope of experiments and the conclusions that can be drawn with respect to “productivity” **(most important)**\n - More thorough explanation of how the web interface and dataset can be used by the research community in the future.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RsRC1fZltS", "reviewer": "Reviewer_Zsge", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The method proposed in this paper offers a new direct way to observe how LLM advancements translates to programmer productivity. \n\n2. The paper discovered the correlations with existing LLMs Eval benchmarks, which helps to understand the quantified advancement of coding LLMs.\n\n3. The presentation is overall good.\n\n**Weaknesses**\n\n1. The proposed method requires human running the evaluations, which limits the scalability", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper uses real human programming procedure to evaluate the helpfulness of coding LLMs. It augment the HumanEval and other static benchmarks by measuring the human interaction metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The literature reviews have complete coverage, the method itself is a good supplement of current static coding LLM benchmarks, and analysis are performed. The paper looks good in current form. The scalability limitations comes from the inherent method itself which is not required to change.", "broader_impact_concerns": "This paper presents no detected ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE also resonates with one of the reviewers' comments on the benchmark name. RealHumanEval is a bit overclaiming than what the benchmark actually entails. Given that there are so many (static) benchmarks in various open-source platforms, such as HuggingFace, the AE believes that the name should entail the scope.
Decision: Accept (poster)