Note great overview and a lot of additional references! frequently updated list: https://github.com/codefuse-ai/Awesome-Code-LLM

Note introduces HumanEval and pass@k

Note human judgement doesn't agree with static metrics (BLEU, ChrF, RUBY ...)

Note perturbations on docstrings/prompts

Note phi-1 model, novel evaluation problems that combine two tasks to make data contamination less likely.

Note model context is whole file up untill funciton header, ground turth the source file, first AST, then Linter, errors in context cause errors in generation, undefined/name error most common, EOF error due to generation length

Note introduces Mostly Basic Programming Problems (MBPP)

Note "non functional" -> focusses on instruct models, has classification tasks and non functional requirements like efficiency, security and also maintability. Still relies on gold labels using DiffBLEU. filters examples larger than 3k tokens using starcoder tokenizer GPT4 seems to be really good on these tasks. might be a lot of prompt engineering tho

Note this paper has been withdrawn!

Note has already been used for marketing: Devin

Note HumanEvalPlus with additional test cases has a great figure to rank HumanEval problems on passrate, showcasing that some of them are much easier than others.

Note Verilog generation. creativity here means ". This term refers to the capacity to think innovatively—the ability to formulate new solutions or connections that are effective and unconventional [11]." where the reference is: https://doi.org/10.1080/10400419.2012.650092

Note Annotate problems by months and spot potential contamination