Before revision:  Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate Python code fulfilling this specification . Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases.
After revision:  Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code . Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases.