Omkar2118/datasets huggingface dataset:
Code Lingua

Code Lingua is a benchmark to compare the ability of language models to understand what the code implements in the source language and translate the same semantics in the target language. It comprises 1,700 code samples across five programming languages, over 10,000 tests, 43,000 translated code snippets, 1,748 manually labeled bugs, and 1,365 bug-fix pairs.

Introducing paper: Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
OOP

The OOP benchmark features 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods¹². The authors of the paper argue that current evaluation frameworks largely neglect OOP in favor of functional programming (FP), such as HumanEval and MBPP¹. To address this, they introduced this OOP-focused benchmark¹².

In addition to the benchmark, they also propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k measures¹². This metric offers a more relevant and comprehensive assessment for OOP code generation¹.

The evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights¹²:
1) pass@o offers a more relevant and comprehensive assessment for OOP code generation¹².
2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT¹².
3) The poor performance of all advanced LLMs on the OOP benchmark highlights a critical need for improvements in this field¹².

(1) OOP: Object-Oriented Programming Evaluation Benchmark for Large .... https://arxiv.org/abs/2401.06628.
(2) OOP: Object-Oriented Programming Evaluation Benchmark. https://arxiv.org/html/2401.06628v2.
(3) OOP: Object-Oriented Programming Evaluation Benchmark .... https://www.x-mol.com/paper/1747067589511319552.
(4) OOP：大型语言模型的面向对象编程评估基准,arXiv - CS .... https://www.x-mol.com/paper/1747067589511319552/t.
(5) undefined. https://doi.org/10.48550/arXiv.2401.06628.
(6) undefined. https://github.com/alphadl/OOP-eval.

Introducing paper: OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models
NaturalCodeBench

NaturalCodeBench (NCB) is a comprehensive code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks¹². It comprises 402 high-quality problems in Python and Java, meticulously selected from an online coding service, covering 6 different domains¹². 

The seed problems of NCB are cleaned from the queries in coding online services, spanning across 6 domains: Artificial Intelligence, Data Science, Algorithm and Data Structure, Front-End, Software Engineering, and System Administration¹. 

Here is a summary of the number of problems in each domain¹:
- Software Engineering: 132 problems
- Data Science: 100 problems
- Algorithm and Data Structure: 95 problems
- System Administration: 33 problems
- Artificial Intelligence: 28 problems
- Front-End: 14 problems

The development set of NCB, which contains 140 problems (70 in Python and 70 in Java), is released for research purposes¹. The data format includes a unique identifier for each question, the problem description, testcases, setup code, a reference solution, and the domain of the problem¹.

(1) GitHub - THUDM/NaturalCodeBench. https://github.com/THUDM/NaturalCodeBench.
(2) NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval .... https://arxiv.org/abs/2405.04520.
(3) 清华、智谱AI 团队推出代码评测基准 NaturalCodeBench .... https://blog.csdn.net/www3300300/article/details/138752139.
(4) NaturalCodeBench: Examining Coding Performance .... https://www.x-mol.com/paper/1788507497390862336.
(5) undefined. https://doi.org/10.48550/arXiv.2405.04520.

Introducing paper: NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts