Omkar2118/datasets huggingface dataset:
Turbulence
Full name: Turbulence

$\textbf{Turbulence}$ is a new benchmark for systematically evaluating the correctness and robustness of instruction-tuned large language models (LLMs) for code generation. Turbulence consists of a large set of natural language question templates, each of which is a programming problem, parameterised so that it can be asked in many different forms. Each question template has an associated test oracle that judges whether a code solution returned by an LLM is correct. Thus, from a single question template, it is possible to ask an LLM a $\textit{neighbourhood}$ of very similar programming questions, and assess the correctness of the result returned for each question. This new benchmark systematically and automatically identifies cases where LLMs are able to solve some problems in a neighbourhood but do not manage to generalise to solve the whole neighbourhood. Therefore, this method is effective at highlighting robustness issues.

Introducing paper: Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code
2D-ATOMS

Official dataset for **Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models.** *Ziqiao Ma, Jacob Sansom, Run Peng, Joyce Chai. EMNLP Findings, 2023.*

We introduce **2D-ATOMS** dataset, a novel text-based dataset that evaluates a machine's reasoning process under a situated theory-of-mind setting.

Our dataset includes 9 different ToM evaluation tasks for each mental state under ATOMS framework, and 1 reality-checking task to test LLMs’ understanding of the world. It is important to acknowledge that our experiment serves as a proof of concept and does not aim to cover the entire spectrum of machine ToM, as our case studies are far from being exhaustive or systematic. Here we release the zero-shot version of our dataset, which is used in our paper.

Introducing paper: Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models