ModeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
Authors: Nathan Andrew Chi, Teodor Malchev, Riley Kong, Ryan Andrew Chi, Lucas Huang, Ethan A Chi, R. Thomas McCoy, Dragomir Radev


Reviewer paper summaries: This paper introduces a multilingual linguistic reasoning dataset consisting of 272 questions. The main contribution claims to be no worry of data leakage. The paper sets up baselines for both open-source LLMs and OpenAI models. Experiment results show the challenge of this dataset.
Summary of strengths: 1. A newly released dataset is always welcoming.
2. The dataset seems quite challenging.
Summary of weaknesses: 1. The size of the dataset is rather small.
2. There is no description of how the dataset is collected. 
3. Writing can be enhanced to increase clarity.
4. The contribution has been insignificant.
Comments, suggestions and typos: 1. **Data source**: Is it from free sources or a crowdsourcing work? If it is annotated by linguistic experts hired by the work, what are the annotation protocols and payments?

2. **Data leakage**: I believe there are many ways to tackle data leakage, how do you think your way of fabricating a new dataset solves this problem? A new dataset is not necessarily an OOD dataset, nor hasn't been seen by a model. There is not enough discussion in your paper on how to guarantee a newly written dataset is not seen by a model. For open-sourced LLMs, can we just check their training data? In line 163 to line 173, I haven't quite got the idea of how this can be a way of checking data leakage.

3. **Contribution**: How do you distinguish yourself from previous work, especially given that PuzzLing is a predecessor featuring the same question style? If the data leakage issue is the main concern, I would say it is a rather trivial contribution.

4. **Writing**: The writing is rather unsatisfying. There are multiple parts of your arguments that make me feel confused. For example, L035-L038: what do you mean by "apparent cases"?  L059-L062: "unlike prior datasets, it consists entirely of problems written specifically for this work, a further measure against data leakage", a key argument of your work, but loosely written.

5. **Figures**: I think Figure 2 takes too much space and gives too little information.

6. **Name of the Dataset**: I wonder how "ModeLing" is named.
Overall assessment scores: 2