Kati8/ICLR_2023_2025 huggingface dataset:
authors: Serge Berger
reviews: 
Comments: The paper presents theoretical results demonstrating the limitations of GPT/LLM architecture for reasoning.
Strengths: It is a nice topic.
Weaknesses: The presentation e.g. the theorem and proof on lines 167-171 is unconvincing.
Recommendation: 1
Reviewer confidence: 3
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This paper investigates the reasoning abilities of large language models (LLMs), examining their potential limitations in producing truly novel and rigorous proofs. The authors argue that while these models can mimic logical structures and provide plausible responses, they lack the capacity for original, rigorous proof generation.
Strengths: The problem of assessing reasoning capabilities of LLMs versus memorization is very relevant to the current machine learning community.
Weaknesses: - The paper seems to be not polished: in the introduction there is a repeated paragraph (lines 69-73), several concepts (e.g. literal learning, first order model graph, generic 0-1 law, hallucination) should be defined formally or explained better, there is no literature review paragraph, the flow is hard to follow. Section 2.1 cannot be followed without reading the appendix, a re-organization of the flow would be helpful.
- The formatting in Section 2.1 is not clear: Q.E.D in line 171 does not seem to be the end of the proof of Theorem 1, and it is not mentioned where the proof of Theorem 1 can be found. Same for lines 176 and 191. In Corollary 3 it is not clear where the statement ends. 
- Because the lack of formalism in the presentation, the results seem unconvincing in this version of the paper.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 1
reviews: 
Comments: The paper studies the reason why today’s LLM models struggle with multi-hop reasoning tasks and produce hallucinations in the process. It presents an argument for why the fundamental nature of inference on the transformer architecture prevents them from being able to generate correct mathematical proofs or output false statements in other settings which require rigor.
To do so, the paper considers the setting of first-order predicate statements on random graphs.
Under this model, the paper claims that a GPT model has exponentially decreasing probability of finding the correct proof unless it has already seen the proof in the train set.
Strengths: - The paper aims to understand fundamental limitations of autoregressive next-token predictors which is an important problem today for ML.
Weaknesses: -	The presentation in the paper severely lacks a level of formalism necessary a technical conference. The writing is at times vague and this makes it hard to verify or critique the main Theorems and Corollaries presented. Some of the issues with the formalism are listed in the questions section below.
-	Theorem 1 seems to be wrong (?) It is well-known that GPT/LLM models can generalize and produce novel proofs so the statement that for almost all proofs, the model is performing literal learning appears to be incorrect?
-	The paper relies heavily on “randomization in inference” of GPT/LLM models. This randomization is not clearly quantified or defined anywhere which makes it hard to follow the argument.
-	Moreover, there are many post-training techniques one could employ to increase the chances of producing correct output. The paper’s claim seems to potentially hold only in the limit where the size of reasoning chains tend to infinity and we wish to be correct over all possible novel statements on the graph.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 1