Title: ARC-AGI Without Pretraining
Venue: NeurIPS 2025
Accepted: no
Keywords: ARC-AGI

pretraining

CompressARC

compression

MDL
Abstract: Conventional wisdom in the age of LLMs dictates that solving IQ-test-like puzzles from the ARC-AGI-1 benchmark requires capabilities derived from massive pretraining. To counter this, we introduce CompressARC, a model without any pretraining that solves 20\% of evaluation puzzles by minimizing the description length (MDL) of the target puzzle purely during inference time. The MDL endows CompressARC with extreme generalization abilities typically unheard of in deep learning. To our knowledge, CompressARC is the only deep learning method for ARC-AGI where training happens only on a fraction of one sample: the target inference puzzle itself, with the final solution information removed. Moreover, CompressARC does not train on the pre-provided ARC-AGI "training set". Under these extremely data-limited conditions, we do not ordinarily expect any puzzles to be solvable at all. Yet CompressARC still solves a diverse distribution of creative ARC-AGI puzzles, suggesting MDL to be an alternative, highly feasible way to produce intelligence, besides conventional massive pretraining.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a test-time learning approach that can solve 20% of the evaluation puzzles from the ARC-AGI benchmark. The approach involves minimizing the minimum description length of the target puzzle at inference time. Further, the approach is demonstrated to work with a model that is not pre-trained. While the approach does not produce SOTA performance on ARC-AGI benchmark it is still interesting in that it demonstrates the strength of unsupervised learning at inference time. The paper analyzes a number of cases where the approach works and does not work as expected.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is clearly written and the approach is well presented.\n- The contribution and the work are significant.\n- The approach tackles an important problem of test-time learning which is becoming more and more relevant in language model research.\n- Extensively studies how the unsupervised training approach works on the ARC-AGI benchmark by providing cases and scenarios. Also, presents failure cases where the approach cannot solve the given task.\n\nWeaknesses:\n- The approach is validated on one benchmark and that brings out the question of whether the approach and insights can transfer / scale to more domains.\n- Does not present the pseudocode and loss functions relevant to the approach. These would be useful for reproducing the results from the paper.\n\nQUESTIONS:\n- The experiments could be strengthened if an appropriate baseline could be added. Specifically, it would be useful to see how a network that is trained on the test-time puzzle would work on the ARC benchmark.\n\n- The paper and contribution could be strengthened further if evaluated on another domain where the test-time performance is crucial. Are there other domains where the authors think it would be useful to evaluate their approach? And would it be possible to add them into the main text or in the appendix? \n\n- The presented can benefit from presenting the loss functions and the pseudocode related to the approach in the main text. \n\n- The approach involves minimizing the MDL on the test time puzzle. The paper does not seem to discuss or experiment with other unsupervised learning approaches (reconstruction objectives). \n\n- The experiment results seem to be coming from a single training seed. It would be useful to understand whether the approach can produce similar results when trained with different random seeds. \n\nI would be happy to increase the scores if baselines and other unsupervised approaches are compared in the experiments.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CompressARC, a novel model designed to solve ARC-AGI-1 benchmark puzzles without any pretraining. It uses Minimum Description Length (MDL) to solve puzzles directly at inference time. It achieves a 20% success rate on evaluation puzzles, demonstrating surprising generalization given its extremely data-limited setting—using only the incomplete test puzzle itself.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The core idea and task formulation presented in this work are both innovative and well-motivated. The use of MDL (Minimum Description Length) to guide inference-time learning is particularly novel and conceptually elegant.\n2. The empirical results are impressive and substantiate the effectiveness of MDL in tackling the ARC-AGI benchmark. The demonstrated generalization performance under minimal data and training constraints is especially compelling.\n\n**Weaknesses:**\n\n1. **Presentation and Clarity:**\n   The paper suffers from presentation issues that make it difficult to follow in key sections.\n\n   * Lines 110–123 in Section 3 would benefit greatly from being reformulated as an explicit algorithm or pseudocode subsection to enhance clarity.\n   * Section 4, which describes the architecture, is underdeveloped. Figure 2 is difficult to interpret, and the caption does little to clarify its content. More detail should be provided on the model components and their interaction.\n   * The paper would benefit from allocating more space to Sections 3 and 4, which are central to understanding the methodology, rather than dedicating extensive discussion to Section 2.\n\n\n2. **Unclear Learning Objective:**\n   While the paper claims to leverage MDL principles, the actual optimization appears to minimize a reconstruction loss. This raises an important question: how exactly does the reconstruction loss relate to MDL in this context? The connection between the two should be made more explicit—ideally with a theoretical or empirical justification. As it stands, the objective being optimized does not clearly reflect MDL, which could undermine the central premise of the approach.\n\n\n3. **Missing Baselines:**\n   The paper lacks comparisons to relevant baselines, particularly *Test-Time Training (TTT)* \\[1], which is closely related in spirit to the proposed method. Including such baselines would strengthen the empirical claims and better contextualize the contribution.\n\n\n\n**Reference:**\n\n\\[1] Akyürek, Ekin, et al. \"The surprising effectiveness of test-time training for few-shot learning.\" *Forty-second International Conference on Machine Learning*, 2024.\n\nQUESTIONS:\nSee Strengths and Weaknesses.\n\nLIMITATIONS:\nSee Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper approaches the ARC-AGI benchmark as one of finding a small program to explain the relationship between the input and the output examples. This is done by training on the examples themselves and, thus, does not require any pre-training. The method learns the mean and co-variance matrix of a normal distribution, samples from this and gives this as input to a neural network which outputs the input/output pairs in the problem. The neural network is trained to reconstruct the input/output pairs and KL divergence is used to make the noise distribution as close to a standard normal as possible.\n\nThe paper shows that more training time results in a greater percentage of puzzles being solved. While training required to solve 20% of test instances requires 138 hours, no pre-training is required.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\nThe paper shows that ARC-AGI puzzles can be solved without any pre-training. The only data needed is the data from the puzzle.\n\n**Weaknesses**\nThe methods in the paper are not clearly presented and many details are buried in the appendix. It is unclear how the parameters of the normal distribution are learned. There is pseudocode given starting at line 126 that shows a seed_z and seed_error. It says these are picked by relative entropy coding (REC). However, this appears to conflict with the overall method of learning a mean and co-variance matrix for a Normal distribution. In fact, this is the only time seed_error is mentioned in the main paper.\n\nThe paper also does not contextualize the results in the broader research landscape. For someome unfamiliar with the ARC-AGI benchmark, one cannot know how these results compare to other existing results, especially for those that do not rely on pre-training. Is this method the only one that does not rely on pretraining?\n\nQUESTIONS:\nIn Figure 1, what are the different layers shown in the left two plots?\nWhere does seed_error come from?\n\nLIMITATIONS:\nThe results appear to show that training is done at about a rate of 15 iterations per hour. Is this for a single example or across all examples? Even so, 15 iterations per hour seems slow. Why is this?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIntroduces a minimum description length approach/model capable of solving a significant subset of ARC-AGI-1 evaluation puzzles without pre-training. Their method trains a neural net on individual puzzles to predict an output grid from a parameterized latent/learned noise and suggests an pre-training alternative to modeling intelligence. Key advantages include lower computation requirements from foregoing pretraining.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- This work introduces a novel test-time approach to low-data/few-shot learning regime.\n- Includes an in-depth analysis of a target puzzle’s training run to demonstrate some measures of interpretability/transparency.\n- Detailed discussion of future avenues for improving the work.\n\n## Weaknesses\n- Highly engineered pipeline for ARC-AGI. As useful as ARC-AGI is for measuring a necessary but not sufficient rapid adaptation ability we would expect from a system truly claiming AGI, targeting it directly as a downstream task with bespoke architecture/layers/representations may hurt the broad appeal of this method.\n- Per-puzzle learning. This system is currently unable to carry forward anything between puzzles, and any adaptations are non-reusable. The authors acknowledge this and include it as a future direction in the appendix.\n\nQUESTIONS:\nQuestions and Suggestions on 3-5 key points\n1. Does only including Z in the “description length” proxy underestimate the efficiency/compactness of the learned program by not including the “decoding” program given by the neural net? I could be misunderstanding but it certainly seems that the neural net is deeply load bearing for learning the transform.\n2. Is this methodology easily transferable to other downstream tasks? MDL approaches on the whole are definitely more generalizable, but are there key takeaways/innovations from this work that enables new use cases? Perhaps this is more of a proof of concept for MDL approach on something abstract like ARC’s spatial reasoning task.\n3. Are there early results available for the joint-compression/cross-puzzle weight-sharing ideas mentioned in the appendix? Other test-time approaches to ARC-AGI (e.g. Akyurek et al. 2024’s Test-Time-Tuning) seem to conclude that reusing weight adaptations across puzzles is counterproductive compared to having separate weights for each.\n\nLIMITATIONS:\npotential negative social impact?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-06"}