Title: Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representational Alignment; Model Recovery; Deep Neural Networks; Similarity Judgments;Representational Geometry;Cognitve Computational modeling
Abstract: Linearly transforming stimulus representations of deep neural networks yields
high-performing models of behavioral and neural responses to complex stimuli.
But does the test accuracy of such predictions identify genuine representational
alignment? We addressed this question through a large-scale model-recovery study.
Twenty diverse vision models were linearly aligned to 4.5 million behavioral judg-
ments from the THINGS odd-one-out dataset and calibrated to reproduce human
response variability. For each model in turn, we sampled synthetic responses
from its probabilistic predictions, fitted all candidate models to the synthetic data,
and tested whether the data-generating model would re-emerge as the best predictor of the simulated data. Model recovery accuracy improved with training-set
size but plateaued below 80%, even at millions of simulated trials. Regression
analyses linked misidentification primarily to shifts in representational geometry
induced by the linear transformation, as well as to the effective dimensionality
of the transformed features. These findings demonstrate that, even with massive
behavioral data, overly flexible alignment metrics may fail to guide us toward artificial representations that are genuinely more human-aligned. Model comparison
experiments must be designed to balance the trade-off between predictive accuracy
and identifiability—ensuring that the best-fitting model is also the right one.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that when pretrained neural network representations are fitted to predict human similarity judgements, these representations lose their identifiability and recoverability. While the situation is improved by an increase in dataset size, the ceiling caps at 80%.\n\nFurther, the authors conduct a set of interesting analyses trying to understand the factors that underlie the problem. They find that a high intrinsic dimensionality and a large change in representational geometry are linked to lower recoverability. They also show that one successful model (CLIP) is recovered from the synthetic data generated by several other models.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nI think this is an excellent paper. The problem addressed is of high relevance to the representational alignment community, and the results are very interesting (also concerning). While the main result is already exciting and accessible, I think the authors do a great job of digging into the causes behind the main findings. The writing is clear, and the fitting and evaluation methods are sound.\n\nAs an additional plus, the code provided by the authors is very well documented. I haven't had a chance to run it, but it definitely increases my belief in the accuracy of the reported results.\n\n# Weaknesses\n\n## Interpretation of Results in High-Data Regime\n\nWhile both recoverability and identifiability are poor in low-data regimes, they are actually very good given a lot of data (ie, Figure 1D). The models that are not recovered well also seem to be outdated models (eg VGG19, EfficientNet, NasNet, Inception, BarlowTwins). If these models are not included, you would have almost a diagonal matrix in 1D. I think it's good you included these models, and I am not asking you to remove them. However, this needs to be carefully interpreted when thinking about how much of a practical problem unrecoverability is. e.g., if I compare a CLIP model, Imagenet ViT-L, and a DINO v2, the risk does not appear to be high. Nevertheless, I agree with the authors that such a high data regime is often not achieved in most studies. I just wanted to point this out for discussion.\n\n## Measure of Effective Dimensionality\n\nI know the PC based effective dimensionality metric is commonly used. However, I have my doubts about it. This paper [1] has shown that DNNs' representations lie on curved manifolds, which makes the PC based method inadequate due to its linear nature. I would recommend the authors use the Two-NN or the Gride method [2, 3], which does not make the linearity assumption. Either method can be easily used [with this package](https://github.com/sissa-data-science/DADApy), and would provide more accurate estimates.\n\n\n---\n\nAlthough I have outlined two weaknesses above, I think these are minor. The reason I'm giving a 5 and not a 6 is because I think the results are highly relevant to the representational alignment community, but I'm not sure if it has an impact in other areas. **I'd be willing to increase my score if the authors address the minor concerns above (the first one simply in writing, and the second one through simple analyses) and can make a convincing argument for why this work is relevant to a wider community.**\n\n\n\n# References\n\n[1] - Ansuini, A., Laio, A., Macke, J. H., & Zoccolan, D. (2019). Intrinsic dimension of data representations in deep neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\\textquotesingle Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 32).\n\n[2] - Facco, E., d’Errico, M., Rodriguez, A., & Laio, A. (2017). Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7(1), 12140. doi:10.1038/s41598-017-11873-y\n\n[3] - Denti, F., Doimo, D., Laio, A., & Mira, A. (2022). The generalized ratios intrinsic dimension estimator. Scientific Reports, 12(1), 20005. doi:10.1038/s41598-022-20991-1\n\nQUESTIONS:\n- I see that you fit $W$ several times in model recovery across seeds, but it seems to be fitted just once in the actual model fitting. Could some of the unrecoverability be attributed to stochastic optimisation? How similar would the learned representations be if you fitted the same linear probe several times?\n- Is $W$ initalised as an identity matrix in the model fitting as well?\n- 80% recoverability is deemed not good in the paper. Intuitively, I agree with this. However, I wonder if you gave the same triplets to human participants several times, what the test-retest reliability would look like. I know it's not the same as recovery, and that there are other factors here, but I think it could be a good and interesting proxy. I know it's not in the scope of this paper, but it might be interesting work for the future.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors conduct large-scale model-recovery simulations on the THINGS odd-one-out behavioral dataset to show that linear probing of DNN representations (essentially flexible evaluations that allow linear reparamerizations of the feature space) fails to reliably recover the true data-generating model. Things improve with more samples but the identifiability of the true model is still below 80% with millions of trials. They show a trade-off between predictivity of human behavior and identifiability of the true model as the flexibility of the probe increases (going from simple scaling to a rectangular to a full linear transformation matrix). The authors also try to identify factors causing misidentifiability and find that representations with higher effective dimensionality and those that incur higher geometry shifts after alignment with behavioral responses are the ones that are more misidentified.\n\nSTRENGTHS AND WEAKNESSES:\nThe results are intuitive and the paper has a nice take-home that also seems actionable: that whenever you use highly flexible, data-driven alignment (linear or otherwise) to compare models to brain or behavior, you risk mistaking best fit for true fit (predictability vs identifiability trade-off). \nThe paper is well written, the experiments are sound and the conclusions are straightforward.\n\nQUESTIONS:\nMy main concern is about the generality of findings (that more flexible mappings reduce identifiiability) beyond behavioral tasks. All simulations use the THINGS odd-one-out behavioral paradigm. Yet the paper’s rhetoric (“flexible evaluation may incur a hidden cost”) implies broad applicability to any linear alignment metric—most pressingly those applied to neural data . It is hard to know if the finding obtained with one dataset would generalize to broader settings, particularly neural data. Without at least a toy simulation using a realistic neural noise model, it’s unclear whether the same identifiability collapse holds, improves, or worsens in neural-alignment settings. \n\nThe authors should also link their finding that flexible evaluations reduce identifiability with other findings in neuroscience. for eg that of conwell et al which show that when flexible evaluations are employed (eg linear predictivity), many different candidate models achieve similar brain predictivity. So in some sense, this issue has already been flagged in the field. \n\nThe paper could also benefit from discussion of other potential criteria that should be used for evaluating methods of identifying candidate models of brain/behavior. Identifiability is an important (almost necessary one) but are there other criteria that we want our evaluation methods to satisfy? e.g., interpretability of weights, robustness to noise etc\n\nThe experiments sweep from zero-shot through diagonal, low-rank, and full linear transforms—but do not examine any class of nonlinear mapping models. It would be valuable to see whether moderately nonlinear mappingsfurther amplify or mitigate identifiability failures. Beyond diagonal/low-rank/full, it would also have been useful to introduce other structured‐linear probes (e.g. sparse at different sparsity levels) and chart their place on the accuracy–identifiability curve. The current probe evaluation (diagonal vs low-rank vs full seems too limited)\n\nAlso the paper flags the main issue of flexible models as their misidentification rate. But which studies use model-behavior comparisons for predictivity vs using the most human behavior like model  to make conclusions about what shapes human behavior.  It would help to cite a few examples of such articles whose inference pipelines depend on the reliability and identifiability of flexible linear probes.\n\nOverall, this paper issues an important warning about over-flexible linear probes in behavioral model comparison, and suggests that similar pitfalls likely afflict neural-alignment studies. However, to substantiate its broader implications and to offer concrete remedies, it needs (1) at least a small neural-data simulation and (2) a thorough exploration of structured and mildly nonlinear probes.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether linear probing can reliably identify which model generated behavioral data in human perception tasks. The authors conduct large-scale model recovery simulations using 20 diverse vision models on the THINGS odd-one-out dataset. They find that even with massive datasets, model recovery accuracy plateaus below 80%, meaning the best-fitting model often isn't the data-generating model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Very thorough evaluation in terms of testing 20 diverse models across multiple dataset sizes (50 to 4.2M trials), with noise calibration to match human response variability. \n- The regression analysis identifying specific factors (representational geometry shifts, effective dimensionality) that drive misidentification provides useful mechanistic insights.\n- I really enjoyed the discussion section “Navigating the accuracy–identifiability tradeoff” at the end of the paper.\n\nWeaknesses\n- The study is restricted to one behavioral task of odd-one-out judgments and one dataset. The quantitative results are specific to this particular model set and may not generalize broadly. The paper does draw attention to this in its limitation section, but still, for a venue like NeurIPS, the scope feels narrow. \n- The core insight that flexible metrics can overfit isn't entirely novel; this has been recognized in various forms across ML. The specific quantification in this behavioral context is useful but somewhat incremental.\n- I wonder if this work would be better suited for a more specialized venue focused on computational cognitive science or model evaluation methodologies rather than NeurIPS.\n\nQUESTIONS:\nYour study only looks at odd-one-out similarity judgments, which is pretty narrow. Do you think these problems with model recovery would show up in other kinds of behavioral tasks or even neural data? It would be helpful to see some theoretical reasoning or even small pilot studies showing this isn't just a quirk of this particular dataset. I'd say this is my primary qualm with the manuscript at this moment.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the hypothesis that good brain data predictivity in neural network models is reflective of a corresponding architectural similarity with the entity that generated said brain data, i.e. that being good at predicting brain data reflects mechanistic alignment between models and human subjects. Standard assumption is that this is the case, but drawing conclusions from such approaches is made hard by the presence of noise in the measurements and the limited data available for model identification. The authors study whether model identification is possible even when data availability is not an issue and the data generating procedure is known. They do this by substituting human participants with a neural network, recording its responses and checking if model identification can pick out the data generating architecture from a set of candidates at evaluation time. They find that in general, even with millions of examples, such identification is not possible. These results cast doubt on the possibility of identifying the mechanisms used by humans to represent data using brain-predictivity methods alone.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well motivated, addressing a key question that has generated significant debate recently.\n- The method is a clever but simple way to side-step the issue of comparing to humans, by just focusing on whether models can identify each other first, which is a much simpler task.\n- The methods are well detailed, explaining how the experimental setting is matched as much as possible to what we would encounter when comparing to humans.\n\n\nWeaknesses:\n- The study is restricted to behavioral data, though I agree with the authors that this is a good starting point.\n- The explanation on how the noise ceiling was determined could be a bit more clear.\n\nQUESTIONS:\nI have no interesting questions at this point. If some come up I will post them.\n\nLIMITATIONS:\nThe authors clearly state the limitations of the study, but I think what they have done is more than sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper questions the common assumption that high predictive accuracy of human behavior is a good proxy for internal representational alignment. To test this, the authors take 20 pre-trained vision models, freeze their weights, and learn a linear transformation of each model's features to best predict judgments from the THINGS dataset. These aligned models, which could be called \"teacher\" models, then generate synthetic datasets. In the recovery phase, the same 20 frozen models (\"student\" models) are fitted to the synthetic data by learning a new linear transformation. The authors then tests if the original data-generating model can be correctly identified based on predictive accuracy.\n\nThe authors find that while model recovery improves with dataset size, it plateaus below 80% accuracy. The errors are systematic, with certain models often being misidentified as the best predictor. They find that that greater flexibility in the alignment metric improves predictive accuracy but hurts model identifiability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe model recovery setup is novel, as it allows for evaluation against a known ground truth.\n\nThe approach is applied to a large set of 20 vision architectures and, calibrates the synthetic data to realistic human noise levels, making the setup more realistic. The authors go beyond simply identifying the problem and use regression analysis to find predictors of misidentification.\n\n**Weaknesses / Feedback**\n\nSome of the presentation was difficult to parse. Adopting the \"teacher/student\" model analogy, as the setup is similar to distillation, would make the simulation process much more intuitive for readers.\n\nThe conclusion could benefit from proposed avenues of research to address the issue the paper identifies.\n\nQUESTIONS:\nRecent work (e.g., Lampinen et al., 2024) has focused on how training objectives create distinct representational biases. How do you see these biases factoring into your analysis? It seems they would determine the initial representational geometries that your linear probe then attempts to align.\n\nYour work shows that DNNs can be confused for other DNNs. How do you think these claims extend to the much harder problem of comparing DNNs to the brain? It seems your findings represent a best-case scenario -- the problem of misidentification may be even more severe in the neuroscience setting.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}