Title: Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations
Venue: NeurIPS 2025
Accepted: yes
Keywords: representational similarity

neural networks

neural coding

decision variable

computational neuroscience
Abstract: Previous studies have compared neural activities in the visual cortex to representations in deep neural networks trained on image classification. Interestingly, while some suggest that their representations are highly similar, others argued the opposite. Here, we propose a new approach to characterize the similarity of the decision strategies of two observers (models or brains) using decision variable correlation (DVC). DVC quantifies the image-by-image correlation between the decoded decisions based on the internal neural representations in a classification task. Thus, it can capture task-relevant information rather than general representational alignment. We evaluate DVC using monkey V4/IT recordings and network models trained on image classification tasks.
We find that model–model similarity is comparable to monkey-monkey similarity, whereas model–monkey similarity is consistently lower. Strikingly, DVC decreases with increasing network performance on ImageNet-1k. Adversarial training does not improve model–monkey similarity in task-relevant dimensions assessed using DVC, although it markedly increases the model–model similarity.
Similarly, pre-training on larger datasets does not improve model–monkey similarity. These results suggest a divergence between the task-relevant representations in monkey V4/IT and those learned by models trained on image classification tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for measuring the similarity of decision strategies between two observers, as is frequently done between DNNs and primate brains in the field of NeuroAI. The method is called decision variable correlation (DVC), which aggregates the trial-level correlations between the decision variables of the observers, which is basically the distance a sample in embedding space has to the separatrix of a linear classifier.\nUsing this method, the authors find low between-group similarity between monkeys and models despite high within-group similarity. This low between-group similarity is not increased by increased classification performance on ImageNet, by more training data, or by higher adversarial robustness, which suggests profound representational differences between monkey brains and image classification models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper addresses a relevant problem and is fairly clearly written.\n- The idea of only using those parts of the representation that are relevant for the task is valuable.\n- The conducted analysis is very thorough, sufficiently many models were investigated and the selection of models is reasonable.\n\n## Weaknesses\nMy main gripe with the paper is that they propose yet another method for measuring brain-model similarity, and immediately jump to deriving conclusions, instead of first sanity-checking the method itself. I propose a few of these sanity checks in the Questions-block.\n\n- The description of the method lacks rigor, the authors never properly define what the decision variable is. I understood it to be the length of the projection onto the separatrix, but I'm not even sure if this is correct.\n- I find the split-normalization highly dubious. The authors just assume that within the two randomly drawn subpopulations of neurons of a layer, the same signal is represented, plus different noise, which can essentially be averaged out. I don't share this intuition, because if the different subgroups represent different features, the signal may be different.\n- The comparison to Cohen's Kappa (especially how figure 5e was obtained) is missing details, e.g. why does a low decision bias enforce a low kappa? \n- I have some concerns about the dependence of DVC on the accuracy of the two classifiers, which are not addressed by the paper. I suspect that two classifiers with very different accuracy cannot have a high DVC, just like two classifiers with different accuracies cannot have a high Error Consistency (as measured by Cohen's kappa over correctness values). It does not surprise me that increased accuracy on ImageNet-1k does not yield increased DVC to monkeys, because I doubt that an optimal linear classifier on top of the monkey representations would perform very well. I fear that such a dependency on accuracy could also explain the findings in figure 2d, because (as seen in figure 2c) models from the same family tend to have similar accuracy. This is problematic, as it would mean that I gain no more knowledge from DVCs than I would trivially get from accuracy.\n\n## Conclusion\nOverall, I find the method interesting and could imagine accepting the paper, because it meaningfully adds to the discourse. But there are too many issues that make me feel uneasy about the method, so I don't trust the results yet. If the authors address these concerns and other reviewers agree, I'm open to increasing my score.\n\nQUESTIONS:\n- What is the effect of the split normalization (sec. 3.2 second paragraph / Appendix A) on results? How would results look like without this normalization? If the point of this step is to reduce noise, wouldn't it be better to split the images into different sets, rather than the neurons?\n- What is the motivation behind checking all pairwise decision variables, rather than just looking at the class-vs-all-others separatrix, which would scale much nicer to large numbers of classes, like 1k ImageNet classes?\n- What is the uncertainty in a DVC-value? You currently just report a point estimate without specifying a CI around these values. This could be achieved by bootstrapping, i.e. if a value is calculated over 1,000 images, draw 1,000 images from the set with replacement for N times and re-calculate DVC for each set, then report the range in which 95% of values end up.\n- What is a model's self-DVC? For example, train a ResNet-50 from scratch on ImageNet with two different random seeds and evaluate their DVC on the validation set.\n- Similarly, what is the DVC between two randomly initialized networks, is it really 0 as expected, and what is the variance?\n- What is the classification accuracy that an optimal linear classifier on top of the monkey representatins achieves?\n- (section 4.5, second paragraph) I'm surprised that logistic regression on top of network representations does not beat the aggregation strategy proposed by Geirhos et al,  shouldn't it be at least as good?\n\nLIMITATIONS:\nThe authors have written a limitation-section, but I think missed one point which they maybe thought was obvious: The method is limited to datasets that have class labels, like ImageNet. This is not true for all image datasets, and other methods like RSA work independently of known class labels, hence I think it should be explicitly mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Decision Variable Correlation (DVC) as a new method to measure task-relevant similarity between neural representations in brains and deep networks.  DVC focuses on the trial-by-trial correlation of inferred decision variables, isolating task-relevant information while being robust to noise and decision bias. Applying DVC to monkey V4/IT and various deep networks, the authors find that model–model consistency is comparable to monkey–monkey, but model–monkey consistency is lower and decreases with higher ImageNet accuracy. Neither adversarial training nor larger datasets improve model–brain alignment, suggesting a divergence between artificial and biological representations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- **Clarity and Motivation:** The paper is clearly written and well-motivated. It introduces a conceptually principled method (DVC) that isolates task-relevant similarity, avoiding confounds like accuracy and decision bias. The derivation from signal detection theory is elegant and intuitive.\n- **Sound Methodology:** The method is grounded in theory, and the implementation accounts for practical issues such as noise correction and high-dimensional instability. The authors take care to validate that DVC behaves sensibly under these conditions.\n- **Conservative Claims:** The paper avoids overclaiming and discusses its limitations transparently, including dataset size and generalizability beyond monkeys.\n\n**Weaknesses**\n\n- **No Empirical Comparison to RSA/CKA:** While the paper argues that DVC improves on representational similarity methods like RSA, it does not empirically compare DVC to RSA, CKA, or CCA. This weakens the case for DVC as a superior alternative, especially since RSA-based methods have also reported similar qualitative conclusions (e.g., limited brain-model alignment).\n- **Overlap with Prior Findings:** Many key takeaways—such as the failure of adversarial training or larger datasets to improve alignment—have been reported previously using other similarity metrics. Without a side-by-side comparison, it remains unclear how much new insight DVC offers.\n\nQUESTIONS:\n**Questions and Suggestions for the Authors**\n\n1. **Comparison to RSA/CKA**: The abstract and introduction motivate DVC by highlighting the limitations of standard representational similarity methods (e.g., RSA, CKA). However, the paper does not empirically compare DVC to these methods. Can the authors include such comparisons—ideally using the same model and brain representations? These metrics can also be applied to the output (decision) layer, allowing for a fairer comparison to DVC, which is decoded via LDA from the same representations.\n\n2. **Clarifying Novelty**: Several of the qualitative findings (e.g., that adversarial training or dataset scale does not improve brain alignment) have been previously reported using other similarity metrics. Can the authors clarify what new insights DVC provides beyond replicating existing conclusions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new representational similarity measure for stimulus-driven neural activity. The method is based on a multidimensional generalization of signal detection theory. Neural responses to different stimuli with a known class structure are projected onto the LDA (linear discriminant analysis) axis. The Pearson correlation between the projections of these stimuli for two observers (modulo a noise correction factor) is the proposed similarity score between the neural representations of the corresponding observers. Some advantages of the proposed measure over purely behavioral measures (Cohen’s kappa) are discussed (e.g. insensitivity to decision biases, dependence on internal (neural) representations) and a seeming discrepancy with an earlier study using a purely behavioral measure (Cohen’s kappa) is resolved by taking into account the decision biases affecting Cohen’s kappa.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n\nI think this is a technically solid paper overall. The experimental demonstration in section 4.5 that Cohen’s kappa used in earlier analyses can be affected by decision biases is a nice contribution. It’s also nice to see error bars in a NeurIPS submission for a change.\n\n**Weaknesses:** \n\nI guess my main concern about this work is that I’m not sure if we really needed another representational similarity measure. In the first two sections of the paper, the authors complain about the multiplicity of representational similarity measures and how they can sometimes give seemingly contradictory results. But they are making this problem worse by introducing (without any good justification) yet another similarity measure that is not going to always agree with previous measures. \n\nWhat exactly is the problem with RSA or regression-based metrics (like Brain-Score), for example, that this new method is addressing? In section 3, the authors write (lines 96-98):\n> ... in contrast to methods for analyzing the similarity of two neural representations (such as representational similarity analysis), our methods focus on the dimensions that are relevant for a behavioral task and is invariant to variability along other task-irrelevant dimensions.\n\nI’m not sure if this is a win for the proposed approach. The proposed approach can’t be applied to cases where there is no obvious class structure in the stimuli or where the class structure is irrelevant (*e.g.* passive viewing of natural videos or images). \n\nEven in cases where there is a class structure and it is relevant to the task, somewhat counterintuitively, the proposed method actually doesn’t say anything about the observer’s actual behavior in the task (it only says something about the behavior of an ideal linear classifier based on the neural responses, which may be completely different from the observer’s own behavior). Whatever its problems, Cohen’s kappa at least doesn’t suffer from this particular issue: it says something about the observer’s actual behavior. \n\nSo, it appears that the proposed method is intended more as an alternative to neural similarity measures, such as RSA or regression-based metrics, as opposed to behavioral similarity measures like Cohen's kappa, but the authors haven't really shown any particular problems with these neural similarity measures that their approach is addressing. The problem they identify only affects Cohen's kappa. So, we're left wondering what exactly is the pressing need or the real motivation behind this new neural representational similarity measure.\n\nIf I’m interested in quantifying the similarity between neural responses and model responses, I can use RSA or a regression-based metric. These are general methods I can use for any type of stimuli or task. Their application is not limited to classification tasks. If I don’t have access to neural responses or if I just want to quantify the behavioral similarity between an observer and a model, I can use kappa or some bias corrected version of it. I haven’t really read anything in this paper that made me question these choices and convinced me that a new similarity measure was desperately needed.\n\nQUESTIONS:\nPlease address the issues raised under the \"weaknesses\" section above. In particular, the authors need to do a much better job motivating their approach: why exactly it is needed and what it offers over the multitude of current methods.\n\nLIMITATIONS:\nI think the issues raised above are limitations of the work that aren't adequately addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. New approach (Decision Variable Correlation) to characterize the similarity of the decision strategies of two modalities referred to as observers in this paper\n2. Model-monkey similarity is consistently lower than monkey-monkey or model-model relationship\n3. Adversarial training , Improved accuracy on Imagenet or pretraining on larger dataset does not increase model-monkey similarity \n\nThe study demonstrates divergenece between task relevant representations in monkey V4/IT and models trained on image classification tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. DVC quantifies task relevant similarities and is insensitive to the observer’s biases and independent of behavioral accuracy. It’s a nice bridge between methods purely based on behavior and methods comparing neural representations and it focuses only on those dimensions which are relvant for the behavioral task\n2. Comprehensive Evaluations:\n    1. Large set of models to identify correlation between imagenet accuracy and model-monkey DVC\n    2. Robustly trained models vs. representative models\n    3. Dependency of monkey-model DVC on scale of data used for model training\n3. Interesting findings:\n    1. Data rich and robustly trained model show lower DVC than representative models. The finding is surprising and reveals difference between how monkeys and models solve classification tasks\n    2. Comparison with Cohen’s Kappa measure shows that DVC is insensitive to decision biases.\n4. Clear description of methods, results and figures\n\n**Weakness**\n\n1. A theoretical/data-driven experiment using some simulated data to demonstrate the key differences between different measures would have revealed the pros and cons of different behavioral and representational similarity methods.\n\nQUESTIONS:\nDid you consider applying DVC to opensource human neuroimaging datasets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method, Decision Variable Correlation (DVC), to measure the similarity between the decision strategies of different observers (e.g., brains and deep neural networks). Grounded in Signal Detection Theory, DVC quantifies the trial-by-trial consistency of task-relevant representations by correlating the decision variables decoded from two observers for a binary classification task. Applying this method to compare monkey V4/IT neural recordings with a wide range of deep learning models, the authors uncover several surprising findings. They report that model-monkey similarity is consistently lower than monkey-monkey similarity and, counter-intuitively, decreases as models' ImageNet performance increases. Furthermore, neither adversarial training nor pre-training on larger datasets appears to close this gap. These results lead the authors to suggest a fundamental divergence between the representations learned by current models and those in the primate visual cortex.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n**Novel and Principled Method:** The paper introduces DVC, a methodologically sound approach for comparing representations that is well-grounded in Signal Detection Theory. The careful consideration of potential confounds, such as the use of PCA to stabilize the decoder and the development of a split-half procedure to correct for measurement noise, demonstrates rigor (Section 3).\n\n**Comprehensive Experimental Scope:** The study's conclusions are supported by an extensive evaluation across a diverse set of 65 models, including standard ImageNet-trained networks, adversarially robust models, and models trained on web-scale datasets (Section 4). This breadth makes the findings less likely to be an artifact of a specific architecture or training scheme.\n\n**Significant and Counter-intuitive Findings:** The paper's central results challenge common assumptions in the field. The strong negative correlation between ImageNet accuracy and DVC to monkey V4/IT (Fig. 2c) and the observation that adversarial training actually reduces model-monkey similarity (Fig. 3a) are important, surprising findings that should stimulate further research.\n\n**Strong Baselines and Contextualization:** The use of monkey-monkey DVC provides a clear biological benchmark, giving crucial context to the model-monkey similarity scores. Furthermore, the direct comparison to Cohen's Kappa, a related behavioral metric, helps to situate DVC within the literature and highlights its specific advantages, such as its insensitivity to decision bias (Section 4.5, Fig. 5e).\n\n### Weaknesses and Possible Improvements\n**Insufficient Contextualization with Recent Literature:** The paper's findings, particularly regarding adversarial training, align with a growing body of work suggesting that standard robustness metrics do not guarantee better alignment with biological vision. Citing recent, highly relevant papers (e.g., Subramanian et al., 2023; Linsley et al., 2023) would strengthen the authors' claims and better situate their contribution within the current scientific discourse.\n\n**Limited Neural Dataset:** The strong claim of a \"fundamental divergence\" between models and the primate brain is based on neural recordings from only two macaque monkeys. While the authors acknowledge this in the limitations section, the forcefulness of their main claims could be tempered by this constraint. The potential for inter-subject variability in the primate data might be a larger factor than is currently accounted for.\n\n**Sensitivity to Decoder Choice:** The entire DVC framework relies on using an optimal linear classifier (LDA) to find the \"decision axis\" for each observer. The paper does not explore how sensitive the final DVC values are to this specific choice. It is possible that a different decoder (e.g., a non-linear one, or even a different linear method) could yield different correlation patterns.\n\n**References:**\n\n1. Linsley, D., Feng, P., Boissin, T., Ashok, A. K., Fel, T., Olaiya, S., & Serre, T. (2023). Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception. arXiv preprint arXiv:2306.03229.\n\n2. Subramanian, A., Sizikova, E., Majaj, N., & Pelli, D. (2023). Spatial-frequency channels, shape bias, and adversarial robustness. Advances in neural information processing systems, 36, 4137-4149.\n\nQUESTIONS:\n**Missing Citations:** The findings that adversarial training increases model-model similarity but decreases model-brain alignment are consistent with other recent work. Have the authors considered citing papers like Subramanian et al. (2023) and Linsley et al. (2023) to better contextualize their results and show how they fit into a broader emerging consensus?\n\n**Generalizability of Claims:** The claim of a \"fundamental divergence\" is quite strong. How confident are the authors in this conclusion given that the neural data comes from only two monkeys? Could the observed model-monkey gap be partially explained by inter-animal variability that a larger sample of primate data might clarify?\n\n**Robustness of the DVC Method:** The DVC calculation relies on an optimal linear classifier (LDA) to define the decision axis. How sensitive are the reported DVC results to this specific methodological choice? Have the authors explored whether using a non-linear decoder or a different linear classifier would significantly alter the observed correlations and the paper's main conclusions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}