Human-like cognitive generalization for large models via mental representation-guided supervision

Domain: biology
Published: 2026-04-01
Editorial outcome: major_revision

Abstract
Recent advancements in deep neural networks (DNNs), particularly large-scale language models, have
demonstrated remarkable capabilities in image and natural language understanding. Although scaling up
model parameters with increasing volume of training data has progressively improved DNN capabilities,
achieving complex cognitive abilities—such as understanding abstract concepts, reasoning, and adapting
to novel scenarios, which are intrinsic to human cognition—remains a major challenge. In this study, we
show that mental representation-guided supervised learning, utilizing a small set of brain signals, can
effectively transfer human conceptual structures to DNNs, significantly enhancing their comprehension
of abstract and even unseen concepts. Experimental results further indicate that the enhanced cognitive
capabilities lead to substantial performance gains in challenging tasks, including few-shot/zero-shot
learning and out-of-distribution recognition, while also yielding highly interpretable concept
representations. These findings highlight that mental representation-guided supervision can effectively
augment the complex cognitive abilities of large models, offering a promising pathway toward developing
more human-like cognitive abilities in artificial systems.

Introduction
Humans live in a world full of ever-changing objects, which requires us to constantly learn new concepts
from a variety of sensory information and systematically structure our experiential knowledge of the
physical world1,2. This ability helps us organize our knowledge and make sense of the world, allowing us
to recognize new things and adapt to new situations. For years, artificial intelligence (AI) researchers have
tried to build systems that understand the world like humans do3–7. Although Fodor and Pylyshyn8
suggested that neural networks are not plausible cognitive models because they lack the systematicity of
mental representations, deep neural networks (DNNs), especially with the help of large-scale datasets and
large models (LMs)9–11, did achieved higher systematic level ability. The question of whether and how
neural networks could achieve human-like cognitive ability has sparked a vigorous debate spanning
several decades1.

Current DNNs or LMs still struggle to match human-like cognition12. Scaling up data and model size has
primarily improved performance in well-defined tasks like classification, object detection, and content
generation13–15. However, when it comes to human-like cognitive abilities—particularly the understanding
and generalization of novel concepts—current models still fall short1,12,16. Moreover, increasing model
size does not consistently enhance these capabilities, and in some cases, performance even plateaus or
degrades12,17, highlighting a fundamental limitation in achieving high-level cognition18. Recalling Fodor
and Pylyshyn's opinion8, the systematicity of mental representations might be an important but overlooked
component for neural networks to obtain human-like cognition.

Here, we proposed to enhance the cognitive ability of DNNs by leveraging the structural organization of
the human brain as a supervisory signal. Our initial observations revealed that while increasing the
parameter size of DNNs improved their understanding of concrete concepts (Fig. 1a), it had little effect
on abstract concepts (Fig. 1b, c). This limitation stems from DNNs' inability to capture the internal
relationships among concepts and their lack of abstraction and conceptualization capabilities (Fig. 1d, e).
Motivated by this insight, we developed a mental representation-guided supervision framework to directly
transfer the brain's cognitive structure to DNNs. Remarkably, we found that aligning the structural
representations of DNNs with the brain using only a small set of objects significantly enhanced their
ability to understand abstract concepts (Fig. 1b, c), with the learned cognitive structures generalizing well
to unseen concepts. This process led to the emergence of human-like conceptual hierarchies in DNNs,
closely resembling WordNet19—even without explicit constraints to match it. The proposed framework
consistently improved performance across a range of challenging tasks, including few-shot/zero-shot
learning20,21 and out-of-distribution recognition22, while also producing highly interpretable concept
representations. These results were robust across diverse DNN architectures, including SimCLR23, CLIP10,
and DINOv29. Our findings suggest a promising new direction for developing AI systems that think more
like humans, moving beyond the limitations of current approaches that rely solely on scaling data and
model complexity.

Results
Increasing the parameter size of DNNs enhances their understanding of concrete but not abstract
concepts

To assess the concept understanding capabilities of DNNs, we first conducted one-shot learning tasks for
both concrete and abstract concepts. The concrete concepts comprised 50 classes, including objects such
as sock and swan, while the abstract concepts consisted of 8 higher-level categories, such
as clothing and bird (Supplementary Fig. S1). We employed the one-shot learning task, where for each
class, one image was randomly selected for training, and the remaining images were used for testing (see
Supplementary for details). The abstract concept task posed a significantly greater challenge, as it required
models to generalize to unseen objects within a broader category. For instance, the model had to infer that
an owl belongs to the class bird when provided with a swan as the only example.

We evaluated several DNN architectures, including SimCLR23, DINOv29, and CLIP10, across varying
parameter scales (see Supplementary for details). For concrete concepts, we observed a consistent
improvement in one-shot classification performance as the number of parameters increased (Fig. 1a). In
contrast, for abstract concepts, increasing model parameters yielded only marginal improvements or even
led to performance degradation (Fig. 1b, c). For instance, with the DINOv2 model, performance on
concrete concepts significantly improved from 74.94% to 85.87% (P = 1.78 × 10−76, two-tailed paired t-
test) as trainable parameters increased from 22.06 million to 304.37 million. However, for abstract
concepts, the performance gain from scaling parameters was negligible or even negative, with
performance dropping significantly from 54.37% to 52.82% as parameters increased from 22.06 million
to 304.37 million (P = 6.66 × 10−4, two-tailed paired t-test). Similar trends were observed across SimCLR,
and CLIP models. These results revealed that while increasing the parameter size of DNNs improved their
performance on concrete concepts, it did not substantially enhance their ability to understand abstract
concepts.

To investigate the underlying reasons for the performance disparity between concrete and abstract
concepts, we analyzed the representational dissimilarity matrices (RDMs; see Supplementary for
computational details) to visualize the conceptual relationships (representational distances) learned by
each model. We first examined the brain's RDM, derived from fMRI voxels from higher visual cortex
(HVC) areas24 including the lateral occipital complex (LOC), fusiform face area (FFA), and
parahippocampal place area (PPA), as a reference for a well-structured conceptual framework (Fig. 1d,
see Supplementary Fig. S2a, b for RIDMs derived from other brain regions).

We observed that the brain's RDM using voxels from HVC revealed two distinct clusters (Fig. 1d)
corresponding to living beings (red square) and non-living objects (purple square), reflecting a robust
semantic understanding of the concepts. When comparing the RDMs of DNNs (using the final output
representations), only the CLIP model exhibited partial alignment with the brain's RDM, showing a
discernible cluster for living beings (Fig. 1e). In contrast, the other models, while demonstrating strong
discrimination of concrete concepts (evidenced by superior classification capabilities), failed to capture
the broader semantic clusters of living and non-living entities. This suggests that without a meaningful
understanding of the relationships between concrete concepts, DNNs struggle to generalize to abstract
concepts, highlighting a critical limitation in their cognitive capabilities. We also found that using fMRI
signals from HVC to guide the model yielded significantly higher classification performance compared
to using voxels from the lower visual cortex (LVC) or the entire visual cortex (Supplementary Fig. S2c).
These results provide further evidence that the semantic structures in higher-level visual areas are critical
for model optimization.

Mental representation-guided supervision enables DNNs to learn human-like conceptual
representations

Previous analysis indicated that the inability to capture internal relationships between concepts might be
a key factor limiting DNNs' understanding of abstract concepts—a capability inherently embedded in
human cognition2. Instead of further scaling up model parameters, we proposed a mental representation-
guided supervision framework to transfer human conceptual structures to DNNs.

We hypothesized that the conceptual structure of objects could be represented with a graph, and both the
pretrained LMs and the human brain have formed their own conceptual structures based on learning. Then,
given a set of objects, we can align the LMs' conceptual graph to the human brain's with a graph
matching25 approach. Here we built an iterative optimization framework (Fig. 2a) to reshape the
conceptual structure of LMs through two main steps that are performed in an alternative manner: 1)
seeking the optimal correspondences between LMs and the human brain representations; 2) updating
representations to maximize structural similarity (Fig. 2b). To this end, our architecture consists of three
learnable modules, i.e., a fMRI encoder, an image encoder, and a graph neural network26. By differentiable
design, gradients can be backpropagated from the matching layers to the encoders during the training
procedure (see Methods for details).

While the number of concepts is large in the real world, we found that by aligning the conceptual graph
with a small set of objects, the alignment was well generated to unseen concepts. Here, using the CLIP
model as an example, we performed alignment between the brain and the CLIP model across 150 concrete
object categories, and evaluated the distances between the representations of the brain and DNN with 50
unseen classes (see Methods). As shown in Fig. 2c, as the mental representation-guided supervision
process evolves, the distance between brain and DNN representations decreased consistently for both the
concepts in the training concepts and unseen concepts in mostly linear processes (R2 = 0.86, Pearson's r
= –0.93 for training, and R2 = 0.83, Pearson's r = –0.91 for unseen), revealing good generalization to
unseen objects. The RDM of the 50 unseen classes also demonstrated similarity to brain's RDM
(Supplementary Fig. S3a), with a better conceptual structure of objects (Supplementary Fig. S3b, c). These
results suggested that mental representation-guided supervision learning approach effectively transferred
human's conceptual structures to DNNs.

Enhancement of abstract concept understanding

We next evaluated the model's ability to understand abstract concepts with and without mental
representation-guided supervision. As illustrated in Fig. 1a and b (Bottom), models trained with mental
representation-guided supervision demonstrated significant improvements in one-shot learning tasks for
abstract concepts (P < 0.0001, two-tailed paired t-test, n = 100 independent runs).

To further assess the understanding of abstract concepts at different levels of abstraction, we employed
the Silhouette Coefficient (SC)27 to measure the quality of representations on 50 unseen classes (n =
66,067 test samples). The SC score, ranging from −1 to 1, evaluates how well each sample fits within its
own cluster (intra-cluster distance) and how well it is separated from other clusters (nearest-cluster
distance). A higher SC indicates that objects within the same concept have similar representations, while
objects from different concepts are well-separated, reflecting a better understanding of conceptual
relationships. As shown in Fig. 3a, without mental representation-guided supervision, models exhibited
strong representation ability for concrete concepts (SC = 0.231 for the 50 leaf nodes) but showed a marked
decline in performance for middle-level (SC = 0.144 for the 8 middle-level nodes) and high-level concepts
(SC = 0.044 for the 3 high-level nodes). This suggests that while models excel at understanding concrete
concepts, they struggle with abstract ones.

In contrast, with mental representation-guided supervision, concept representation improved significantly
across all three abstraction levels (Fig. 3a, P < 0.0001, two-tailed paired t-test), particularly for abstract
concepts. The SC increased from 0.144 to 0.239 for middle-level concepts and from 0.044 to 0.127 for
high-level concepts. Notably, we observed a nearly linear correlation between the degree of structural
alignment (measured by Gromov-Wasserstein (GW) distance28, which quantifies the dissimilarity
between two metric spaces by comparing the pairwise relational structures; see Methods for
computational details) and the SC score (Fig. 3b). This strongly suggests that the enhanced conceptual
understanding stems from the alignment of the model's representational structure with the brain's
conceptual framework. Further improvements could be achieved by pushing the models' representational
structures to be more consistent with the brain. Together, these findings demonstrate that mental
representation-guided supervision effectively enhances models' capacity to conceptualize abstract
categories.

Emergence of human-like conceptual hierarchy

We further assessed the similarity between the conceptual hierarchy structures of humans and models
using the 50 unseen concepts. To this end, we utilized WordNet19, a comprehensive lexical database that
encodes conceptual-semantic relationships, as a representation of human conceptual hierarchy (Fig. 4a).
We compared this structure of the models with the 50 unseen concepts. Our analysis revealed that models
trained with mental representation-guided supervision exhibited concept similarity matrices (CSMs)
highly aligned with WordNet, whereas models without such supervision showed no such alignment (Fig.
4b; see Supplementary Tab. S1 for the row/column labels of CSMs). Note that each entry of the CSM
quantifies the similarity between a pair of concepts (see Supplementary for computational details). These
findings were consistent across different DNN architectures and brain signals from multiple subjects (Fig.
4b-d and Supplementary Fig. S4).

Additionally, we found that mental representation-guided supervision enabled models to exhibit human-
like judgments in concept similarity tasks. Specifically, we employed a triplet odd-one-out task, where
participants (human or DNN model) were presented with three images and asked to select the least related
one (Fig. 4e). Human judgment data were sourced from the THINGS database29, which contains 4.7
million triplet odd-one-out judgments across 1,854 objects from 12,340 participants. We compared the
similarity between human and model responses (see Supplementary for details). Results showed that
mental representation-guided supervision significantly improved the alignment between human
judgments and model behaviors (Fig. 4f), with accuracy increasing from 43.74% to 49.94% (P = 7.56 ×
10−97, two-tailed paired t-test).

Many current DNN models can be easily deceived by targeted pixel-level perturbations (adversarial
attacks), whereas humans are remarkably robust against such perturbations. The lack of robustness
highlights that DNNs fall short of human-like understanding concepts. Therefore, we used projected
gradient descent (PGD) attacks30 to evaluate the robustness of mental representation-guided supervised
models (Fig. 4g, see Supplementary for details). We observed that human-like conceptual hierarchy leads
to increased adversarial robustness for CLIP-base on abstract concepts, with adversarial accuracy
increasing from 50.82% to 68.95% (3 abstract concepts; P = 4.01 × 10−71, two-tailed paired t-test) and
10.98% to 14.79% (8 abstract concepts; P = 5.94 × 10−18, two-tailed paired t-test). Together, these results
demonstrate that mental representation-guided supervision enables models to develop human-like
conceptual hierarchy structures, significantly enhancing their ability to conceptualize abstract categories.

Human-like conceptual hierarchy enhances performance in complex cognitive tasks

We demonstrated that models endowed with human-like conceptual hierarchy structures achieve
significant performance improvements across a variety of complex cognitive tasks.

One-shot learning (OSL). We first evaluated the one-shot learning capabilities of the models at different
abstraction levels. As was mentioned, a central challenge in one-shot classification, is to rapidly
understand the concepts from only one example31. The problem can be more difficult when the concepts
become more abstract (Fig. 5a, see Supplementary). When there are only two abstract concepts of living
thing and non-living thing, CLIP-base with mental representation-guided supervision achieved an average
improvement of 20.5 ± 1.3% (mean ± s.d., averaged across S1-S3, P < 0.0001 for all subjects, two-tailed
paired t-test) in OSL performance, outperforming baseline models with 4.9 times more parameters (Fig.
5b). Mental representation-guided supervision achieved consistent performance gains across different
DNN architectures (CLIP10, SimCLR23 and DINOv29) and supervision using brain signals from different
subjects (Fig. 5b and Supplementary Fig. S5). Moreover, we found a nearly linear correlation between the
degree of structural alignment (measured by GW distance28) and the one-shot classification accuracy,
consistent across supervision using brain signals from different subjects (Supplementary Fig. S6).

Generalized out-of-distribution (OOD) recognition. Next, we examined how mental representation-
guided supervision enhanced the robustness of generalized OOD recognition (Fig. 5c, see Supplementary).
We defined a series of OOD recognition tasks (31 tasks, see Supplementary Tab. S2), which assess the
conceptual extrapolation ability of DNNs from minimal category supervision, such as determining
whether unseen categories (e.g., "bowling" and "carriage") should be classified as a broader category (e.g.,
"non-animal"). These tasks require models to capture high-level semantic structures that align with the
hierarchical structure of human conceptual knowledge. As shown in Fig. 5d, mental representation-guided
supervision significantly improved OOD robustness, yielding performance increases of 11.5 ± 4.7%, 13.6
± 9.3%, 9.1 ± 8.3% (mean ± s.d., averaged across 3 subjects and 31 tasks, P < 0.0001 for all subjects,
two-tailed paired t-test) for CLIP-base, SimCLR-base and DINOv2-base, respectively, without the need
for extensive supervision labels. Similar improvement trends can be found in different architectures across
varying parameter scales and supervision using brain signals from different subjects (Fig. 5d).

Brain-to-image and image-to-image retrieval. To intuitively assess the quality of the learned
representations, we evaluated the effectiveness of mental representation-guided supervision for both
brain-to-image (Fig. 5e, see Supplementary) and image-to-image (Fig. 5g, see Supplementary) retrieval
tasks. Brain-to-image retrieval, a common benchmark for neural decoding models24,32,33, evaluates the
ability to reconstruct perceived stimuli (n = 50 visual stimuli) from brain signals. Our approach
consistently improved decoding performance, achieving an average increase of 7.34%–26.67% in 50-way
retrieval across all subjects compared to existing decoding methods33–35 (Fig. 5f). Qualitative analysis of
brain-to-image retrieval on a subset of ImageNet-21K36 (n = 66,067 test samples, Supplementary Fig.
S7a-c) indicated the concept representations of DNNs can be predicted from fMRI patterns derived using
our mental representation-guided supervision framework. For example, the fMRI pattern evoked from an
image of "a man lying on a hammock" can be the image of "a child lying on the baby carriage" (the 5th
row, Supplementary Fig. S7a). In the case of "harp" (the 6th row, Supplementary Fig. S7a), although the
top retrievals fall into the wrong concrete categories, they are all related to "musical instruments". Similar
results can be found in most instances across different subjects (Supplementary Fig. S7b, c).

In image-to-image retrieval tasks (Fig. 5g, h), mental representation-guided supervision enabled models
to retrieve semantically similar objects despite significant visual differences, even when these objects
belonged to different predefined categories. For example, an image of a "beer mug" successfully retrieved
images sharing attributes like "cup" or "jug" from distinct categories (Fig. 5h, top row). Similar results
were observed across most instances, highlighting the model's enhanced ability to capture semantic
relationships beyond superficial visual features. Collectively, these findings demonstrate that human-like
conceptual hierarchy structures, facilitated by mental representation-guided supervision, significantly
enhance model performance in complex cognitive tasks.

Cognitive-level consistency of the learned conceptual manifold

To evaluate whether the latent embedding spaces, shaped by mental representation-guided supervision,
capture typical judgements for concepts, we generated a low-dimensional PCA-based space (conceptual
manifold; see Supplementary for details), and trained a GPT-2-based37 text decoder on the COCO
dataset38 (see Supplementary for details). This conceptual manifold allows us to assess cognitive-level
consistency for both seen and unseen points in the latent space, while the decoder reconstructs the
semantics of a given representation point as a sequence of words, enabling a detailed analysis of the
manifold's properties.

Local semantic consistency. We first assessed the local semantic consistency of the manifold, which
measures whether similar concepts are clustered together in the representation space. To this end, we
designed an out-of-distribution semantic coherence test (see Supplementary for details), evaluating the
semantic consistency and similarity between newly generated samples and their nearest real neighbors in
PCA space (Fig. 6a). This task can also be considered a zero-shot test of semantic coherence, since the
evaluations are performed in novel categories without explicit training. Note that in this study, zero-shot
(learning) means that performing a task on unseen samples, where the model is trained only on known
samples—not learning without any data. As shown in Fig. 6b, we found that the 2-D PCA points sampled
from different conceptual regions show strong semantic correlations with the real samples. For example,
we can reconstruct the semantics of T-shirt and tie within the "clothing" region and generate the semantics
of truck near the region associated with "vehicles". In contrast, models without brain supervision produced
semantically meaningless reconstructions (Supplementary Fig. S8a). By using 8 abstract categories as
ground-truth, we found that mental representation-guided supervision significantly enhanced local
semantic coherence, with semantic consistency accuracy improving by 15.9% and semantic similarity
scores increasing by 6.2% on average compared to the model without supervision (P < 0.0001, two-tailed
paired t-test, Fig. 6c).

Intersections between different concepts. We further investigated whether intersections between
different concepts yield meaningful reconstructions. By generating new points through interpolation
between two real samples from distinct concepts (Fig. 6d), we observed that the reconstructed semantics
aligned well with human intuition, exhibiting smooth and realistic transitions (Fig. 6e). For instance,
transitioning between "iPod" and "video recorder" resulted in a logical semantic shift from "television" to
"video recorder" (Fig. 6e, left), while interpolating between vehicles and ungulate animals produced the
concept of "horse" (Fig. 6e, right). Qualitative analysis indicates most of the key features of these concepts
can be effectively captured with a limited number of principal components, which is also reflected in the
reconstruction error and explained variance ratio (Supplementary Fig. S8b). These findings suggest that
mental representation-guided supervision ensures the conceptual manifold is semantically continuous and
smooth, enhancing the model's cognitive generalization capabilities.

Concept summarization and arithmetic. Finally, we evaluated the original latent space using two highly
challenging tasks: concept summarization (Fig. 6f) and concept arithmetic (Fig. 6h), which involves linear
operations on latent embeddings (see Supplementary for details). For concept summarization, the model
was tasked with generating a sentence to describe multiple input images. As shown in Fig. 6g, the
regulated manifold enabled the model to infer shared behavioral details and the number of objects. For
example, given a set of images depicting different types of animals lying on grass, the decoder with mental
representation-guided supervision was able to reconstruct the sentence of "animal laying on a grassy area".
Given a set of images, each depicting a close-up of a pair of different animals standing on grass, the
decoder successfully reconstructed the text "two animals standing in a grassy area".

For concept arithmetic (Fig. 6h), we tested whether mental representation-guided supervision could steer
generated captions toward desired conceptual directions through subtraction and addition operations. For
example, subtracting "motorcycle" and adding "horse" shifted the semantics from "a man riding a
motorcycle" to "a man riding a horse" (Fig. 6i, first case). Similarly, removing "dog" from "a dog playing
with a frisbee" and adding "a young girl" yielded "a young girl is playing with a frisbee" (Fig. 6i, second
case). In the third case, subtracting "banana" and adding "cell phone" transformed "a young girl holding
a banana" into "a young girl holding a cell phone."

These results demonstrate that mental representation-guided supervision produces a low-dimensional
conceptual manifold that aligns with human judgments, enabling accurate and intuitive descriptions of
objects along semantically meaningful dimensions. This advancement highlights the potential of brain-
guided approaches in creating models with enhanced cognitive generalization and interpretability.

Methods
Datasets

fMRI data. BOLD signals are obtained from an open dataset DIR45, which are collected using a Siemens
MAGNETOM Verio scanner 3.0-Tesla from three healthy subjects. In image presentation experiments,
each subject viewed 1,250 unique images of natural scenes. These image stimuli are drawn from 200
representative object categories of the ImageNet dataset36, with 150 categories used for training and 50
for testing. Each training image stimulus (8 images per category, totaling 1,200 images) is repeated 5
times, and each test image stimulus (1 image per category) is repeated 24 times, leading to 6,000 and
1,200 image-fMRI instances, respectively. More fMRI data preprocessing details can be found in the
original paper. In this study, we focused on voxels from brain visual areas including the lateral occipital
complex (LOC), fusiform face area (FFA), and parahippocampal place area (PPA), which are known to
be part of the higher visual cortex (HVC)24.

Natural images. The natural images used in this study were drawn from three datasets: ImageNet-21K36,
the THINGS object concept and image dataset46, and the Common Objects in Context (COCO) dataset38.
1) Corresponding to the BOLD signals of the DIR45, we collected natural images including 200 object
categories from Image-21K for mental representation-guided supervision of models, where 150 categories,
comprising 206,848 object images, were used as the training set, while the remaining 50 categories,
totaling 66,067 object images (275–2115 images per category), constitute the test set. Note that the object
categories are the same as those in the DIR, and there is no overlap between the training and test sets. 2)
The images of 1,854 diverse object concepts, collected from the THINGS dataset, were leveraged to
perform triplet odd-one-out tasks. 3) The COCO dataset, a large-scale image dataset including detailed
annotations of the images such as image captions, was used to train the text decoder in our manifold
analysis experiments.

Overall, we leveraged four public datasets to evaluate the effectiveness of our approach. The visual stimuli
used in the fMRI dataset (DIR) are drawn from ImageNet and are therefore independent of the THINGS
and COCO datasets. This ensures that the test samples in triplet odd-one-out tasks (Fig. 4e) and semantic
reconstruction (Fig. 6) are out-of-distribution (OOD). For mental representation-guided supervised
models, we only employed 150 categories in the DIR dataset for training. The THINGS dataset provides
4.7 million human triplet odd-one-out responses across 1,854 object concepts. This greatly surpasses the
200 object categories in the DIR dataset, making it highly suitable for testing whether mental
representation-guided supervision enhances the alignment between models and humans (Fig. 4f). Unlike
ImageNet, which only provides simple category labels, the COCO dataset includes detailed image
description, enabling the training of a text decoder to reconstruct semantic information from mental
representation-guided supervised models.

Mental representation-guided supervised learning

To integrate the human conceptual knowledge into LMs, here we propose a new mental representation-
guided supervised learning framework to bridge the gap of conceptual structures between DNNs and the
brain. Our framework leverages non-invasive neural responses (fMRI) to reshape DNNs, aligning their
latent representation structure of concepts with that of the brain (Fig. 2a, b, and Supplementary Fig. S9).
Note that the mental representation-guided supervised learning adopts a per-subject-per-model paradigm
(training a separate DNN model for each individual subject).

Problem formulation and notation. We will denote the model (image) embedding and the fMRI
embedding as 𝑥 and 𝑦, respectively. Together, (𝑥,𝑦) form a positive or negative pair based on whether
their semantic labels match. Given an arbitrary set of image embeddings 𝕏 = {𝑥 }𝑛 and neural patterns
𝑖 𝑖=1
𝕐 = {𝑦 }𝑚 , our goal is to find a mapping 𝑓∗ such that 𝑓∗ = 𝑎𝑟𝑔𝑚𝑖𝑛𝑓 (𝕏;𝕐,𝑑), where 𝑑 represents a
𝑖 𝑖=1 𝜃 𝜃 𝜃
𝑓𝜃
structural distance measure that assigns smaller values to more similar structures.

By representing image embeddings and fMRI signals as entities (nodes) and modeling them as graphs,
our target can be naturally formulated as an instantiation of graph matching (GM), a technique that aims
to find meaningful node correspondences between graphs by considering a structural similarity metric34.
Our learning procedure is intuitive, comprising two key steps executed alternately: 1) constructing a set
of putative matches (correspondences) 𝑫 = {(𝑥 ,𝑦 )}𝑁 from (𝕏,𝕐) ; 2) optimizing the structural
𝑖 𝑖 𝑖=1
similarity between corresponding entities (Supplementary Fig. S9).

Correspondence construction via optimal transport (OT). OT provides a means to measure distances
between probability distributions over the domain. It is well known that, within the basic form of OT,
probability distributions are interpreted as piles of sand. In this context, the distance between two piles of
sand can naturally be formulated as the amount of work required to transform one pile into the other,
also known as the Earth Mover’s Distance (EMD) or Wasserstein Distance (WD)47. From a maximum entropy
perspective, Cuturi48 proposed an entropy-regularized OT formulation, allowing the optimal transport
plan can be efficiently solved on GPU by Sinkhorn’s matrix scaling algorithm. The Sinkhorn algorithm
is an iterative method for computing an entropy-regularized optimal transport plan between two
probability distributions efficiently. Below, we introduce how to incorporate the OT tools into our mental
representation-guided supervised learning framework to construct a set of putative matches, which can be
regarded as a node matching procedure.

Formally, suppose 𝕏 and 𝕐 be sample points in two complete separable metric spaces respectively, and
their discrete probability measures can be formulated as 𝝁 = ∑𝑛 𝑝 𝛿 and 𝝂 = ∑𝑚 𝑞 𝛿 , where 𝛿
𝑖=1 𝑖 𝑥𝑖 𝑖=1 𝑖 𝑦𝑖 𝑥𝑖
denotes the Dirac function centered on 𝑥 , [𝑝 ] ∈ Δ and [𝑞 ] ∈ Δ belong to the n- and m-
𝑖 𝑖 1:𝑛 𝑛 𝑖 1:𝑚 𝑚
dimensional simplex. Based on this definition, we adopt OT tools47 to produce a set of putative
correspondences by seeking the least costly transport plan 𝜞 , which can be compactly written as:
𝑊
𝒟 (𝝁,𝝂):= 𝑖𝑛𝑓𝔼 [𝑐(𝕏,𝕐)] =𝑚𝑖𝑛 ⟨𝜞 ,𝑪⟩ (1)
𝑤 T(𝕏,𝕐)~𝛾 𝜞𝑊∈Π 𝑊
𝛾∈Π
where Π(𝝁,𝝂) ≔ {𝜞
𝑊
∈ ℝ+ 𝑛×𝑚 | 𝜞
𝑊
𝟙
𝑚
= 𝝁,𝜞
𝑊
T𝟙
𝑛
= 𝝂} , 𝑪 ∈ ℝn
+
×m is a cost matrix between 𝕏 and 𝕐
(calculated by the cosine distance function 𝑐), and ⟨⋅,⋅⟩ denotes Frobenius inner product.

Representational structure alignment. Suppose we have obtained a set of putative correspondences
𝑫 = {(𝑥 ,𝑦 )}𝑁 between image embeddings 𝕏 and neural patterns 𝕐 using the Sinkhorn algorithm,
𝑖 𝑖 𝑖=1
together with the local structure associated with each correspondence (see Supplementary for a formal
definition), structure similarity ℒ can be quantified from two aspects: 1) the number of correct putative
correspondences within local neighborhoods, and 2) the topological similarity among these matching
nodes (Supplementary Fig. S9). The alignment process can be written as:
𝑁 𝑁
ℒ(𝕏,𝕐;Θ) = ∑|𝑔 (𝑥 ,𝑦 ;𝑫)|+∑𝒟 (𝑥 ,𝑦 ;𝑔 ) (2)
𝑘 𝑖 𝑖 𝑔𝑤 𝑖 𝑖 𝑘
𝑖=1 𝑖=1
where 𝑔 returns the k nearest neighbor for each node (|⋅| returns the cardinality of correct putative
𝑘
correspondences), 𝒟 denotes a topological metric function. Due to the non-differentiability of the |⋅|
𝑔𝑤
operation, we achieved a similar effect by aligning the sampled distributions between 𝕏 and 𝕐 .
Accordingly, our final cost function can be optimized through the update of model parameters Θ during
end-to-end training. Moreover, for this work, we consider the Gromov-Wasserstein (GW) distance28 as
the topological metric function, which can be used to evaluate the structural similarity between different
metric spaces without directly calculating distances between entities across domains. Note that our
optimization process does not rely on task-specific loss functions such as classification. Mathematical
definitions and implementation details of GW-based representational structure alignment are provided in
Supplementary Text.

Architecture and optimization

Model architecture. Technically, like most representation learning practices, our first component
employs two nonlinear encoders (here, a ViT50 and a multilayer perceptron but can take any other form)
to map brain signals (fMRI) and image embeddings to new latent spaces (Fig. 2a). Next, a GNN34,51,52
with multi-head attention, which learns node representation for both fMRI and images (parameters are
shared across modalities; see Supplementary for details) is leveraged to predict an assignment matrix with
minimal Wasserstein distance47, resulting in initial correspondences between brain and model
representations. The representations in latent spaces are considered as graph nodes (entities). To reshape
embedding spaces of DNNs, we leverage a differentiable sampler (based on re-parametrization technique
Gumbel Softmax49) with the similarity scores between entities to construct local structures for each
putative correspondence. It is worth noting that the procedure of structure (edge) construction is
"dynamic" since node representations are evolving through the update of neural network parameters
during training. See Supplementary for the architecture configuration and hyperparameter settings.

By using relaxation techniques48,49, all calculation and optimization processes in the pipeline are
differentiable, enabling gradients to be backpropagated from the matching layers to the encoders. We
conduct mental representation-guided supervision on the DIR dataset. The stimuli (150 training categories
and 50 test categories) in DIR dataset are identical to those used in the GOD dataset24 but contain a larger
number of evoked fMRI recordings (6,000 training samples and 1,200 test samples). See Supplementary
for details of training procedures.

Loss function. Given an image-fMRI dataset (𝕏,𝕐), we first minimized the negative log-likelihood of
the transport plan 𝜞 , and a cross-entropy item to seek a set of putative correspondences 𝑫:
𝑊
ℒ = − ∑ 𝑙𝑜𝑔[𝜞 ] − 𝜆 ∑ 𝑙𝑜𝑔[𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑪)]
𝑊 𝑊 𝑖,𝑗 1 𝑖,𝑗 (3)
(𝑖,𝑗)∈𝑉 (𝑖,𝑗)∈𝑉
where 𝑉 contains ground-truth matching labels, 𝜆 = 0.1 is used to balance the two terms, and 𝑪 is the
1
cosine similarity matrix derived from the outputs of the image and fMRI encoders. The transport plan
𝜞 = diag(𝒂)𝐊diag(𝒃) was obtained through the solution of an entropy-regularized OT problem
𝑊
𝑚𝑖𝑛⟨𝜞 ,𝟏−𝑪⟩−𝜀𝐻(𝜞 ), where 𝐻(𝜞 ) = −∑ [𝜞 ] 𝑙𝑜𝑔[𝜞 ] is an entropy constraint term (trade-
𝑊 𝑊 𝑊 (𝑖,𝑗) 𝑊 𝑖,𝑗 𝑊 𝑖,𝑗
off parameter 𝜀 = 1), 𝑲 = 𝑒−(1/𝜀)𝑪 ∈ ℝ𝑛×𝑚 is a Gibbs kernel, 𝒂 ∈ ℝ𝑛 and 𝒃 ∈ ℝ𝑚 can be computed by
+ + +
Sinkhorn iterations48.

Let 𝒔 ∈ ℝ𝑁−1 and 𝒛 ∈ ℝ𝑁−1 be the sampled probability vectors for 𝑥 and 𝑦 , respectively, computed
𝒊 𝒊 𝑖 𝑖
using Gumbel-Softmax. Then, we minimized the cross-entropy loss between 𝒔 𝒊 and 𝒛 𝒊 , and GW distance
between dynamically constructed local structures, which can be written as:
𝑁 𝑁−1 𝑁
1 𝜆
ℒ = − ∑∑[𝒛 ] 𝑙𝑜𝑔[𝒔 ] + ∑∑𝐿([𝑪 ] ,[𝑪 ] )[𝜞 ] [𝜞 ] (4)
𝐺𝑊 𝑁 𝒊 𝑗 𝒊 𝑗 𝑊 𝑥𝑖 𝑐,𝑘 𝑦𝑖 𝑑,𝑙 𝐺𝑊 𝑐,𝑑 𝐺𝑊 𝑘,𝑙
𝑖=1 𝑗=1 𝑖=1 𝑉
Similarly, the transport plan 𝜞 (GW distance) was derived by solving an entropy-regularized
𝐺𝑊
optimization problem using the Sinkhorn algorithm. 𝑪 and 𝑪 are two measured self-similarity
𝑥𝑖 𝑦𝑖
matrices for 𝑥 and 𝑦 (see Supplementary for details). In the second item, we penalize only the
𝑖 𝑖
dissimilarity between true correspondences 𝑉, where 𝑊 denotes the number of true correspondences
within constructed local structures (trade-off parameter 𝜆 = 100). Together, ℒ and ℒ constituted
2 𝑊 𝐺𝑊
the final loss functions of the mental representation-guided supervision framework across the two iterative
steps.

Statistical analysis

In this study, we applied the two-tailed paired t-test and repeated random sub-sampling validation (also
known as Monte-Carlo cross-validation) for statistical analysis. For one-shot classification and OOD
recognition tasks, a run corresponds to an independent evaluation with different training samples and
different test samples, with the sample categories randomly selected. Data visualizations were
accomplished using seaborn, a Python data visualization library, and statannotations, which is used to add
statistical significance annotations on seaborn plots. The detailed P values for all comparisons are
summarized in Tab. 1.

Data availability

All data used in this study are publicly available. 1) DIR dataset: https://open
neuro.org/datasets/ds001506/versions/1.3.1. We used the officially preprocessed fMRI data (v2 data) from
https://figshare.com/articles/dataset/Deep_Image_Reconstruction/ 7033577; 2) THING database:
https://osf.io/f5rn6/; 3) Microsoft COCO 2017 dataset: https://cocodataset.org/; 4) ImageNet-21K dataset:
https://image-net.org/download. Download the images of a single synset: https://image-
net.org/data/winter21_whole /[synsetid].tar (replace [synsetid] with the actual synset ID). We used
Natural Language Toolkit (NLTK) in python to acquire WordNet. Source data are provided with this paper.

Code availability

The source code for mental representation-guided learning is publicly available at
https://github.com/JxuanC/mental-representation-guided-learning.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "(Remarks to the Author)\n(Remarks on code availability)\nThere is a link to the source code used in the study\nbut when attempting to access it, one gets the following message: “The record is\npublically available but files are restricted to users with access”. I did not request\naccess.", "reviewer_2": "(Remarks to the Author)\nThe paper reports some examples of pattern recognition tasks to be learned by deep networks, also called 'large models',\nwhere some learning abilities are improved by providing the network with additional inputs generated from fMRI data from\nsome brain areas of some subjects. From this the authors draw wide ranging conclusions, namely that the fMRI input allows\nthe network to 'reshape' its representations and thereby to acquire 'human conceptual structures'. The most relevant\nargument for this is the the similarity of the 'response dissimilarity matrices' (RDMs) of the human fMRI response and the\nrepresentation in some layer of the network, when it is provided with the fMRI input, which entails a similar cluster structure\nof these representations. In the article it is neither clarified which area is used for the fMRI (some higher visual area) nor\nwhich layer of the network is used. Also, the task is a rather simple classification problem based on a simple two-level\nhierarchy of visual objects. This experiment does not substantiate the wide ranging consequences drawn by the authors,\ninvolving, for example, completely different tasks, or just other hierarchies, other human subjects or other brain areas. The\narchitecture of the network is not explained in the article. Is it important which area to use for computing the RDM? Is a signal\nwith low temporal resolution, like fMRI, sufficient for other tasks? Can we learn something about the area or areas involved?\nDo the extensive statistics make any sense (in particular, what is the N)? Could the simple hierarchical structure be\npresented more directly to the network or does it need human fMRI? Would this be practical and for which kind of tasks?\nIn summary, many important details are not explained or mentioned in the manuscript. In addition, a lot of technical jargon is\nused concerning the computational methods without a comprehensible explanation (e.g. RDM, SC, GW distance, CSM,\nconcept arithmetic, conceptual manifold, structural similarity metric, Frobenius inner product, Sinkhorn algorithm, Gumbel\nsoftmax trick, shared multiplex attention GNN).\n(Remarks on code availability)", "reviewer_3": "(Remarks to the Author)\n“I co-reviewed this manuscript with one of my doctoral students, and Early Career Researcher.\nThis is part of the Nature Communications initiative to facilitate training in peer review and to provide appropriate recognition\nalso for ECRs who co-review manuscripts.”\nNoteworthy results: (reviewers' summary)\nRecent advances in deep neural networks (DNNs), especially large models (LMs) have skyrocketed their image and text\nunderstanding as well as generation and classification performance. However, imitating the in depth interpretation and\nabstract reasoning capacities of humans remains a major challenge.\nThis study challenges the ability of large neural network model architectures to capture and learn generalised concepts.\nSince DNNs are usually trained to perform well on a clearly defined task, such as classification, they tend to lack the human-\nlike abstraction capacity. To this end, a brain-in-the-loop supervision framework was introduced, which aligns the DNN\nembeddings with human brain activation structure. Specifically, fMRI brain signals were compared to the DNN image\nembeddings via an encoder and graph neural network architecture. The improvement of generalization capabilities was\ndemonstrated in a series of tailored experiments providing evidence and supporting the claim that the semantic coherence of\nDNNs can be enhanced by incorporating human conceptual structure into their training. Improved performance also involved\none-shot learning and out-of-distribution recognition.\nValidity and Significance\nThis paper introduces a novel method of leveraging human brain-based representations to improve AI systems toward\nhuman-level understanding. An iterative graph matching routine is introduced utilising Wasserstein distance and Gromov-\nWasserstein (GW) distance based loss functions (line 630 and 642 in the main text). The validity of the results is tested on\nseveral state-of-the-art models including CLIP, SimCLR and DINOv2 showing that one-shot classification accuracy\nimproves significantly when brain-in-the-loop supervision is deployed, particularly for abstract concepts (see Fig.1 a-c). The\nstatistical robustness of the results is confirmed by two-tailed paired t-tests with adequate P-value scores.\nNotably, the framework paves the way for integrating human-like abstraction capabilities into AI by means of its structural\nalignment with biological neural activity.\nOverall, the paper is very well written, with a lot of numerical results, providing a consistent picture about what has been\ngained by including sparse brain-in-the-loop information into AI models.\nMethodology and Data\nIn order to test the concept understanding of DNNs (without brain-in-the-loop) a one-shot learning experiment (Fig.1, a-c)\nwas conducted for concrete and abstract concepts and three different architectures with varying sizes. Increasing the model\nparameter number yielded an improvement in the concrete concept classification, but only a negligible advantage in abstract\nconcept classification.\nFor the remaining set of experiments the networks were integrated into a composite architecture together with the fMRI\nencoder of brain signals, training the encoders across 150 training categories and evaluating with 50 unseen ones. The\nSilhouette Coefficients were computed to measure representation quality for concrete vs. abstract concepts demonstrating\nenhanced conceptual alignment with brain-in-the-loop (Fig.3).\nFurthermore, conceptual similarity matrices (CSM) of DNN’s representations with brain-in-the-loop demonstrate higher\nsimilarity to WordNet’s conceptual hierarchy and odd-one-out accuracy rises by ~6% compared to the base model (Fig.4).\nOne shot classification and out-of-distribution recognition with brain-in-the-loop display higher accuracy for all considered\nDNNs (Fig.5) and consistency tests on a PCA manifold also corroborate improved semantic robustness with brain-in-the-\nloop supervision (Fig.6). Moreover, multiple tests, such as semantic summarisation, focusing on the conceptual manifold\nconfirmed its cognitive-level consistency and ordered organization of concepts. (also Fig. 6).\nAltogether, the initial assumption that incorporating brain-in-the-loop optimization into large DNN image classifiers promotes\nthe model’s abstract conceptual understanding is more than well supported by the experimental evidence.\nClarity and Reproducibility - Questions\nAlong with the paper, both the code and data are provided which should enable good reproducibility of the results.\nRegarding the clarity, a few questions seem to remain open after reading. During training of the encoders and GNN 206,848\nobject images but only 6,000 fMRI signals were used (see Methods line 492 and 501 and Supplementary Text). Even\nthough the batch size is kept constant for both branches of the framework (fMRI and LM), the graph neural network\neventually receives an imbalanced distribution of training data to be matched. Does this discrepancy affect the brain-in-the-\nloop learning process and if so how?\nLine 130 of main text mentions that during brain-in-the-loop training the image encoder, fMRI encoder and graph neural\nnetwork are optimized, does this imply that the vision model (large neural network in Fig.2 a) has its parameters frozen?\nThe batch size of 256 matches the embedding size (dimension) of the image encoder (see Supplementary Text /\nImplementation details of brain-in-the-loop supervised learning). However, the data points can in principle still be linearly\ndependent (i.e. not spanning the whole space). In the main article the discrete probability measures of the sample points are\nassumed to be n/m-simplices (line 543), which is in general not guaranteed for random in-distribution batch.\nDuring brain-in-the-loop optimization of the DNNs, only the image encoder (MLP), fMRI encoder (ViT) and the shared GNN\nwere trained. How were these framework components later integrated into the models in order to perform the classification\nexperiments? For example in Supplementary / One-shot classification, a linear classifier was trained, but it is not clear where\nit connects to the brain-in-the-loop architecture… could you briefly specify how the architecture components were\nincorporated in the experimental phase (e.g. one-shot classification)?\nZero-shot learning is mentioned in the abstract, introduction and discussion parts, but it is missing in the methods and\nexperimental results.\nSuggestions\nUnify the font for scalars, vectors and matrices. E.g. in line 638, z_i is a vector, but in line 636 a is also a vector. In line 631, V\nshould not be bold.\nIn statistical analysis, would it be possible to present numerical results in a more transparent form such as a table (rather\nthan one huge paragraph)?\nMaybe a question also to the editor: is it necessary to report all P-values in a complete detail? (Perhaps simplification to, e.g.\nP<10^{-4} (for all smaller values), would be sufficient.\nWe suggest adding the reference: Zhu et al (2000) Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike\nCommon Sense. Engineering, 6, 310-345, https://doi.org/10.1016/j.eng.2020.01.011. It introduces a wide approach toward\nhuman-like AI, largely different from the brain-in-the-loop framework presented here.\nMinor details\nL331: Is this sentence grammatical? “Our results support that human-like cognition…” (‘support’ is a transitive verb)\n“... suggest/indicate that…” “... support the idea/claim/assumption that…” “... demonstrate that…”\nUse ‘–’ instead of ‘-’ for mathematical minus symbol\nReferences contain a few typing errors (e.g. missing commas, small first letters in conference name, …).\nLimitations - or points for the general discussion\nOne could argue that utilizing fMRI data within a hybrid system is a constraint in itself in the sense that this data is not easily\naccessible for an average stakeholder or developer. Probably in the future this line research will be enabled by availability of\nnumerous public datasets usable for brain-in-the-loop supervision framework (or similar approaches).\nThe paper talks about understanding images (and text) based on the excellent performance (including generalization) of\ntrained DNNs on various tasks. However, all DNNs in general are subject to adversarial vulnerability, which implies the lack\nof robustness, see, e.g. Ren et al (2020), https://doi.org/10.1016/j.eng.2019.12.012 and the issue regarding ViT, e.g.\nFavole and Rawat (2025), https://doi.org/10.1145/3729167. This implies that the term understanding should be used with care,\nsince the lack of robustness is not compatible with true understanding of concepts. On the other hand, improving the\nDNNs using human conceptual knowledge, as shown in this paper, may contribute to increased robustness in DNNs which can be\nexperimentally tested.\nThe paper talks about abstract concepts, but it should be acknowledged that the approach only considers a subclass of\nabstract concepts, namely those involving a superordinate level of categorization of concrete objects (as mentioned in Suppl\nFig.S1). The world of abstract concepts is very heterogeneous, see, e.g. Borghi et al (2018),\nhttp://dx.doi.org/10.1098/rstb.2017.0121m so it would be interesting to see whether the brain-in-the-loop approach could be\nextended to abstract concepts that do not provide good visual exemplars (e.g. truth, democracy), yet images (situational\ncontexts) exist where they are relevant. The success would depend on the properties of brain patterns.\n(Remarks on code availability)", "reviewer_4": "(Remarks to the Author)\nThis work sets out to investigate whether Brain-in-the-Loop supervision can elevate the performance of cognitive\ngeneralization for Large Language Models.\nIt proposes to enhance the cognitive ability of Deep Neural Networks (DNNs) by using the structural organization of the\nhuman brain as a supervisor. Initial findings showed that increasing the parameter size of DNNs improved their\nunderstanding of concrete concepts but there was little effect on abstract concept understanding.\nIncreasing parameter sizes enhances DNN's understanding of concrete but not abstract concepts was observed in DNN\narchitectures SIMCLR, DINOv28 and CLIP9 across varying parameter scales.\nHypothesising that that both the pretrained LMs and the human brain have built their own conceptual structures the authors\nset out to align the LMs conceptual graph with the human brain’s via graph matching. This is done through an iterative\noptimization framework using two steps: finding the optimal correspondences between the LM and the brain representations\nand then updating the LM representation to maximize structural similarity. The architecture for this comprises an fMRI\nencoder, an image encoder and a graph neural network.\nThe details of the architecture and the algorithms used for graph matching and distance (Gromov Wasserstein) are\nthoroughly described in the Methods section of the paper.\nThe authors say that they found that in aligning the conceptual graph with a small set of objects there was good\ngeneralization to unseen concepts, but it is not clear how extensive this generalisation is beyond the test set of unseen\nobjects. The alignment was done using 150 object categories and evaluated the distances between the brain\nrepresentations and the DNN with 50 unseen categories.\nThe paper then reports on the major results that can be set out as:\n• Brain-in-the-loop supervision enables DNNs to learn human-like conceptual representations - as the brain-in-the-loop\nsupervision process evolves, the distance between brain and DNN representations decreased for both the training and\nunseen concepts;\n• Enhancement of abstract concept understanding with brain-in-the-loop\n• Emergence of human-like conceptual hierarchy - models trained with brain-in-the-loop supervision exhibited concept\nsimilarity matrices highly aligned with WordNet while the others were not\n• Human-like conceptual hierarchy enhances performance in complex cognitive tasks – achieve significant performance on\none-shot learning, generalized out-of-distribution recognition, brain to image and image to image retrieval\n• Cognitive-level consistency of the learned conceptual manifold\nThese results are all clearly described although the reader is often referred to information in the supplementary information.\nThe supplementary information is very clear. The experimental setup and the datasets are quite complex. One concern that I\nhave is how the fMRIsfrom the 3 subjects are used to give a single brain model? Additionally, the detailed experiments\nwould sit more comfortably if the connections between the BOLD fMRI data set and the THINGS and COCO datasets were\nmade clearer.\nThis is a very good piece of research that is well motivated and quite clearly communicated. The statistical results are well\nexecuted and in general the experimental results are clear, however there are many results, and the graphs and tables could be\nindividually clearer. The theoretical development in the methods is sound however the way in which the fMRI models are\nlinked to the other datasets used needs some additional clarification.\nPertinent to the overall claim of the paper is the extent to which the claim that the Brain-in-the-Loop improves cognitive\ngeneralisation capability of LMs is whether this work with a relatively small number of concept categories will extend and\nwhat further things would need to be done to match the extent of LLMS? How extensible is the approach to achieve human\nbrain aligned LMs?\nThere are some minor typographic errors that can be fixed by a thorough proof reading.\n(Remarks on code availability)"}, "author_response": "Response to the Editors and Reviewers\nNote: reviewers’ comments appear in black text. Our replies appear in blue text, and the\nrevised manuscript text appears indented (with old text shown in black and new edits in\nred).\nOverview:\nWe thank the reviewers for their careful reading of the manuscript and their insightful and\nhelpful suggestions. Before our point-to-point responses, we first give a brief overview\nof the four major modifications that resulted in new figures and/or significant changes\nto the presentation of the work.\nFirst, we added new analyses and results on how brain region selection affects our method,\nwhich provides new insights into the role of different brain areas in shaping performance.\nSecond, we conducted new experiments to evaluate representational robustness, which\nfurther demonstrate the effectiveness of our approach.\nThird, we provided more detailed technical clarifications—such as network architecture,\noptimization, and relationships between datasets—to address all reviewers’ concerns.\nFinally, we added new discussions (e.g., limitations and future research) in accordance\nwith the reviewers’ suggestions.\nWe believe these revisions help make the findings more general and stronger. Many\nthanks!\nPoint-by-point Responses\nReviewer #1 (Remarks on code availability):\nThere is a link to the source code used in the study but when attempting to access it, one\ngets the following message:“ The record is publicly available but files are restricted to\nusers with access”. I did not request access.\nThank you. The issue may have been caused by a line break in the raw URL in the original\nsubmission, which prevented direct access. We apologize for the inconvenience. For\neasier access, we now provide a direct hyperlink: source code. Many thanks.\nReviewer #2 (Remarks to the Author):\nThe paper reports some examples of pattern recognition tasks to be learned by deep\nnetworks, also called 'large models', where some learning abilities are improved by\nproviding the network with additional inputs generated from fMRI data from some brain\nareas of some subjects. From this the authors draw wide ranging conclusions, namely that\nthe fMRI input allows the network to 'reshape' its representations and thereby to acquire\n'human conceptual structures'. The most relevant argument for this is the similarity of the\n'response dissimilarity matrices' (RDMs) of the human fMRI response and the\nrepresentation in some layer of the network, when it is provided with the fMRI input, which\nentails a similar cluster structure of these representations.\nWe thank the reviewer for the insightful and helpful questions and suggestions.\nThe modified manuscript makes a lot of progress due to your valuable suggestions. The\ncomments are response point by point below, and the revised text is highlighted in the new\nmanuscript. Thanks again.\nIn the article it is neither clarified which area is used for the fMRI (some higher visual area)\nnor which layer of the network is used.\nThank you for this question. In our study, we used fMRI voxels from brain visual cortex\nareas including the lateral occipital complex (LOC), the fusiform face area (FFA), and the\nparahippocampal place area (PPA), as described in the datasets part of Methods section.\nFor clarity, we have further clarified this point in the main text, which reads:\nWe first examined the brain's RDM, derived from fMRI voxels from higher visual cortex\n(HVC) areas22 including the lateral occipital complex (LOC), fusiform face area (FFA),\nand parahippocampal place area (PPA), as a reference for a well-structured\nconceptual framework (Fig. 1d, see Supplementary Fig. S2a, b for RDMs derived from\nother brain regions).\n22. Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using\nhierarchical visual features. Nat. Commun. 8, 15037 (2017).\nRegarding the network layer, we consistently leveraged the final output representation of\nthe models throughout all experiments. Because these representations are widely applied\nin downstream tasks. We have updated the main text to explicitly clarify this point.\nAlso, the task is a rather simple classification problem based on a simple two-level\nhierarchy of visual objects. This experiment does not substantiate the wide ranging\nconsequences drawn by the authors, involving, for example, completely different tasks, or\njust other hierarchies, other human subjects or other brain areas. The architecture of the\nnetwork is not explained in the article.\nThank you for the valuable comments. Besides simple classification tasks based on two-\nlevel hierarchy of objects, we also conducted experiments across different tasks, including\none-shot classification (Fig. 5a, b, Fig. S5a) across eight-level hierarchy, generalized out-\nof-distribution (OOD) recognition (Fig. 5c, d, Fig. S5b), brain-to-image retrieval (Fig. 5e, f,\nFig. S7 a-c), image-to-image retrieval (Fig. 5 g, h), text generation including semantic\ninterpolation, summarization, and arithmetic (Fig. 6). Most experiments are conducted\nacross three subjects. Furthermore, the detailed architecture of our network is provided in\nthe Supplementary text (the implementation details part), where we have also added the\npreviously missing details as requested. Many thanks.\nRegarding the analysis of different brain areas, we have added new supplemental results,\nwhich are reproduced below:\nSupplementary Fig. S2 | Analysis of how different brain areas contribute to\nperformance gains. a, b, Representational dissimilarity matrices (RDMs) of different\nvisual cortex (VC) areas (averaged across three subjects S1-S3). Here lower visual\ncortex (LVC) includes V1, V2, and V3. Only the RDMs from higher visual cortex (HVC)\nareas (LOC, FFA, PPA) exhibited semantic clusters of living and non-living entities. c,\nOne-shot classification performance on 50 concrete concepts and 3 abstract concepts\nusing brain-in-the-loop supervision with different visual cortex voxels (S3). Among the\nregions of interest (ROIs) in the VC, HVC-guided brain-in-the-loop supervision (CLIP-\nbase) demonstrated significant improvements in one-shot learning tasks for abstract\nconcepts (P < 0.0001, two-tailed paired t-test). The one-shot accuracy of abstract\nconcepts increased from 62.32 ± 15.36% (VC) and 65.57 ± 16.37% (LVC) to 74.00 ±\n9.64% (mean ± s.d., averaged across 100 independent runs; P < 0.0001, two-tailed\npaired t-test).\nWe believe the new results do help make the results more general and stronger. Many\nthanks.\nIs it important which area to use for computing the RDM?\nThank you for this insightful question. The choice of brain area is important for computing\nRDMs. In the revised manuscript, we included new supplementary results of RDMs\ncomputed from different visual cortex (VC) areas including V1-V4, LOC, FFA, and PPA\n(see Supplementary Fig. S2a, b). Notably, only the RDMs from higher visual cortex (HVC)\nareas (LOC, FFA, PPA) exhibited semantic clusters of living and non-living entities, likely\nbecause these areas are closely correlated with high-level visual semantic representations.\nBy contrast, lower visual cortex (LVC) areas mainly encode low-level visual features such\nas edges and orientation. Many thanks.\nIs a signal with low temporal resolution, like fMRI, sufficient for other tasks?\nThank you for this insightful question. While fMRI indeed has limited temporal resolution,\nit offers high spatial resolution and reliable measurements of distributed activity patterns,\nmaking it particularly effective for tasks that depend on robust representational structures,\nsuch as classification, semantic clustering, neural decoding, and content generation.\nWe agree that fMRI is not well suited for tasks requiring fine-grained temporal precision.\nHowever, it is sufficient for the current scope of our study, which focuses on semantic\nstructure alignment at the level of spatial representations. Looking forward, integrating\ncomplementary modalities with higher temporal resolution (e.g., MEG or EEG) represents\na promising direction to extend our method to temporally fine-grained tasks. Many thanks.\nCan we learn something about the area or areas involved?\nThank you for this valuable question. We have conducted new supplemental experiments\nto analyze the contributions of brain areas to performance gains (see Supplementary Fig.\nS2c). Our analyses further show that using fMRI signals from higher visual cortex (HVC)\nto guide the model yielded significantly higher classification performance compared to\nusing voxels from the lower visual cortex (LVC) or the entire visual cortex, suggesting that\nLVC does not yet form efficient semantic structures, as supported by the RDMs in\nSupplementary Fig. S2a, b. Overall, these findings provide further evidence that the\nimprovement arises broadly from HVC (LOC, FFA, and PPA). That is, the semantic\nstructures in higher-level areas are critical for model optimization. Many thanks.\nDo the extensive statistics make any sense (in particular, what is the N)?\nThank you for pointing out this issue, which we should have explained more clearly. In our\nexperiments, the meaning of N varies depending on the context. For example, N refers to\nthe number of independent trials for diverse downstream tasks (Fig. 5) and the semantic\nconsistency analyses (Fig. 6a–c), whereas for representational evaluation (SC score) N\ndenotes the number of test samples.\nWe have carefully checked all instances of N throughout the manuscript, corrected any\ninconsistencies, and explicitly defined its meaning in the Methods section or relevant figure\ncaptions. Many thanks.\nCould the simple hierarchical structure be presented more directly to the network or does\nit need human fMRI?\nThank you for this insightful question. Our brain constantly learns concept representations\nfrom the physical environment, which enables visual scenes to convey information beyond\nthe mere identity of the objects present. It remains unclear how to represent the semantic\ninterrelations learned by the brain in a form that can be directly presented to computational\nmodels1. In practice, existing research typically relies on fMRI signals to reveal brain\nrepresentations of natural objects2. Therefore, using human fMRI to guide the network in\nlearning human-like conceptual structures provides a practical and effective approach.\n1. Doerig, A. et al. High-level visual representations in the human brain are aligned with\nlarge language models. Nat. Mach. Intell. 7, 1220–1234 (2025).\n2. Hebart, M. N., Zheng, C. Y., Pereira, F. & Baker, C. I. Revealing the multidimensional\nmental representations of natural objects underlying human similarity judgements. Nat.\nHum. Behav. 4, 1173–1185 (2020).\nWould this be practical and for which kind of tasks?\nThank you for this question. Our method does not require any task-specific loss or manual\nannotation, making it highly practical. On the other hand, brain-in-the-loop supervision can\nbe used to train a foundation model that provides transferable, high-quality representations,\nmaking it particularly effective for tasks that depend on robust representational structures\nsuch as few-shot classification, object recognition, image retrieval, neural decoding, and\ncontent generation.\nTo further validate the effectiveness of our approach, we have conducted a new adversarial\nexperiment to evaluate the robustness of brain-in-the-loop supervised models, as shown\nbelow\ng. Evaluation of adversarial robustness of models through projected gradient descent\n(PGD) method (L , ϵ = 1/1020, steps = 10). Brain-in-the-loop supervision with CLIP-\n∞\nbase significantly improved adversarial accuracy for abstract concepts, resulting in\nperformance gains of 18.13 ± 3.69% and 3.80 ± 3.58% (mean ± s.d., averaged across\n100 independent runs; P < 0.0001, two-tailed paired t-test, n = 128 adversarial\nexamples per run).\nIn Fig. 4g, we quantitively evaluated adversarial robustness of model representations\nwithout and with brain-in-the-loop supervision using projected gradient descent (PGD)\nmethod. We found that brain-in-the-loop supervision resulted in increased adversarial\naccuracy on abstract concepts, providing additional evidence that our method yields\nstable and robust representational structures.\nIn addition, a lot of technical jargon is used concerning the computational methods without\na comprehensible explanation (e.g. RDM, SC, GW distance, CSM, concept arithmetic,\nconceptual manifold, structural similarity metric, Frobenius inner product, Sinkhorn\nalgorithm, Gumbel softmax trick, shared multiplex attention GNN).\nThank you for pointing out the parts that may not have been sufficiently clear. Many of the\nterms mentioned are already defined or briefly explained in the Methods section and/or the\nSupplementary text. To improve clarity, we have carefully re-checked the manuscript to\nensure that each technical term is properly introduced. In addition, we have revised several\ndescriptions to make the explanations more accessible to a broader readership. Many\nthanks.\nReviewer #3 (Remarks to the Author):\nSummary\nRecent advances in deep neural networks (DNNs), especially large models (LMs) have\nskyrocketed their image and text understanding as well as generation and classification\nperformance. However, imitating the in depth interpretation and abstract reasoning\ncapacities of humans remains a major challenge. This study challenges the ability of large\nneural network model architectures to capture and learn generalised concepts. Since\nDNNs are usually trained to perform well on a clearly defined task, such as classification,\nthey tend to lack the human-like abstraction capacity. To this end, a brain-in-the-loop\nsupervision framework was introduced, which aligns the DNN embeddings with human\nbrain activation structure. Specifically, fMRI brain signals were compared to the DNN\nimage embeddings via an encoder and graph neural network architecture. The\nimprovement of generalization capabilities was\ndemonstrated in a series of tailored experiments providing evidence and supporting the claim that the semantic coherence of\nDNNs can be enhanced by incorporating human conceptual structure into their training.\nImproved performance also involved one-shot learning and out-of-distribution recognition.\nValidity and Significance: This paper introduces a novel method of leveraging human brain-\nbased representations to improve AI systems toward human-level understanding. [...]\nNotably, the framework paves the way for integrating human-like abstraction capabilities\ninto AI by means of its structural alignment with biological neural activity. Overall, the paper\nis very well written, with a lot of numerical results, providing a consistent picture about what\nhas been gained by including sparse brain-in-the-loop information into AI models.\nMethodology and Data: [...] Altogether, the initial assumption that incorporating brain-in-\nthe-loop optimization into large DNN image classifiers promotes the model’s abstract\nconceptual understanding is more than well supported by the experimental evidence.\nWe are gratified that the reviewer expresses that this study provides novel methods and\npaves the way for integrating human-like abstraction capabilities into AI. We thank the\nreviewer for the thorough reading of the manuscript and the insightful and helpful questions\nand suggestions.\nThe modified manuscript makes a lot of progress due to your valuable suggestions. The\ncomments are response point by point below, and the revised text is highlighted in the new\nmanuscript.\nQuestions\nRegarding the clarity, a few questions seem to remain open after reading. During training\nof the encoders and GNN 206,848 object images but only 6,000 fMRI signals were used\n(see Methods line 492 and 501 and Supplementary Text). Even though the batch size is\nkept constant for both branches of the framework (fMRI and LM), the graph neural network\neventually receives an imbalanced distribution of training data to be matched. Does this\ndiscrepancy affect the brain-in-the-loop learning process and if so how?\nThank you for this insightful question. Even though the number of object images used for\ntraining is much larger than the number of available fMRI signals, this discrepancy does\nnot adversely affect brain-in-the-loop learning. First, both the images and the fMRI signals\nare drawn from the same set of concept categories. Second, our optimization objective is\nformulated to align the high-level visual semantic structure with the fMRI semantic structure.\nTogether, these factors ensure that no semantic distribution mismatch arises between the\ntwo modalities.\nIn addition, this setting (the larger number of object images) can be regarded as a form of\ndata augmentation, which encourages the model to be less sensitive to low-level visual\nfeatures and to focus instead on the semantic structure shared with fMRI.\nLine 130 of main text mentions that during brain-in-the-loop training the image encoder,\nfMRI encoder and graph neural network are optimized, does this imply that the vision model\n(large neural network in Fig.2 a) has its parameters frozen?\nThank you for the valuable questions. Yes, the vision model (depicted in Fig. 2a) is frozen\nduring brain-in-the-loop training. Because it has been pretrained on large-scale datasets\nand provides generalizable visual representations.\nThe batch size of 256 matches the embedding size (dimension) of the image encoder (see\nSupplementary Text / Implementation details of brain-in-the-loop supervised learning).\nHowever, the data points can in principle still be linearly dependent (i.e. not spanning\nthe whole space). In the main article the discrete probability measures of the sample points\nare assumed to be n/m-simplices (line 543), which is in general not guaranteed for random\nin-distribution batch.\nThank you for the valuable question. Yes, the sampled data points may not always span\nthe entire space, and thus the discrete measures do not strictly form n/m-simplices. In OT-\nbased learning framework, the discrete empirical distributions are estimated from finite\nrandom batches, which may contain linearly dependent samples. Nevertheless, this does\nnot affect the computation of the discrete OT, as the solution of OT (e.g., via the Sinkhorn\nalgorithm) only requires a valid probability measure over the sampled points—that is, an\nempirical distribution defined on the probability simplex. Many thanks.\nDuring brain-in-the-loop optimization of the DNNs, only the image encoder (MLP), fMRI\nencoder (ViT) and the shared GNN were trained. How were these frameworks components\nlater integrated into the models in order to perform the classification experiments? For\nexample in Supplementary / One-shot classification, a linear classifier was trained, but it is\nnot clear where it connects to the brain-in-the-loop architecture… could you briefly specify\nhow the architecture components were incorporated in the experimental phase (e.g. one-\nshot classification)?\nThank you, we should have been clearer. After brain-in-the-loop training, we used the\nfrozen vision model together with the trained image encoder to extract visual embeddings.\nThen, a linear classifier was trained directly on these visual embeddings. We have clarified\nthis point in the Supplementary by adding the following sentence:\nSpecifically, the inputs (images) were passed through the vision model (CLIP) and the\nfMRI-guided image encoder (MLP). Then, the resulting representations served as the\nfeatures for the linear classifier.\nZero-shot learning is mentioned in the abstract, introduction and discussion parts, but it is\nmissing in the methods and experimental results.\nThank you for pointing out the unclear points, which we should have made clearer. Our\nbrain-to-image experiment (Fig. 5 e, f) and semantic consistency analysis (Fig. 6a-c) can\nbe regarded as zero-shot tasks, since their evaluations are performed in novel categories\nwithout explicit training. We modified the main text to make it clear by adding the following\nsentence:\nThis task can also be considered a zero-shot test of semantic coherence, since the\nevaluations are performed in novel categories without explicit training.\nSuggestions\nUnify the font for scalars, vectors and matrices. E.g. in line 638, z is a vector, but in line\ni\n636 a is also a vector. In line 631, V should not be bold.\nThank you for pointing out the inconsistency of notations. We have carefully revised the\nmanuscript to ensure consistent notation for scalars, vectors, and matrices throughout the\ntext. Specifically, scalars are now denoted by lowercase letters, vectors by bold lowercase\nletters, and matrices by bold uppercase letters. The inconsistencies mentioned here (i.e.,\nlines 638, 636, and 631) have been corrected accordingly.\nIn statistical analysis, would it be possible to present numerical results in a more\ntransparent form such as a table (rather than one huge paragraph)?\nThank you for this suggestion. We have revised the presentation of the statistical analysis\nby restructuring the results into tables (i.e., Tab. 1), which makes the numerical results\nclearer than the original long paragraph. Many thanks.\nMaybe a question also to the editor: is it necessary to report all P-values in a complete\ndetail? (Perhaps simplification to, e.g. P<10-4 (for all smaller values), would be sufficient.\nThank you for the helpful suggestion. According to the Nature Communications author\nguidelines, statistical information should be reported with exact P-values. In accordance\nwith these requirements, we provided the specific P-values in the manuscript. Many thanks.\nWe suggest adding the reference: Zhu et al (2000) Dark, Beyond Deep: A Paradigm Shift\nto Cognitive AI with Humanlike Common Sense. Engineering, 6, 310-345, https://doi.org/\n10.1016/j.eng.2020.01.011. It introduces a wide approach toward human-like AI, largely\ndifferent from the brain-in-the-loop framework presented here.\nThank you for highlighting the classic work by Zhu (2000). We have included a citation to\nZhu (2000) in the revised manuscript. Many thanks.\nMinor details\nL331: Is this sentence grammatical? “Our results support that human-like cognition…”\n(‘support’ is a transitive verb), “... suggest/indicate that … ”, “... support the\nidea/claim/assumption that…”, “... demonstrate that…”\nThank you for pointing out the inaccurate sentence. We have corrected this sentence in\nthe new manuscript. Many thanks.\nUse ‘–’ instead of ‘-’ for mathematical minus symbol.\nThank you for this suggestion. We have carefully revised the manuscript and replaced all\noccurrences of ‘-’ with the proper minus symbol ‘–’ in mathematical expressions.\nReferences contain a few typing errors (e.g. missing commas, small first letters in\nconference name, …).\nThank you for pointing out the typing errors in the references part. We have thoroughly\nchecked the reference list and corrected the typing errors.\nLimitations - or points for the general discussion\nOne could argue that utilizing fMRI data within a hybrid system is a constraint in itself in\nthe sense that this data is not easily accessible for an average stakeholder or developer.\nProbably in the future this line research will be enabled by availability of numerous public\ndatasets usable for brain-in-the-loop supervision framework (or similar approaches).\nThank you for the thoughtful comment. We agree that fMRI data is not easily accessible to\nthe broader research community, which indeed poses a practical limitation for the brain-in-\nthe-loop framework. Nevertheless, with the growing availability of large-scale and publicly\naccessible neuroimaging datasets, such frameworks are expected to become increasingly\nfeasible and scalable in the future. We have added a discussion in the Conclusion section\nto explicitly acknowledge this potential limitation and to highlight future opportunities for\nbrain-in-the-loop supervised learning. The new discussion reads as follows:\nMoreover, the reliance on brain signals may currently limit the practicability of our\napproach. The increasing availability of large-scale public neuroimaging datasets is\nexpected to make brain-in-the-loop supervision (or similar approaches) more\naccessible in the future.\nThe paper talks about understanding images (and text) based on the excellent\nperformance (including generalization) of trained DNNs on various tasks. However, all\nDNNs in general are subject to adversarial vulnerability, which implies the lack of\nrobustness, see, e.g. Ren et al (2020), https://doi.org/10.1016/j.eng.2019.12.012 and the\nissue regarding ViT, e.g. Favole and Rawat (2025), https://doi.org/10.1145/3729167. This\nimplies that the term understanding should be used with care, since the lack of robustness\nis not compatible with true understanding of concepts. On the other hand, improving the\nDNNs using human conceptual knowledge, as shown in this paper, may contribute to\nincreased robustness in DNNs which can be experimentally tested.\nThank you for your valuable suggestion. We have conducted additional experiments to\nevaluate the robustness of the brain-in-the-loop supervision by using adversarial attacks,\nas shown below.\ng. Evaluation of adversarial robustness of models through projected gradient descent\n(PGD) method (L , ϵ = 1/1020, steps = 10). Brain-in-the-loop supervision with CLIP-\n∞\nbase significantly improved adversarial accuracy for abstract concepts, resulting in\nperformance gains of 18.13 ± 3.69% and 3.80 ± 3.58% (mean ± s.d., averaged across\n100 independent runs; P < 0.0001, two-tailed paired t-test, n = 128 adversarial\nexamples per run).\nIn Fig. 4g, we quantitively evaluated adversarial robustness of model representations\nwithout and with brain-in-the-loop supervision using projected gradient descent (PGD)\nmethod. We found that brain-in-the-loop supervision resulted in increased adversarial\naccuracy on abstract concepts.\nWe have also added relevant descriptions to clarify the correlation between robustness\nand understanding, which reads:\nMany current DNN models can be easily deceived by targeted pixel-level perturbations\n(adversarial attacks), whereas humans are remarkably robust against such\nperturbations. The lack of robustness highlights that DNNs fall short of human-like\nunderstanding concepts. Therefore, we used projected gradient descent (PGD)\nattacks28 to evaluate the robustness of brain-in-the-loop supervised models (Fig. 4g,\nsee Supplementary for details). We observed that human-like conceptual hierarchy\nleads to increased adversarial robustness on abstract concepts, with adversarial\naccuracy increasing from 50.82% to 68.95% (3 abstract concepts; P = 4.01 × 10−71,\ntwo-tailed paired t-test) and 10.98% to 14.79% (8 abstract concepts; P = 5.94 × 10−18,\ntwo-tailed paired t-test).\n28. Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards Deep\nLearning Models Resistant to Adversarial Attacks. in International Conference on\nLearning Representations (2018).\nThe paper talks about abstract concepts, but it should be acknowledged that the approach\nonly considers a subclass of abstract concepts, namely those involving a superordinate\nlevel of categorization of concrete objects (as mentioned in Suppl Fig.S1). The world of\nabstract concepts is very heterogeneous, see, e.g. Borghi et al (2018), http://dx.doi.org/\n10.1098/rstb.2017.0121 so it would be interesting to see whether the brain-in-the-loop\napproach could be extended to abstract concepts that do not provide good visual\nexemplars (e.g. truth, democracy), yet images (situational contexts) exist where they are\nrelevant. The success would depend on the properties of brain patterns.\nThank you for this insightful comment. We treated superordinate-level categorizations of\nconcrete objects as abstract concepts, since this form provides a well-defined conceptual\nstructure (e.g., WordNet) and allows us to systematically examine our brain-in-the-loop\nsupervision frameworks.\nWe also fully agree that abstract concepts are highly heterogeneous. Exploring how our\napproach can be applied to other types of abstract concepts (e.g., truth) is an important\nand exciting direction for future research. To reflect this point, we have revised the\nConclusion section to highlight this future direction, which reads:\nIncorporating a broader variety of categories and introducing more heterogeneous and\nabstract concepts such as democracy and kindness41 would allow for a more\ncomprehensive alignment between the representational spaces of the human brain\nand LLMs.\n41. Borghi, A. M., Barca, L., Binkofski, F. & Tummolini, L. Varieties of abstract\nconcepts: development, use and representation in the brain. Philos. Trans. R. Soc. B\nBiol. Sci. 373, 20170121 (2018).\nReviewer #4 (Remarks to the Author):\nThis work sets out to investigate whether Brain-in-the-Loop supervision can elevate the\nperformance of cognitive generalization for Large Language Models. […] The details of the\narchitecture and the algorithms used for graph matching and distance (Gromov\nWasserstein) are thoroughly described in the Methods section of the paper. […]\nThe paper then reports on the major results that can be set out as:\n• Brain-in-the-loop supervision enables DNNs to learn human-like conceptual\nrepresentations - as the brain-in-the-loop supervision process evolves, the distance\nbetween brain and DNN representations decreased for both the training and unseen\nconcepts;\n• Enhancement of abstract concept understanding with brain-in-the-loop;\n• Emergence of human-like conceptual hierarchy - models trained with brain-in-the-\nloop supervision exhibited concept similarity matrices highly aligned with WordNet\nwhile the others were not;\n• Human-like conceptual hierarchy enhances performance in complex cognitive tasks\n– achieve significant performance on one-shot learning, generalized out-of-\ndistribution recognition, brain to image and image to image retrieval;\n• Cognitive-level consistency of the learned conceptual manifold.\nThese results are all clearly described although the reader is often referred to information\nin the supplementary information. The supplementary information is very clear. The\nexperimental setup and the datasets are quite complex. This is a very good piece of\nresearch that is well motivated and quite clearly communicated.\nWe are gratified that the reviewer expresses that this study is a very good piece of research\nand well-motivated and quite clearly communicated. We thank the reviewer for the\nthorough reading of the manuscript and the insightful and helpful questions and\nsuggestions.\nThe modified manuscript makes a lot of progress due to your valuable suggestions. The\ncomments are response point by point below, and the revised text is highlighted in the new\nmanuscript.\nOne concern that I have is how the fMRIs from the 3 subjects are used to give a single\nbrain model?\nThank you for the valuable questions. In this study, we did not merge fMRI from different\nsubjects into a single model. Instead, we adopted a per-subject-per-model paradigm, i.e.,\ntraining a separate DNN model for each individual subject. To avoid potential confusion,\nwe have clarified this point in the Methods part by adding the following sentence:\nNote that the brain-in-the-loop supervised learning adopts a per-subject-per-model\nparadigm (training a separate DNN model for each individual subject).\nAdditionally, the detailed experiments would sit more comfortably if the connections\nbetween the BOLD fMRI data set and the THINGS and COCO datasets were made clearer.\nThank you for pointing out our unclear presentation. We modified the Methods section to\nexplicitly describe the relationship between the datasets used in experiments, which now\nread:\nOverall, we leveraged four public datasets to evaluate the effectiveness of our\napproach. The visual stimuli used in the fMRI dataset (DIR) are drawn from ImageNet\nand are therefore independent of the THINGS and COCO datasets. This ensures that\nthe test samples in triplet odd-one-out tasks (Fig. 4e) and semantic reconstruction (Fig.\n6) are out-of-distribution (OOD). For brain-in-the-loop supervised models, we only\nemployed 150 categories in the DIR dataset for training. The THINGS dataset\nprovides 4.7 million human triplet odd-one-out responses across 1,854 object\nconcepts. This greatly surpasses the 200 object categories in the DIR dataset, making\nit highly suitable for testing whether brain-in-the-loop supervision enhances the\nalignment between models and humans (Fig. 4f). Unlike ImageNet, which only\nprovides simple category labels, the COCO dataset includes detailed image\ndescription, enabling the training of a text decoder to reconstruct semantic information\nfrom brain-in-the-loop supervised models.\nThis is a very good piece of research that is well motivated and quite clearly communicated.\nThe statistical results are well executed and in general the experimental results are clear,\nhowever there are many results, and the graphs and tables could be individually clearer.\nThank you for the valuable suggestions. According to the journal’s submission guidelines,\nthe number of figures in the main text cannot exceed six, and therefore we are unable to\nincrease the number of figures. Nevertheless, we have revised several figures and tables\nto improve readability. Specifically, we simplified the layouts and increased font sizes,\nwhich aim to make each result easier to interpret individually. Many thanks.\nThe theoretical development in the methods is sound however the way in which the fMRI\nmodels are linked to the other datasets used needs some additional clarification.\nThank you for pointing out our unclear presentation. The visual stimuli used in the fMRI\ndataset (DIR) are drawn from ImageNet and are therefore independent of the THINGS and\nCOCO datasets. The THINGS dataset provides 4.7 million human triplet odd-one-out\nresponses across 1,854 object concepts, far exceeding the 200 object categories in the\nDIR dataset. It was therefore used to test whether brain-in-the-loop supervision enhances\nthe alignment between models and human judgments (Fig. 4f). The COCO dataset\nincludes detailed image descriptions, enabling the training of a text decoder to reconstruct\nsemantic information from brain-in-the-loop supervised models (Fig. 6). We have revised\nthe Methods section to explicitly describe how the fMRI dataset relates to other datasets.\nMany thanks.\nPertinent to the overall claim of the paper is the extent to which the claim that the Brain-in-\nthe-Loop improves cognitive generalisation capability of LMs is whether this work with a\nrelatively small number of concept categories will extend and what further things would\nneed to be done to match the extent of LLMS? How extensible is the approach to achieve\nhuman brain aligned LMs?\nThank you for the valuable question. Although our model was trained on a relatively limited\nset of concept categories, our experiments demonstrate that brain-in-the-loop supervision\nenhances cognitive generalization of LMs to unseen categories. For example, we observed\nperformance gains in few-shot classification of abstract concepts, and more human-like\nrepresentational structures for unseen concepts. These results indicate brain-in-the-loop\nyields promising generalization capability beyond the training distribution, and these\nproperties may be transferable to broader DNN models (e.g., LLMs).\nOn the other hand, recent studies1, 2 have also reported that the representations of LLMs\nexhibit strong similarity to high-level visual representations in the human brain, which\nprovides a promising foundation for extending brain-in-the-loop supervision to LLMs.\nIncorporating a broader variety of concept categories would allow for more comprehensive\nalignment between human brain and LLMs’ representational spaces.\nIn the revised manuscript, we have added a new discussion of potential extensions of our\napproach, which reads:\nRecent studies4, 40 have shown that LLM representations exhibit strong similarity to\nhigh-level visual representations in the human brain, which may facilitate the\nalignment of conceptual structures, thereby providing a promising foundation for\nextending the brain-in-the-loop supervision to LLMs. Incorporating a broader variety\nof categories and introducing more heterogeneous and abstract concepts such as\ndemocracy and kindness41 would allow for a more comprehensive alignment between\nthe representational spaces of the human brain and LLMs.\n4. Mischler, G., Li, Y. A., Bickel, S., Mehta, A. D. & Mesgarani, N. Contextual feature\nextraction hierarchies converge in large language models and the brain. Nat. Mach.\nIntell. 6, 1467–1477 (2024).\n40. Doerig, A. et al. High-level visual representations in the human brain are aligned\nwith large language models. Nat. Mach. Intell. 7, 1220–1234 (2025).\n41. Borghi, A. M., Barca, L., Binkofski, F. & Tummolini, L. Varieties of abstract\nconcepts: development, use and representation in the brain. Philos. Trans. R. Soc. B\nBiol. Sci. 373, 20170121 (2018).\n1. Mischler, G., Li, Y. A., Bickel, S., Mehta, A. D. & Mesgarani, N. Contextual feature\nextraction hierarchies converge in large language models and the brain. Nat. Mach.\nIntell. 6, 1467–1477 (2024).\n2. Doerig, A. et al. High-level visual representations in the human brain are aligned with\nlarge language models. Nat. Mach. Intell. 7, 1220–1234 (2025).\nThere are some minor typographic errors that can be fixed by a thorough proof reading.\nWe thank the reviewer for pointing this out. We have carefully proofread the manuscript\nand corrected the typographic errors throughout the text. Many thanks."}, {"round": 2, "reviewer_comments": {"reviewer_2": "(Remarks to the Author)\nThe basic idea proposed in the paper is to combine methods of modern artificial intelligence, in particular DNNs, with fMRI\nrecordings from 'higher visual' areas of 3 subjects. The paper shows that these recordings can effectively be used as a kind\nof teacher signal to improve the performance of the tested DNNs in several respects. This is demonstrated by using a large\ndiversity of measures and methods. The most important improvement made in the revised manuscript is to provide short\nsuperficial and intuitive explanations of most of these methods, which makes the paper much easier to read and understand.\nHowever, given this diversity, I don't think it helps a lot to provide more detailed explanations including mathematical\nformulas of all these methods. Only very few people will benefit from these equations and they are probably better explained\nin the original papers introducing these methods. The situation would be different, if (perhaps in some cases) the existing\nmethods were significantly improved or modified for this paper, but this doesn't seem to be the case.\nI also doubt whether the extensive statistical tests are really necessary and always justified in order to substantiate the clear\nimprovements made by using the fMRI data. Some of these astronomically low significance probabilities are somewhat\nsuspicious. Are you sure that the 'independent runs' are independent in the statistical sense? What is the source of\nrandomness in these cases?\nAbove all, the paper definitely needs a better introduction which explains more clearly the goals of this research and puts it\nin the context of some existing scientific communities. My first review was much less detailed and more negative because I\napparently (and hopefully) misunderstood these goals. In the abstract you write 'achieving complex cognitive abilities —such\nas understanding abstract concepts, reasoning, and adapting to novel scenarios, which are intrinsic to human cognition —\nremains a major challenge'. This implies that the 'human in the loop' will help to improve these abilities. This reminds me of\nthe old and well-taken criticism that neural networks have problems with 'symbolic reasoning', logic, and other high level\ncomputational taks, e.g. by Fodor and Pylyshyn (1988) and the extensive efforts in this direction in the 'neurosymbolic' (nesy)\ncommunity. The 'adapting to novel scenarios' reminds me of the research in 'artificial general intelligence' (AGI). Do you\nreally think that your work relates to the problems tackled in these communities? If yes, I believe you have not yet provided\nevidence for that; if no, you should at least mention some of these papers and clarify your relation to their goals. My\nimpression now is that your goal is to make 'artificial intelligence' more human, i.e. not to surpass humans, but to come\ncloser to natural human performance. This goal is consistent with the choice of mostly pattern recognition like tasks (with\nsome 'higher' components, like a hierarchical structure) where the 'gold standard' is the human performance. In such tasks it\nis no great surprise that the provision of additional human signals will improve the network's performance.\nOn the other hand, the combination of fMRI with deep networks is a rather indirect way to a better understanding of human\nconceptualization in terms of brain signals. (There is also no such thing as the human brain, since human brains are almost\nas divers as humans, and there may be differnces between your 3 subjects and their brains, and their conceptualizations\nmay also depend on their cultural background.)\nAnother disclaimer nay be necessary towards the neuroscince community where non-invasive recording techniques (like\nMEG, EEG, oe fMRI) are used to understand the neural representations in higher visual areas, in particular also in infero-\ntemporal (IT) areas. Here similar methods (e.g. (dis)similarity matrices) are used and similar things are shown more directly;\nsee for example the work of N. Kriegeskorte and J.D. Haynes.\nAnd of course, a better neuroscientific or psychological understanding of human conceptualization could also be used\ndirectly to make the pattern recognition and related capabilities of DNNs more human, without the need of recording task\nspecific fMRI from several humans to be used as teacher signal for various visual, auditory and other tasks.\nIn summary, what I suggest is\n1) more citations and disclaimers in the introduction,\n2) less or even no formulas,\n3) less (and better documented) statistical tests.\nSome detailed remarks:\nThe term 'human in the loop' is a bit misleading because it suggest a direct feedback loop between the human and the DNN,\nwhich is not possible when the fMRI recordings are performed offline.\nThe term 'zero-shot learning' is misleading because it can be understood as learning from no examples, which is not really\nlearning.\nGiven its central importance, there should be a clearer detailed explanation of how the fMRI 'guides the model' to higher\nclassification performance.\n(Remarks on code availability)", "reviewer_3": "(Remarks to the Author)\nWe carefully checked the reviewed version of the paper, as well as authors' responses. We acknowledge that the authors'\nseriously considered all comments from the reviewers and correspondingly improved the manuscript.\nAt this point, we do not have further comments and agree with accepting the paper for publication.\n(Remarks on code availability)", "reviewer_4": "(Remarks to the Author)\nThe authors have thoroughly addressed the reviewer's concerns and the paper has been considerably improved. The paper\nreads well and makes a significant contribution to understanding LM DNN structures and how introducing human knowledge\norganization and conceptual structures can be used to extend the recognition and reasoning with more abstract concepts.\nThe results are dense in places but the presentation has been improved considerably.\nThe first heading in the Results section needs to be corrected by rephrasing\n\"Increasing parameter sizes of enhances DNN's understanding of concrete but not\nabstract concepts\"\nIrecommend acceptance of the manuscript.\n(Remarks on code availability)"}, "author_response": "Response to the Editors and Reviewers\nNote: reviewers’ comments appear in black text. Our replies appear in blue text, and the\nrevised manuscript text appears indented (with old text shown in black and new edits in\nred).\nOverview:\nWe thank the reviewers for their careful reading of the manuscript and their insightful and\nhelpful suggestions. Before our point-to-point responses, we first give a brief overview\nof the three major modifications that resulted in significant changes to the presentation\nof the work.\nFirst, we have rewritten the Introduction section to better clarify the goals of our research.\nSecond, we have revised the Methods section to retain only the essential descriptions\nnecessary to understand the proposed method, while moving more detailed explanations\nto the Supplementary Text.\nFinally, we have added additional discussions and explanations in accordance with the\nreviewers’ suggestions.\nWe believe these revisions help make the findings more general and stronger. Many\nthanks!\nPoint-by-point Responses\nReviewer #2 (Remarks to the Author):\nThe basic idea proposed in the paper is to combine methods of modern artificial intelligence,\nin particular DNNs, with fMRI recordings from 'higher visual' areas of 3 subjects. The paper\nshows that these recordings can effectively be used as a kind of teacher signal to improve\nthe performance of the tested DNNs in several respects. This is demonstrated by using a\nlarge diversity of measures and methods. The most important improvement made in the\nrevised manuscript is to provide short superficial and intuitive explanations of most of these\nmethods, which makes the paper much easier to read and understand.\nWe thank the reviewer for the thorough reading of the manuscript and the insightful and\nhelpful questions and suggestions. We have modified the manuscript following your\nsuggestions, and we believe that the revisions have further improved the rigor and clarity\nof the paper.\nThanks again!\nHowever, given this diversity, I don't think it helps a lot to provide more detailed\nexplanations including mathematical formulas of all these methods. Only very few people\nwill benefit from these equations and they are probably better explained in the original\npapers introducing these methods. The situation would be different, if (perhaps in some\ncases) the existing methods were significantly improved or modified for this paper, but this\ndoesn't seem to be the case.\nThank you. To improve the overall readability, we have retained only the essential\ndescriptions necessary to understand the proposed framework in the revised version, while\nmoving the more detailed explanations (including mathematical formulas) to the\nSupplementary Text. Thanks.\nI also doubt whether the extensive statistical tests are really necessary and always justified\nin order to substantiate the clear improvements made by using the fMRI data. Some of\nthese astronomically low significance probabilities are somewhat suspicious. Are you sure\nthat the 'independent runs' are independent in the statistical sense? What is the source of\nrandomness in these cases?\nThank you. Indeed, there are extensive statistical tests employed in the experiment.\nHowever, we think they are still necessary to ensure that the observed improvements are\nconsistent and robust. To allow verification, we have provided all the raw data used in our\nanalyses.\nFor the statistical tests, a run corresponds to an independent evaluation with different\ntraining samples and different test samples, with the sample categories randomly selected.\nAveraging performance over multiple random episodes is a standard practice (such as [1-\n2]) for evaluating learning models (e.g., neural networks), and this procedure is commonly\nreferred to as repeated random sub-sampling validation (also known as Monte-Carlo cross-\nvalidation).\nWe have added additional explanations of statistical tests in the Statistical analysis section.\nMany thanks.\n1. Lake, B. M. & Baroni, M. Human-like systematic generalization through a meta-learning\nneural network. Nature 623, 115–121 (2023).\n2. Schneider, S., Lee, J.H. & Mathis, M.W. Learnable latent embeddings for joint\nbehavioural and neural analysis. Nature 617, 360–368 (2023).\nAbove all, the paper definitely needs a better introduction which explains more clearly the\ngoals of this research and puts it in the context of some existing scientific communities. My\nfirst review was much less detailed and more negative because I apparently (and hopefully)\nmisunderstood these goals. In the abstract you write 'achieving complex cognitive abilities\n—such as understanding abstract concepts, reasoning, and adapting to novel scenarios,\nwhich are intrinsic to human cognition —remains a major challenge'. This implies that the\n'human in the loop' will help to improve these abilities. This reminds me of the old and well-\ntaken criticism that neural networks have problems with 'symbolic reasoning', logic, and\nother high level computational taks, e.g. by Fodor and Pylyshyn (1988) and the extensive\nefforts in this direction in the 'neurosymbolic' (nesy) community. The 'adapting to novel\nscenarios' reminds me of the research in 'artificial general intelligence' (AGI). Do you really\nthink that your work relates to the problems tackled in these communities? If yes, I believe\nyou have not yet provided evidence for that; if no, you should at least mention some of\nthese papers and clarify your relation to their goals. My impression now is that your goal is\nto make 'artificial intelligence' more human, i.e. not to surpass humans, but to come closer\nto natural human performance. This goal is consistent with the choice of mostly pattern\nrecognition like tasks (with some 'higher' components, like a hierarchical structure) where\nthe 'gold standard' is the human performance. In such tasks it is no great surprise that the\nprovision of additional human signals will improve the network's performance.\nMany thanks. Yes our goal is to make 'artificial intelligence' more human. Thanks for\npointing out the literature of Fodor and Pylyshyn (1988) and a possible better way to\npresent in the introduction.\nFollowing your suggestion, we have rewritten the Introduction section to better clarify the\ngoals of our research, which now reads:\nHumans live in a world full of ever-changing objects, which requires us to constantly\nlearn new concepts from a variety of sensory information and systematically structure\nour experiential knowledge of the physical world1,2. This ability helps us organize our\nknowledge and make sense of the world, allowing us to recognize new things and\nadapt to new situations. For years, artificial intelligence (AI) researchers have tried to\nbuild systems that understand the world like humans do3–7. Although Fodor and\nPylyshyn8 suggested that neural networks are not plausible cognitive models because\nthey lack the systematicity of mental representations, deep neural networks (DNNs),\nespecially with the help of large-scale datasets and large models (LMs)9–11, did\nachieved higher systematic level ability. The question of whether and how neural\nnetworks could achieve human-like cognitive ability has sparked a vigorous debate\nspanning several decades1.\nCurrent DNNs or LMs still struggle to match human-like cognition12. Scaling up data\nand model size has primarily improved performance in well-defined tasks like\nclassification, object detection, and content generation13–15. However, when it\ncomes to human-like cognitive abilities—particularly the understanding and\ngeneralization of novel concepts—current models still fall short1,12,16. Moreover,\nincreasing model size does not consistently enhance these capabilities, and in some\ncases, performance even plateaus or degrades12,17, highlighting a fundamental\nlimitation in achieving high-level cognition18. Recalling Fodor and Pylyshyn's opinion8,\nthe systematicity of mental representations might be an important but overlooked\ncomponent for neural networks to obtain human-like cognition.\nHere, we proposed to enhance the cognitive ability of DNNs by leveraging the\nstructural organization of the human brain as a supervisory signal. […]\n8. Fodor, J. A. & Pylyshyn, Z. W. Connectionism and cognitive architecture: A critical\nanalysis. Cognition 28, 3–71 (1988).\n18. Muttenthaler, L. et al. Aligning machine and human visual representations across\nabstraction levels. Nature 647, 349–355 (2025).\nWe believe the revised version presents the motivation and contributions more clearly.\nThanks again!\nOn the other hand, the combination of fMRI with deep networks is a rather indirect way to\na better understanding of human conceptualization in terms of brain signals. (There is also\nno such thing as the human brain, since human brains are almost as divers as humans, and\nthere may be differences between your 3 subjects and their brains, and their\nconceptualizations may also depend on their cultural background.)\nThank you for the insightful comment. A recently published paper [1] used human behavior\nas the guidance, which is a more direct signal. However, using human behavior also has\nlimitations such as the task can be specific. We think using brain signals is a more\ngeneralizable mental representation. But we admit that it might be better ways to construct\nhuman-based signals, which are worth lots of thinking in future studies.\nAbout the inter-individual variability (e.g., cultural background), it indeed poses a practical\nlimitation for our approach. Currently we are trying to learn individual-invariant mental\nrepresentations with a larger population, which might be a possible solution. As noted in\nthe Conclusion section of the manuscript, we have highlighted and discussed this potential\nlimitation.\nMany thanks.\n1． Muttenthaler, L. et al. Aligning machine and human visual representations across\nabstraction levels. Nature 647, 349–355 (2025).\nAnother disclaimer may be necessary towards the neuroscince community where non-\ninvasive recording techniques (like MEG, EEG, or fMRI) are used to understand the neural\nrepresentations in higher visual areas, in particular also in inferior-temporal (IT) areas.\nHere similar methods (e.g. (dis)similarity matrices) are used and similar things are shown\nmore directly; see for example the work of N. Kriegeskorte and J.D. Haynes. And of course,\na better neuroscientific or psychological understanding of human conceptualization could\nalso be used directly to make the pattern recognition and related capabilities of DNNs more\nhuman, without the need of recording task specific fMRI from several humans to be used\nas teacher signal for various visual, auditory and other tasks.\nThank you for this insightful suggestion. Non-invasive recording techniques have indeed\nbeen extensively used to study neural representations. As noted in the Conclusion section,\nwe also acknowledge that using task-specific fMRI as a teacher signal for deep networks\nis not the only way to reshape the model representation structure. In the revised manuscript,\nwe have added a disclaimer, which reads as follows:\nIt is worth noting that methods2,42 using measured neural activity patterns to reveal\nmental representations (e.g., representational similarity analysis42) have been impactful\nin neuroscience, but how human-like representational structures reshape the behavior\nof DNNs remains largely unexplored.\n2. Hebart, M. N., Zheng, C. Y., Pereira, F. & Baker, C. I. Revealing the multidimensional\nmental representations of natural objects underlying human similarity judgements. Nat.\nHum. Behav. 4, 1173–1185 (2020).\n42. Kriegeskorte, N. Representational similarity analysis – connecting the branches of\nsystems neuroscience. Front. Syst. Neurosci.\nSuggestions\nIn summary, what I suggest is 1) more citations and disclaimers in the introduction, 2) less\nor even no formulas, 3) less (and better documented) statistical tests.\nThank you. As suggested, we have revised: 1) We have rewritten the Introduction section\nto better clarify the goals of our research; 2) We have revised the Methods section to retain\nonly the essential descriptions necessary to understand the proposed framework, while\nmoving more detailed explanations to the Supplementary Text; 3) We have added\nadditional explanations of statistical tests in the Statistical analysis section.\nAll the revised text is highlighted in the new manuscript. We believe that the modified paper\nmakes a lot of progress. Many thanks.\nDetailed remarks\nThe term 'human in the loop' is a bit misleading because it suggest a direct feedback loop\nbetween the human and the DNN, which is not possible when the fMRI recordings are\nperformed offline.\nThank you for pointing out this issue. We have replaced the wording 'human-in-the-loop'\ninto 'mental representation-guided', which we think more accurately express our approach.\nThanks.\nThe term 'zero-shot learning' is misleading because it can be understood as learning from\nno examples, which is not really learning.\nThank you. In machine learning studies, ‘zero-shot learning’ is a widely used task, meaning\nthat performing a task (such as classification) on unseen samples, where the model is\ntrained only on known samples—not learning without any data. We have added\nexplanations about the concept, which read as follows:\nThis task can also be considered a zero-shot test of semantic coherence, since the\nevaluations are performed in novel categories without explicit training. Note that in this\nstudy, zero-shot (learning) means that performing a task on unseen samples, where\nthe model is trained only on known samples—not learning without any data.\nGiven its central importance, there should be a clearer detailed explanation of how the\nfMRI 'guides the model' to higher classification performance.\nThank you for the valuable suggestion. We discussed the possible reason why the mental\nrepresentation could 'guides the model' to higher classification performance. We think that\ntraditional DNNs obtain the classification ability via statistical learning from large data, such\nthat their abilities remain rooted in statistical correlations rather than conceptual abstraction,\nresulting in lacking of understanding of abstract concepts nor the generalization ability\namong concepts.\nHuman cognition benefits from lifelong, interactive learning—a process that integrates\ncontinuous sensory input, active exploration, and real-world engagement to refine\nconceptual relationships. These mechanisms, which are inherently difficult to replicate in\nDNN training paradigms, underscore the limitations of purely scale-driven approaches in\nachieving human-level cognitive flexibility.\nTherefore, with the mental representation structure as the guidance, DNNs could learn\nbetter class representations as well as well-defined concept structures, thus benefiting their\nclassification ability.\nWe hope the explanation could answer your question. Thanks!\nReviewer #3 (Remarks to the Author):\nWe carefully checked the reviewed version of the paper, as well as authors' responses.\nWe acknowledge that the authors' seriously considered all comments from the reviewers\nand correspondingly improved the manuscript. At this point, we do not have further\ncomments and agree with accepting the paper for publication.\nMany thanks for the helpful and constructive comments!\nReviewer #4 (Remarks to the Author):\nThe authors have thoroughly addressed the reviewer's concerns and the paper has been\nconsiderably improved. The paper reads well and makes a significant contribution to\nunderstanding LM DNN structures and how introducing human knowledge organization\nand conceptual structures can be used to extend the recognition and reasoning with more\nabstract concepts. The results are dense in places but the presentation has been improved\nconsiderably.\nThe first heading in the Results section needs to be corrected by rephrasing\n\"Increasing parameter sizes of enhances DNN's understanding of concrete but not\nabstract concepts\"\nI recommend acceptance of the manuscript.\nWe have rephrased the first heading in the Results section, which now reads:\n“Increasing the parameter size of DNNs enhances their understanding of concrete\nbut not abstract concepts”\nMany thanks for the helpful and constructive comments!"}, {"round": 3, "reviewer_comments": {"reviewer_2": "(Remarks to the Author)\nAlthough not many changes have been made, the manuscript now reads much better and the results are very interesting.\nWith respect to the still ongoing debate about the appropriateness of artificial neural networks for symbolic reasoning it is to\nme still unclear what conclusions to draw from this. In this sense the presented results are only a beginning, of course. From\na neuroscience perspective, it would be very interesting to learn more about which cortical areas help more or less with\nwhich tasks. I still feel a bit uneasy about the many statistics, but I don't have the time to look deeper into this matter and I\ndon't think they are necessary to show that the results are real. The effects are clearly strong.\n(Remarks on code availability)"}, "author_response": "Response to the Editors and Reviewers\nNote: reviewers’ comments appear in black text. Our replies appear in blue text, and the\nrevised manuscript text appears indented (with old text shown in black and new edits in\nred).\nPoint-by-point Responses\nReviewer #2 (Remarks to the Author):\nAlthough not many changes have been made, the manuscript now reads much better and\nthe results are very interesting. With respect to the still ongoing debate about the\nappropriateness of artificial neural networks for symbolic reasoning it is to me still unclear\nwhat conclusions to draw from this. In this sense the presented results are only a beginning,\nof course. From a neuroscience perspective, it would be very interesting to learn more\nabout which cortical areas help more or less with which tasks. I still feel a bit uneasy about\nthe many statistics, but I don't have the time to look deeper into this matter and I don't think\nthey are necessary to show that the results are real. The effects are clearly strong.\nMany thanks for the helpful and constructive comments!"}]

Decision letter