A Concept-Centric Approach to Multi-Modality Learning
Authors: Yuchong Geng
Ao Tang
Venue: Accepted by TMLR ()

Abstract: Humans possess a remarkable ability to acquire knowledge efficiently and apply it across diverse modalities through a coherent and shared understanding of the world. Inspired by this cognitive capability, we introduce a concept-centric multi-modality learning framework built around a modality-agnostic concept space that captures structured, abstract knowledge, alongside a set of modality-specific projection models that map raw inputs onto this shared space. The concept space is decoupled from any specific modality and serves as a repository of universally applicable knowledge. Once learned, the knowledge embedded in the concept space enables more efficient adaptation to new modalities, as projection models can align with existing conceptual representations rather than learning from scratch. This efficiency is empirically validated in our experiments, where the proposed framework exhibits faster convergence compared to baseline models. In addition, the framework’s modular design supports seamless integration of new modalities, since projection models are trained independently yet produce unified outputs within the shared concept space.

We evaluate the framework on two representative downstream tasks. While the focus is not on task-specific optimization, the framework attains competitive results with a smaller training footprint, no task-specific fine-tuning, and inference performed entirely within a shared space of learned concepts that offers interpretability. These findings point toward a promising direction for developing learning systems that operate in a manner more consistent with human cognitive processes.

Published in Transactions on Machine Learning Research (01/2026)
A Concept-Centric Approach to Multi-Modality Learning
Yuchong Geng
yg534 Gcornelleol
School of Electrical and Computer Engineering Cornell University
Ao Tang
atangScornell.co edu
School Electrical and Computer Engineering
Cornell University
Reviewed on OpenReview: https: // openreview. net/ formfo id= SWAAPP32c7
Abstract
Humans possess a remarkable ability to acquire knowledge efficiently and apply across diverse modalities through coherent and sharod understanding of the world. Inspired by this cognitive capability, we introduce concept-centric multi-modality learning framework built around modality- concept space that captures structured, abstract knowledge, alongside set of modality-specifie projection models that map raw inputs onto this shared space. The concept space is decoupled from any specific modality and serves as a repository of universally applicable knowledge. Once learned, the knowledge embedded in the concept space enables more efficient adaptation to new modalities, as projection models can align with existing conceptual representations rather than learning from scratch. This efficiency is empirically validated in our experiments, where the proposed framework exhibits faster convergence compared to bascline models. In addition, the framework's modular design supports seamless inlegration of new modalities, since projection models are trained independently yet produce unified outputs within the shared concept space.
We evaluate the framework two representative downstream tasks. While the focus is not on task-specific optimization, the framework attains comparable results with smaller training footprint, no task-specific fine-tuning, and inference performed entirely within shared space of learned concepts that offers interpretability. These findings point toward a promising direction for developing learning systems that operate in a manner more consistent with human cognitive processes.
1 Introduction
Humans are capable of acquiring knowledge at remarkable speed even from a young age, which stands in stark contrast to most learning frameworks that require substantial resources to achieve human-like intelligence on specific tasks. Morcover, human cognition grounded a shared and coherent understanding of the world that spans across different modalities. For instance, when learning new language, we do not build an entirely separate system of knowledge for it. Instead, existing understanding of the world, in other words, to our common sense. We believe concept-centric approach to multi-modality learning could be key to not only bridging the efficiency gap, but also enabling learning frameworks that are more structured, reusable, and aligned with how humans organize knowledge across modalitics.
intuitively connect new linguistic elements to
At the center of our framework modalities, resembling the common sense embedded in the human mind. Recent inspiring works on Concept Learning often focus linking concepts to specific neurons (Liu et al. 2023b and encoded embedding
concept space that carries universal knowledge applicable diverse
Code available http://///ttttte...e///cc///aaaa
learning

---

Published in Transactions on Machine Learning Research (01/2026)
Indicates
PG: (/2 0.55
red small cubber sphere
Divene midii-ddlly inpus
Fxip
POr bid 0.12
Datuser:
Reflested
Independent projections
Y'
Unitted knowledge space where informnce conducted
Knowledge Spoce
Learning Concept Space
Learning Projection Models
Downstream Applications
Overall structure the proposed concept-centrie multi-modality learning framework. A modality-agnostic concept space is trained to reflect the relations between the set of concepts y observed in training dataset D (left). Modality-speciffc projection models are trained to create projections Q for their inputs based on the inputs' associations with concepts (middle). The modular design of the framework offers great flexibility and adaptability to wide range of downstream tasks (right).
Figure 1:
vectors Kalibhatt al. 2023 Wang et al. 2023b) of model or injecting specific concepts neurons into a model's structure (Shoth & Kahou 2023 Koh et al.) 2020). Compared to these works, our proposed framework takes a systematic approach by organizing modality- abstract concepts in an interpretable knowledge space and establishing connections to different modalities by projecting modality-specific inputs onto the same space.
While it is common in multi-modality learning create shared representation space for multiple modalities [Radford al. 2021 Li et al. 2022 Ramesh et al. 2022) or utilize projections to align features from different modalities (Liu et al. 2023a
Hsiung et al. 2022 such approaches primarily aim to co-locate modality- specific features in a common embedding space. In contrast, our shared concept space serves as an explicit intermediate layer of abstract concepts that can be reused and composed across modalities, enabling more efficient learning and seamless incorporation of new modalities, as demonstrated in our experiments, We believe the proposed framework takes step toward matching key aspects of human learning, where we build cohesive concept-leved understanding and connect multiple modalities (e.g., vision and language) shared knowledge.
Specifically, outlined in Fig. D the proposed multi-modality learning framework features an abstract concept space and set of modality-specifi projection models. The modality-agnostic concept space, inspired by prior works on structured embedding spaces (Vilnis et al. 2018 Li et al. 2018). optimally reflects real- world relations between concepts via entailment probabilities (Fig. left). Probing this concept space can also be achieved through simple queries of concept pairs of interest, providing interpretability into the learned knowledge.
Complementing the concept space, modality-specific projection models process inputs from different modalities and map them into shared domain, which we refer to as the knowledge space (Fig. [ middle). This knowledge space hosts both the abstract knowledge encoded in the concept space and the specific information extracted from individual inputs. By decoupling the projection models from the concept space, the framework enables efficient and modular learning. Fach projection model only required to produce consistent outputs within the knowledge space, allowing flexibility in architocture and optimization for different modalities. Although the projection models operate independently, their outputs unified in the knowledge space, where they interact with each other and with the learned concept representations, resulting in structure that supports probabilistic reasoning and cross-modality interactions.
The proposed design, characterized by a shared concept space with universally applicable knowledge and flexible projection mechanisms, naturally facilitates the reuse of learned knowledge across diverse modalities and task domains. Such a design cuhances the generalizability of our framework and enables straightforward adaptation various downstream tasks, with all inference processes conducted within the knowledge space (Fig. right).
2

---

Published in Transactions on Machine Learning Research (01/2026)
Contribution. Our contributions are three-fold. First, we propose novel approach to multi-modality learning that centers around concept space embedded with universally applicable knowledge. To our knowledge, this idea of a concept-focused learning scheme has been underexplored in the field of multi- modality learning (Sec. 3). Second, we offer a clear motivation and justification for the proposed framework. Leveraging knowledge learned from the concept space, our framework demonsttates more efficient learning curves compared to traditional methods (Fig. 2 Additionally, although our projection models are trained independently, they still exhibit strong cross-modality aligument without joint training because they rely the same concept space that encodes shared abstract knowledge. (Sec. 4.2) This design also makes the framework naturally scalable for incorporating new and diverse modalities (Sec. 4.3). Third, we evaluate the performance of our framework on two downstream tasks. We show that the proposed framework, with a modest pretraining foolprint, achieves comparable performance to benchmarks out-of-the-box without fine-tuning, while conducting all inference within a shared knowledge space containing interpretable concept reprosentations (Sec. 4.4
2 Related Work
Multi-Modality Learning. Vision and language modalities remain at the forefront of multi-modality
learning research, with some works exploring alternative modalities like audio (Akbari et al., 2021 Shi et al, b and biomedical data (Masood ct al. 2025). Within the vision-language area, CLIP by Radford et al. [2021] h employs two modality-specific encoders to learn a joint representation through image-text matching. Subsequent work by Ramesh et. al.
(2022) introduces a text-lo-image generation framework, using a text encoder and an image decoder to generate high-quality images from textual descriptions. Transformer-basse
architectures (Vaswani et al.
2017] have been widely explored for cross-modality information exchange and
learning (Singh et al. 2022a Bao al. 2022 Kim et al. 2021a Wang el. al. 2023a Lu el al. 2023
Beyond combining and relating modalities, research has delved into diverse areas such as multi-modality
few-shot learning (Alayrac et al. 2022 Li ct al.
2021 and visual-textual pattern mining (He & Peng] 2020). Some studies propose generalized learning frameworks applicable across various modalities (Jaegle el al.
2021 Bacvski et al. 2022a b). While these frameworks showcase strong capabilities in tasks like text-to-image
generation and -isual-language few-shot learning, our work addresses a distinct and important issue: creating a universally applicable concept space with abstract knowledge reflecting real-world observations. [Baevski (022b) present. versatile representation learning framework, yet it isolates modalities, impeding cross-modality interactions. In contrast, our proposed method directly combines modalities by projecting modality-specifie inputs onlo unified concept space, eliminating the information barrier between them.
et al.
Concept Learning. Early approaches to Concept Learning utilized Boolean logic for defining concepts based on relationships with other concepts (Angluin 1988) and their associated attributes (Mitchell 1997)
Lake et. al. (2055) propose a Bayesian Program Learning framework that represents concepts as probabilistic programs. Nowadays, a prevalent method involves placing concepts within structured embedding space. Concept learning frameworks such as those proposed by Mao al.] (2019) and Li et al. (2020b) construct embedding spaces that align concept representations with corresponding visual feature vectors. Lee et al. (2024h propose framework that learns concept embeddings via. distillation from pre-trained vision-language models. Methods from |iinis et al.] (2018) and Mei et al. (2022) emphasize entailment relationships between concepts in learned embedding spaces, while the work from Sinha et al. (2024) captures hierarchical information.
In departure from structured concept embedding spaces, the Concept Bottleneck Model (CBM) (Koh al. 2020 has become popular framework that represents concepts as intermediate neural network outputs. CBM first predicts set of pre-defined concepts aligned with human annotations and then produces a classification decision based on those concept predictions. Extending the idca of CBM, Dominici et al. (2023) propose a framework that learns a multimodal concept space jointly across modalities, using CBM- style supervision to inject interpretability into the shared concept space. Liu et al. (2023b) propose a method for identifying small subset of model paramcters responsible for generating specific concepts in a diffusion model. Kong et al.] (2024) propose a thcoretical view of concept learning as an identification problem of a discrete latent. hierarchical model.
3

---

Published in Transactions on Machine Learning Research (01/2026)
While we acknowledge that some motivating works adopt similar strategy involving concept embedding space, our approach stands out for several reasons. The primary distinction lies in the organization of our concept space, which reflects real-world knowledge by providing meaningful numerical entailment probabilities that mirror relationships among actual concepts. Furthermore, no barrier in our concept space prevents concepts belonging to different groups, such as red in color and cube in shape, from interacting with each other. More importantly, instead of being fitted to a specific modality, our concept space is designed to be abstract and modality-agnostic, thus allowing interactions between inputs from different modalities.
3 Method
Our proposed multi-modality learning framework consists of a modality-agnostic concept embedding space that models underlying relationships between concepts via entailment probabilities and set of modality- specific projection models that extract representation from single-modality inpuls and project them onto the domain where the concept space resides, i.e, the knowledge space.
Learning abstract knowledge in the concept space ensures generality, which makes its domain a good landing place for extracted representations from different modalities. Decoupled from the concept space and each other, modality-specifie projection models can be tailored for adaptation to their unique inputs, while modality-spocific knowledge remains connected after the projection.
We describe the design of the concept space in Sec. 3.1 and projection models in Sec. 3.2
Further
implementation details can be found in Sec. 4.1
3.1 Learning Concept Space
Davis et al. 1993 describe a knowledge reprosentation as surrogate that both carries the thing existing in the real world and serves as medium for pragmatically efficient computation. Building upon their definition of knowledge representation, we adopt probabilistic box embedding space (Li et al. 2018) to organize the learned representations of abstract concepts. In this formulation, each concept is represented as an axis-aligned hyperrectangle (box) in a high-dimensional space, and geometric relations between boxes correspond directly lo semantic relations. Intuitively, the strength of the relationship between two concepts is reflected in the amount of overlap between their boxes: a larger intersection indicates stronger semantic inclusion entailment, while boxes with smaller overlap correspond to weaker or less related concepts.
Like mental ins of specific knowledge in our brains, where we can relate concepts to each other, abstract entities in this concept space should be capable of interacting with each other, allowing reasoning inferences. With probabilistic box embeddings, these interactions take geometric form: relations between concepts are expressed through the overlap structure of their boxes, which determines their entailment probabilities. In the proposed framework, we focus on entailment relations between concepts depicted by these entailment probabilities to allow interactions between concepts. Contrary to latent spaces or learned ML model parameters, probing into the learned knowledge of this concept space can be easily achieved by querying the entailment probabilities of concept pairs of interest. Furthermore, efficiency of learning and referencing this concept space, facilitated by its compact parameter size, which qualifies il as medium for progmalically efficient compulation.
experiments demonstrate the
Defining Concept Space. We first define a knowledge space C Rd as a d-dimensional embedding space. Let y be a set for mgnasity- concopts. Each concept y e y is represented in K. by a box embedding (the surrogale), defined by pair of vectors Gy = (Wmin.y: Wmax.y); (.miin..:.aax.).. where Wmin.ys Wmax.y E K. correspond to the minimum and maximum boundaries of the box in K. We use C = (Sy yy E e C K. to denote set of box embeddings for every concepts in y and we call C the concept space whose parameters are optimized to reflect real-world knowledge.
softplus(w") A smoothing function mina (w) softplus(GG baac G is introduced on each dimension i of K. so a joint probability between two disjoint concepts can still be obtained. G G Ginn terms are the global maximum and minimum values at the i dimension among all 2ys in C. More details of mast be found in
4

---

Published in Transactions on Machine Learning Research (01/2026)
Appendix A.1 The probability of single concept y is calculated as P(v) = P(2.) II Mion (Wmax.y wmin.y). The joint probability between two concepts and y/2 is calculated as
d II
P (22) P(2y, (g
m sot (minn(waax.3) Wmax, max maax(miinyy yi Wmin.
Embedding Knowledge. Let X, denote a sample space of an unspecified modality marked by *, where each sample can be associated by a subset of madasicy- concepts in V. A training dataset given D. = ((t ( where at X, and y; = {リı yo E y and yj describes x;) This set of concepts that describe z can include both attribute concepts, like fluff and blue, as well category concepts, like dog and sky.
Modality-agnostic abstract knowledge can be extracted from D. by examining entailment probabilities between concepts indicated by {y Specifically, the ground-truth probability of a single concept and and the entailment probability of a concept pair (2), are calculated by P(y) Eyey countlyy)
count(y)
Ply 122 = coint((yi as they appear in D.. count(1z)
To drive the concept space to reflect real-world relationships between concepts via entailment probabilities, the objective for pretraining C is naturally defined as minimizing the KL divergence between predicted probabilities obtained from C' and the true probabilities observed in D.. In addition to the true concepts in y, for each data point, we sample a set of negative concepts and append them to y:- This negative sampling is necessary because the supervision available in D. contains only positive co-occurring concept pairs; without explicit counterexamples, the model would have evidence that unrelated concepts should exhibit low zero entailment. Negative samples therefore provide the missing contrastive signal, ensuring that the concept space learns to represent both strong entailments (via overlapping boxes) and non-entailments (via small or negligible overlap), rat than collapsing into an overly permissive representation. The specific sampling protocol varies across datasets, and further details are provided in Sec. 4
For each sample and its set of concept labels (z 8:) E D., we compute the entailment probability Q(y1 y2) given by the concept space for every possible combination of concept pairs U2) and for sampled negative concept pairs in Vs, and we compare these values with the corresponding true entailment probabilities P(yi 12). We refer readers to Appendix A for more details about the concept space.
3.2 Learning Projection Models
Defining Projection Models. Decoupled from the abstract concept space, each mppality- projection model can be viewed as mapping function f. : X, -> KC that generates a box representation K. for each input from its modality-specifii sample space X, of an unspecified modality denoted by *, This projection onto K. allows interactions between specific objects from X, and abstract concepts in C. Specifically, given a modality-specific input x. X., its representation in K. can be obtained by fx(z;;0) = 22 where O c K. follows the same definition of Py C C. With this representation made available, the probability that an object is associated with a concept can be naturally described by an entailment probability of Pyy | x;)) = P(2y (2;).
Adapting to the Concept Space. Given the training set D. corresponding a modality marked by *, the projection produced for an input xf should entail not only single concept y but also all other concepts associated with of. In other words, the projection O2 for o should lie at the intersection of the set of concepts describing x7 Thus, the optimal projection for ot should maximize the entailment probability P(Ng) Ev. Øj x.).
In our experimental datasets, concepts fall into two types: attribute concepts and category concepts. A sample may possess multiple positive attributes but associated with exactly one mutually exclusive category. As a result, attribute concepts require independent binary classification, whereas category concepts require single multi-class decision. Naturally, a binary cross-entropy loss is applied to attribute concepts,
5

---

Published in Transactions on Machine Learning Research (01/2026)
and SoftMax cross-entropy loss is applied to category concepts. Further details about the datasets and the nature of these concept types are provided in Sec.
To drive projection models to produce the optimal projection described above, we use a combination of a binary cross- loss on attribute concepts yatrrr C):
(yttr = D
Σ
I(y E y)l log P(Yy [2.]
y
(1)
+ lly & y) P(Yy) [2.)]
(where w is a weight assigned to positive attribute concepts)
and multi-class cross-entropy loss with SoftMax category concepts yeat V:
exp P(P (2.))
(y, = - -log Ey' yoor yyor exp P(Pyy' (2,)
(2)
(where yeat y
Now, given specific modality denoled by A and its training dataset DA. The training objective for SA is formally described as minimizing:
CA(OA:
1
Σ
(y, (y, fa(2;0A)) + (y. 0A))
DAA
(I:
While the training objective and projection outputs remain consistent across different modalities, projection models can be customized to accommodate unique modality-spoeific inputs, such as images or sequences of texts, bringing flexibility and versatility to the proposed framework.
3.3 Adapting to Downstream Tasks
With an abstract concept space and decoupled projection models, our proposed learning framework naturally accommodates various downstream tasks involving single multiple modalities. Regardless of the specific downstream tasks, their inference process consists of two stages: creating projections and relating them to learned knowledge. This approach more closely resembles human learning than traditional black-box models. In our daily interactions with objects, we process external stimuli like vision by creating abstract mental entities for objects we see. We then comprehend these mental entitics using our understanding of the world, or, in other words, our concept space (Gärdenfors) 2014). Section 4 we use an Image- Text Matching lask involving multi-modality and Visual Question Answering task with single-modality-foor focusedd approach illustrate the functionality of the proposed framework.
4 Implementation and Experiments
We base evaluation three existing datasets: CLEVR (Johnson el al. 2017a),, COCO (Lin et al. 2014), and GQA (Hudson & Manning 2019)
where their concepts are formed from original and supplemental COCO and GQA contain both attribute and category concepts, while CLEVR provides only attribute concepts. For consistency across datasets, the shape attribute family in CLEVR is reinterpreted as category concepts when required.
annotations [Patterson & Hays 2016
In COCO and GQA, many annotated attributes appear extremely infrequently (sometimes only few dozen instances), which provides insufficient training signal. To ensure meaningful supervision, we focus on the most. frequent and well-represented concepts, retaining the top 35 attribute concepts for COCO and

---

Published in Transactions on Machine Learning Research (01/2026)
equally sized set of 35 attribute concepts for GQA, together with their associated categories. This selection yields 64 total concepts for COCO and 68 for GQA.
To evaluate the scalability and generalization capability of our framework, we also construct Unified dataset that aggregates all concepts and mopality-specific representations from CLEVR, COCO, and GQA. This dataset contains overlapping concepts across domains, enabling us to examine how well the framework transfers and reconciles abstract concepts when they appear in different visual or linguistic forms. More details on the datasets and preprocessing steps can be found in Appendix [ Our experiments follow the same train and validation splits as the original datasets: the framework is pretrained on the train sets and evaluated on the validation sets.
4.1 Pretraining
Concept Space. To ensure that each concept box always has a valid set of lower boundarics smaller than its upper boundaries, we use two vectors, (Wmin.y) WA,y) Sy. instead of (Wmin: Wmax) to represent box in our actual experiments, where wA K2o is restricted to non-negative values. A box's upper boundaries be obtained by Wmax Wmin + wa. We set the dimension of K to 50, based empirical experiments (see Appendix A.5 for an ablation study on concept space dimensionality). Initial parameters for C are sampled from two uniform distributions. As for the negative sampling method, in CLEVR, the only negative concept pairs come from combinations of concepts residing in the same-attribute families, such as (red, blue) in the color family. For COCO and GQA, negative samples are randomly selected from all concepts. The concept space is trained for just two epochs for each dataset with a batch size of 256 using an AdamW optimizer (Loshchilov & Hutter 2017) with learning rate of 10-3 The training of this concept space be completed quickly as there are only thousands of parameters for moderalely-sized concept space.
Projection Models. In adapting our framework to the datasets featuring vision and natural language modalitics, we incorporate a vision projection model fvision basod on a Vision Transformer encoder (Dosovitskiy el al. 2020) and a natural language projection model JNL based on BERT encoder (Devlin) et al. 2018). Both models utilize their encoders' outputs on [CLS] tokens to generate projection boxes K. The outputs e with dimension of 768 are divided into two equal chunks, Amin and ha, each with a dimension of 384. These chunks are then imput into two fully connected layers to produce wmin and Wa for their respective projection boxes. To ensure WA is always non-negative vector, an additional ReL.U layer is applied. The complete projection process for inputs from the vision modality is outlined in Algorithm
Algorithm Illustration a ViT-based projection model fvision which projects vision modality inputs the knowledge space K.
input modality-speeific input Rvision
Ensure: wxision E KCoo, @vision C
Evision <- (VT((ivision)
A vvision h Aivision
vision + split(evisiou)
w wiin
Linearyision (hmin)
wissonn
ReI.U[innear vision (h^))
output vislion (uxlalon, A vislon)
For each object i in the CLEVR dataset, its attribute prediction for specific attribute family (e.g.) color) is generated by गe = argmax P(S.)99)) For each object i in COCO and GQA, threshold is applied to P(Ω 22),y y E yattr to obtain attribute predictions, and category prediction is generated by geat = argmax P(Z,, (2).
We establish a bascline by replacing the concept space with a traditional Multilayer Perceptron (MLP) at the classification head of fvision. Additionally, we implement the vision-modality projection model using ResNet model (He et. al. 2015) the backbone to showcase the flexibility of the proposed framework.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Results summarized in Table 1 show that proposed framework achieves comparable performance to traditional models while leveraging a novel concept space with interpretable learned knowledge.
CLEVR
COCO
GQA
Unified
Backbone
Method
Acc
Acc
mAP
Acc
mAP
Acc
mAP
Baseline
0.997
0.894
0.532
0.725
0.202
0.897
0.447
ResNet
fvision
0.990
0.892
0.520
0.727
0.191
0.892
0.439
Baseline 0.999
0.960
0.594
0.842
0.335
0.949
0.512
VIT
fvision 0.999
0.957
0.590
0.844
0.352
0.949
0.502
Table 1: Comparison with baseline models on vision-modality classification. CLEVR uses Accuracy. COCO, GQA, and Unified evaluate category concepts using Accuracy and attribute concepts using mean Average Precision (mAP).
Apart from featuring a concept-centric learning scheme, the proposed framework can also learn modality- specific knowledge faster by referencing learned knowledge from the modality-agnostic concept space as indicated in Fig. 2 This more natural learning process of our framework bridges the efficiency gap ween traditional machine learning methods, which often demand extensive data, and human learning, which excels at adeptly and efficiently extracting modality-specifie representations and associating them with mental entities of ract knowledge. To fully evaluate the impact of this transparent, modality-agnostic concept space on the learning of modality-specific projection models, we conduct an ablation study on it in Sec. A.4
COCO
GOA
Unified
0.25
0.5
0.4
Validaion mAP
0.4
0.15
0.3
0.2
0.10
0.05
0.1
170
340
510
680
850
170
340
510
680
850
190
380
570
760
950
Training Step
Training Step
Trinnng Scep
Validalion Ascuracy
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0
170
340
$10
680
850
170
340
$10
680
850
0
190
380
570
760
950
Training
Training Step
Training
MLP Bascline
Figure 2: Learning curves of the proposed projection models and the baseline model. The plot displays the evaluation accuracy on category concepts and the evaluation mAP attribute concepts measured every 50 training steps. During the learning process, the proposed vision-modes to the baseline thanks learning process of our framework bridges the efficiency gap between traditional machine learning methods, which require a large amount of data, and human learning that excels at extracting modality-specifie representations and linking them to structured abstract knowledge.
projection model improves more quickly compared universal concept space that already has abstract knowledge embedded in This faster
Projection models for the natural-language modality achieve highly accurate performance (2 99%) thanks the clearly structured description sentences. Further implementation and training details of projection models can be found in Appendix

---

Published in Transactions on Machine Learning Research (01/2026)
Model
Init. Acc.
Best Acc.
Time to 95% Acc. (s)
CLIP
0.767
0.9701
42.7
FLAVA
0.499
0.9848
103.1
ViLT
0.747
0.9568
265.1
Ours
0.954
0.9568
(already >95% at step 1)
Table 2: Multimodal alignment comparison across vision-language models. We image-text matching accuracy for validation split the Unified dataset as a proxy for evaluating cross-modality alignment. Our framework achieves high alignment initialization (over 95% accuracy at step 1) due to the shared knowledge, and requires significantly less computation than CLIP, FLAVA, or ViLT to reach their best performance.
Valuation Accuracy
0.8
Ours
0.6
ViLT
CLIP
FLAVA
20
40
80
100
120
140
160
180
200
Training GPU time (s)
Figure 3: Comparison of training efficiency for mross-modality alignment. Validation accuracy for image-text matching task of the Unified dataset used as a proxy for evaluating cross-modality aligument. Our proposed method reaches near-saturated aligument without finetuning and attains its final accuracy within significantly shorter period, while ViLT, CLIP, and FLAVA require substantially more computation time to achieve comparable alignment.
4.2 Cross Modality Alignment
A key distinction between our framework and most existing multinodal models is that cach modality-specific projection model is trained independently, yet they remain inherently compatible and aligned because they share a common concept space that already encodes abstract knowledge. As result, the projections from different modalitics are well aligned without roquiring any extensive joint training typically needed in models such ViLT, CLIP, FLAVA. We assess alignment between the vision and language modalities from decoupled projection models by measuring the agreement between their knowledge space projections and by comparing the results with strong multimodal benchmark models. Because cross-modality alignment cannot measured directly, we use accuracy on an image-tess text matching task (Section [.4.1] from the Unified dataset with a large pool of 130 concepts as a proxy, and we initialize all comparison models with their pretrained weights ensure fairness.
Before any fine-tuning, measure each method's initial alignment. Table 2 shows that our independently trained projection models already achieve strong modal alignment at initialization, reaching over 95% accuracy without further training. We then apply joint training objective from Appendix B to finetune the two projection models. This joint training is used only to obtain a direct comparison of cross-modality alignment efficiency against existing multimodal models and not required for our framework to function. FigureBplote plots image-text matching accuracy every 50 steps against the GPU time elapsed during finetuning. Our franework not only starts with strong alignment but also converges significantly more quickly than existing multimodal models during joint training.

---

Published in Transactions on Machine Learning Research (01/2026)
Modality
Image
English 0.955
Chinese
Spanish
French
Image
0.968
0.961
0.935
English
0.955
0.974
0.982
0.876
Chinese
0.968
0.974
0.970
0.887
Spanish 0.961
0.982
0.970
0.878
French
0.935
0.876
0.887
0.878
Table Cruss-modality matching accuracy image, English, Chinese, Spanish, and French. Each modality- specific model is trained independently with no joint training.
4.3 Incorporating New Modality
Thanks the decoupled training scheme of each individual modality-specifi projection model and the inherent compatibility and alignment that result from sharing unified knowledge space, incorporating new modalities into our framework is easy and scalable. To add a new modality to functioning system with existing projection models, the new projection model only needs to be trained using the same projection- model training objective, and none of the existing modality-specific models need to be adjusted or jointly finctuned.
To demonstrate this scalable design, we translated the natural language descriptions originally used to train the BERT-based projection model into three new languages, including Chinese, French, and Spanish. Each translated language represents
modality with distinct input space added system that previously contained only vision and English (previously referred inputs are natural language, each language is treated as an independent modality because it is mapped into the shared knowledge space via its separately trained modalityyspecific projection model, rather than sharing parameters with the English text projection as in multilingual encoders. We follow the same training process as for the English projection model and train these new models independently while adapting them the shared knowledge space. We then directly measure cross-modality alignment using cross- moda-mmydl eppresentation-pall matching task made from the Unified dataset proxy. Results in Table show strong alignment across all modalities. To gain further insight into the agreement between different projection models, Figure reports the average entailment probabilities for all positive mods-ito representation pairs (two representations correspond to the same set of concepts) and negative ones (perturbed concepts) across all modality combinations.
the natural language modality). Although all
This ability to incorporate new modalitics without any joint finetuning with existing modalities highlights the flexibility and scalability of our framework. In addition, the spedlit projection models for these modalities demonstrate the same efficient training behavior shown in Figure
4.4
Downstream Tasks
Now, we focus on our proposed framework's adaptation to two downstream tasks: Image-Texx Matching involving cross-modality references and Visual Question Answering with a higlle-odalltyyfoeussee approach.
4.4.1 Image- Text Matching
Image-text matching is binary classification task whether natural language sentence describes image. Our framework can naturally adopt a common approach involving creating representations for sentences and images in a shared latent space. In contrast. to those works, however, our latent space is a knowledge-embedded concept space that supports efficient probing. Specifically, given an image-text pair vision Ir, NL, their representations in the learned concept space are generated by Exsion(2)ii (2visioo) = Qvision and fNL (2 DN.. The probability that (xvviionn ...T.) ) is a positive pair can be determined by the cross entailment probability follows:
10

---

Published in Transactions on Machine Learning Research (01/2026)
Positive Pairs
1
Negative Pairs
Chinese
Chinese
0.00
-2.12
-6.05
-3.43
-2,42
0.00
-7.19
-9.60
-6.15
-7.50
English
English
-2.12
0.00
-6.19
-2.18
-1.48
-7.19
0.00
-9.47
-5.31
-6.87
French
French
-6.05
-6.19
0.00
-7.41
-6.53
-9.60
-9.47
0.00
-8.99
-9.81
Image
Image
-3.43
-2.18
-7.41
0.00
-2.57
-6.15
-5.31
-8.99
0.00
-5.57
Spanish
Spanish
-2.42
-1.48 English
-6.53 French
-2.57
0.00
-7.50
-6.87
-9.81
-.55
0.00
Chinese
Image
Spanish
Chinese
English
French
Image
Spanish
-6
-2
Entailment Log Probability
Figure 4: Modality alignment measured by entailment probabilities between positive and negative cross-modality representation pairs across image, English, Chinese, Spanish, and French modalities. Left: higher values (1) indicate stronger projection overlap for positive concept pairs. Right: lower values (L) indicate stronger separation for negative pairs. All mopelity-ppecific models are trained independently.
P(matched (xyision (LL) 1 P(Oyision
12 P [yisioo))
This inference process is demonstrated in Fig. in Appendix.
In our experiments, we employ two methods to create negative image-text pairs: swapping whole description sentences and swapping attributes. Specifically, for the first method, replace 50% of images' description sentences using random sampling. For example, original description sentence of CLEVR object might be changed from 'There is a large, metal, red cube" to 'There is rubber, small, yellow sphere." On the other hand, swapping altributes involves changing only a subset of attributes that describe an object, creating a more challenging image-text matching task. For instance, the same description sentence would be changed to "There small, metal, red cube.'
To compare our framework's performance, we implement other benchmark multi-modalit models with applications in the Imago-Text Matching task. The results summarized in Table 4 In contrast to those models with traditional black-box architectures, our framework displays more efficient learning process and adopts a more transparent inference process without sacrificing its performance. Details of this experiment can be found in Appendix
CLEVR
COCO
GQA
Method
Fine-tuned?
sent.
attr.
sent.
attr.
sent.
attr.
BLIP
Li et a
0.999
0.999
0.992
0.536
0.979
0.576
CLIP
Radford et al.
x
0.997
0.997
0.974
0.587
0.945
0.532
FLAVA [iighh et a
0.998
0.998
0.992
0.505
0.980
0.536
ViLT Kim et al.)
0.994
0.994
0.985
0.515
0.965
0.555
Ours
0.995
0.995
0.970
0.550
0.924
0.531
Table 4: A comparison with state-of-the-art. multi-modality models on the Image-Text Matching Task. We test these models and our framework using two variants of matching task: swapping whole sentences (sents.) swapping attributes (attr.), Classification accuracy (%) is reported.
11

---

Published in Transactions on Machine Learning Research (01/2026)
4.4.2 Visual Question Answering
Visual Question Answering (VQA) evaluates an Al system's ability to reason about images by answering questions related those images in a natural language format.. For this task, we focus on the CLEVR dataset, whose questions are designed to include attribute identification, counting, comparison, spatial relations, and logical operations. Recently, several works (Johnson et al. 2017b Yi et al. 2018 Mao et al. 2019 Li et al., 2020a; Mei et al. 2022 have focused on neural-symbolic reasoning approach, using chains of symbolic programs to predict answers to these questions. Our framework's adaptation to VQA involves using a similar set of symbolic programs, but these programs operate on the knowledge space XC containing interpretable concepts in instead of the high-dimenniooall latent spaces used by previons works.
Problem Formulation. Given an imago-question pair {Xylsion, :9:} where Xylalon is an original CLEVR image as shown in Fig. and qi is a natural language question such as "Are there more cubes than yellow things?', an AI system needs to generate an answer o, in the natural language format such "Yes'.
Symbolic Programs. We design our symbolic programs as deterministic functions operating on K. Precisely, we follow the same program definitions as proposed by Johnson et al. (2017a).
Program Generator. An LSTM model т used to process questions into sequences of programs: & (q:). We follow the same pretraining procedure used in (Johnson et al.) 2017b) train this program generator. However, as there is no line-tuning stage in our adaptation, the parameters in are frozen once pretraining is finished.
Object Detection and Projection. Similar to our pretraining process, we use fdetection to obtain set of single-object images ziiision from Xyision which are then fed into fvision so their projections can be obtained. Additionally, each single object's coordinates predicted by fdetection are attached to its projection box so questions involving spatial relations can be inferred.
Inference Process. A correctly predicted program sequence &, starts with a Scene function that returns all objects in an image and ends with program that outputs the answer o,- Intermediate programs take the output from previous programs inputs, which is recurring process until the final function. Our concept space C is mainly involved in attribute identification, following the same procedure used when evaluating projection models in Sec. 4.1 Spatial information for each object is obtained from the bounding- box coordinates predicted by faetection and is passed to the corresponding symbolic functions that handle spatial reasoning. The complete inference process is also demonstrates in Fig. in Appendix.
Method
Accuracy
Finc-tuned?
SA+MLP (Johnson et al.) Dependency Tree [Cao et al.)
73.2
89.3
Human (Johnson et al.
92.6
N/A
RN Santoro et al.]
95.5
IEP
(Johnson et a..)
96.9
V
MDETR
(Kamath et al.)
99.7
NS-VQA
Yi et a
99.8
V
Ours
96.5
X
Table 5: A comparison between our framework's performance and state-of-the-art models.
Results. We perform no fine-tuning on the concept space C and vision-modality projection model fvision for the VQA task. A comparison to benchmark models summarized in Table shows our framework achieves performance levels on par with those fine-tuned benchmark models.
5 Discussion
A Cognition-Inspired Learning Paradigm. Most current multi-modee multi-modality learning frameworks, and even the broader landscape of machine learning systems, rely on learning paradigm that differs substantially
12

---

Published in Transactions on Machine Learning Research (01/2026)
from those observed in human cognition. When exposed to new knowledge, we instinetively form concept and associate it with the external stimuli tied to that information. This newly formed concept is then integrated into our existing body of knowledge and stored as persistent memory in the mind. In contrast, most machine learning frameworks encode knowledge into large sets of model paramcters that are difficult to interpret without specific model input. As a result, the activalion of learned knowledge such systems is often transient and dependent on specific inputs. This fundamental difference presents significant challenge in designing systems that can explicitly form, retain, and reason over interpretable concepts in manner analogous to human cognition.
The inclusion of structured concept space and the use of concept-grounded inference may initially appear restrictive, particularly when compared with conventional models that rely on dense, task-specific representations optimized end to end for performance. However, we view this design as a deliberate and principled choice. By introducing a concept space that reflects structured, abstract knowledge similar how humans form and retain concepts, the framework gains several benefits that are otherwise difficult achieve. These include more efficient learning, natural generalization across modalities, and interprelability through explicit probing. The concept space acts an inductive bias consistent with human cognition, enabling machine learning systems to operate in more principled and cognitively grounded manner. We believe this work highlights a compelling direction for rethinking learning systems to more closely mirror human intelligence.
Addressing Bias. Hidden biases learned from datasets often hinder the trustworthiness of ML systems (Amodei et al. 2016 Lederer 2023 Kaur et al., 2022; Knott et al.) 2023). For example, NLP models often tend to associate the word "monarch" more with the word "male" than "female," " reflected, for instance, in higher similarity scores between embeddings of "monarch" and "male." Our proposed framework facilitates effective probing into the model's learned knowledge and offers the capacity to rectify such biases.
Concopt
Concept
Concept Space
Ground Truth
Orange
Bus
0.043
0.043
Old
Building
0.032
0.048
Smiling
Person
0.074
0.073
White
Snow
0.910
0.974
Parked
Car
0,228
0.244
Cloudy
Sky
0.173
0.192
Table 6: Sample Entailment Relation Queries of Concepts in Learued GQA Concept Space
Table 0 shows probing of a learned concept space fitted to the GQA dataset action. Our framework enables easy querying of targeted concept pairs, which would be computationally expensive, if not infeasible, in traditional latent spaces. Further demonstrations of probing into the learned concept space can be found in Appendix A.3
Revisiting the earlier example of the concept pair "monarch" and gender, the bias can be addressed directly in our framework by adjusting the ground-truth entailment probabilities. Specifically, ensuring equal entailment probabilities between "monarch- "monarch-male" male" and "monarch " mitigates representational bias, a correction that can be easily applied through user-guided specification.
Scalability of the Concept Space. In our experiments, the concept space constructed reflect ground-truth entailment probabilities observed in training data. This approach can scale larger and more diverse sets of concepts. Prior work (Vilnis et al.) 2018 Li et al. 2018 Lai & Hockenmaier 2017 has shown that similar embedding structures can learn entailment relations for large ontologies such as WordNet (WordNet). Although scaling introduces challenges in generating ground-truth probabilities, textual corpora offer a promising resource for extracting such relations, as demonstrated by He & Peng (2020). To assess scalability, we fitted a concept space the full set of WordNet noun entries, totaling 10,765 concepts. The resulting space achieved a KL divergence of 0.1308 with respect to the ground truth, compared to 0.1172 for the GQA concept space.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Call for Concept-Focused Datasets. A major bottleneck in concept-centric learning is the lack of high- quality datasets with accurate concept annotations. In our experience, even after preprocessing, concept and attribute labels in datasets such as COCO and GQA contain substantial noise. This limits not only the performance of our framework but also that of other systems. Recent works (Pham et al., 2021 Saini et al., 2022 Bravo el. al. 2023 Vedantam et al. 2020) have begun addressing this gap by improving the collection and annotation of aitribute-focusser datasets. We believe that further efforts to build datasets with richer and more reliable concept annotations will greatly support the development of interpretable and trustworthy AI systems.
Integrating Concept Spaces with Vision-Language Models. Compared to conventional vision- language foundation models, our cognitively inspired framework maintains an explicit and interpretable concept space for retaining and reasoning with knowledge, but its reasoning capabilities are currently limited to more controlled and simpler task domains. In contrast, modern VLMs, while operating largely as black boxes, offer strong flexibility and broad gencralization across diverse downstream tasks through largo-scale pretraining. A promising direction for future work is to explore hybrid architectures that combine these strengths. A concept space, which could take a form similar to that adopted here or extend beyond capturing only entailment relations, could be incorporated into transformer-basee VLM as a set of Iearned model parameters. Attention mechanisms can then be used to associate model inputs with the knowledge embedded in the concept space. These concept, parameters could be kept frozen, allowing the model to use them as a stable semantic memory, or updated during downstream training, enabling the model to refine or even discover new concepts through interaction with task data.
Scope and Limitations. In the current framework, the concept space supports only two types of concepts: attributes and categories. Future work should explore methods to expand this space to broader range of concepts, such as abstract or relational concepts for more complex reasoning, and action-oriented concepts to accommodate modalities beyond vision and language, such as robot planning.
6 Conclusion
Human cognition exhibits a remarkable ability to form a coherent and structured understanding of the world, and to apply this knowledge efficiently across diverse tasks and modalities. Inspired by this capability, we propose concept-centric multi-modality learning framework centered around modality-agnostic concept space that captures universally applicable knowledge.
The primary technical contribution lies in the design of a modular framework that integrates a shared concept space with a flexible set of modality-specific projection models. This design enables knowledge reuse across modalities and task domains, supporting learning that is interpretable, generalizable, and modular. Unlike traditional end-to-end learning systems that encode knowledge implicitly within dense parameter spaces, the proposed framework embeds knowledge explicitly into a structured concept embedding space, enabling interpretability through efficient probing of concept entailment probabilities.
Empirically, the experiments demonstrate that the proposed framework supports more efficient learning. In the vision modality, the projection model converges significantly faster than a baseline model built on a traditional architecture. This gain in efficiency is attributed to the fact that the concept space already encodes structured, abstract knowledge that the projection model can adapt to. Additionally, we evaluated the framework on two downstream tasks, Imago-Text Matching and Visual Question Answering, and demonstrated that our method achieves performance comparable to state-of-the-art methods even without tasc-fpecific fine-tuning. While our goal is not to surpass existing benchmarks in raw performance, these results support the viability of a cognitively inspired learning paradigm. Rather than optimizing solely for accuracy, our framework emphasizes learning efficiency, interpretability, and structural alignment with human cognition. These qualities are increasingly important as machine learning systems are deployed in more complex and dynamic environments.
More broadly, the work motivates a rethinking of how machine learning systems acquire and represent knowledge. By introducing concept space as an inductive bias, this framework opens a promising research
14

---

Published in Transactions on Machine Learning Research (01/2026)
direction toward building systems that more naturally align with human reasoning. Such systems may offer greater transparency, flexibility, and the ability to generalize knowledge across tasks and modalities.
Looking forward, several directions can further extend the proposed framework. First, scaling the concept space to support larger vocabularies and richer relational structures beyond entailment, inchuding compositional and causal relations, would expand its expressive power. Second, applying the framework to new task domains such as concept-grounded Text-to-Image generation presents a natural extension. Beyond these extensions, the results point toward meta-learning approaches that enable learning systems to discover, update, and organize concepts directly from multimodal imputs, including vision and language. Rather than treating concept vocabularies as fixed and externally provided, future systems should be able to infer and refine conceptual structures from raw data as models interact with new modalities and tasks. Advancing adaptive concept learning mechanisms for concept discovery, concept organization, and concept application is an important step toward more autonomous, scalable, and cognitively grounded machine learning systems. Finally, explicit concept-level representations may serve as a semantic interface for semantic communication, in which agents exchange meaning the level of concepts and relations rather than raw signals low-level features.
Acknowledgements
This work was supported in part by the National Science Foundation under Grant No. 2133403, and by computing credits provided by Amazon Web Services and Microsoft Azure. We thank the reviewers and the action editor for their thoughtful feedback and constructive suggestions.
References
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shil-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for Multimodal Self- Superviuee Learning from Raw Video, Audio and Text. Advances in Neural Information Processing Systems, 34:24206 24221, 2021.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. arXio preprint arXiv:1606 06566, 2016.
Dana Angluin. Queries and concept learning. Machine learning, 2:319-344 342, 1988.
Alexci Bacyski, Arun Babu, Wei-Ning Hsu, and Michael Auli Efficient self-supervised learning with contextualized (arget representations for vision, speech and language. ArXiv, abs/ abs/2212.00725. 2022a.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language. In International Conference Machine Learning, 2022b.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. VLMo: Unified Vision-Language Pre- Training with Mixture-of-Modee Experts. Advances in Newral Information Processing Systems, 35:32897-32912, 2022.
María A. Bravo, Sudhanshu Mittal, Simon Ging, and Thomas Brox. Open-vocabulary attribute detection, 2023. URL
12914
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin. Visual question reasoning on general dependency tree. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition, pp. 7249-7777 1259, 2018.
Randall Davis, Howard Shrobe, and Peter Szolovits. What knowledge representation? AI mogazine, 14 (1):17-17, 1993.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl urXiv:1810.04: 04805, 2018.
Gabriele Dominici, Pietro Barbiero, Lucie Charlotte Magister, Pietro Liò, and Nikola Simidjievski. Shares:
Shared concept space for explainable multimodal learning, 2023. URL https://arxiv httpaaawww...irar//aaaaaaaaaaaea org/abs/2307,
00316
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999, 2020.
Peter Gärdenfors. The Geometry of Meaning: Semantics Based on Conceptual Spaces The Geometry of Meaning: Semantics Based on Conceptual Spaces. The MIT Press, 2014.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abss1512.00335,, 2015. URL http://aaxxiv..orraab/////2222 03385
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE international conference on computer vision, pp. 2961 2961-2969, 2017.
Xiangteng He and Yuxin Peng. Fine-grained visual-textual representation learning. IEEE Transactions on Circuits and Systems for Video Technology, 30:520-531, 531100533,110 2020.
Eric Hsiung, Hiloni Mehta, Junchi Chu, Xinyu Liu, Roma Patel, Stefanie Tellex, and George Konidaris. Generalizing to new domains by mapping natural language to lifted Itl. In 2022 International Conference on Robotics and Automation (ICRA), pp. 3624-3630, 2022. doi:
Drew A. Hudson and Christopher D. Manning. GQA: a new dataset for compositional question answering over world images. CoRR, abs/ htee/1902099500 2019. URL http:///axxiv..ggabbb//////s 09506,
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General Perception with Iterative Attention. In International Conference on Machine Learning, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C LLwwrence Zitnick, and Ross Girshick. Clevr: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Inferring and exccuting programs for visual reasoning. In Proceedings of the IEEE international conference on compuler vision, pp. 2989 2998, 2017b.
Neha Kalibhat, Shweta Bhardwaj, Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. In Proceedings of the 40th International Conference on Machine Learning, 1CML'23. JMLR.org, 2023.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. MDETR- modulated detection for end-to-end multi-modal understanding, CoRR, abs/2104.12763, 2021.
URL https://arxiv.ortt abss//bb/211 12763
Davinder Kaur, Suleyman Uslu, Kaley Rittichier, and Arjan Durresi. Trustworthy Artificial Intelligence: a Review. ACM Computing Surveys (CSUR), 55(2):1-38, 2022.
Wonjae Kim, Bokyung Son, and Tddoo Kim. ViLT: Vision-and-Tangueee Transformer Without Convolution or Region Supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021a.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languuage transformer without convolution
region supervision. In International Conference on Machine Learning. 2021b. URL hattps://apt
semanticscholar.oom htt.s//c....rr..oo///oooaaa/aaaa
16

---

Published in Transactions on Machine Learning Research (01/2026)
Alistair Knott, Dino Pedreschi, Raja Chatila, Tapabrata Chakraborti, Susan Leavy, Ricardo Bacza- Yates, David Eyers, Andrew Trotman, Paul D. Teal, Przemyslaw Biecek, Stuart Russell, and Yoshua Bengio. Generative ai models should include detection mechanisms as a condition for public release. Ethics and
Inf. Technol., 24), October 2023. ISSN 1388-1957. doi: 111107//106676-022-8777-585 URL https:
//doi.org/// 10. http://0..0000010000000--0000000
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models, 2020.
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric P. Xing, Yuejie Chi, and Kun Zhang. Learning discrete concepts in latent hierarchical models. In The Thirty-eighth Annual Conference on Newral Information Processing Systems, 2024. URL httzp:://peerrevee wtt.:pparrrrreeeeeeo/rourreleeaa net//oorumPLLe-0SSXXvlllll
Alice Lai and Julia Hockenmaier. Learning to predict denotational probabilities for modeling entailment. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 1, Long Papers, pp. 721-730, Valencia, Spain, April 2017. Association for Computational Linguisties. URL
org/E17-1088
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-ILevel Concept Learning through Probabilistic Program Induction. Science, 350(6266):1332 1338, 2015.
Edith M. Lederer. Exec Tells First. UN Council Meeting that Big Tech Can't be URL https://apnews. com/article/ haa.rrreaaaaaa111117833333333333
Trusted to Guarantee AT Safety, Jul 2023.
artificial-inellliennee
Sharon Lee, Yunzhi Zhang, Shangzhe Wu, and Jiajun Wu. Tangunge-informed visual concept learning. In
The Twelfth International Conference on Learning Representations, 2024. URL httpe://openrevvie:
juuyw8BB81
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-inage pre- training for unified vision-language underslanding and generation. CoRR, abs abs/2201.12086, 2022. URL
https://arxiv http:://aaxiv..o/ab//////a/ wrg/ 2atp:/220...1.0 12086
Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Zhu. A competence-aware curriculum for visual concepts learning via question answering. In European Conference on Computer Vision, pp. 141-157. Springer, 2020a.
Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Chun Zhu. A Competence-Aware Curriculum For Visual Concepts Learning via Question Answering. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, pp. 141-157, Berlin, Heidelberg, 2020b. Springer- Verlag. ISBN 078-3-030-58835--8 doi: 10.1007 -70007997---000-5553---- 9. URL. https: 10tp://00.1000/7---00000000
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Improving task adaptation for cross-domain few-shot learning. CoRR, fos/2107.00335, 2021. URL http::////axxxi.o/////////////// 00358
Xiang Lorraine Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the Geometry of Probabilistic Box Emboddings. In International Conference on Learning Representations, 2018. URL atttp::/appi.eeenatticcchooa.. wtt.://oo....ttto/oorrpaaa...101
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongic, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva. Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014. URL http::/arxiv orgg/bs///00
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept Neurons in Diffusion Models for Customized Generation. ArXio, abs/2303.05125,
2023b.
17

---

Published in Transactions on Machine Learning Research (01/2026)
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ar preprint arXiu:1711.0511 2017.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action, 2023. URL https: https://arxiv: http://arwvvorrg./a..// 2tt.//ab/2221.117//
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B.
Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes Words and Sentences from Natural Supervision. ArXiv, abs/1194.12584, 2019.
Muhammad Arslan Masood, Markus Heinonen, and Samuel Kaski. Multi-modal representation learning for molecules. In Learning Meaningful Representations of Life (LMRL) Workshop at TCLR 2025, 2025. URL
Lingjic Mei, Jiayuan Mao, Ziqi Wang, Chuang Gan, and Joshua B. Tenenbaum. FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations. ArXin, abs/2203.16633) 2022.
Tom M. Mitchell. Machine Learning, International Edition. McGraw-Hill Series in Computer Science. McGraw-Hill, 1997. ISBN 978-0-07-042807-2. URL httpps:/wwwx http://wwwwwwww.......oooo http://wwwo.llaa..ooooooo//oo/// 0reieat.orr///1c/111110000
Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. Tn European Conference on Computer Vision, 2016. URL http:::/pppp..eaatttccccccoooooo
14849501
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. CoRR, abs/2106.09700 2021. URL https://arxiv.
org/ 0tt/aab/2210..5555
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramosh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents, 2022.
Nirat Saini, Khoi Pham, and Abhinav Shrivastava, Disentangling visual embeddings for attributes and objects. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 18658-13666 June 2022.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances én neural information processing systems, 30, 2017.
Ivaxi Sheth and Samira Ebrahimi Kahou. Auxiliary losses for learning generalizable concept-based models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevierr net/forum?id=jyYXnnnee
Bowen Shi, Abdelrahman Mohamed, and Wei-Ning Hsu. Embeddings with AV -uBBRT 2022.
Learning Lip-Based Audio-Visual Speaker
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kicla. FLAVA: A Foundational Language And Vision Aligument Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1663--15600, 2022a.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638 15650, 2022b.
Aditya Sinha, Siqi Zeng, Makoto Yamada, and Han Zhao. Learning structured representations with hyperbolic embeddings. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances in Newrol Information Processing Systems, volume 37, pp. 91220 91259. Curran Associates, Inc., 2024. URL httpps://prooceedingg. httpa//ss..oorippp..opppaaeeee wwwes//appppseee paales//pppr/222// 111256228.00b77755aaaaaafftaaaa2
pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Usckoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. Advances in neural information processing systems, 30,
2017.
Ramakrishna Vedantam, Arthur Szlam, Maximilian Nickel, Ari Morcos, and Brenden M. Lake. CURT: A benchmark for productive concept learning under uncertainty. CoRR, abs/2010.02855, 2020. URL
sttps://arxiv.oot btrg/abs//0n httggabs/2010...oo 0085.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowledge graphs with box lattice measures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 263 272, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1V118663/11/P18-1225
11/118-1025 URL
Wen Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiangbo Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for vision and vision-language tasks. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19175-19186, 2023a. URL 260068316
hwtpcchhollarooooooouuueuse org/CorpusID:
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. In Thirth- Conference on Newral Information Processing Systems, 2023b. URL
WordNet. Wordnet, a lexical database for english. URL https://sordneet.priia www.ceton.edeeeeeee
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmect Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in newral information processing systems, 31, 2018.
A Concept Space Details
A.1 Preliminary
A smoothing function for the concept space is defined
softplus(w")
m (W) softplus( Gt maaa G (min
(4)
where the denominator normalization term with Gmazi Gman being the global maximum and minimum values at dimension. In short, this smoothing function introduced so a valid joint probability can be caleulated even if two concepts/boxes are disjoint and we refer readers to Li al. (2018) for its complete proof.
19

---

Published in Transactions on Machine Learning Research (01/2020)
Predicated
Ground
1.00
0.08 0.05
0.08
0.07 0.05 0.08 0.08 0.33 0.34 0.33 0.30 0.50
0.51
1:00
0.00
0.00
0.00
0.00 0.00 0.00 0.33 0.34 0.35
0.50 0.50
0:47
0.53
blue
0.08 1.00 0.08
0.07
0.08 0.07 0.08 0.08 0.33 0.35 0.33 0.50 0.50 0.49 0.51
0.00
1.00
0.00
0.00
0.00
0.00 0.00 0.00 0.34
0.33 0.35
0.50
0.50
0.47
0.53
brown
0.07
0:88
1.00
0.08 0.08 0.08 0.07 0.08 0.34 0.35 0.35 0.:50 0.50 0.49 0.51
-0.00
0.00
1.00
0.00 0.00 0.00 0.00 0.00
0.31 0.55 0.35
0.50 0.50
0.47
0.53
yellow
0.08
0.03
0.07
0.33
0.50
0:51
-0.00
0.00
0.00
0.00
0.33
0:53
0.08
0.33
0:51
0.00
0.33
0.33
0:33
0.50
00,00
0.00
0.33
0:50
0.53
piople
0.08 0.08 0.05 0.08 0.08
0.08
1.00
0.08
0.34
0.33
0.33
0.50 0.50
0,48
0.52
0.000 0,00 0.00
0.00 0,00
0.00
1,00
0.00
0.33
0.33
0.34
0.50
0.50
0:47
0.53
0.08 0L08 0.D5 0.08 0.07 0.08
0.08
1.00
0.34
0.35
0.33
00.50
0.50
0.49
0.51
-0.00 0,00 0.00 9.00 0.00 0.00
0.00
1.00
0.33
01.33
0.34
0.50
0150
0:47
0.53
cylndder
0.12 0.12 0.13 0.12 0.12 0.12 0.15
0.12
1,00
0.16 0.16
0.550
0.50
0.48
0.52
-0.12 0.13 0.13 0.12 0.12 0.12 8.13
0.12
1.00
0.00 0.00
1.50
0.50
0.47
0.53
nafoo
0.13 0L13 0.13 0.12 0.13 0.12 0.12 0.12
0.16
1.00
0.16
0.50
0.50
0:47
0.33
0.13 0.13 0.13 0.13 0.12 0.12 0.13 0.12
0.00
1.00 0.00
0.50
0.50
0:47
0.53
Vihoro
0.16
-0.2
0.52
0.13
0.12 0.12 0.12 0.35 0:35
1.00
-0.13 0.12
0.12
0:31
0:59
0,12 0.13 0.13 0.13 0.12 0.12
0.34 0.35
0.13
1.00
0.51
0..2 0.13 0.12 0.12 0.13
0.33 0.33
0.00
0.53
Tango
0.13 0.13 0.13 0.12 0.13 0.12 0.12 0.13 0.33 0.35 0.34 0.50 0.50
1.00 0.14
00.12 0.13 0.13 0.13 0.12 0.12 0.13 0.15 0.33 0.33 0.34 0.50 0.50 1.00
0.00
maal
0.12 0.13 0.13 0.12 0.13 0.12 0.12 0.12 0.34 0.34 0.35 0.50
0.50
0.12 1.00
0.12 0.13 0.13 0.12 0.12 0.12 0.13 0.12 0.34 0.34 0.33
0.50 0. 1.50
0.00 1:00
Bray
but
brown
vellas
groa
pupple
cyfinder
sifle
splirr
modal
mirler
hage
mall
Baw
blue
brown
Jallo
Brean
parple
5hai
cylieker
kale
sphore
metal
rabber
Iage
small
02
0.6
1.0
Figure 5: A comparison between the learned concept space's understanding of the CLEVR world and the ground truth relations illustrated via entailment probabilities of concept pairs. Such comparison allows simple probing into the knowledge learned by this abstract concept space. A SoftMax function is applied on entailment probabilities of same-attribute concepts conditioned on a single concept so Eyeestrr Ply'ly) = 1 is satisfied.
A.2 Concept Space Training Objective
We define a KL-divergence measure between a predicted conditional probability distribution ((v1112) and a target p(y1 as:
P(y (y2) = [y log Q (82)
(5)
Let (2) denote set of all concept pairs created from 2-combination from y The objective for training the concept space is formally described as the following:
Croncepee (C:D.)
Σ
[D.]
(a.
(6)
1
Σ
D Daa(P((noo))(Q(()ll))) + DKL (1= P()1 Q (22))
2
(*)
()))
A.3 Probing into Concept Space
Figure 5 shows an example of probing into learned knowledge of the concept space exposed to CLEVR. Benefited from such efficient probing mechanism, this concept space offers more interpretability compared to traditional latent spaces model parameters of previous learning frameworks.
20

---

Published in Transactions on Machine Learning Research (01/2026)
CLEVR
GQA
COCO
1.0
0.8
1.0
Training Accuracy
0.8
0.8-
0.6
0.4
0.6
0.4
0.2
0.4
0.0
0.0
400
100
200
300
step
stop
stop
with Pretrained Concept Space
with Fresh Concepl Space
Figure 6: Ablation study the pretrained concept space. We cut projection models' access to the pretrained concept space and the learning of this concept space is combined into training processes of the projection models. Shaded area in plots represents 2-sigma compare the ablated version and the original framework.
five trails of experiments. Their classification accuracy used to
Dataset.
Dim (K)
Accuracy
mAP
24
0.99898
CLEVR
50
0.99900
96
0.92914
24
0.95899
0.57212
COCO
50
0.95653
0.58961
96
0.56060
0.56024
24
0.84752
0.21271
GQA
50
0.84419
0.35202
96
0.61458
0.32331
Table 7: Ablation on concept space dimensionality (K). CLEVR reports accuracy only; COCO and GQA report accuracy and mAP. Bolded values indicate the selected dimensionality (K 50).
A.4 Ablation on Concept Space
We discover that using a pretrained concept space with learned abstract knowledge helps modality-specific projection models converge faster compared to the ones without the access. Specifically, we cut our framework's 's access to the pretrained concept space C. Instead, the framework is only provided with a freshly initialized concept space C' and the loss function during pretraining of the vision- modaaiyy projection model is changed to Cvision = Lvision + Le. Fig. G shows that the original framework's projection models can converge laster than the ablated version. Based on this evidence, we conclude that the abst knowledge shared by the protrained concept space streamlines the lcarning process of modality-spccific projection models.
A.5 Ablation on Concept Space Dimensionality
To assess how the dimensionality of the concept space affects downstream performance, conduct ablation study using K (24, 50, 96). For each value of K, we follow the same pretraining protocol: we first train the concept space to convergence and then train the ViT-based projection model to adapt to that space. All runs use identical hyperparameters, optimization settings, and training steps to ensure a fair comparison. Results are reported in Table 7
Empirically, K 50 achieves the strongest and most consistent performance across datasets, showing the highest mAP both COCO and GQA while matching the near-saturated accuracy CLEVR. These
21

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset
# Attributes
# Categories
GQA
35
33
CLEVR
12
3
COCO
35
29
Unified (All)
70
60
Table 8: Summary of concept subsets used each dataset.
observations provide practical support for choosing 50 as the concept-space dimensionality used in all main experiments.
We also observe that increasing the dimensionality to K 96 leads noticeable performance degradation, particularly on COCO and GQA. A plausible explanation is that larger box dimensions introduce complex geometric structure, which makes the concept space harder to learn effectively and can lead to diminishing returns or overfitting. Conversely, very small dimensions, such as K = 24, limit expressivity and may restrict performance on semantically richer datasets. The intermediate dimensionality of 50 therefore offers a favorable balance between expressiveness and learnability.
B Cross Modality Joint Training
To allow probabilistic analysis for cross-modality tasks, an optional joint training stage can be used encourage different projection models to produce projections that overlap with cach other's for the same object. This joint training stage is optional because each individual projection model is already adapted to the shared knowledge space, and it is computationally lightweight, as the modality-specific projection models have already been trained and aligned with the unified concept space. It requires very modest resources, with convergence occurring within a few hundred training steps, as demonstrated in Sec. 4.2 Subsequently, this design with demonstrated efficiency allows the effortless incorporation of new projection models into our proposed framework, mirroring humans' ability to learn and link knowledge across modalities efficient manner. Specifically, consider system with modalities, and B, an example. The training dataset would be denoted DAUB {x a H and the training objective for this joint training stage is defined as:
fast and
Cjoine (0x, AUBB) 2DD (
Σ
P(fA(ZA:0A) fB(2;; 0B)) P(f((z;; 0B) (((=A;
The overall training objective becomes a combination of modalityyspeciific projection losses and this joint training loss.
C Evaluation Datasets and Preprocessing
C.1 Dataset Details
We base our evaluations three datasets and a world dataset aggregating all concepts and representations together:
CLEVR datasct comprises synthesized images paired with intricate questions testing system's visual reasoning capabilities. We choose CLEVR for evaluation because it provides a highly controlled mini-world, where concepts are easily drawn from visual objects, and relationships between concepts are clearly defined. Each CLEVR image displays scene with random number of objects, each described by color, shape, material, and size, which produces 15 unique values such as blue and cube, forming attribute concepts related to specific objects.
22

---

Published in Transactions on Machine Learning Research (01/2026)
Type
CLEVR
COCO
GQA
Attributes
blue, brown, cyan, gray, green, large, metal, purple, red, rubber, small, yellow
adult, appetizing, athletic, busy, casual, cloth, cooked, enjoying, lamily-friendly, female, flully, fresh, functional, furry hairy, healthy, holding, horizontal, laying, male, metal/ /metalicc moving, parked, participating, public, sitting, socializing, soft, sporty, standing, tame, lasty/delicious, useful, vertical, watching/looking
black, blue, brick, brown, clear, cloudy, conerete, dark, glass, gray, green, happy, large, long, metal, old, open, orange, parked, pink, red, round, short, silver, sitting, small, smiling, standing, striped, tall, tan, white, wood, yellow, young
Categories
cube, cylinder, sphere
airplane, apple, banana, bear, bicycle, bird, boat, broccoli, bus, cake, car, carrot, cat, cow, dog, donut, elephant, girafle, horse, hot dog, motorcycle, orange, person, pizza, sandwich, shoep, train, truck, zebra
bed, boy, building, bus, car, chair, fence, field, floor, giraffe, girl, grass, ground, hair, head, jacket, man, person, plate, road, shirt, sidewalk, sky, snow, street, table, train, tree, trees, wall, water, window, woman
Table Exact attribute and category concepts present each dataset.
Concept
Type
Datasets
blue
attribute
CLEVR, GQA
brown
attribute
CLEVR, GQA
gray
altribute
CLEVR, GQA
green
attribute
CLEVR, GQA
large
attribute
CLEVR, GQA
metal
attribute
CLEVR, GQA
parked
attribute
COCO, GQA
red
attribute
CLEVR, GQA
sitting
altribute
COCO, GQA
small
attribute
CLEVR, GQA
standing
attribute
COCO, GQA
yellow
attribute
CLEVR, GQA
bus
category
COCO, GQA
car
category
COCO, GQA
giraffe
category
COCO, GQA
person
category
COCO, GQA
train
calegory
COCO, GQA
Table 10: Overlapping concepts appearing in multiple datasets. These concepts are merged in the Unified dataset.
COCO datasel exposes our framework to a knowledge world resembling the real world better than computer- generated images from CLEVR. We use attribute annotations proposed by Patterson & Hays to establish attribute concepts such as soft, cooked, and parked (sce Fig. in (2016)). The original COCO object categories are used category concepts. We focus our evaluation on the top 35 frequent attributes and their associated categories to gain meaningful insights, resulting in 64 concepts (see Table gh.
23

---

Published in Transactions on Machine Learning Research (01/2026)
GQA dataset is similar to COCO, providing controlled sandbox mimicking real-world settings. We use the original attribute and category labels in GQA as concepts and filter out rare attributes and classes, resulting in the same total number of concepts as COCO, Example attribute and category concepts include happy, old, gray, and boy. Exact concepts can be found in Table g
Unified dataset aggregates all attribute and category concepts appearing in CLEVR, COCO, and GQA into single world dataset, resulting in 130 unique concepts (70 attributes and 60 categories; see Table 8 Overlapping concepts across datasets (listed in Table 10 merged into unified concepts. This dataset allows to evaluate the scalability of the proposed framework and its ability to generalize abstract concepts across different visual and linguistic domains. To avoid the Unified datasct being dominated by CLEVR, which contains significantly more samples than COCO and GQA, we balance the dataset sizes during construction. In the raw training splits, CLEVR contains 455,632 samples, compared to 78,898 in GQA and 91,667 in COCO. We therefore cap the CLEVR training split to match the size of the largest non- CLEVR dataset (COCO), resulting in 91,667 CLEVR samples after filtering. This ensures that the Unifiod dataset remains balanced and diverse across data sources.
C.2 Dataset Preprocessing
Since each image in these datasets contains multiple objects, a preprocessing step is essential to isolate single objects. This isolation allows focused learning targeted objects, reducing ambiguity. This process mirrors human learning, where attention naturally centers novel object while ignoring the surrounding environment Gärdenfors (2014).
Both COCO and GQA datasets already include object segmentation data. For the CLEVR dataset, we employ a MASK R-CNN model eHe al. 2017), denoted as Sdetection: trained on small amount of annotated data as an object detection model to generate segmentation. Visual object inputs are created by cropping original images include only the objects of interest, as illustrated in Fig.
In addition to object isolation, language as a new modality in the dataset. Fach sentence of an object has the structure "There is a" followed by a sequence of values indicated by its attribute concepts in random orders to ensure diversity. Category concept values are added last to the sequence, except for CLEVR, where values from the shape attribute family are placed last for naturig--oundig sentences.
generate descriptive sentence for each object, introducing natural
Model Details
D.1 Backbone Architectures
D.1.1 Vision Modality
VIT. A Vision Transformer Dosovitskiy et al. (2020) pretrained on 2mageNet-21k is used as a backbonc. The pooled embedding at the [CLS] token is passed to either the projection head (our framework) or the MLP head (baseline).
ResNets. A ResNet-50 He al. (2015) pretrained on ImageNet-21k used a backbone, Its final fully connected layer is replaced by either our projection head or the baseline MLP head.
D.1.2 Natural Language Modality
BERT. pretrained BERT-Base encoder Devlin et al. (2018) used as the backbone for the natural- language modality. Analogous to the VIT sctup, the pooled representation the [CLS] position used input to both the projection head and the MLP head.
D.2 Baseline MLP Heads
To provide consistent and comparable baselines for our projection models, we attach a simple three-layer MLP head to each backbone, whereas projection heads consist of only single linear layer. Unless otherwise
24

---

Published in Transactions on Machine Learning Research (01/2026)
specified, the two intermediate layers contain 128 units each. For ResNet--basd models, however, the bascline ML.P head uses 512 and 256 units in the intermediate layers to accommodate the higher dimensionality of ResNet's feature representation.
MLP Head Output Dimension. of concepts present in the dataset:
The output dimension of each baseline ML.P is set to the total number
dim(out) Deategory + Dattribute
Final predictions are obtained by applying an argmax over the category neurons and a threshold over the attribute neurons.
D.3 Projection Heads
As shown in Algo. 1 our projection head is implemented using two parallel linear layers. Conceptually, the projection head a single linear mapping from the backbone feature vector into the knowledge space K. For computational convenience, however, we split the backbone feature vector into two equal-sized chunks and apply an independent linear layer to each chunk. This is equivalent to applying one linear layer to the full feature vector, but allows us to separately obtain (Wmin wa) K. in our framework.
D.4 Training Details
Vision modality projection models are trained for 10 epochs with a batch size of 256 with an exception of CLEVR whose models are only trained for epoch. An AdamW optimizer with a learning rate of 10-4 is used. Learning rate schedulers are used to achieve warm-up for first epoch and then a process of 10-1 linear decrease over the remaining epochs.
Natural-language modality projection models are trained for epoch using the same setup and hyper- parameters as used by the vision ones.
Thresholds for attribute identification are selected based on performances from training splits. Thresholds producing the best fl score on training sets are used in tests.
E
Image- Text Matching Experiment Details
E.1
BLIP
We follow the training method as stated in Li et al. (2022) and fine-tune the pretrained BLIP model directly the Image-Texx Matching task (swapping-sentencee split) using both the image-text contrastive loss and task-specific image-text matching loss produced by the image-text matching classification head in BLIP. We use a greater batch size of 512 as the calculation of image-text contrastive loss requires large number of samples.
E.2 CLIP
We follow the training method as stated in Radford et al. (2021) and adapt the pretrained CLIP model the general three datasets using the symmetric loss that favors larger similarity scores between positive image- text pairs and smaller scores for negalive ones. We use batch size of 512 as in BLIP during pretraining. Similar to our framework, CLIP model is not directly trained on the Image- Text Matching task.
E.3 ViLT
Similar BLIP, we follow the training method as stated in Kim et al. (2021b) and fine-tune the pretrained ViLT model directly on Imago-Text Text Matching task (swapping-sentenne split) using a binary cross-entropy loss the matching classification head.
25

---

Published in Transactions on Machine Learning Research (01/2026)
E.4 FLAVA
We use the same procedures used in ViLT to fine-tune a pretrained FLAVA model on the data domains appeared.
F Additional Figures
Original CLEVR Image
fdtectioo
Figure 7: The segmentation masks generated by faetection are applied to the original CLEVR images to isolate each object from its surroundings environment. This preprocessing step enables our proposed framework to replicate the way we, humans, naturally focus attention novel objects during the learning process.
Vision Modality
Abstract Concept Space
Natural Language Modality
yellos small ey cylinder
N
fo
of
friston
Fut
fo
DNG
A red nubber sphero
X
x
Figure 8: Application of the propased framework the Image-text matching task. An image vision of yellow, small rubber cylinder and two description sentences z Nt. are processed by their modality-speciff models fviston and fNL which project modality-speciffc imputs onto learned abstract concept space C. We use the cross-entailment probability between projections of image and sentence to determine if they form a positive pair. While creating representations of images and sentences task, our shared representation space is knowledgs-embeelded concept space offering interpretability, which is in drastic contrast. the commonly used latent space with black-box structure.
shared latent space a common approach for the image-text matching
26

---

Published in Transactions on Machine Learning Research (01/2026)
What color the cylinder object?
visiin
Soene
Quiston
Original CLEVR Image
1 Piniioo
filter shape
X
fdatection
x2
Niston
fulston
1T.
Q pisiion
unique
query_colar
"Yellow"
xg
mision
Q
Qptaton
Dj
Concept Space
Symbolic Programs
Figure 9: Application of the proposed framework to Visual Question Answering task. We reuse the object detection model fdstection from the pretraining stage, which extracts set of single objects from original CLEVR image X,. The vision-modality projection model fviston then projects æ, onto the K. A program generator æ is used to predict sequence of symbolic programs é, based input question z in natural language format. Programs & operate the concept space and produce answer to q;-
27


Reviews:
[{"review_id": "drbzwVSQfL", "reviewer": "Reviewer_jDaK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a modality-agnostic concept space designed to capture structured, abstract knowledge. This is accompanied by modality-specific projection models (ViT, ResNet) that map raw inputs from different modalities onto this single shared space.\nThe paper claims the shared embedding space is modality agnostic and that this approach allows the shared concept space to possess abstract knowledge, which in turn facilitates more efficient learning and effortless incorporation of new modalities. The authors also claim that their framework, with a modest pretraining footprint, achieves comparable performance to benchmarks out-of-the-box without fine-tuning.\n\nThe authors first describe their approach using a mathematical basis of entailment probabilities, then describe the parameterization of the concept space as box probabilities. The authors then detail how modality projections could be learned by fitting to the bounds of the box probabilities using standard MLP + relu blocks. They then demonstrate the performance of their approach on downstream tasks, specifically image-text matching and VQA on CLEVR, GQA and COCO datasets, and compare it to existing competitive models to showcase the effectiveness of their framework, while highlighting the efficacy of adaptation and ease of interpretability in their approach.\n\nStrengths: \n- Shows a learned concept space that is interpretable and specifies projections into this space demonstrating the efficacy of introducing new modalities\n- Substantiates their claims through multiple ablations and downstream tasks on text-matching and VQA, specifically noting that beating state of the art is not their emphasis, rather it's the interpretability provided by a concept learning approach.\n\nWeakness (See more below):\n- Does not provide sufficient evidence around **cross modal** entailment/interpretability\n- Baselines for learning the shared space seem weak, with not enough ablations to showcase the efficacy of their chosen embedding space compared to any learned embedding space of similar size.", "claims_and_evidence_yn": "No", "claims_explanation": "Several claims in the paper are not fully supported by the evidence provided or require clarification.\n1. On Interpretability: The paper claims that inference is \"performed entirely within a shared space of learned concepts that offers interpretability.\" The evidence for this (\"Probing this concept space can also be achieved through simple queries of concept pairs...\") is limited. The paper positions interpretability as a key advantage of their approach, but most of the evidence is moved to the Appendix section. The interpretability demonstrated is only shown along pairwise relationships on text (specifically words and short tokens). This does not sufficiently back the broader claim of multimodal interpretability. I would recommend adding visualizations (T-sine or other) of the learned concept space, or showcase vision-text entailments to strengthen this claim. \n\n2. On Efficient Learning and Adaptation: The paper claims \"more efficient learning curves\" and adaptation to new modalities in \"a few 100 steps.\"\n- This \"faster learning\" argument is a bit weak. The experiment compares the proposed method (which uses a pretrained embedding space) to a baseline MLP - that likely does not have access to the pretrained space. A model leveraging a pretrained space will naturally learn faster. Consider adding a few more ablations around models that have access to pretrained embedding spaces.\n- The claim of adapting in a few hundred steps also seems tied to the dimension D of the box embedding space, which is not discussed. The output dimension of the MLP is not specified, so there’s no way to accurately compare the parameterization of the learned concept embedding space, v/s any learned latent space.\n- The claim of efficiency via \"referencing a (compact parameterized) concept space\" is not supported by evidence. What is specifically efficient about the concept space v/s other learned spaces of a similar dimension?\n\n3. On Novelty and Related Work: The authors claim that \"this idea of a concept-focused learning scheme has rarely been explored in the field of multimodality learning.\"\n- Sharcs (https://arxiv.org/pdf/2307.00316) appears to be a very close piece of related work, where shared concepts are learned and mapped to a semantically meaningful space.\n- Unified-IO Line of work (https://arxiv.org/abs/2312.17172 ) also explores unified representations using patch sequences.\n\n4. On Empirical Evidence:\n- The CLEVR dataset and it’s associated metrics seem to be very saturated (upper 0.99 accuracy), limiting the conclusions that can be drawn. While I appreciate the call to action for more datasets for concept learning, the authors should consider more recent datasets (e.g CURI - https://arxiv.org/abs/2010.02855 )\n- No confidence intervals (CIs) are reported in Table 2, making it hard to know if the reported numbers are statistically significant or if the vision encoders are doing the heavy lifting in the metrics.\n- In Table 1, a regular MLP baseline actually performs better on some tasks on CLEVR than the proposed method, undermining the claim of being \"competitive.\" Consider a few more experiments other than a regular MLP to provide a range of baselines in learning the concept space for Table 1.\n- The paper does not demonstrate how discrete tokens (text) can be blended with continuous tokens (image patches). This is a critical challenge in unified projection. The authors mention projecting vision and text embeddings into a continuous probabilistic space but do not comment on the efficacy of projecting continuous tokens. While the box diagrams shown in Figure 7, are helpful, consider also showing vision-vision and text-text entailments as a way to demonstrate modality agnostic relationships learning in the concept space. \n\n5. Misc Unsupported Claims:\nThere's a minor claim in the paper that states in Section 3.3 on the training loss - \"Optionally, the optimization can also include parameters from C, so that the abstract knowledge learned in the concept space is adjusted based on modality-specific information.\" There is no experiment to support this claim. Consider specifying this in future work or adding experimental evidence.\n\n6. On the Motivational Analogy: The paper uses the analogy of learning a new language, where \"we intuitively connect new linguistic elements to our existing understanding of the world.\" This analogy is flawed; learning a new language is still (initially) within the text modality. Humans sometimes translate first, then build new concepts over time, which is a different process. While the comparison is noted, it weakens the argument. Consider another comparison, particularly around learning concepts from different modalities, since that seems to be one of the underlying claims of the paper.", "audience_yn": "Yes", "audience_explanation": "Yes, the concept-centric approach and interpretability arguments can be topics of interest to TMLR audience.", "requested_changes": "1. Clarifications:\n- Provide a clear justification for the mathematical formulation of $P(y)$ as a product of $m()$ over all dimensions for the d-dimensional embedding K. Why does this need to be a product of probabilities *across* the dimension, rather than leave it as is or a summation? How does this change the outcome of the experiments as the dimension D varies?\n- **Clarify what is novel in the multimodal projection and embedding space, given that the concept space itself is adopted from Li et al (2018) (Section 3.1 in the paper)**. You can explicitly call out the changes made to adapt it to a multimodal concept space.\n- State the output dimension of the baseline MLP used in the experiments.\n2. Experiments and Evidence:\n- Interpretability: **The paper should show evidence of multimodal relationships in the concept space, not just pairwise text relationships.**\n- Projections: Consider a demonstration and/or discuss how discrete tokens (text) are blended with continuous tokens (image patches). Comment on the efficacy of the continuous token projections, where both vision and text embeddings are split to learn parameters of the box embedding. \n- Baselines:\n  - **The \"faster learning\" claim must be substantiated with a fair comparison**. The baseline MLP should also have access to a pretrained embedding space that serves as its projection.\n  - Consider adding other baselines that use the same vision and language backbones. The BEIT line of work (e.g., https://arxiv.org/abs/2208.10442) could serve as a baseline, as it also learns a common embedding space and uses the same vision and text encoders.\n- Metrics: Consider showing the F-1 scores and other metrics mentioned in Table 1 for the training accuracy comparison in Figure 3. It can strengthen the claim that the concept space enables faster learning.\n- Ablations: Consider adding an ablation study on the dimension of the concept space embedding (K) to justify the choice of 50.\n\n3. Scope and Limitations:\nThe paper suggests projections along \"attributes\" and \"categories.\" Many modalities (e.g., robotic actions, alpha channels) do not fit this construction. Consider explicitly calling out this limitation, it is fine to leave this as future work.", "broader_impact_concerns": "There is no Broader Impact statement and is not required.", "additional_comments": "Thank you for your submission to TMLR, I look forward to hearing your response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "zlJw8xaNpj", "reviewer": "Reviewer_KSiN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an alternative approach to multimodal representation learning that uses structured concept spaces rather than the dense vector representations employed in methods like CLIP. The framework consists of two stages: (1) learning a modality-agnostic concept space using probabilistic box embeddings that encode relationships between concepts via entailment probabilities, and (2) training modality-specific projection models that map inputs from different modalities into this shared concept space.\n\nThe core idea—using structured, interpretable concept spaces as an intermediate representation for multimodal learning—is interesting and well-motivated. However, the experimental validation does not convincingly support the broad claims made about efficient and modular multimodal learning and adaptation. The experiments are conducted in small-scale, controlled settings (CLEVR, simplified subsets of COCO and GQA with only 64 concepts) where the multimodal aspect is particularly weak: the natural language modality consists of synthetically generated, structured descriptions that are essentially concatenated lists of attributes, making it difficult to assess true cross-modal alignment capabilities.\n\n## Strengths\n\n- **Strong motivation and positioning:** The paper provides a compelling argument for concept-centric learning inspired by human cognition, with a thorough literature review connecting to relevant work in multimodal learning and concept learning.\n- **Interesting technical approach:** The synthesis of structured embedding spaces (box embeddings), concept learning, and multimodal projection is novel and the use of supplementary concept annotations from datasets like COCO attributes and GQA scene graphs is well-executed.\n- **Interpretability:** The framework demonstrates clear benefits for interpretable reasoning, particularly in the VQA experiments where symbolic programs operate over learned concept representations, and the ability to probe entailment probabilities between concepts is genuinely useful.\n- **Thoughtful discussion:** The paper includes good consideration of future work, limitations, and broader implications including bias mitigation.\n\n## Weaknesses\n\n1. **[W-1] Claims Not Convincingly Supported by Experiments**\n    - Multimodal claims are overstated: The paper claims to enable \"efficient and modular multimodal learning\" that bridges \"the efficiency gap between traditional machine learning methods, which require extensive data, and human learning.\" However the text modality in all experiments consists of synthetically generated, highly structured descriptions (e.g. \"There is a yellow small rubber cylinder\") on small (<=64) concept spaces. While well motivated, it is hard to support claims of this breadth for multimodal learning.\n    - Ablation studies do not sufficiently support core claims: The ablation study in Section 5 receives significant emphasis but does not effectively support the paper's central claims about efficient multimodal learning.\n\n2. **[W-2] Presentation and Missing Details**\n    - Important details are missing or difficult to find: Core technical and experimental details in Sections 3, 4 are either absent (impacting reproducibility) or inconsistently distributed between main text and appendices\n\n3. **[W-3] Limited Scope and Missing Contextualization**\n    - Relationship to current VLM paradigm not directly addressed: The paper does not discuss how this approach relates to, could complement, predominant vision-language model paradigm. For instance:\n        - What would it take to handle more general scene understanding with complex natural language queries (as in full GQA) rather than controlled symbolic reasoning (as in CLEVR)?\n        - Could this concept-space approach be integrated with or benefit from pretrained vision-text-language models?\n\n**Overall** \n\nThe work would benefit from either (1) repositioning the claims to focus on the interpretability and structured reasoning benefits that are well-demonstrated, or (2) conducting additional experiments to make the claims more convincing (described below).", "claims_and_evidence_yn": "No", "claims_explanation": "## Claims\n\nCore claims of the paper (in Section 1 and 6)\n- [CL-1]: Efficient multimodal learning \n- [CL-2]: Modular learning and adaptation \n\n[CL-S]: Softer claims made through the paper: lower footprint, no fine tuning, interpretability\n\n## Core Issues (CI)\n\n### [CI-1] Ablation studies (Figures 2-3) do not convincingly support efficient multimodal learning claims\n\nThe ablation comparing pretrained vs. fresh concept spaces is presented as central evidence for CL1, but has several limitations:\n- (a) Convergence benefit may reflect initialization rather than conceptual efficiency. The learning curves for COCO start around 0.4 accuracy, which is unsurprising given that attributes in COCO are highly correlated (e.g., \"jumping, catching, exercising\" co-occur frequently, as shown in Fig. 1 of Patterson & Hays 2016). Random chance baselines achieve high average precision for frequent attributes (Fig. 6, Patterson & Hays 2016), and the COCO Attributes paper explicitly discusses \"benefits of exploiting multi-label occurrence.\". The paper reports average thresholded F1 scores. Being able to directly compare AP/mAP scores would help assess whether the observed benefit comes from the concept space structure or simply from exploiting attribute correlations.\n- (b) This is not a multimodal alignment task. The ablation tests single-modality projection to attributes, not cross-modal efficiency, yet is used to support claims about multimodal learning.\n- (c) No external efficiency comparison. The ablation is entirely internal (pretrained concept space vs. fresh concept space). There is no direct comparison of training footprint, convergence speed, or data efficiency against baseline multimodal methods like CLIP or BLIP.\n\n### [CI-2] Evidence for modular adaptation (CL2) is limited \n\nThe claim that \"projection models can align with existing conceptual representations rather than learning from scratch\" and that the framework enables \"effortless incorporation of new modalities\" is not empirically validated:\n- No experiments demonstrate assimilation of genuinely new concepts or modalities to an existing system\n- Each projection model is individually finetuned per dataset/task (COCO, CLEVR, GQA separately), which undermines the modularity claim\n\n### [CI-3] Contradiction regarding finetuning\n\nSection 4.2 states the model is \"not finetuned\" (Table 2), but Appendix D describes joint training. This inconsistency needs clarification - it appears the method does involve joint training on the image-text matching task? If so should be acknowledged claims about fine-tuning in Section 1 \"Contribution\".\n\n### [CI-4] Limited experimental scope insufficient for broad claims\n\n- Experiments use small-scale, controlled settings: CLEVR (synthetic), simplified COCO and GQA subsets (only 64 concepts)\n- The natural language modality consists of synthetically generated, structured descriptions (essentially concatenated attribute lists), making it difficult to assess true cross-modal alignment capabilities vs natural language from a truly new text modality\n\n\n## What is well supported\n\nThe paper does provide convincing evidence for softer claims (SC):\n- **Interpretability:** The ability to probe entailment probabilities (Table 4, Figure 4) is genuinely useful and well-demonstrated\n- **Structured reasoning:** The VQA experiments (Section 4.3) show that purely symbolic programs can operate over learned concept representations with a high accuracy\n- **Competitive performance:** Tables 1-3 show the framework achieves reasonable performance with limited task-specific optimization", "audience_yn": "Yes", "audience_explanation": "The paper addresses several questions of significant interest to the multi-modality learning and interpretable AI communities:\n1. **Cognitively-inspired learning paradigm:** The concept-centric approach that mirrors human cognitive processes (forming abstract concepts and connecting them across modalities) represents a fundamentally different direction from current end-to-end learning systems. This perspective alone would interest researchers exploring alternatives to standard deep learning paradigms.\n2. **Interpretability and probing:** The explicit concept space with queryable entailment probabilities (Table 4, Figure 4) offers a practical mechanism for model introspection that is notably absent in conventional latent spaces. This directly addresses growing concerns about ML system transparency and trustworthiness.\n3. **Compositional and controllable representations:** The framework's treatment of concepts as modular, reusable entities aligns with important open questions in compositional generalization and systematic generalization. Moreover, the explicit concept space could enable better control and steering of generative models—a natural extension the authors note for concept-grounded text-to-image generation, where interpretable concept manipulation (cf. Liu et al. 2023b on concept neurons in diffusion models) remains an active area of interest.", "requested_changes": "## [RC]: Requested Critical Changes (Required for Acceptance)\n\n### RC0: Reposition claims \n\nSuggestion is to either reposition claims to focus on interpretability and structured reasoning benefits (which are well-demonstrated), or proceed with RC1 onwards below.\n\n### RC1: Address multimodal efficiency [CI-1]\n\nAddress concerns in [CI-1]. Or instead focus on the clearly multimodal image-text matching task via joint fine-tuning (Section 3.3). There appear to be promising results in Appendix D \"joint training converges in 600 steps with significantly less resources\". Expanding on details with a direct comparison vs (BLIP, VILT, CLIP) on training footprint, or efficiency of fine-tuning would make the claim more convincing.\n\n### RC2:  Empirically validate modular adaptation claims [CI-2]\n\nShow actual modality/concept acquisition: Conduct experiments demonstrating addition of new concepts or modalities to an existing system. For example:\n- Modality extension: Test robustness to perturbed text descriptions (language translations, different phrasings) to show decoupled projections are more robust than end-to-end finetuned models in image-text matching\n- Concept extension: Experiment with held-out concepts (e.g, CLEVR-CoGenT experiment from NS-VQA Yi 2019)\n- Explain per-dataset finetuning: Projection models appear to be individually finetuned per dataset and task. Demonstrate joint training of shared projection models across all datasets (COCO, CLEVR, GQA) with individually trained concept spaces OR explicitly discuss why individual finetuning is necessary and how this impacts the modularity claim.\n\n### RC3: Address fine-tuning contradiction [CI-3]\nRefer above for Section 4.2 and Table 2\n\n## [RS]: Requested Strengthening Changes (Non-critical, improve submission)\n\n### RS1: Clarity of presentation and missing details \n\n1. Section 4: Cite COCO attributes (Patterson, 2016) and consider referring readers to Fig 1 from the paper to make it clearer to readers unfamiliar with notion of categories vs attributes in such datasets/tasks.\n2. Section 4 / Appendix B: appears like modified subsets of COCO and GQA were used with only 35 attributes/64 concepts (out of 196) for COCO and 64 concepts (our of 620 attributes) for GQA. Why was this necessary? This is an important detail for readers to assess results and should be called out in main contents. Exact attributes/concepts should also be listed in appendix so results are reproducible and comparable with other published results (Patterson, 2016).\n3. Section 3.1: For benefit of the reader unfamiliar with this technique when you \"adopt .. by Li et al\", name it explicitly \"probabilistic box embeddings\" and provide geometric intuition i.e. overlap of d-dimensional boxes.\n4. Section 3.1: Explain the motivation for negative concept sampling: \"why are negative concepts/noise added to entailment probabilities? Is it to address noise/sparsity similar to n-gram smoothing?\" \n5. Section 3.2: Refer readers to Section 4 for attribute vs category distinction and why treated differently in loss\n6. Section 4.1: Consider using threshold independent mAP instead of F1, for comparison against (Patterson, 2016) and reproducibility without needing to provide attribute-specific thresholds. Ideally at least one external baseline comparison to establish reasonable attribute prediction.\n7. Section 4.1: How does one \"fit\" a concept space to 15 concepts in CLEVR using batch size 256 when there are only 15C2=105 pairs? This should be explained coherently in one place with cross-references. This should be explained clearly in one place with minimal cross references (Section 3.1, 4.1, Appendix A)\n7. Section 4.2 / Appendix D: It seems like baselines were finetuned only on sentence swapping and not attribute swapping? What about the joint trained results from the paper?\n8. Section 4.3: Specify whether you used provided pretrained models from IEP (Johnson 2017b) or trained your own. If training your own, explain why this was necessary and provide sufficient details for reproduction\n9. Section 4.3: Are all question types from CLEVR evaluated with the functional programs? Explain how spatial comparisons (e.g., \"in front\" / \"behind\") are handled using only 2D object detection coordinates. Reference to Fig. 2 from Johnson 2017b might be helpful.\n\n\n### RS2: Structural improvement\n\n1. To provide room for technical details from appendices, consider shortening redundant claims and information in Sections 1, 6, 7 and moving the ablation in Section 5 to appendix.\n2. Consider moving Fig 7 and Fig 8 to main contents as they are critical to understand the application of the framework. Fig 6 also seems relatively redundant w.r.t. Fig 8\n\n### RS3: Broader context and discussion\n\n1. Cite relevant attribute datasets: Section 6 calls for attribute datasets but should cite: VAW (Pham 2021), LSA/TAP (Pham 2022), and OVAD (Bravo 2023). Note that OVAD also evaluates zero-shot attribute prediction of VLM foundation models, which is relevant to this work.\n2. Relationship to VLM paradigm: Directly address whether/how this approach relates to or could complement current vision-language models. Specifically:\n    - What would it take to handle more general scene understanding with complex natural language queries (as in full GQA) as opposed to controlled symbolic reasoning (CLEVR)?\n    - Could the concept-space approach be integrated with or benefit from expansive knowledge in pretrained vision-language models?\n    - Natural language as concept space: How do you address the perspective that natural language itself already serves as a generalizable concept space (as leveraged by models like CLIP)?", "broader_impact_concerns": "No additional broader impact statement is required. The authors already provide a thoughtful discussion on bias mitigation in Section 6 (\"Addressing Bias\"), demonstrating how the framework's interpretable concept space enables direct probing and correction of learned biases - such as gender associations - through adjustment of ground-truth entailment probabilities.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xYD2KKJ1Ve", "reviewer": "Reviewer_5zbu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Strengths:\n1. The submission proposing a new direction to align knowledge from different modalities by aligning them in a common space with concepts. The motivation of this concept-centered approach itself is promising and reasonable.\n\nWeaknesses:\n1. The work straightforwardly extends the previous box embedding solution to the multimodal case. Unfortunately, some key challenges about multimodality are not well-explained. For example, 1) How to evaluate whether the learned concepts are modality-agnostic; 2) How to design general decoders to utilize the modality-agnostic concepts.\n2. The experimental results and evaluations are not comprehensive. Some baselines are relatively out-of-date. For example, in the VQA task, the latest baseline in published in 2021. Meanwhile, the performance of the proposed method doesn't show better/stronger gains, which greatly limits the illustration of the proposed method. \n3. The presentation can be further improved, and lots of details are missing, such as the specific design choices for various potential downstream applications.", "claims_and_evidence_yn": "No", "claims_explanation": "The submission claims a very general and intuitive framework for multimodal learning. However, the experimental results on two simple tasks and comparisons with out-of-date baselines don't demonstrate the claimed strengths of the general concept-centric framework.", "audience_yn": "Yes", "audience_explanation": "The idea of concept-centric multimodal learning itself is an interesting direction. Thus, it may still be of interest for some individuals.", "requested_changes": "1. More comprehensive evaluation to demonstrate the effectiveness and generalization of the proposed framework.\n2. Polish the presentation by adding more details and descriptions about the methodology.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)