Object-Centric Concept-Bottlenecks
Authors: David Steinmann
Wolfgang Stammer
Antonia Wüst
Kristian Kersting
Venue: NeurIPS 2025 poster ()
Keywords: Concept Learning
Object Centric Learning
Interpretability
Primary area: deep_learning

Abstract: Developing high-performing, yet interpretable models remains a critical challenge in modern AI. Concept-based models (CBMs) attempt to address this by extracting human-understandable concepts from a global encoding (e.g., image encoding) and then applying a linear classifier on the resulting concept activations, enabling transparent decision-making. However, their reliance on holistic image encodings limits their expressiveness in object-centric real-world settings and thus hinders their ability to solve complex vision tasks beyond single-label classification. To tackle these challenges, we introduce Object-Centric Concept Bottlenecks (OCB), a framework that combines the strengths of CBMs and pre-trained object-centric foundation models, boosting performance and interpretability. We evaluate OCB on complex image datasets and conduct a comprehensive ablation study to analyze key components of the framework, such as strategies for aggregating object-concept encodings. The results show that OCB outperforms traditional CBMs and allows one to make interpretable decisions for complex visual tasks.

Object-Centric Concept-Bottleecks
David Steinmann' 1,2
Wolfgang Stammer' 1.2
Antonia Wüst!
Kristian Kersting!.2 1.2.3,
CComputer Science Department, TU Darmstadt; "Hessian Center for Al (hessian.Al), 3 German Research Center for Al (DFKI); *Centre for Cognitive Science, TU Darmstadt
Abstract
Developing high-performing, yet interpretable models remains critical challenge in modern AI. Concept-based models (CBMs) attempt to address this by extracting cuman-understandabll concepts from global encoding (e.g., image encoding) and then applying linear classifier on the resulting concept activations, enabling transparent decision-making. However, their reliance on holistic image encodings limits their expressiveness in object-centric real-world settings and thus hinders their ability to solve complex vision tasks beyond single-label classification. To tackle these challenges, introduce Object-Centric Concept Bottlenecks (OCB), a framework that combines the strengths of CBMs and pre-trained object-centric foundation models, boosting performance and interpretability. We evaluate OCB on complex image datasets and conduct comprehensive ablation study to analyze key components of the framework, such strategies for aggregating object-concept encodings. The results show that OCB outperforms traditional CBMs and allows one make interpretable decisions for complex visual tasks.
Introduction
In recent years, the field of interpretable machine learning has made significant progress, particularly models (Chen et al., 2019; Koh et al., 2020; Rudin
through the development of interpretable-by- al., 2021; Chattopadhyay al., 2023b,a). These models are designed to provide explanations that faithfully reflect the model's internal reasoning, thereby improving user understanding and control. One of the most prominent lines of research in this area concept bottleneck models (CBMs) (Koh et al., 2020; Stammer et al., 2021), which aim to ground a model's internal representations in semantically meaningful high-level concepts. The core idea is to decompose complex inputs, such as images, into an interpretable concept encoding and to make predictions based on this concept-level abstraction. To improve the practical applicability of this approach, many recent CBM methods have leveraged the power of pretrained models, reducing the amount of required annotations and prior knowledge (Oikarinen et al., 2023; Yang et al., 2023; Bhalla et al., 2024; Yamaguchi et al., 2025).
While such recent advances have enhanced CBMs, their deployment has thus far been largely confined simpler tasks, such single-label image classification. One reason for this limitation is that CBMs typically extract concepts from an entire image, leading to a single. holistic encoding. However, addressing more complex visual reasoning tasks (e.g., as illustrated Fig. 1) requires shift towards object- based processing. Objects provide a natural and intuitive abstraction for visual reasoning, as decomposing complex scenes into discrete, meaningful components simplifies the analysis of structure and relations. Importantly, object-centric representations also enhance transparency by supporting more detailed, human-aligned explanations.
Despite this, object-centricity has largely been overlooked in recent CBM research, particularly in natural image domains. While earlier concept-based models working with synthetic data have successfully integrated object-centric representations (Stammer et al., 2021, 2024b; Delfosse et al., 2024), most SOTA CBM approaches for natural images continue to rely on image-level encodings.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we aim to bridge this gap by integrating the advantages of object-centric modeling into CBMs, with focus on models that can handle complex, natu- ral images while maintaining interpretability through structured, object-level representations.
Image-level Concept Reasoning
of concepts:
helmet gloves person wheels license ".
To this end. introduce Object-Centric Concept Bottlenecks (OCB), In this framework, an object pro- posal module first identifies relevant objects in an image, which are then represented by high-level con- cepts from concept discovery module. For the final task prediction. concept activations from both the original image and individual objects are aggregated and passed through linear prediction layer. Follow- ing recent trends in interpretable-by -byaasss------ign research, OCB utilizes pretrained models for both object de- tection and concept discovery, thereby reducing the amount of required human supervision. As the addi- tion of object-level representations expands the con- cept space, effectively aggregating these becomes new challenge. Thus, our evaluations include a de- tailed analysis of the critical components of OCB. We conduct experiments across multiple datasets, includ- ing our newly introduced COCOLogic benchmark, based on MSCOCO (Lin et al., 2014), which requires models to classify images according to concepts defined by logical combinations of objects. Importantly, we also evaluate CBMs via OCB for the first time on multi-label image classification benchmarks.
This image depicts "biker"
Object-level Concept Reasoning
Because of concepts:
wheels
gloves
person
person
rearlight
wheels
handlebars
license
00
This image
depicts
"tw bikers"
Figure 1: Reasoning and explaining on the object-level requires object representations.
Overall, our work shows that Object-Centric Concept Bottlenecks represents important step towards more competitive, yet interpretable Al models, with particular focus on object-centricity and its application to CBMs for natural images. Our contributions can be summarised as:
(i) Introducing object-level representations into the framework of CBMs.
(ii) Extending concept-based based models for the first time to more complex classification settings, including multi-label classification.
(iii) Providing detailed performance analysis of OCB's components, e.g., aggregation strategies.
(iv) Introducing COCOLogic, a new benchmark dataset for complex object-based classifications real-world images.
The remainder of the paper is structured as follows. We begin with review of related work, highlighting recent developments in the field. We then introduce our proposed framework, Object- Centric Concept Bottlenecks, and present its formal description. This is followed by comprehensive experimental evaluation, including several ablation studies that examine key components of the framework. Finally, discuss findings and conclude the paper.
2 Related Work
Concept-Based Models. The introduction of Concept Bottleneck Models (CBMs) (Koh al., 2020) marked an important moment in the growing interest concept-based models (cf. (Yeh et al., 2021; Fel et al., 2023) for overviews concept-based explainabilityy. Their appeal lies in promise of interpretable predictions and structured interface for human interaction. While the original CBM framework relied on fully supervised concept annotations, subsequent research has relaxed this requirement by introducing unsupervised concepts (Sawada and Nakamura, 2022), leveraging pretrained lision-languaa models like CLIP for concept extraction (Bhalla et al., 2024; Yang et al., 2023; Oikarinen et al., 2023; Panousis et al., 2024; Yamaguchi et al., 2025), or employing fully unsupervised concept discovery methods (Stammer et al., 2024b; Schrodi et al., 2024; Schut et al., 2025). Efforts have also focused on improving the bottleneck interface by mitigating concept
'Code and data available https://gitub..com/Dii htt.eeee//////// Centre Centric-Coneenne ConcepptBottleeeecsss Bottlenecks/

---

leakage (Havasi et al., 2022) and enabling dynamic expansion of the concept space (Shang et al., 2024). Despite this progress, prior work has not explored the potential of explicit object-centric representations. Although some studies have incorporated such representations into bottleneck models (Yi al., 2018; Stammer et al., 2021; Wüst et al., 2024; Delfosse et al., 2024), these approaches have thus far been limited to synthetic data. In contrast, our work introduces object-centric bottlenecks for real-world image data. While Prasse et al. (2025) utilizes foundation models to extract an object-based concept bank from images, they compute single concept activation vector per image, in contrast to OCB, which explicitly combines image and object-level concept activations.
Object-Centris Representations. Object-centric representations (decomposing scenes into discrete, object- based components) have emerged as a powerful inductive bias, facilitating compositional reasoning, transferabilityy and sample-efficien learning across domains such as robotics (Shi et al., 2024), video understanding (Tang et al., 2025), and vision-language tasks (Assouel et al., 2025; Didolkar et al., 2025; Karimi-Mamaghan et al., 2025). Early slot-based approaches demonstrated the promise of object-centric learning in synthetic settings (Eslami al., 2016; Burgess et al., 2019; Lin et al., 2020; Greff et al., 2019; Locatello et al., 2020), while region-based models, such as the R-CNN family (Girshick, 2015; He et al., 2017), established object-level reasoning through supervised detec- tion and segmentation. More recent developments, including GENESIS-V2 (Engeleke et al., 2021) and DINOSAUR (Seitzer et al., 2023), have extended slot-based decomposition to natural images, making object-centricity increasingly practical for real-world data. Object-centric learning also remains of interest in the context of foundation models, e.g., regarding compositional generalization. For example, object-centric models outperform foundation model baselines in compositional visual question answering tasks Kapl et al. (2025); Carion et al. (2020). Furthermore, recent large-scale models such as SAM (Kirillov et al., 2023) and DeiSAM (Shindo al., 2024) adopt object-level interfaces, underlining the growing role of structured, object-aware representations in scalable Al sys- tems. Following this trend, OCB integrates object-centric representations into interpretable-by-eett models, aiming to improve both predictive performance and model transparency.
Utilizing Pre-trained Models for Interpretability. A growing body of work investigates how large pretrained models, such as vision- laiguaee models (VLMs) and large language models (LLMs), can be leveraged to enhance interpretability,, particularly through concept bottlenecks. Many recent approaches incorporate features from pretrained models into inherently interpretable architectures to avoid the need for manual concept supervision. For instance, Yang et al. (2023) and Rao al. (2024) extract concepts from language models or detection backbones, while Ismail et al. (2024) introduce CB-pLMs for controllable protein design. In natural language processing, Tan et al. (2024) retrofit LLMs with lightweight bottlenecks, and Sun et al. (2024) scale this to classification and generation with competitive performance and built-in interpretability. Further, Chen et al. (2025) improve self-explanations by aligning internal LLM representations with explicit concept subspaces. In line with these works, OCB utilizes pretrained models build object-centric concept representations.
3 Object-Centric Concept Bottlenecks (OCB)
In this section, we introduce our novel framework Object-Centric Concept Bottlenecks (OCB). It consists of three different components: (I) an object proposal module that detects and crops relevant objects from an image, (II) concept discovery module that transforms images into a set of human- understandable concepts and (III) a predictor module that solves the task based on theses concepts in an interpretable fashion. With that, OCB provides a competitive and interpretable architecture to handle complex visual tasks an object-centric way. Before going into the details of the individual components, we first introduce the necessary background notation.
Background. To build a general concept- based model, let us assume we have access to some image data X' E R-Nx NDD which consists of N images of dimension D. Additionally, we have some labels y E IR N for example, class or category labels for multiclass or multilabel classification. We want to develop a concept-based model ff that predicts the labels given the input f X - V, with f()) = ğ E M for a given input x. For an interpretable prediction, CBMs commonly split into two stages: predicting human- unnnannnndssttddaalle concepts from the input and then performing the task based on these concepts. Thus, a discovery module h X + predicts the presence of high-level concepts C E N.C from the input with a concept space of dimension C. Then, an interpretable classifier (e.g., a linear model) g : C > y' predicts the task labels based on the concept activations from h. Together, g and h represent the concept- based model f(z) g(h(x)).
3

---

Full Image
(II
(II)
Crop +
h
9
①
B
B
Cat
*Petting Zoo*
Concept Discovery
Aggregate
Predict
Object Proposals
Proposal
Refinement
Figure 2: Object-Centric Concept Bottlenecks combine object-centric representations with concept- based modeling three-stage pipeline: (I) An object proposal module identifies and refines object candidates within an image. (II) A concept discovery module encodes the entire image and its object crops into human-understandable concept activations.
(III) These activations are aggregated and passed to simple, interpretable predictor to generate the final output. This architecture enables interpretable, object-aware reasoning for complex visual tasks.
From Images to Objects. To incorporate object representations into concept-based models, OCB first identifies relevant objects within an image (see (I) in Fig. 2). This is achieved using an object proposal model o: X' - (B, S), which maps an input image x to a set of bounding boxes and associated certainty scores S, indicating the confidence that each box contains a meaningful object. Since may generate numerous proposals of varying quality, OCB filters these object proposals to remove noisy object proposals. Bounding boxes that are too small (low resolution and less likely to carry meaningful concepts) or too large (too similar to the full image) are removed based on configurable size thresholds Emin and tmaz. Additionally, boxes with certainty scores below the threshold feer discarded. As many abject-proposia models have a tendency to generate multiple bounding boxes for the same object, all object proposals with high Intersection over Union (IoU)= > tou filtered out, inspired by non- mon-maximum suppression (Neubeck and Van Gool, 2006). For this, OCB first orders the abject-proposale according to their certainty scores and removes all bounding boxes that have an ToU higher than /1o with any remaining bounding boxes of a higher certainty (presented here as (b, B) lott). Lastly, restrict the number of object proposals to maximum of bounding boxes, which can, for example, be set based on the complexity of the task and the images (cf. full filtering pscudocode in Alg. in the appendix). Based on this filtering process, we obtain a refined set of object proposals:
B, {bl(b, s) o(x) A Emin S sbze(b) tmax A 2 teer A loU B) S too I. k (I) where size(b) represents the total size (in pixels) of the bounding box b. By applying these steps to refine the set of object proposals, OCB avoids adding too many or noisy object proposals, which, in turn, would lead to noisy and uninformative concepts in the following steps.
From Objects to Concepts. In the next step. OCB generates the concept activations from the input image and the refined object proposals B,. Given an image E RDD, the concept discovery module h(1) c; E RCC maps that image into vector of C concept activations. To retain both global and object-centric information, we first generate concepts based on the whole image x, obtaining C₂ h(x). Then, for each object proposal b; E Br, we crop and resize the corresponding region Th, = resize(crople bz)) E IR" and compute its concept activations via Cb = h(Ir,) (see (II) in Fig. 2). Since the number of object proposals can vary across images, pad with zeros as concept activations to ensure fixed number is of object-centric concept representations per image:
[l(Ta,)) if B 2 (10. otherwise
Vi S k co.
(2)
Together, our enriched concept activations for an input image consist of the activations for the whole
image and all object proposals: c = {cr} J {CB} = (Czs cu. cx} E IR (k++IJC By default, the concept space for image-level concepts and object-level concepts is the same. If this is the case, the underlying assumption of OCB is that these concept spaces are sparsely activated, so that not every concept is active (to some extent) for every image and object.
Aggregating Object Concept Encodings. To enable prediction based on the discovered concepts, OCB aggregates concept activations from the full image and object proposals. A simple approach is

---

concatenation, which preserves per-object information but scales linearly with the number of objects k. limiting scalability. Next to (i) concatenating the encodings, we further suggest the use of the aggregation forms (ii) sum, (iii) max, (iv) count, and (v) sum + count. In the following, we use subscripts to indicate the object-proposal (or main image) from which a concept activation is obtained and superseript to denote the index of the concept in the activation space.
The max aggregation takes the element-wise maximum over the concept-activations max(c)' arg max,e _k) d. This maintains the same value range for all activations but loses information about activation strength or number. The sum aggregation maintains the information about the individual activation strengths: sum(c)' = Eje(z,o.... .A cy However, this aggregation still does not have explicit information about the number of times a concept has been activated, and thus about counts of objects in the image. The count aggregation is explicitly suited for situations where numbers of objects are important: count(c)' = _)E(,,0,--..) k) Ic))), where I(a) equals 1 if a o 0 and 0 otherwise. Compared to concat, all these aggregations have the advantage that the output space remains RC, independent of the number of objects that concept activations computed for. However, as all of these aggregations also lose some kind of information, the final aggregation method we introduce is sum + count. This aggregation combines sum and count: sum + count(c) = (sum(c)', count(c)', with an output space of IR2C. This aggregation retains information about the concept activation strength as well as explicitly keeping track of concept activation counts, while still having an output space that is constant in size relative to the number of objects. We provide evaluations on the different forms of aggregations in Sec. 5.
From Object Concepts to Decisions. After the concept activations from the original image and the object proposals have been combined with one of the aggregations above, the last step is to compute a final prediction based on c ((III) in Fig. 2). This is done with a predictor g(+) where we follow the typical setup of concept-bottle bottleneck models and use a linear layer predictor due to its inherent interpretability. The input space of g is either RC (for max, sum and count), R2C (for sum + count), or R (R++1]C (for concat), depending on the aggregation method. Overall, this leads to the joint processing of OCB shown in Fig. 2: Finding and refining object proposals (I), computing and aggregating concept activations (II) and finally predicting the task output (III).
Utilizing Pretrained Models. OCB relies on two core components: the object proposal model and the concept discovery module h. While both can be trained from scratch, doing so typically requires extensive annotations. Instead, OCB leverages pre-trained models for both, eliminating the need for costly supervision. In this setup, only the predictor network g is trained, while o and remain fixed.
4
The COCOLogic Dataset
Pair of Pets
Previous CBM research has primarily focused on simple single-label classification tasks, often using datasets where success depends largely on detecting the presence of a specific object type. In contrast, our work extends the evaluation of CBMs to both more complex multi-label settings and, for the first time, to a novel single-label classification task that demands richer reasoning capabilities on natural images. For this, we introduce COCOLogic, new benchmark based on the MSCOCO dataset (Lin et al., 2014), that is designed to evaluate a model's ability to perform structured visual reasoning in a single-label classification context.
bid
Animal Meets Traffic
COCOLogic is intended as a proxy task for structured reason- ing in vision, capturing aspects such as multi-object compo- sition, negation, and hierarchical categories while still being grounded in real-world data. Concretely, COCOLogic re- quires models to classify images based on high-level semantic concepts defined by logical combinations of objects. These concepts incorporate disjunctions, conjunctions, negations, and counting constraints over detected object categories, far beyond simple object presence. Each image is assigned one
STEO2
Figure 3: Two examples from the COCOLogic dataset with relevant ob- jects for class decision.
5

---

of ten mutually exclusive labels, determined by logical rule applied to COCO object annotations. Two examples are shown in Fig. (cf. Sec. for more details).
The dataset construction process ensures exclusive labeling, such that each image satisfies exactly one class definition. This setting allows for controlled comparison of model expressiveness on complex visual reasoning tasks, while still formulated in the context of single-label classification. COCOLogie thus serves as bridge between logical reasoning and visual understanding within simple training setup: it remains grounded in realistic visual scenes while being inherently challenging from a logical reasoning perspective.
5 Experimental Evaluations
In our evaluations, we investigate the potential of object representations for performant, interpretable decision making via our novel OCB framework. We investigate the overall predictive performance, analyse the framework's individual components, and conduct qualitative inspection of OCB's object-centric concept space. The evaluations are guided by the following research questions:
(RQ1) Do object-centric representations allow for more competitive CBMs?
(RQ2) How impactful is the choice of object extractor?
(RQ3) Which aggregation strategy best supports task performance?
(RQ4) How much does the number of objects affect OCB's performance?
Datasets. We evaluate OCB on both single- and multi-label image classification tasks. For multi- label settings, we use PASCAL-VOC (Everingham et al., 2010) and MSCOCO (Lin al., 2014), distinguishing between low-level categories (COCO()) and high-level super-categories (COCO(h)). For single-label classification, we use SUN397 (Xiao al., 2010, 2016) and our novel COCOLogic dataset (see Sec. 4).
Metrics.
When comparing downstream performance of the investigated models on the single-label datasets, we follow recent work and report test accuracy (balanced accuracy for COCOLogic due to its strong class imbalance). For the multi-label datasets, we report mean Average Precision (mAP). In all experiments, we report average and standard deviation over 5 seeds.
Models. For evaluations, do not rely any concept object-level supervision but leverage the potential of pretrained models for instantiating the different components of OCB. For the object- proposal model we explore the use of the "narrow- pairroospprpse" pretrained model MaskRCNN (He et al., 2017) (denoted as RCNN), which has been trained to detect objects of the COCO (Lin et al., 2014) dataset. As second, general-purposss 2023), which has been pretained on SA-1B and, in principle, allows to segment arbitrary images. To instantiate the concept discovery module h, is principle possible use any of the recent pretrained CBM approaches. We revert to the practical and quite general SpLiCE (Bhalla et al.. 2024) approach, in particular due its task-independent concept vocabulary. Moreover, SpLiCE can be seen as generalized framework that subsumes a broad family of recent concept bottleneck models that leverage pretrained vinion-language representations such as CLIP (Yang et al., 2023; Oikarinen et al., 2023; Panousis et al., 2024; Yamaguchi et al., 2025). This generality allows SpLiCE to serve as a representative instantiation for studying concept-bass based models in a unified framework. Additionally, SpLiCE generates sparse concept activations, which aid understandability and thus downstream interpretability.
proposal model, we revert to SAM (Kirillov et al.,
We first evaluate a non-interpretable model that maps CLIP image embeddings to class predictions via a two-layer MLP, serving as an upper bound on performance using CLIP features alone. For interpretable basclines, we use a SpLiCE-based CBM, where a linear classifier predicts multiclass or multilabel outputs from whole-image SpLiCE encodings. SpLiCE's inherently sparse concept space aids interpretability despite its large vocabulary. Our method (OCB) generates concepts from multiple object proposals, yielding slightly less sparse concept space than the base CBM. To test if this increased capacity drives performance gains, we include a CBM (equal capacity) variant with reduced SpLiCE sparsity regularization to match OCB 's number of active concepts. We also compare to the Coarse-to-Fine CBM (C2F-CBM) (Panousis et al., 2024), which uses hierarchical, patch-based concepts. For fairness, we adopt the same LAION- vocabulary as OCB and disable OCB sparsity constraints. Since C2F-CBM supports only single-label classification, it is evaluated

---

Table OCB shows improved task performance even for complex tasks. OCB achieves superior
performance compared to non-object-centrie CBMs (based on SpLiCE) with both object-propose generators, even compared to an equal concept capacity CBM. The best ("*") and runner-up ("o") results are bold.
Multi-label COCO(h)
Single-label
Model
PASCAL-VOC
COCO()
SUN397
COCOLogic
CLIP+MLP
89.60 0.05
89.67= 0.03
68.20 0.13
79.62:
65.95# 1.00
CBM
82.42+ 0:1
84.73= 0.00
59.19 0.00
74.79
58.84+ 0.9
CBM (equal capacity)
82.90+ 0.01
86.31 0.00
61.70
74.85+ 0.01
60.01 0.09
OCB (RCNN)
85. 75 0.01
.7. a 0.00
64.12 0.00
77..88 0.04
688.84 0.11
OCB (SAM)
+84.37 0.01
c86.91 0.01
o63.33 0.00
o75.13 = 0.02
c62 12.42:22 :: 0.10
on SUN397 and COCOLogic. Lastly, we compare to opaque model that maps CLIP image embeddings directly to class predictions via a two-layer MLP (CLIP+MLP).
5.1 Enhancing Task Performance via Objects (RQ1).
To investigate OCB's general predictive performance, we compare the performance of CBMs with and without objects on the five different datasets. As the results in Tab. 1 show, the performance of OCB is superior to the non-object-centrr CBM over all datasets, i.e., both across the multi-label well as single-label settings (we hereby focus on the OCB (RCNN) results and compare these to OCB (SAM) in the next subsection). Interestingly, OCB outperforms the base CBM even on SUN397, despite that dataset consisting of many classes like "sky", "desert sand", or "mountain", " where an object-centrie approach intuitively would not provide many benefits. However, the performance increase is rather small compared to the improvement on the more object-based tasks. The largest improvement can be seen on COCOLogic, which requires object-based visual reasoning. For COCO(), observe a large boost via object representations, indicating that OCB successfully extends the concept space by information about specific objects in the image. To better contextualize our results, we also compare against a non non-interpretable upper bound that uses the same image backbone as our models (CLIP+MLP). We observe that CLIP+MLP achieves higher accuracy, particularly than vanilla concept-based approaches, at the cost of offering meaningful explanations for its predictions. However, our object-centric CBM (OCB) substantially narrows this gap: by leveraging structured object-level concept representations, OCB recovers much of the predictive power of the opaque model while retaining transparency.
also
As Tab. indicates that more concept capacity improves the baseline performance, we compare base CBM and OCB with different concept capacities (i.e., the average number of non-zero concepts, cf. Fig. 8 in the appendix). Overall, increasing concept capacity improves the performance of the baseline and OCB, but on all object- based tasks, adding object-level concepts is consistently superior to adding more image-level concepts, again indicating that OCB allows for more competitive CBMss
Lastly, in Tab. 2 we evaluate against the recent Coarse- to-Fine CBM (Panousis et al., 2024), which intro- duces hierarchical, patch-based image processing. As C2F does not support multi-label classification, we perform these evaluations on the SUN397 and CO- COLogic datasets and, for fair comparisons, utilise a common LLION-based concept vocabulary with no sparsity constraints. We observe that our OCB model consistently outperforms C2F under these conditions, highlighting the benefit of object-centric representa- tions over patch-based hierarchies.
Table 2: Comparison between patch vs ob- ject encodings. Results for SUN397 and CO- COLogic datasets. Best results are in bold.
Model
SUN397
COCOLogic
C2F-CBM
70. 70.12 0.24
05.81 1.77
OCB (non-sparse)
89.60
68.63.100 0.42
5.2 Impact of the Object-Proposa. Model (RQ2).
As the object-proposal model is an important component of OCB, we compare two different instanti- ations of this component. Mask-RCNN (He et al., 2017) (RCNN) is pretrained on a rather narrow and specific domain, while Segment Anything (Kirillov et al., 2023) (SAM) can be considered
7

---

85
80
Performance
65
60
PASCAL-VOC
COCOBN)
COCO(I)
SUN397
CoCO-Logic
Geometric Mean
max
sum count
Figure 4: Different datasets benefit from different aggregation methods. While the performance comparison of different aggregation strategies with OCB (RCNN) and k = 7 does not show a one- fits-all choice, max sum always solid choices.
general-purpose object-proposse model. The results in Tab. 1 show that OCB outperforms the CBM baselines with either object detector. Additionally, we observe that the performance of in-domain object-proposal models surpasses the performance of the general-purpose object-proposal model. In particular, on COCOLogic, which requires the detection of only several specific object classes, the performance OCB (RCNN) shines. Interestingly, the performance of OCB (RCNN) even surpasses OCB (SAM) on SUN397, which is mostly out-of-domain for -Ack- However, the difference between the two models minimal, which can be explained by the general lack of object-centricity of SUN397. Overall, the results show that access to an in-domain object-proposal model can be beneficial for OCB, using a general general-purpose purpose model instead also leads good improvements.
5.3 Impact of Aggregation Mechanism (RQ3).
second key element of OCB is the aggregation of the object-concepts and the image-level concepts. To investigate the impact of the different aggregation methods, we compare the task performance of OCB using Busk-RCNN as object encoder and a maximum number of objects 1: The results can be found in Fig. 4, with detailed numerical values in Sec. E. We observe that the concatenation of concept encodings generally does worse than sum max. This is most likely due the increased size of the concept space via concatenation (vocab size x num objects), and thus the input to the predictor network, making training the linear layer less effective. max and sum, on the other hand, perform quite similarly for all datasets. While max performs slightly better on the datasets that only require general information about the presence or absence of objects, sum has slight advantage when object counts are also relevant. count and sum + count also perform very similarly, with the latter being consistently bit better than the former, indicating that the additional continuous concept activations provide only small benefit over the discrete counting information. On SUN397 and COCOLogic, the more discrete nature of count leads to overfitting of the predictor and, in turn, to lower performance. On the other hand, for PASCAL-VOC, this property leads to substantial increase in performance over the other baselines.
Although max aggregation leads to the best overall performance, the results indicate that there is no one-fits-all choice the aggregation method. sum and max perform generally well, while sum count can have the potential for good performance, but with the caveat of more difficult training of the predictor network. Additionally, it noteworthy that concat retains full traceability between objects and concepts for model explanations, which might outweigh the smaller performance gain depending on the application (cf. discussion in Sec. 6).
5.4 Impact of the Object Proposal Refinement (RQ4).
Let us now turn the influence of the number of object proposals for OCB. The most influential element of the refinement is the limitation of the number of objects to the top k. We investigate the impact of this parameter on the performance
OCB in Fig. 5, where the best-performing aggregation method is shown for each dataset. In general, we observe trend indicating that adding more objects improves performance, in particular for PASCAL-VOC and COCO(h). On the other hand, performance beyond certain point degrades again for COCOLogie and SUN397.

---

Due to the structure of COCOLogic, only the recognition of few key objects is necessary for correct classification, while several classes re- quire category counts or exclude certain categories. Thus, additional low-quality proposals, such as faulty or redundant detections, can lead to in- correct predictions. A similar effect occurs on the scene-centric SUN397, where extra object
85
Performance (%)
80
PASCAL-VOC
75
CO
COCO(I)
70
SUN397
COCO-Logic
65
crops add noise rather than informative concepts. We also investigate the effect fmin- the threshold limiting the minimum object size in Tab. 10 in the appendix. Here, it can be observed that a lower threshold generally increases performance, with the exception of SUN397, where the scene- centric nature of this dataset does not benefit from the detection of small objects. In summary, al- lowing more object proposals and the detection of smaller objects generally improves on object- centric datasets, careful filtering is essential mitigate noise in tasks sensitive to false positives.
0
2
7
Number of Objects
Figure 5: Adding more object proposals im- proves performance on object- tasks. However, for complex tasks like COCOLogic, adding to many (noisy) object proposals can re- sult in reduced performance. OCB (RCNN) per- formance for different values k.
5.5 Interpreting Object-centric Concepts.
In the previous sections, we investigated whether the addition of object-centric concepts to CBMs can increase task performance. However, another important aspect of CBMs is their inherent inter- pretability via inspections the concept space. Indeed OCB's concept space also allows for more fine-grained and modular analysis of these concepts. As each object proposal is directly asso- ciated with the discovered concepts, is possible provide conceptual description for each found object. Fig. 6 (top) shows an example where the object-centric concepts enrich the conceptual ex- planation of the image via previously undiscov- ered concepts, like the presence of fork wine glass. In contrast, Fig. 6 (bottom) highlights that the object-centric concepts do not always have extend the concept space by additional concepts. Even if the main concept (Pelican) is already iden- tified in the full image encoding, the object-centric explanations reveal that there is not just one, but two pelicans in the image. Overall, the object-
Image
Object
Object
Object
dnner, broccoli. seafood.
Wine, label, martini, patent, champagne
plated, dinner, cutlery, rhinestone
Kale, dinner, recipe, broccoli, asperagus
vegetarien lan
Full Image
Object #1
Object #2
pelican, Swarts scassape.
pelican, heron, mer,
pelican, heran, mar. swimming
joined
Figure 6: Object-centric concept representa- tions allow for a more fine-grained and mod- ular concept space. Object-centric representa- tions include previously undiscovered concepts (top) provide detailed information about ob- ject counts (bottom).
centric concept space allows for a more fine-grained representation of the image, also including information like object counts, which is an important basis for object-centric explanations.
Discussion
Overall, observe that integrating object representations into concept bottlenecks via OCB provides substantial performance improvements, but also explanatory value. However, there are few points to take into consideration, which we wish to discuss here.
The aggregation step in OCB enables concept representations that are both fixed in size, regardless of the number of detected objects, and invariant to the order in which objects are detected. However, this comes at potential cost: the ability to clearly map concepts back individual objects may be lost. If two objects share overlapping concept representations, it becomes unclear which object a concept refers to. This limitation especially pronounced when using aggregation methods like sum, max, count, and sum count. In contrast, the concat aggregation preserves object-level associations

---

but suffers from linearly growing encoding space with increasing number of objects. Overall, this highlights a trade-off between producing a compact concept representation and maintaining direct object identifiability; an important consideration in the context of object-level traceability.
To find suitable object proposals and extract meaningful concepts from them, OCB utilizes pre- trained models to avoid annotations and additional training costs, While this is a big ad advantage, the performance of OCB is also dependent on the performance of these pretrained models. As shown in our experimental evaluations, the utilization of SAM as a general-purpose object proposal model less performant compared more specific models like RCNN. Similarly, the concept discovery module in our evaluations is based on SpLiCE, and thus on CLIP. While this approach can provide reasonable concept descriptions for full image, cropping the image to an object proposal sometimes leads to noisy concepts, in particular if the object is small. While exploring other concept discovery models or other ways of encoding the image encodings with CLIP (ie., via pointing (Shtedritski et al., 2023)) could lead to more robust concept discovery even for smaller objects, the overall performance remains bound by CLIP. In this context, some recent studies have raised concerns about the uneritical use of context concept-base -aaed models for interpretability. Debole et al. (2025) find that VLM-supervised CBMs can produce low-quality concepts with poor alignment to expert annotations, even when achieving strong downstream performance. Other works (Marconato et al., 2023; Bortolotti et al., 2025) raise concerns regarding shortcut learning in concept- models. While OCB's ss more modular and object-centric concept space does not mitigate these limitations, it allows for more detailed control and evaluation of the learned concepts.
Lastly, we wish to highlight the importance of the global image encoding in overall task performance. As shown in an ablation study (Tab. 4), removing the image-level representation leads to substantial drop in accuracy on Pascal VOC, despite the presence of rich object-centric features. These results reinforce a central message of our work: both object-centric and global features are necessary to achieve high predictive performance in complex visual reasoning tasks.
7 Conclusion
In this work, we show the importance of object- encodings in the context of recent interpretable, concept-based models. The strength and novelty of our novel Object-Centric Concept Bottlenecks lie in making CBMs more structured, scalable, and interpretable by rethinking their core repre- sentation; from flat, image-level representations to grounded, object-level ones without requiring custom supervision or architectures. Our experimental evaluations show that this allows to solve more complex object-driven tasks such as multi-label classification and single-label object-level reasoning on our novel COCOLogic benchmark. Future avenues moving forward include investi- gating relational representations (Wüst et al., 2024: Delfosse al., 2024), but also the potential of object-centric representations in other forms of interpretability research (Wäldchen et al., 2024), self- refinement (Stammer et al., 2024a) and more complex visual tasks such as visual question answering.
Acknowledgments
This work was supported by the "ML.2MT" project from the Volkswagen Stiftung, the Priority Pro- gram (SPP) 2422 in the subproject "Optimization of active surface design of high-speed progressive tools using machine and deep learning algorithms" funded by the German Research Foundation (DFG) and supported by the DFG under Germany's Excellence Strategy (EXC 3066/1 "The Adaptive Mind". Project No. 533717223). It has further benefited from the HMWK projects "The Third Wave of Artificial Intelligence 3AI", and Hessian.AI, the Hessian research priority program LOEWE within the project WhiteBox, the EU-funded "TANGO" project (EU Horizon 2023, GA No 57100431), and from early stages of the Cluster of Excellence "Reasonable AI" funded by the German Research Foundation (DFG) under Germany's 's Excellence Strategy- EXC-3057; funding will begin in 2026.
10

---

References
Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, and Adriana Romero-Sorinn Object- centric binding in contrastive language-image pretraining. CoRR, abs/2502. 14113, 2025.
Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Information Processing Systems (NeurIPS), pages -099888-428. 2024.
Samuele Bortolotti, Emanuele Marconato, Paolo Morettin, Andrea Passerini, and Stefano Teso. Shortcuts and identifiability in concept-based models from neuro-symbolic lens. CoRR, abs/2502. 1245, 2025.
Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. CoRR, abs/1901. 1390, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision (ECCV), pages 213-229, 2020.
Aditya Chattopadhyay, Kwan Ho Ryan Chan, Benjamin David Haeffele, Donald Geman, and Rene Vidal. Variational information pursuit for interpretable predictions. In International Conference Learning Representations (ICLR), 2023a.
Aditya Chattopadhyay, Stewart Slocum, Benjamin D Haeffele, Rene Vidal, and Donald Geman. Interpretable by design: Learning predictors by composing interpretable queries. Transactions on Pattern Analysis & Machine Intelligence, ((100)):430-7443) 2023b.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Guanxu Chen, Dongrui Liu, Tao Luo, and Jing Shao. SEER: self-explainability enhancement of large language models' representations. CoRR, abs/2502.05242, 2025.
Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passeggini, Stefano Teso, and Emanuele Marconato. If concept bottlenecks are the question, foundation models the answer? CoRR, abs/2504.19974, 2025.
Quentin Delfosse, Sebastian Sztwiertnia, Mark Rothermel, Wolfgang Stammer, and Kristian Kersting. Interpretable concept bottlenecks to align reinforcement learning agents. Advances in Neural Information Processing Systems (NeurIPS), pages 06826-66855 2024.
Aniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, and Aishwarya Agrawal. Ctrl-o: sanguage-controlalee object-centric visual representation learning. CoRR, abs/2503.2 2025.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. HENESIS-V2. inferring unordered object representations without iterative refinement. In Advances in Neural Information Processing Systems (NeurIPS), pages 8085-8094 2021.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Mark Everingham, Luc Van Gool, Christopher K. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, pages 303-338, 2010.
Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation, In Advances in Neural Information Processing Systems (NeurIPS), 2023.
11

---

Ross B. Girshick. Fast R-CNN. In International Conference on Computer Vision (ICCV), pages 1440-1448, 2015.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning (ICML), 2019.
Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems (Neur/PSS,
2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
Aya Abdelsalam Ismail, Tuomas Oikarinen, Amy Wang. Julius Adebayo, Samuel Stanton, Taylor Joren, Joseph Kleinhenz, Allen Goodman, Héctor Corrada Bravo, Kyunghyun Cho, and Nathan C. Frey. Concept bottleneck language models for protein design. CoRR, (tbs/2411.0600 2024.
Ferdinand Kapl, Amir Mohammad Karimi Mamaghan, Max Horn, Carsten Marr, Stefan Bauer, and
Andrea Dittadi. Object-centric representations generalize better compositionally with less compute. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling, 2025.
Amir Mohammad Sarimi-Mamaghan Samuele Papa, Karl Henrik Johansson, Stefan Bauer, and Andrea Dittadi. Exploring the effectiveness of object-centric representations in visual question answering: Comparative insights with foundation models. International Conference on Learning Representations (ICLR), 2025.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In International Conference on Computer Vision (ICCV), pages 3992-4003, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning (ICML), pages 5338-5348, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations (ICIR), 2020.
Francesco Locatello, Dirk Weissenborn Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini. Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In International Conference on Pattern Recognition (ICPR), pages 850-855, 2006.
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. CoRR, abs/2304.061 29, 2023.
Konstantinos Panousis, Dino Ienco, and Diego Marcos. Coarse-to-fine concept bottleneck models. Advances in Neural Information Processing Systems (NeurIPS), pages 105171-105199 2024.
Katharina Prasse, Patrick Knab, Sascha Marton, Christian Bartelt, and Margret Keuper. Dcbm: Data-efficient visual concept bottleneck models. International Conference on Machine Learning (ICML), 2025.
12

---

Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover sescover-then-eee name: Task-agnostic concept bottlenecks via automated concept discovery. In European Conference on Computer Vision (ECCV), pages 441-461, 2024.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. CoRR, abs/2103.11251, 2021.
Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758 41765, 2022.
Simon Schrodi, Julian Schur, Max Argus, and Thomas Brox. Concept bottleneck models without predefined concepts. CoRR, abs/2407.03921. 2024.
Lisa Schut, Nenad Tomašev, Thomas McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-ai knowledge gap through concept discovery and transfer in alphazero. Proceedings of the National Academy of Sciences, (22(13))e2406675522 2025.
Maximilian Seitzer, Max Horn. Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabrin Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In International Conference on Learning Representations (ICLR), 2023.
Chenming Shang. Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. In- cremental residual concept bottleneck models. Conference on Computer Vision and Puttern Recognition (CVPR), pages 1030-110400 2024.
Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre-trained object- centric representations for robotics from "what" and "where" foundation models. In International Conference Roboties and Automation (ICRA), pages 15424- 15432, 2024.
Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Devendra Singh Dhami, Patrick Schramowski, and Kristian Kersting. Deisam: Segment anything with deictic prompting. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. In International Conference on Computer Vision (ICCV), pages 11987- 2023.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Conference on Computer Vision and Pattern Recognition CVPR, pages 3619-3629, 2021.
Wolfgang Stammer, Felix Friedrich, David Steinmann, Manuel Brack, Hikaru Shindo, and Kristian Kersting. Learning by self-explaining. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856.
Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder. In Advances in Neural Information Processing Systems (NeurIPS), pages 71792-71830 2024b.
Chung- -nung-En&o Sun, Tuomas P. Oikarinen, Berk Ustun, and Tsui-Wei Weng. Concept bottleneck large language models. CoRR, abs/2412.07992, 2024.
Zhen Tan, Lu Cheng, Song Wang, Bo Yuan, Jundong Li, and Huan Liu. Interpreting pretrained language models via concept bottlenecks. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 56-74, 2024.
Zitian Tang, Shijie Wang. Junho Cho, Jaewook Yoo, and Chen Sun. How can objects help video- language understanding? CoRR, abs/2504.07454, 2025.
Stephan Waldchen, Kartikey Sharma, Berkant Turan, Max Zimmer, and Sebastian Pokutta. Inter- pretability guarantees with Merlin-Arthur classifiers. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1963-1971. PMLR, 2024.
13

---

Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Pix2code: Learning compose neural visual concepts as programs. In Uncertainty in Artificial Intelligence (UAI), pages 3829-3852 2024.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492, 2010.
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119(1): 3-22, 2016.
Shin' ya Yamaguchi, Kosuke Nishida, Daiki Chijiwa, and Yasutoshi Ida. Zero-shot concept bottleneck models. CoRR, abs/2502.09018, 2025.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Gallison-Burche and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 19187-19197, 2023.
Chih- Ych, Been Kim, and Pradeep Ravikumar. Human-centered concept explanations for neural networks. In Neuro-Symbolic Artificial Intelligence: The State of the Art, volume 342 of Frontiers in Artificial Intelligence and Applications, pages 337-352. IOS Press, 2021.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic VQA: disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems (NeurIPS), pages 1039-1050, 2018.
14

---

Supplementary Materials
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The main claim of this paper is that integrating object-centric representations into the concept-bottleneck framework leads to improved performances in downstream visual reasoning tasks. This claim is underscored by our results across several benchmarks, indicating higher performance via our method than non-object-centric baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We highlight the potential limitations of our proposed approach in our Discus- sions section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: We provide details on the evaluation setup (model parameters, training pa- rameters, etc. in the Appendix (Additional Details for Evaluations) and our linked code repository.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
16

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes].
Justification: We provide all scripts and details for working with our novel COCOLogic dataset in our linked code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/guides Seglliccguidee//oddeetmmiiiss CrwoodeSumisssioooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https:
//iiss.co. wttppblcc//aaaetcccccccccccccccc /eee//giieeee / cetmimssionpee ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We provide these in the linked code repository well the Appendix (Additional Details for Evaluations).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]..
Justification: All experiments were conducted using 5 seeds and we report average and standard deviation for all experiments, also all figures.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
We provide this in Additional Details for Evaluations of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tttp:://neurps... cc wwwbllcc/thhcsSuideeee
Answer:
[Yes].
Justification:
All points of the guidelines have been adhered
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We have done in the impact statement section in the appendix.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA].
Our novel dataset only contains a subset of the already public and accessible MSCOCO dataset; therefore, does not require any additional safeguards. Our model is also based on previously published models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We provide references to all assets used in our work, e.g., MSCOCO, Mask- RCNN and SAM.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswitheeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We provide a detailed description of the novel model and dataset throughout the main text and appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpa::///eriipie for what should or should not be described.
21

---

A Impact Statement
This paper introduces Object-Centree Concept Bottlenecks (OCB), novel framework that advances the capabilities of concept-based models (CBMs) by incorporating object-centric representations. By leveraging pretrained object detection and concept discovery modules, OCB enables interpretable, structured, and high-performing Decision-making on complex visual tasks, such as multi-label and logic-based image classification, without requiring extensive manual annotations. Notably, it extends the applicability of CBMs beyond traditional single-label settings, improving both predictive accuracy and explanation granularity.
Ethically, the approach supports greater transparency and control in Al systems by producing explanations tied to discrete objects and their associated concepts. This aligns with responsible Al principles, especially in safety-critice also inherits limitations from the pretrained models builds upon. These models may encode biased or misaligned representations (e.g.. due to skewed training data in foundation models like CLIP or SAM), potentially resulting in misleading culturally insensitive explanations. The framework's reliance on such components emphasizes the need for careful auditing and user-in-the-loop validation in real-world deployments.
applications such as autonomous systems. However, OCB
In summary, OCB presents an important step toward more interpretable and modular visual reasoning systems, while raising critical questions about the trustworthiness and accountability of explanations generated by models grounded in potentially flawed foundation systems.
B Additional Details for OCB
Alg. I outlines the procedure used by OCB to refine the object proposals generated by an object- proposal model. These proposals consist of pairs of bounding boxes and certainty scores. They are first filtered by bounding box size and certainty threshold to discard uninformative or low- confidence candidates. The remaining proposals are sorted by certainty score and filtered based on intersection-over-
(IOU), removing boxes that significantly overlap with higher-scoring proposals. Finally, the number of proposals limited a predefined maximum k.
Algorithm Refine Object Proposals
Require: Image x, object-proposal model o, thresholds fmin (max, leers trou, max proposals k
1: (B, S) + o(x)
2: F F+l
filter by size and certainty
3: for all s:) E (B. do
4: if Imin S. size(h:) S max and 8; 2 beer then
5:
Append (bi, 8i) F
6: end
7: end for
8: Sort F in descending order of score
9: N I]]
remove overlapping boxes
10: for all (bs, 8;) E F do
if for all by E IoU(b,, by) s too then
12:
Append by to N
end if
14: end for
B, <- (nI,".. mk)
D first elements of N
16: return B,
C Additional Details for Evaluations
In our evaluations, we compare OCB against a baseline CBM and evaluated the performance of both on several different datasets. As concept extractor for OCB and the baseline CBM we use SpLiCE (Bhalla et al., 2024). We followed their default settings, for model, vocabulary and 11 I1-regularization. The only exception that is CBM (equal capacity), where we reduced the regularization from 0.25 to 0.2, wheih led to comparable number of non-zero concepts to OCB.
22

---

Ambiguous Pair
Pair of Pets
Rural Animal Scene
Leash Licence
Animal Meets Traffic
Occupied Interior
Empty Seat
Odd Ride Out
Personal Transport
Breakfast Guests
Figure 7: Example images of the ten classes from COCOLogic. The relevant objects for the classes marked green.
OCB also introduces several hyperparameters: While we provide ablations on the number of objects- porposals & and the choice of aggregation, we kept the hyperparameters tmin mas and feer fixed. For OCB (RCNN), imin 0.01, mex 0.85 and beer = 0.2, where the size-related parameters consider the bounding box size relativ to the full image size. For OCB (SAM), miin = 0.02, mmaz 0.85 and feer = 0.94, and we used the stability factor of SAM as certainty score. Additionally, we used grid-like prompting scheme for SAM with 16 points per iamge to generate object-proposales Hoo = 0.5 is kept the same for both object proposal models.
For the predictor network, we used a single linear layer. For its training, we optimized the parameters learning rate and the number of epochs the layer was trained for, using the validation sets. The parameter configurations for each dataset and aggregation can be found in the code repository. All experiments were conducted using T single GPUs from Nvidia DGX2 machines equipped with A100-40G and A A100-80G graphics processing units.
D COCOLogic
The COCOLogic dataset comprises ten semantically rich classes derived from COCO images. Examples of each class are shown in Fig. 7. Each class is defined by specific logical rule, detailed Tab. alongside the number of training and test examples per class. Images are selected such that exactly one class applies per image; any image that does not satisfy exactly one rule is discarded. This ensures that the class labels are mutually exclusive and unambiguous.
These rules are designed to be semantically meaningful while introducing logical complexity that makes the classification task non-linearly separable the input space. Consequently, COCOLogic serves as a challenging benchmark for assessing the representational capacity of both linear and non-linear classifiers, as well as symbolic and neuro- symbolic models. In this work we mainly evaluated models that had linear classifiers (both OCB and the baseline CBM) on COCOLogic. While this showed that incorporating object-level concepts are essential to solve this dataset, utilizing more powerful classifiers is equally as important, as linear classifiers cannot resolve the more complex concept-class relationships.
E
Additional Evaluations
In Tab. 4, we ablate the influence of the whole image encoding versus object encodings in OCB. We observe that removing the image-level encodings from OCB leads to a substantial performance drop (Pascal VOC), confirming that object-centric and global representations provide complementary information. These results support key message of our work: both object-centric and global features necessary for achieving high predictive performance.
23

---

Table 3: COCOLogic class definitions and sample sizes.
Class Name
Class Rule
Samples
Ambiguous Pairs
(cat XOR dog) AND (bicycle XOR motorcycle)
Pair of Pets
Exactly two categories of (cat, dog, bird) are present
56
Rural Animal Scene
At least one of (cow, horse, sheep) AND no person
2965
Leash Licence
dog XOR car
4188
Animal Meets Traffic
At least one of (horse, Cow, sheep) AND at least one of (car, bus, traffic light)
24
Occupied Interior
(couch OR chair) AND at least one person
8252
Empty Seat
(couch OR chair) AND no person
4954
Odd Ride Out
Exactly one category of [bicycle, motorcycle, car, bus)
3570
Personal Transport
person AND (bicycle XOR car)
279
Breakfast Guests
bowl AND at least one of (dog, cat, horse, cow, 169 sheep]
Table 4: Performance comparison on Pascal VOC showing the contribution of object-centric and image-level encodings.
Full (Object Image) Only Image Encodings
Only Object Encodings
VOC
85.75 + 0.01
82.42 + 0.01
77.48 + 0.03
E.1
Analysis of CBM Capacities
As the evaluation table Tab. shows, increas- ing the capacity of CBM generally also im- proves its performance. capacity as the average number of non-zero con- cepts in its concept space. In Fig. 8, we conduct more thorough evaluation of the effect of con- cept capacity on model performance. The results confirm that adding more concept capacity gen- erally also improves performance, with the ex- ception of COCOLogic, where there is dropoff after certain concept capacity. While this trend true both for the base CBM and for OCB, adding more concept capacity via object-level concepts better on object-bacc based task than just increasing the number of image-level concepts and hope that the objects are going to repre- sented better. For COCOLogic, both OCB and the base CBM have dropoff in performance after certain number of concepts in the con- cept space, which could be explained by the additional noisy concepts making this already challenging task more difficult and do not pro- vide benefits anymore.
Base CBM OCB
Moscal COCOIN)
CO0010 SUN
COCoLogic
that, we refer to the
Performance
20
140
183
Average st of Zero Concapts
Figure 8: Adding object-level concepts is gener- ally better than increasing image-level concepts. Comparison of the base CBM and OCB with dif- ferent levels of sparsity regularization (resulting in different number of non-zero concepts). For all object-based tasks, adding object-level results in bigger performance gains than just adding more image-level concepts.
As both models use SpLiCE as a backbone, the exact number of non-zero concepts cannot directly be set. Instead, SpLiCE uses 11-regularization to enforce the sparsity of the concept space. Setting the -e-reuularization strength does result in different average non-zero concepts which is the reason for the differences in Fig. 8.

---

Table 5: VOC Results OCB (RCNN)
Num Objects
3
Sum
84.15 1 0.01
83.87 1 0.01
83.28 1 0.01
82.84 1 0.00
82.98
1 0.01
83.11
0.00
84.28
0.01
84.22 1 0.01
83.77 0.00
83.26 0.00
83.30 1 0.01
83.34
Concat
84.04
0.01
83.27++000
82.75 0.01
82.41
Sum Count
85.07
0.00
85.62
1 0.00
0.00
85.81 1..11.0000
85.67
1 0.01
85.59
0.01
Count
84.96
1 0.00
85.52
1 0.00
85.66
0.00
85.72 1..22:0000 0.01
85.59
1 0.01
85.51
0.01
Table 6: COCOO) results OCB (RCNN)
Num Objects
3
5
10
Sum
60.78 0.00
62.19 + 0.01
62.85
0.00
63.42 +
63.61 + 0.00
63.82
Max
60.90 0.00
62.18
0.03
62.86
0.01
63.52 + 0.00
63.91 + 0.00
64.12
0.00
Concal
61.14 0.00
62.24
0.00
62.82
0.00
6.022
63.39
0.00
63.43
1 0.00
Sum + Count
61.50 + 0.00
62.44
0.03
62.77
0.01
63.15
63.23 + 0.02
63.29 = 0.03
Count
61.30 + 0.00
62.28 + 0.00
62.57
0.01
0.001
63.08 + 0.04
63.
0.02
Table 7: COCO(h) results, OCB (RCNN)
Num Objects
3
5
10
Sum
85.37 + 0.00
86.26 + 0.00
86.65 + 0.00
87.06 +
87.13
0.00
87.18 + 0.00
Max
85.64 + 0.00
86.28
0.00
86.08
0.00
87.15
87.28 + 0.00
47.33:00 0.00
Concat
85.82 0.00
86.45 0.00
86.79
0.00
87.08 + 0.00
87.18
87.19 0.00
Count
86.01 0.00
86.55 + 0.00
86.91
87.17 0.00
87.16 0.00
87.11
Count
85.89 1: 0.00
86.45 1 0.00
86.81
0.00
87.07 1: 0.00
87.06
1 0.00
87.01
0.00
Table 8: COCO-Logic results, OCB (RCNN)
Num Objects
3
Sum
67.77 0.08
67.42 + 0.11
08.84
66.89 + 0.08
65.44
65.40 0.12
Max
64.31 0.99
66.72
+ 0.09
68.67
65.33
65.23
Concat
65.27 + 0.13
65.67 + 0.08
66.32
0.56
63.07 +
62.16
0.14
60.
0.21
Sum Count
67.37 0.07
63.77 0.06
61.83 0.12
61.56
60.16 0.55
61.38
0.78
Count
67.42 4.12 0.13
03.85 + 0.08
61.97 = 0.10
61.57 +
59.82 + 0.02
61.34 = 0.89
Table 9: No one-fits-all choice of the aggregation method. Comparing the performance of different aggregation strategies with OCB (RCNN) and 7.
Aggregation PASCAL-VOC COCO(h)
COCO()
SUN397
COCO-Logie
Geometric Mean
concat
82.60 0.
87.18= 0.00
63.394 oon
74.32+ 0:11
62.16 0.14
73.25
max
83.30 0.01
87.28 0.00
63.91 0.00
74.02 0.02
65.33 0.13
74.18
sum
82. 92.98 0.01
87.15= nno
63.61 000
73.65 u.0
65.44: um
73.99
count
85.59 0.01
87.06= 0.00
63.08
0.04
68.88 0.03
59.82+ 0.02
72.01
count
85.67
87.16=
63.23
69.32 0.02
60.164 0.55
72.25
E.2
Object Number and Aggregation
In this section we provide further results for varying object numbers and aggregation strategies. In Tab. 9, the numerical values for Fig. 4 are reported. Additionally, we have results forPASCAL-VOC in Tab. 5, COCO(I) in Tab. 6, COCO(h) in Tab. 7 and COCOLogic in Tab, 8 for Mask RCNN as object proposal method.
E.3 Minimum Object Size
Next to the aggregation method and the (i.e. the maximum number of potential objects allowed), we investigate here the impact of the parameter fmin on the performance of OCB. This threshold sets the minimum size of object proposal relative to the image size. Tab. 10, we show the performance of OCB on the different datasets when varying the minimum size of an object proposal. Overall, one can see that lowering this minimum size consistently improves performance. The only exception is SUN, where there is slight improvement with higher minimum object size. This
25

---

Multi-label COCO(h)
Single-label COCOLogic
Emin
PASCAL-VOC
COCOa)
SUN397
Baseline
84.73= 0.00
59.19
74.79 0.01
58.84+ 0.09
0.005
85.87+ 0.17
87.53= 0.00
64.34 0.00
75.24
69.31
0.01
85.70= 0.11
87.32= 0.01
64.11 0.00
75.29 0.03
68.76+ 0.07
0.02
85.21
86.93= 0.00
63.45=
75.35
66.20
0.05
84.62:+ 0.04
86.40 0.00
6 0.00
75.61 0.02
67. 67.70. 0.69
Table TO: Investigating the effect of the minimum object size threshold (fmin) the performance of
OCB. All runs are over 5 seeds and with Mask-RCNN and the best performing settings of aggregation and k.
Figure 9: Typical failure cases of the object-proposal generation of OCB. (Left): Multiple object proposals of the same object get generated and are not filtered out. (Right): Irrelevant objects for the context of the task get detected and result in spurious associations, i.c., a human gets associated with the class "snow field" of SUN.
can be explained by the task of SUN scene understanding often does not need small objects, so increasing the minimum object size removes non-important object proposals. While lower values of Emin are generally beneficial, it thus remains important to consider the data and task at hand and decide whether detecting smaller objects helpful.
E,4 Typical Failure Cases of the Object Proposals
While the pre-trained prrect-proposl generation OCB allows for quite a general detection of potential objects without costly training, there also typical failure cases of these detectors.
Sometimes, it can happen that the object proposal model generates multiple proposals from the same object, despite filtering them for low overlapping IOU. In this case, the same object appears in the concept space multiple times, which can lead to incorrect results if the exact number of concepts is relevant (cf. Fig. (left), where multiple proposals of the airplane are generated and not filtered).
some objects are inherently not relevant for the task, detecting them and adding them to the concept space can introduce spurious correlations. For instance, in the 'snow field" class of the SUN dataset, most samples also contain humans, which are detected as objects (cf. Fig, 9, right). This can lead the model to incorrectly associate the presence of humans with the "snow field" label.
26


Reviews:
[{"review_id": "r2N8DJspdm", "reviewer": "Reviewer_gmMc", "summary": "This work proposes a framework that combines concept-based models with pre-trained object-centric models to improve interpretability and performance for complex computer vision tasks. The authors propose three components to implement this: an object proposal module, a concept discovery module, and a predictor module. For implementation, their approach combines pre-trained object detectors (Mask R-CNN/SAM) with concept extractors (SpLiCE/CLIP), applies concept extraction to object crops, and trains linear classifiers on aggregated features. The authors evaluate OCB across multiple datasets and additionally introduce COCOLogic, a new benchmark dataset designed to evaluate the model’s ability to perform complex reasoning in a single-label classification setting.", "questions": "Please try to address my questions in the Weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n* This work tackles a limitation of current concept bottleneck models (not well-suited for complex tasks like multi-label classification due to image-level encoding) by extending to object-level encoding, which allows concept bottleneck models to handle more complex visual reasoning tasks including multi-label classification and logical reasoning visual tasks. \n* The paper proposes three key components (i.e., object proposal, concept discovery, aggregation) and conducts various ablation studies to analyze the effectiveness of each component.\n* The authors also introduces COCOLogic, a new dataset based on MSCOCO designed to evaluate models' ability to perform complex logical reasoning over natural visual content.\n\nWeaknesses\n* The novelty is a bit unclear. For instance, the 'From Image to Object' module appears similar to standard object detection pipelines with basic filtering. \n* The aggregation mechanisms show no clear winner across datasets (e.g., Figure 4). How does this generalize to real-world applications?\n* Given that the proposed approach heavily relies on pretrained models (Mask-RCNN, SpLiCE), and the authors mention in Section 5.4 that low-quality proposals can hurt performance, a more comprehensive analysis, such as showing and analyzing more failure cases, would be more insightful for future work on developing a better aggregation module.", "was_revised": true, "final_justification": "Thanks to the authors for their response! Some of my concerns have been addressed. The additional baselines look like a good enhancement of the main contribution of this work. However, I do feel that a deeper analysis of the bias/error of the dependency models (e.g., CLIP and Mask-RCNN) would benefit future work in this direction. Accordingly, I have increased my score.", "rebuttal": "We thank the reviewer for their time and questions. We provide clarifications for the points that were unclear below.\n\n> Novelty is a bit unclear.\n\nWe acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.\n\nTo the best of our knowledge, this is the first work that operationalizes object-centric concept representations within CBMs and applies them to real-world, multi-label classification tasks. Traditional CBMs rely on global, image-level features and often suffer from spurious or entangled concept associations [1], especially in cluttered scenes. In contrast, our object-centric bottleneck grounds concepts in specific, localized regions via pretrained object detectors, enabling finer-grained, interpretable, and more faithful concept-level reasoning.\n\nWhile this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.\n\nIn summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation—from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.\n\n\n> \"The aggregation methods show no clear winner accross datasets\" \n\nWe are somewhat puzzled the reviewer considers this as a weakness of the paper as it answers a practically relevant question, namely that the specific choice of aggregation method is often not critical. As discussed in Section 5.3, in the absence of dataset-specific knowledge or validation data, simple methods like *sum* or *max* generally perform best. More complex options like *concat* or *sum+count* can help, e.g., in terms of retaining object-mapping. Importantly, we further observe that on all object-based tasks, OCB outperforms the non object-based baseline **independent of the aggregation method** used. \n\n\n> Reliance on pre-trained models\n\nWe agree that the reliance on pretrained models such as Mask-RCNN and SpLiCE introduces potential failure points, especially when their outputs are of low quality (cf. Section 5.4). However, their usage also brings a major advantage: it enables concept extraction at the object level for real-world data without the need for task-specific, large-scale annotation or supervision. This addresses a well-known limitation of prior CBM approaches, which often require carefully curated concept annotations or extensive domain-specific training [2, 3].\n\n\n> More comprehensive analysis\n\nWe agree a failure-case analysis is beneficial and have added such a section to the paper. As we cannot add images and plots here, we describe some of the failure cases of the proposal generation that we encountered. \n\n**Overlapping proposals:** Sometimes it can happen that the object proposal model generates multiple proposals from the same object, despite filtering them for a low overlapping IOU. In this case, the same object appears in the concept space multiple times, which can lead to incorrect results if the exact number of concepts is relevant.\n\n**Detecting irrelevant objects:** If some objects are inherently not relevant for the task, detecting them and adding them to the concept space can introduce spurious correlations. For instance, in the \"plane interior\" class of the SUN dataset, most samples also contain humans, which are detected as objects. This can lead the model to incorrectly associate the presence of humans with the \"plane interior\" label.\n\n\n> Additional baseline results\n\nWe additionally wish to highlight novel baseline and ablation evaluations resulting from other reviews and strengthen the findings of our work.\n\n### 1. Opaque Upper-Bound Model  \nTo quantify the performance potential of our concept encoder backbone (CLIP), we introduce a non-interpretable model that directly maps CLIP image embeddings to class predictions using a non-linear multi-layer perceptron (MLP) (2 layers). It serves as an upper bound on the performance that can be achieved using CLIP features alone. As expected, it outperforms interpretable CBM variants slightly, but comes at the cost of complete opacity, offering no meaningful explanations. OCB closes the performance gap between CBM and the opaque model, underscoring its value for transparent, high-performing concept reasoning.\n\n|               | Pascal-VOC         | COCO(h)           | COCO(l)         | SUN               | COCOLogic         |\n|---------------|--------------------|-------------------|------------------|--------------------|--------------------|\n| CBM | $82.42 \\pm 0.01$ | $84.73 \\pm 0.00$ | $59.19 \\pm 0.00$ | $74.79 \\pm 0.01$ | $58.84 \\pm 0.09$ |\n| **OCB (RCNN)** | $85.75 \\pm 0.01$ | $87.33 \\pm 0.00$ | $64.12 \\pm 0.00$ | $75.28 \\pm 0.04$ | $68.84 \\pm 0.11$ |\n| Upper Bound   | $89.60 \\pm 0.05$   | $89.67 \\pm 0.03$  | $68.20 \\pm 0.13$ | $79.62 \\pm 0.11$   | $65.95 \\pm 1.00$   |\n\n\nCaption: Upper bound with frozen CLIP encodings and MLP classifier compared to CBM and OCB.\n\n### 2. Coarse-to-Fine CBM (C2F)  \nWe further evaluate against the recent Coarse-to-Fine CBM [4], which introduces hierarchical, patch-based concept processing. Unlike many CBM variants that differ primarily in vocabulary, C2F contributes a distinct architectural inductive bias. To ensure a fair comparison focused on architecture rather than vocabulary, we ran C2F with the same LAION-based concept vocabulary of our OCB setup and compare to OCB without sparsity constraints. Since C2F only supports single-label classification, we evaluate it on the SUN397 and COCOLogic datasets. We observe that our OCB model consistently outperforms C2F under these conditions, highlighting the benefit of object-centric representations over patch-based hierarchies. \n\n\n|               | SUN                | COCOLogic         |\n|---------------|--------------------|--------------------|\n| C2F           | $61.28 \\pm 1.44$                  | $66.27 \\pm 1.09$   |\n| OCB (ours)    | $78.11 \\pm 0.57$   | $68.63 \\pm 0.42$   |\n\nCaption: Comparison to Coarse-to-Fine CBM (with LAION vocabulary).\n\n\n### Sparsity Analysis\n\nWe next analyze the impact of the sparsity constraint on the base CBM and OCB (Pascal-VOC). We evaluate both models (Base CBM and OCB) with varying $\\ell_1$-penalties, which controls the number of active concepts for image and concept-level encodigns. As reference, we also provide the result where both models are not constrained and generate activations for every concept.\n\nThe results below show that both models improve as sparsity is relaxed. However, **OCB consistently outperforms the Base CBM across all sparsity levels**, indicating that object-centric representations yield more informative and efficient concepts.\n\nThese trends confirm that the performance gain of OCB is not due to increased capacity, but rather to the stronger inductive bias of object-centric representations. We have added this experiment and discussion to the main paper.\n\n| Avg. # of non-zero Concepts  | 15 | 25 - 30 | 60 - 70  | 130 - 160   | |10000 (no sparsity) |\n|---------------|--------------------|--------------------|--------------------|--------------------| -|--------------------|\n| Base CBM      | $82.42 \\pm 0.01$   | $83.49 \\pm 0.02$   | $84.07 \\pm 0.01$   | $84.14 \\pm 0.01$   | |$88.27 \\pm 0.17$   |\n| OCB (ours)    |- | $85.75 \\pm 0.01$   | $86.27 \\pm 0.01$   | $86.35 \\pm 0.00$                  | |$90.56 \\pm 0.02$   |\n\n\n### OCB: Contribution of Object vs. Image-Level Encodings\n\nFinally, we observe that removing the image-level encodings from OCB leads to a substantial performance drop (Pascal VOC), confirming that **object-centric and global representations provide complementary information**. \nThese results support a key message of our work: **both object-centric and global features are necessary** for achieving high predictive performance. We thank reviewer YQGn for this insightful suggestion.\n\n|               | Full (Object + Image) | Only Image Encodings | Only Object Encodings |\n|---------------|------------------------|------------------------|------------------------|\n| VOC           | $85.75 \\pm 0.01$       | $82.42 \\pm 0.01$       | $77.48 \\pm 0.03$       |\n\nAll novel results and discussions have been integrated into the main paper.\n\nWe hope that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.\n\n\n[1] Marconato et al. \"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.\" NeurIPS, 2023.\n\n[2] Koh, Pang Wei, et al. \"Concept bottleneck models.\" International conference on machine learning. PMLR, 2020.\n\n[3] Yang, Yue, et al. \"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.\" CVPR, 2023.\n\n[4] Panousis et al. *\"Coarse-to-Fine Concept Bottleneck Models.\"* NeurIPS, 2024."}, {"review_id": "Q4uuAn0STY", "reviewer": "Reviewer_YQGn", "summary": "This paper proposes to modify concept bottleneck models (CBMs) by first performing generic object detection, followed by a cropping of each detected object and an extraction of an object-specific CBM for each object. The concept-based representation of the whole image and the object proposals are then aggregated before being used to solve a downstream task. For all steps, pretrained models are used: MaskRCNN or SAM for object proposals, Splice with CLIP for building the concept bottlenecks.\nA new COCO-based dataset is proposed (COCOLogic) in which the classes correspond to specific combinations of objects.\nThe results show that using object-centric concept bottleneck leads to improved performance with respect to plain CBMs.", "questions": "- What is the behaviour of OCB and CBM for varying levels of sparsity?\n- What is the performance with a pure object-centric representation, without including the whole image concept representation?\n- If I got it well, crops with object proposals are directly given to the pretrained CLIP image encoder. It seems that the smaller the object proposals, the further from the CLIP training data. Did the authors investigate whether the size of the object proposals has an impact on performance?\n- Although the concepts are obtained separately for every object, it seems that all aggregation procedures combine the concepts in a way that the decision module cannot know which concepts belong together (to the same object). Is there a reason for this?", "limitations": "There is a lack of discussion about several of the limitations of the approach, including computational considerations, the fact that the decision is taken over aggregated concepts (removing object information) and the connections between COCOLogic and some real tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Object-centric representations are a sensible choice for many tasks, and this work shows that extracting concepts from image crops obtained via object proposals can lead to improved performance on some downstream tasks.\n- The approach is very simple and uses preexisting pretrained models, not requiring any training except for concept discovery from CLIP using Splice.\n- The paper is clearly written and easy to follow.\n\nWeaknesses:\nAlthough I think the results are worth sharing, it is not clear to me whether they would qualify for NeurIPS:\n- The only baseline is a regular CBM, with no alternatives, competitors or upper bounds considered. This makes the performance assessment relatively poor and does not put the results in their full context. A more competitive baseline that does not require additional forward passes would be to select the token embeddings (after a single forward pass) that correspond spatially to each object proposal.\n- A more detailed evaluation of the “equal capacity” CBM would be required. After all, the sparsity of OCB is likely related to the number of object proposals (and their diversity). Does the “equal capacity” CBM follow the same trends? It would be important to see the sparsity vs performance curves for both approaches.\n- The proposed COCOLogic, where the proposed model performs best, remains a rather artificial dataset. There is no discussion about how this would measure progress on some real task.\n- There is no assessment nor discussion regarding computational considerations at inference time. Although the proposed approach requires very light training (just Splice), it seems to me it requires as many forward passes as there are object proposals.", "was_revised": true, "final_justification": "The authors have addressed my main concerns. However, I concur with reviewer Kyb9 that some of the paper's claims are overstated. I lean slightly towards acceptance, although these issues need to be addressed first.\n\nI quote from a discussion with Kyb9, with whom I agree that the conclusion may be misleading.\n\n> Additionally, the authors wrote in the rebuttal,\n\n>>    This difference is not merely about granularity; it provides explicit grouping of concepts by object, which is not available in global image-level approaches.\n>>\n>>    It allows models to distinguish which concepts belong together, tied to specific objects.\n>>\n>>    It facilitates compositional and interpretable reasoning, aligning more closely with how humans understand scenes.\n\n> However, the concept groupings by object boundaries are not actually employed during the prediction. The concepts are aggregated but the groupings are not maintained (except for concat, but even there the groupings are not conveyed). As such, I find these claims in the rebuttals to also be unfounded.", "rebuttal": "We thank the reviewer for their helpful suggestions and the appreciation of our results. We provide clarifications for the raised points below, starting with a general response regarding the novelty of our method followed by additional baselines.\n\n> Novelty\n\nWe acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.\n\nTo the best of our knowledge, this is the first work that operationalizes object-centric concept representations within CBMs and applies them to real-world, multi-label classification tasks. Traditional CBMs rely on global, image-level features and often suffer from spurious or entangled concept associations [1], especially in cluttered scenes. In contrast, our object-centric bottleneck grounds concepts in specific, localized regions via pretrained object detectors, enabling finer-grained, interpretable, and more faithful concept-level reasoning.\n\nWhile this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.\n\nIn summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation—from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.\n\n\n\n> Additional baselines.\n\nWe agree that adding additional baselines strengthens the contextualization of our results which we highlight below:\n\n### 1. Opaque Upper-Bound Model  \nTo quantify the performance potential of our concept encoder backbone (CLIP), we introduce a non-interpretable model that directly maps CLIP image embeddings to class predictions using a non-linear multi-layer perceptron (MLP) (2 layers). It serves as an upper bound on the performance that can be achieved using CLIP features alone. As expected, it outperforms interpretable CBM variants slightly, but comes at the cost of complete opacity, offering no meaningful explanations. OCB closes the performance gap between CBM and the opaque model, underscoring its value for transparent, high-performing concept reasoning.\n\n|               | Pascal-VOC         | COCO(h)           | COCO(l)         | SUN               | COCOLogic         |\n|---------------|--------------------|-------------------|------------------|--------------------|--------------------|\n| CBM | $82.42 \\pm 0.01$ | $84.73 \\pm 0.00$ | $59.19 \\pm 0.00$ | $74.79 \\pm 0.01$ | $58.84 \\pm 0.09$ |\n| **OCB (RCNN)** | $85.75 \\pm 0.01$ | $87.33 \\pm 0.00$ | $64.12 \\pm 0.00$ | $75.28 \\pm 0.04$ | $68.84 \\pm 0.11$ |\n| Upper Bound   | $89.60 \\pm 0.05$   | $89.67 \\pm 0.03$  | $68.20 \\pm 0.13$ | $79.62 \\pm 0.11$   | $65.95 \\pm 1.00$   |\n\n\nCaption: Upper bound with frozen CLIP encodings and MLP classifier compared to CBM and OCB.\n\n### 2. Coarse-to-Fine CBM (C2F)  \nWe further evaluate against the recent Coarse-to-Fine CBM [2], which introduces hierarchical, patch-based concept processing. Unlike many CBM variants that differ primarily in vocabulary, C2F contributes a distinct architectural inductive bias. To ensure a fair comparison focused on architecture rather than vocabulary, we ran C2F with the same LAION-based concept vocabulary of our OCB setup and compare to OCB without sparsity constraints. Since C2F only supports single-label classification, we evaluate it on the SUN397 and COCOLogic datasets. We observe that our OCB model consistently outperforms C2F under these conditions, highlighting the benefit of object-centric representations over patch-based hierarchies. \n\n\n|               | SUN                | COCOLogic         |\n|---------------|--------------------|--------------------|\n| C2F           | $61.28 \\pm 1.44$                  | $66.27 \\pm 1.09$   |\n| OCB (ours)    | $78.11 \\pm 0.57$   | $68.63 \\pm 0.42$   |\n\nCaption: Comparison to Coarse-to-Fine CBM (with LAION vocabulary)\n\nAll results and discussions have been integrated into the main paper, and we thank the reviewers for this valuable suggestion.\n\n\n> Sparsity vs. Performance curves\n\nWe next analyze the impact of the sparsity constraint on the base CBM and OCB (Pascal-VOC). We evaluate both models (Base CBM and OCB) with varying $\\ell_1$-penalties, which controls the number of active concepts for image and concept-level encodigns. As reference, we also provide the result where both models are not constrained and generate activations for every concept.\n\nThe results below show that both models improve as sparsity is relaxed. However, **OCB consistently outperforms the Base CBM across all sparsity levels**, indicating that object-centric representations yield more informative and efficient concepts.\n\nThese trends confirm that the performance gain of OCB is not due to increased capacity, but rather to the stronger inductive bias of object-centric representations. We have added this experiment and discussion to the main paper.\n\n| Avg. # of non-zero Concepts  | 15 | 25 - 30 | 60 - 70  | 130 - 160   | |10000 (no sparsity) |\n|-|-|-|-|-|-|-|\n| Base CBM      | $82.42 \\pm 0.01$   | $83.49 \\pm 0.02$   | $84.07 \\pm 0.01$   | $84.14 \\pm 0.01$   | |$88.27 \\pm 0.17$   |\n| OCB (ours)    |- | $85.75 \\pm 0.01$   | $86.27 \\pm 0.01$   | $86.35 \\pm 0.00$ | |$90.56 \\pm 0.02$   |\n\n> COCOLogic relevance\n\nWhile we agree that the COCOLogic dataset includes constructed labels, this is by design: COCOLogic is intended as a **proxy task for structured visual reasoning within single-class image classification**, i.e., to capture aspects such as multi-object composition, negation, and hierarchical categories while still using **real-world visual data**.\nFor example, distinguishing a \"personal transport scene\" involving a person on a bicycle versus an image of a bicycle alone without a person is the kind of **entity-based and compositional reasoning** that object-centric models are meant to handle.\nWe have added a discussion of this point in Section 4 of the revised manuscript to clarify its motivation and role in benchmarking progress toward **real-world reasoning tasks**, such as visual question answering and scene understanding.\n\n> Ablation: only object encodings\n\nWe further observe that removing the image-level encodings from OCB leads to a substantial performance drop (Pascal VOC), confirming that **object-centric and global representations provide complementary information**.\nThese results support a key message of our work: **both object-centric and global features are necessary** for achieving high predictive performance. We thank reviewer YQGn for this insightful suggestion.\n\n| | Full (Object + Image) | Only Image Encodings | Only Object Encodings |\n|-|-|-|-|\n| VOC | $85.75 \\pm 0.01$ | $82.42 \\pm 0.01$ | $77.48 \\pm 0.03$ |\n\n> Limitations discussion.\n\nWe agree that one limitation of our approach is the reduced computational efficiency, as multiple CLIP inference runs are necessary. However, as it is possible to batch the runs of multiple objects together, this should not be a major hinderance in practice. Nevertheless, we added this limitation to the discussion section of the paper.\n\n> Ablation: Impact object size.\n\nWe appreciate the reviewer YQGn's observation and agree that very small object crops may fall outside the typical CLIP training distribution. We therefore evaluated OCB on Pascal VOC using different minimum object size thresholds (as a fraction of the image area). The results below show that performance **gradually degrades as smaller objects are excluded**, suggesting that **including smaller object proposals contributes positively**, despite the potential domain gap:\n\n| Minimum Size Threshold | 0.005| 0.01| 0.02 | 0.05|\n|-|-|-|-|-|\n| VOC Accuracy| $85.70 \\pm 0.01$ | $85.59 \\pm 0.01$ | $85.09 \\pm 0.01$ | $84.59 \\pm 0.02$ |\n\n> Aggregation methods object mapping?\n\nWhile four out of the five aggregation methods do lose the explicit mapping of concepts to individual objects, the concat method retains this information and offers a good trade-off—provided memory usage and a fixed object limit are not an issue. We discuss this in Section 5.3. Additionally, gradient-based techniques or attribution methods could potentially help disentangle which objects contribute to a decision, even when using pooled representations.\n\nWe trust that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.\n\n\n[1] Marconato et al. \"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.\" NeurIPS, 2023.\n\n[2] Panousis et al. \"Coarse-to-Fine Concept Bottleneck Models.\" NeurIPS, 2024."}, {"review_id": "gfFyNFnEcS", "reviewer": "Reviewer_Kyb9", "summary": "The work developed a new framework (Object-Centric Concept Bottlenecks--OCB) for visual reasoning tasks, using \"object-level representations\".\nIt identifies objects in an image and then infers concepts from the image and the identified objects before making the final prediction.", "questions": "> Fig. 6 (top) shows an example where the object-centric concepts enrich the conceptual explanation of the image via previously undiscovered concepts, like the presence of a fork or a wine glass.\n- Could this not be viewed as that CBM does not consider them as relevant concepts or as less important concepts?\n\n- Why use the same concept module for the whole image and the resized-cropped object region? Conceptually, if object-level and image-level are supposed to be different granularity of concepts, does it not make sense that the concepts extracted should be from different concept sets?\n\n> Interestingly, OCB outperforms the base CBM even on SUN397, despite that dataset consisting of many classes like \"sky\", \"desert sand\", or \"mountain\", where an object-centric approach intuitively would not provide many benefits.\"\n- What do the concepts look like in these classes? More generally, how does \"object-level concepts\" work for relations, actions, properties etc, that are not object based, in terms of accuracy and the level of interoperability? \n\n- Why stop at two levels? Why not keep going down the object hierarchy to get more specific breakdowns of objects?\n\n> While OCB’s more modular and object-centric concept space does not mitigate these limitations, it allows for more detailed control and evaluation of the learned concepts.\nWhat kinds of control and evaluation?\n\nMinor comments:\n\nLine 112: citations seem mis-formatted\n\nLine 195: missing parenthesis", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Significance: The work presents a new benchmark for evaluating single-label classification requiring visual reasoning. OCB performs better than CBM on image classification tasks that require visual reasoning.\n\nWeaknesses:\n- Clarity: It is not clear how fundamentally object-level concepts are different from image-level concepts beyond that there's some vague notion of level of details. Object-level concepts seem just as possible to be incorporated into the concept space of image-level concepts. It seems like the novelty is having layers of concept detection, with some arbitrary hierarchical grouping, but this is not made explicit or clear. \n- Significance: While OCB could be useful for classification tasks that rely on labels built on top of multiple objects, it does not seem to generalize to other classification tasks, as shown by the minor improvement over SUN, that do not segment nicely into objects with clear boundaries. It also does not perform significantly better over CBM (equal capacity) on tasks that do not need reasoning over multiple objects. \n- Quality: The evaluation is limited. The work does not evaluate against common benchmarks such as ImageNet or CIFAR-100 nor many  other classification tasks in non-object domains aside from SUN.", "was_revised": true, "final_justification": "The additional results are promising, and the authors addressed my main concerns about evaluation and the claims. The only drawback  is the limited application, where OCB only significantly benefit multi-objects-centric classification tasks.", "rebuttal": "We thank the reviewer for their time and suggestions. We provide clarifications for these points below.\n\n> Clarification object-level vs image-level concepts in CBMs \n\nTraditionally, CBMs compute a single concept representation for the entire image, often relying on supervised concept labels [1] or CLIP-based methods that match global image features to textual descriptions [e.g., [2]]. The image-level concepts reflect properties of the overall scene but do not explicitly distinguish between individual entities within.\n\nIn contrast, object-level concepts, as introduced in OCB, are computed separately for each detected object in the image. While both are based on the same backbone encoder, object-level concepts are grounded in **specific, localized entities**, enabling a structured representation of the scene. This difference is not merely about granularity; it provides **explicit grouping of concepts by object**, which is not available in global image-level approaches.\n\nThis structure has important implications:\n- It enables reasoning over **multiple entities** in complex, multi-object scenes.\n- It allows models to distinguish **which concepts belong together**, tied to specific objects.\n- It facilitates **compositional and interpretable reasoning**, aligning more closely with how humans understand scenes.\n\nOur results support this distinction. For example, our **sparsity experiments** show that object-centric representations encode concepts more efficiently and faithfully than image-level representations. Moreover, object-centric models perform better in **multi-label settings**, where distinct object-level reasoning is essential.\n\nWhile object-level concepts could be forced into an image-level framework, this ignores the inductive bias that gives object-centric representations their strength. Our contribution goes beyond hierarchical grouping, it offers a practical, interpretable path from flat concept reasoning to **structured, entity-aware representations**, essential for scaling CBMs to real-world vision tasks.\n\nWe have updated the manuscript to make sure that these points are presented more clearly in the paper. \n\n> 2. What is the novelty of this work?\n\nWe acknowledge that the individual components of our Object-Centric Bottleneck (OCB) framework, such as object proposal generation and concept classification, are not novel in isolation. However, the novelty of our work does not lie in proposing a new object detection mechanism, but in how we integrate object-centric representations into the Concept Bottleneck Model (CBM) framework, making it scalable and applicable to complex, real-world visual data.\n\nWhile this design may appear simple, bridging object-centric representations with CBMs in a real-world setting is non-trivial and has not been demonstrated before. Prior work exploring object-centric inductive biases has largely been limited to synthetic and single-class environments. Our contribution lies in extending these insights to realistic, diverse, and noisy data, removing the need for manual concept annotations, improving cross-dataset generalization, and demonstrating that object-level concept reasoning is viable and beneficial in practical scenarios.\n\nIn summary, the strength and novelty of our approach lie in making CBMs more structured, scalable, and interpretable by rethinking their core representation from flat, image-level concepts to grounded, object-level ones without requiring custom supervision or architectures.\n\n\n> Small improvements on tasks that do not require object information.\n\nThe primary goal of OCB is to **extend the applicability of CBMs to more complex visual tasks**, such as multi-label classification or visual reasoning, where **reasoning over distinct objects is essential**.\nWe had included results on SUN, which is commonly used in CBM evaluations, to demonstrate that **adding object-centric representations does not harm performance** on tasks where object structure is less relevant. \n\n\n> Evaluation against common benchmarks in non-object domains.\n\nAs discussed in the previous point, the goal of OCB is not to improve performance on non-object-centric tasks, but to **enable concept-based models to operate effectively on object-centric, multi-entity domains**, a setting where traditional CBMs struggle.\nFor this reason, our evaluation primarily focuses on datasets with explicit object structure, such as Pascal VOC and COCO. That said, we agree that broader benchmarking is valuable and present additional experiments on CIFAR-100, to expand the evaluation beyond object-segmented scenes. We observe that OCB and CBM perform comparably, indicating that OCB does not incur any performance loss on non-object-centric benchmarks. We have added the results to the paper.\n\n|CIFAR100|CBM|OCB|\n|-|-|-|\n|Acc|$74.21 \\pm 0.01$|$74.47\\pm 0.02$|\n\n### Questions:\n\n\n> Re: Qualitative concept examples:\n1. Does CBM view the object-concepts as less important? \nA: Yes, that is a valid interpretation. The base (image-level) CBM lacks the spatial focus to attend to smaller or less prominent objects like the fork or wine glass. As a result, these concepts may not be activated or may be deemed less relevant, even if they are present and potentially informative. In contrast, OCB explicitly examines individual regions, allowing it to **surface concepts tied to localized entities** that might be overlooked at the global level. \n2. Why same concept module/vocabulary? \nA: We used a shared, general-purpose concept vocabulary to keep the approach task-agnostic and free of additional supervision. This ensures broad applicability without requiring domain-specific concept sets.\nThat said, we agree that using different vocabularies for image-level and object-level representations when task-specific information is available is a promising direction for future work.\n\n> Re: Examples for concepts for SUN\n\n1. Example concepts for SUN:\nFor SUN, many images from natural scenes do not have any recognized objects. However, there are also scenes like \"computer room\", \"stable\" or \"dining room\" which have detectable objects like computers, horses, furniture or food that help identifying the global scene category. This explains the slight performance increase over the base model for this dataset.\n\n2. Relations, Actions, Hierarchies? \nA: We see object-level representations as a foundational step toward more structured reasoning, including relations, actions, and hierarchical object decomposition. Our goal in this work was to establish object-centric concept representations within CBMs.\nWhile we stop at the object level, we agree that further decomposition into sub-parts, relationships, or action-level concepts which often **emerge from interactions between objects or between objects and the global scene** is a natural and valuable next step. Our current design provides a flexible base for such extensions, and we view this as a promising direction for future work.\n\n> Re: What kind of control and evaluation for concepts? \n\n A: OCB enables more fine-grained control and evaluation by explicitly structuring the concept space around individual objects. In particular, when using the `concat` aggregation, the model retains which concepts are tied to which object proposals. This allows for **object-level explanations**, e.g., identifying which specific object triggered a certain prediction, and makes it possible to **evaluate concept activation per object**, rather than just at the global image level. This structured representation also allows for **targeted interventions**, such as modifying or masking concepts for a specific object.\n\n> Additional baseline results\n\nWe additionally wish to highlight novel baseline and ablation evaluations resulting from other reviews and strengthen the findings of our work.\n\n### 1. Opaque Upper-Bound Model  \nWe introduce a non-interpretable MLP classifier on top of frozen CLIP embeddings as an upper bound. As expected, it outperforms interpretable CBMs but lacks transparency. OCB narrows the gap significantly, offering strong accuracy with interpretable object-centric reasoning.\n\n| |Pascal-VOC|COCO(h)|COCO(l)|SUN|COCOLogic|\n|-|-|-|-|-|-|\n|CBM|$82.42\\pm 0.01$|$84.73\\pm 0.00$|$59.19\\pm 0.00$|$74.79\\pm 0.01$|$58.84\\pm 0.09$|\n|**OCB (RCNN)**|$85.75\\pm 0.01$|$87.33\\pm 0.00$|$64.12\\pm 0.00$|$75.28\\pm 0.04$|$68.84\\pm 0.11$|\n|Upper Bound|$89.60\\pm 0.05$|$89.67\\pm 0.03$|$68.20\\pm 0.13$|$79.62\\pm 0.11$|$65.95\\pm 1.00$|\n\n### 2. Coarse-to-Fine CBM (C2F)  \nWe compare OCB to Coarse-to-Fine CBM [4], using the same LAION concept vocabulary. Evaluated on SUN and COCOLogic (single-label tasks), OCB outperforms C2F, highlighting the benefits of object-centric over patch-based representations.\n\n| |SUN|COCOLogic|\n|-|-|-|\n|C2F|$61.28\\pm 1.44$|$66.27\\pm 1.09$|\n|OCB|$78.11\\pm 0.57$|$68.63\\pm 0.42$|\n\n### 3. Sparsity Analysis  \nWe assess both CBM and OCB under varying sparsity constraints (via $\\ell_1$ penalty). OCB consistently outperforms CBM at all sparsity levels, confirming that its gains stem from object-centric structure, not capacity.\n\n|# Non-Zero Concepts|15|25–30|60–70|130–160|No Sparsity|\n|-|-|-|-|-|-|\n|CBM|$82.42$|$83.49$|$84.07$|$84.14$|$88.27$|\n|OCB|–|$85.75$|$86.27$|$86.35$|$90.56$|\n\nWe trust that these clarifications and additional results strengthen the clarity and completeness of our work, and we respectfully encourage the reviewer to reconsider their evaluation in light of these updates.\n\n\n[1] Koh et al. \"Concept bottleneck models.\" ICML, 2020.\n\n[2] Yang et al. \"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.\" CVPR, 2023.\n\n[3] Marconato et al. \"Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.\" NeurIPS, 2023.\n\n[4] Panousis et al. \"Coarse-to-Fine Concept Bottleneck Models.\" NeurIPS, 2024."}]

Author rebuttal:

Meta-review: This paper introduces Object-Centric Concept Bottleneck, an interpretable concept-based model for image classification which emphasizes the model’s ability to distinguish multiple different object types in the same image. First, a pre-trained “object proposal” model is used to identify candidate concepts and bounding boxes. Then, after some post processing tricks (removing duplicates or low-confidence concepts, and cropping and resizing), the newly formed images are aggregated, together with the full image, and fed through a subsequent “concept discovery” module (which is also pre-trained), which outputs a feature representation formed by a score for each of a finite number $C$ of concepts. Lastly, a trainable linear layer is employed on the concept layer to produce the final classification. In experiments on several datasets, the authors demonstrate improved performance compared to an interpretable concept-based model baseline. In addition, a **new dataset, COCOLogic** is introduced which is based on the COCO dataset but creates artificial classes which consist in logical conjunctions of the individual objects present in the image (further details are available in Section D and in table 2 on page 16). For instance, there is a class corresponding to the presence of exactly one of “dog” and “car” (dog XOR car). Similarly, the method outperforms the simple baseline in this case. 


Whilst some of the reviewers were initially skeptical regarding the novelty (e.g. Rev. gmMc) and the lack of comparison with more (not-necessarily interpretable) baselines (Rev. YQGn), **all reviewers eventually agreed** that the reasons to accept outweigh the reasons to reject. Therefore, I am happy to recommend **acceptance**. However, I strongly urge the authors to work hard on the camera-ready revision to address both the reviewers’ and my concerns. 


In particular, more baselines should be added, even if the non-interpretable ones outperform the current method. In addition, some additional details are needed to make the paper clearer.  For instance, the fact that both the object proposal and concept discovery modules are pretrained is only mentioned in lines 207 to 210 at the end of section 3, which makes the paper less friendly to readers unfamiliar with the current craze for pre-trained models and fine-tuning.  (I appreciate reviewer gmMc’s clarification that the modules are typically Mask R-CNN/SAM and SpLiCE/CLIP respectively). Other clarifications on the experimental settings are required in the camera-ready version (including the mode detailed explanation in the rebuttal comment https://openreview.net/forum?id=9lhijvd0fs&noteId=gDfk8Ftwqi). 

Lastly, Reviewers Kyb9 and YQGn both have concerns regarding the meaning of the concepts and groupings generated by the two main pre-trained components. Neither reviewer was fully satisfied by the authors’ answer, though neither reviewer thought that this justified rejecting the paper. I also agree that the **interpretability of the last concept layer is imperfect**: considering the introduced COCOLogic dataset, since XOR relationships are not satisfactorily representable by linear maps, it must mean that either the concept classes at the last pre-trained layer already contain composite features, or *the performance of the introduced method is actually poor on this dataset (on the XOR-type classes) in an absolute sense* and it only outperforms a weak baseline also based on a linear combination of concepts. I suspect the latter is true, in which case the authors should also **investigate the use of a full 1-3 layer MLP at the final prediction layer** and **evaluate the class-wise performance**. 

However, I still argue for acceptance as all the reviewers are positive and I also find that the introduction of datasets such as COCOLogic to be a strong and creative contribution which takes the community in the right direction of explorative research into new tasks and interpretability (as opposed to obsessing over infinitesimal performance improvements on well-established datasets). 


**Minor typos**: 

Line 175 “for in input” should be “for an input”.
Decision: Accept (poster)