Label-Focused Inductive Bias over Latent Object Features in Visual Classification
Authors: Ilmin Kang
HyounYoung Bae
Kangil Kim
Venue: ICLR 2024 poster (2024)
Keywords: Representation Learning
Output-Domain Focused Inductive Bias
Classification
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Most neural networks for classification primarily learn features differentiated by input-domain related information such as visual similarity of objects in an image. While this focus is natural behavior, it can inadvertently introduce an inductive bias that conflicts with unseen relations in an implicit output-domain determined by human labeling based on their own world knowledge. Such conflicts can limit generalization of models by potential dominance of the input-domain focused bias in inference.
To overcome this limitation without external resources, we introduce Output-Domain focused Biasing (ODB) training strategy that constructs inductive biases on features differentiated by only output labels. It has four steps: 1) it learns intermediate latent object features in an unsupervised manner; 2) it decouples their visual dependencies by assigning new independent embedding parameters; 3) it captures structured features optimized for the original classification task; and 4) it integrates the structured features with the original visual features for the final prediction.
We implement the ODB on a vision transformer architecture, and achieved significant improvements on image classification benchmarks. This paper offers a straightforward and effective method to obtain and utilize output-domain focused inductive bias for classification mapping two different domains.

Published as a conference paper at ICLR 2024
LABEL-FOCUSED INDUCTIVE BIAS OVER LATENT OB- JECT FEATURES IN VISUAL CLASSIFICATION
Ilmin Kang', HyounYoung Bae", Kangil Kim Al Graduate School, GISTT, Republic of Korea
[kangilmin0325, bonheur606060, kangilkki)@@gaiiln
ABSTRACT
Most neural networks for classification primarily learn features differentiated by input-domain related information such visual similarity of objects in im- age. This input-domain focused inductive bias, while natural, can unintentionally conflict with unexpressed yet implicitly utilized relations over latent objects in human labeling, referred to Undescribed world knowledge (UWK). Such conflicts can limit generalization of models by potential dominance of the input-domain focused bias in inference. To overcome this limitation without external resources, we introduce Label- Label-focused Latent-object Biasing (LLB) training method that con- structs label label-focused inductive bias over latent objects determined by only labels learns intermediate latent object features in an unsupervised manner; 2) it decouples their visual dependencies by assigning new independent embedding parameters; 3) it captures structured features optimized for the original classification task; and 4) it integrates the structured features with the original visual features for the final prediction. We implement the LLB on vision transformer architecture, and achieved significant improvements im- age classification benchmarks. This paper offers straightforward and effective method to obtain and utilize undescribed world knowledge in classification tasks. The codes are available at htttts://ggtthhhh com GISTT TRR/LLB
as UWK. has four steps:
INTRODUCTION
In many classification tasks, neural network has role of learning latent features from an input to deter- mine its accurate output labels. In case of image clas- sification, the model learns the features based on vi- sual similarity and differentiation depending on their classes. This preference for the input-domain to han- dle similarity of features is a common property in most well- models such convolutional neural network Krizhevsky et al. 2017 He et al.) 2016) and vision transformer (VIT) (Dosovitskiy et al. 2020) as
Quill
Paperknife
Mop
Komondor
their high sensitivity to input specific information (Park & Kim).
This behavior is naturally expected for neural networks, but the relations on the latent objects may be different in human labeling based on world knowledge. Figure shows the clear example of the conflict in image classi- fication. The figures on the left are visually very similar those on the right, but the semantics of the labels different in the real world. These samples may be regarded as just ambiguous and difficult samples for visual classification, but fact we can differentiate them correctly by using unobserved relations in the data, such the stick part of Mop or the ice ground for the Hockey puck.
Crutch
Hockey Puck
Figure 1: Visually similar but semanti-
cally different images when we see the re- lations of all objects that not explicitly described in data
:: corresponding author
1: Gwangju Institute of Science and Technology

---

Published as a conference paper at ICLR 2024
This conflict between input-domain and potential knowledge on labels has been already discussed in various practical tasks, including visual question answering, cross modal learning, and domain adaptation (Hendricks et al., 2018; Rohrbach et al., 2018: Zhou et al. 2019 Zhang et al. 2021 Ca- dene et al. | Lemesle et al. 2022] Min et al., 2020) Radford et al. 2021) in various perspectives. However, these works have primarily focused on the inconsistency of rich information in observed data of two explicitly different domains (i.e. linguistic and visual relations of observed entities in each domain), while the conflict between input-domain information and undescribed relations over internal objects for determining output labels referred to Undescribed World Knowledge (UWK) in a single classification has been less investigated.
Such conffict in a single classification can limit generalization, because training and test data col- lected in the same labeling environment share the same UWK. If model fails to learn such UWK, inference on test data can be dominated by input-domain focused inductive bias and lead to incorrect prediction, This limit of generalization by the conflict has not been discussed as an important issue to control so far. In visual classification, for example, the training data already contain portion of UWK, as labeling can inform the category of internal objects in an image that are sufficiently abstract for the use as basic elements of the knowledge.
To this end, we propose Label-focused Latent-object Biasing (LLB) method for learning label- focused inductive bias over latent object features determined solely by categorization of labels, regarded as UWK. It has four sequential steps: 1) it learns intermediate latent object features in an unsupervised manner; 2) it decouples their visual dependencies by assigning new independent embedding parameters; 3) it captures structured features optimized for the original classification task; and 4) integrates the structured features with the original visual features for final predic- tion. We empirically investigate the conflict and impact of LLB on latent feature distributions and implement it on ViT architecture. Our experiments on an image classification task show that LLB improves performance in both quantitative and qualitative analyses, demonstrating the benefits of regularizing the model without external resources.
Our contribution points are:
We first raise the dominance of input-domain focused inductive bias of neural networks on inference that conflicts with undescribed world knowledge over latent objects by human labeling.
We propose a training strategy, Label-focused Latent-object Biasing (LLB), to obtain UWK and utilize it as label-focused inductive bias, and implement on vision transformer for visual classification.
We verify the proposed method in various image classification benchmarks with quantita- tive and qualitative analysis.
2 RELATED WORKS
Conflict of Input Domain Focused Inductive Bias The conflict between input-domain and po- tential knowledge labels has been discussed in various tasks. However, most works focus on the inconsistency of two different domain resources rather than the conflict of input-domain with undescribed world knowledge over latent objects in human labeling. In image captioning, (Hen- dricks et al.] 2018) addresses the conflict of bias on visual contextual cues with gender-specific texts. (Rohrbach et al.
2018) raises over-reliance issue on language prior, leading hallucination problem. (Zhou et al.
2019; Zhang et al., 202T) also observes that semantic inconsistency in the visual-text domain leads to hallucination, highlighting the issue of scarce aligned visual-text pairs. In visual question answering. (Cadene et al. 2019) introduces the impact of reducing dependencies on single-domain based statistical regularities when using both text and image input information. In cross-modality representation learning, vision networks leverage broader supervision of texts and adopts language bias in visual representation (Radford et al., 2021). However, cross-modal repre- sentations suffer inconsistency of language and visual domains (Pan et al.) 2022: Lemesle et al.) 2022). (Chen et al., 2019 Alberts et al.] 2020 Li et al. 2020 Tan & Bansal 2019) aim to make se- mantic aligned object representation using image- text paired dataset. To adapt CLIP (Radford et al.) 2021) in a different domain, (Ma et al. 2022) made semantic alignments with large collection of
semantic entities paired with images. (Pan et al. 2022 suggests semantic connection between differ-
2

---

Published as a conference paper at ICLR 2024
ent modalities for semantic perceiving and addressed the limitation with external knowledge-gree -gappp based networks.
Conflict Reduction via Knowledge Graph A traditional and direct solution of reducing the input- domain focused bias is to use external resources defined labels as knowledge graphs. In visual understanding. (Marino et al. 2017) learns structured semantic representation over objects (Ren et al. 2015) with structured prior knowledge graph (Krishna et al. 2017 Miller 1995). (Wang al. 2018 Kampffmeyer et al. 2019) tried to inject knowledge graph and biases of language in the vision classifier make zero-shot predictions. For open-world detection, (aao al.] (022) proposed large-scale unified concept dictionary with large-scaled image-text paired dataset to provide prior knowledge on detection. (Zhu al. 2021 also learns dynamic knowledge graph over object class label from language domain bias to overcome the limitation of data scarcity in few-shot object detection. However, all these works focus on the effective use external knowledge while UWK of the target training data has not been utilized and even recognized.
Disconnecting Latent Objects from Inputs In visual understanding, Marino et al. 2017 learns graph structures on internal objects, which are regarded as nodes, and integrates the features with external prior knowledge. To initialize each node feature, the object class label is used, which disconnects the visual dependency in separate phase for graph neural networks. (Zhu et al.) 2021) also learns dynamic semantic knowledge graph on object class label, which disconnects the visual dependency. However, visual disconnection in (Marino et al.) (017) are based on graph extraction and analysis. And (Zhu et al. 2021) may have learn separate embedding input-domain, with the aim of representing language based semantic embedding over object class labels, rather than label-focused inductive bias. (Min al. 2020) highlights the need for managing input-domain focused bias regularizing representation for zero-shot classification, using decoupling effects by contrastive learning. Differently, our method focuses on conflicts in standard classification and direct extraction of UWK to induce decoupling and repositioning bias that is insensitive to input-domain features.
METHOD
3.1 PRELIMINARIES
1000
228 Komondor
800
840
Mop
600
462 Broom
7,4
*840
400
746
Puck
462.
523
200
228
523
Crutch
Centroids of Class-wise Features (1000 classes)
(b) Image Samples for Adjacent Classes
Figure 2: Input-domain focused inductive bias features at the final layer, and its conflict with semantic difference on labels.
Problem Confirmation in Class Distribution The problem to be addressed in this paper is the dominance of the input-domain focused bias on latent object features that conflicts with UWK. In Figure 2 the problem is clearly shown. The dots the leftside figure represents the centroids of all features in each class of ImageNet, extracted from a MAE He al. (2022) trained on the data. We zoomed in on an area where the centroids are closely located and selected almost adjacent five classes. However, in the rightside figure, their class labels are semantically unrelated. Although their visual similarity is sufficiently high, a human can correctly recognize the label. For example, the pole in mop image may be confused with tightened harness of Komondor, but their shape
3

---

Published as a conference paper at ICLR 2024
Input Domain Focused Inductive Bias
Visual Encoder
visual feature V1
V
Label Focused Inductive Bias
(1)
Latent Feature
22)
Visual Dependency
(3) Feature Structuring
No LN
Integration
Extraction
Disconnection
(a) Process of LLB
vo
Li
MLP
A
RNX
RxD
MLP
Transformer
NO
MLPN
Py
EDOL
Po
p
Layer
PN
No
Diversity
(b) LLB Module on Visual transformer
Figure 3: Overview of Lobul-focused Latent-object Biasing in image classification. (blue: visually determined, red: non-visually determined)
and angles associated with the other hairy features are distinguish the two classes. This observation shows evidence for the dominance of the visual input-domain focused bias over the UWK. See Appendix C for additional problem confirmation using CNN and comparison with LLB.
Motivation for Method: Label-based Grouping of Latent Objects Determines Their Similarity in UWK Our motivation is that categorizing latent object features provides their similarity the UWK without relying of external resources. A straightforward approach to utilize this label-focused bias is to train classification model that takes the latent features input. But, input-domain fo- cused inductive bias may dominate over the label-focused bias in determining the feature positions. To address this issue, we propose training separate plug-in classification model capture the label-focused bias. Additionally, we propose disconnecting the direct forward path from an input to latent object features, which will then be fed into the model as new inputs.
Notations for VIT We start with brief recap of ViT (Dosovitskiy et al. 2020). VIT reshape input image RC xW to patches Xp E (PP-CC) where (HI W) is original image's resolution with channel size C. Patches have size of (P, P), and the derived total number N of patches per images is HW/P?. Patches are then projected into hidden dimension of size D with additional positional embedding. Function fwx), encodes patch representations with additional CLS token by stacks of Ly transformer (MHSA+FFNN) layers. We denote visual features as Vi = [ff]] where v E R hidden vector of :th--ptch of !th-layer.
3.2 METHOD DESCRIPTION
Overview of Architecture on Vision Transformer We propose Label-focused Lateedaaaa Latent-object asing (LLB) method for visual classification problems as shown in Figure Our network is based on a typical pre-trained Dosoviiky al. (2020) that uses split image-patches as an input sequence, and is implemented in four sequential steps: 1) latent object extraction, 2) visual dependency dis- connection, 3) object feature structuring, and 4) integration non-visual and visual features for the final prediction. In the first step. the generated features from intermediate 1-th layer of the ViT (visual features features Vi) generates probability to select latent object index among O objects. In the next step for the disconnection, the most probable object index is mapped separate learnable embed-

---

Published as a conference paper at ICLR 2024
ding non-visual features N |m']1/0 E RoxD where D is the equal dimension visual features and i indicates the object index. In the third step, the following transformer stacks (LLB layers) learn the structures of the non-visual embedding. At the end, the visual and non-visual features are integrated as an ensemble for the final decision.
Latent Object Extraction from Visual Features The first step in visual object extraction involves quantizing visual features into set of latent objects. This process is essential in limiting the number of non- vos-visual features that may be associated with UWK, thereby ensuring that computational cost constraints are met. To implement the quantization, we pass a visual feature of each image patch into an MLP. This is then followed by a Softmax operation temperature of (= 0.1). The resulting vector, p', determines the probability of selecting the latent object of index i. The following process applied to all patches generated from an image.
p'
So Softmax (vi))
p' E R
(I)
The clarity of objects is influenced by their number, therefore consider it a hyper-parameter: The effective range for this parameter is detailed in Table 7a
To effectively represent groups of visual features using limited number of latent objects, we in- corporated a loss function that promotes diversity of probabilities, as described in (Van Gansbeke
al. 2020 Mustafa et al. 2022).
N p and H(p) - Σ p' log(p')
L Divereity
= -H(p), where p
(2)
1=0
where Divvrriit makes the averaged probability vectors of all objects uniform, encoursing assignment across all objects. Diverssity is combined with downstream task loss (e.g., cross-entropy loss) using the balancing parameter X(=0.5).
Visual Dependency Disconnection
The core idea of visual disconnection is to assign separate embedding parameters visually determined latent objects. This process interrupts the gradient flow of the embedding parameters stemming from input-based differentiation, ensuring that subse- quent training on the embedding is not overly influenced by this differentiation. To achieve this, we employ straightforward Disconnect network, as illustrated in Figure 3b
No Disconnect (A, NN A x N
A matrix patch-wise probability vectors to select latent objects
N A matrix of non-visual features in disconnected parameters from input
The Disconnect network straightforwardly assigns disconnected embedding parameters to latent objects and subsequently produces patch-wise non-visual features, denoted as No = [m]]] used as inputs of LLB module. It operates matrix multiplication of assign matrix A = [p I, E IR Nxo composed of the generated probability vectors (p's in Equation( I to select latent objects, and the separate trainable non-visual feature N. Rather than using the argmax function, matrix mul- tiplication with the assign matrix is employed. This makes it differentiable while interrupting the input-based differentiation. It is worth noting that we utilize separate CLS token, denoted as ra), for downstream tasks.
Visuvi Feature Structuring Structuring non-vist von-visual features is the core step to redefining the similarity of features built over latent objects via solely the categorization of labels. The structuring method, referred to g( X, W), processes non-visual features No using weights W from structuring module to produce structured non-visual features NLx- We implemented the function g by a stack- ing NN layers of transformer (Vaswani et al.) 2017) layer. We use transformer to extensively discern semantic relations within sequential input, as evidenced by works like (Devlin et al.) 2018 Radford eet al. 20188] Park & Kim), owing to their flexible and scalable architecture. Adhering the original method, we attached a CLS token to the head of non-visual feature sequence, denoted as C,,, from the structured non-visual features for the final decision.
N
g((c,) No). W)
(4)
where N = |n| x Compared to the original transformer layer, we exclude the positional embedding due to the weak dependence of clusters to specific positions in sequence.
5

---

Published as a conference paper at ICLR 2024
(nainr))
(a) visual feature (internal)
(b) non-visual feature (internal)
LENT
Lote
(d) non-visual feature (final) Figure 4: Comparison of object (image patch) feature distribution of each sample for four example classes at the end of training (Impact of visual disconnection: (a) and (b), Impact of structuring: (b) and (d), Impact of label-focused bias: (c) and (d)).
(c) visual feature (final)
Integration of Non-visual and Visual Feature
The primary objective learning structured non- visual features is achieve better generalization, as opposed merely accurate prediction ob- served data, which is effectively achieved by the original visual features. To leverage the strengths of both visual and non- visual features, LLB consolidates the output probabilities from both domains as described below.
Pv = SoftMax(MLPy((c) (c.))
(5)
PN CoftMax(MMPNx(C))
(6)
Po = a x pv + (1 - a) x pn
(7)
To safely preserve the pre-trained probability for the visual features, we employ a separate classifier for predictions from non-visual features. Subsequently, we aggregate the output probability vectors, Py and pv, from each classifier. A balancing parameter, a, is introduced modulate their respec- tive contributions, elaborated in Equation (7). The integrated probability Po determines the final cross-entropy.
3.3 EMPIRICAL ANALYSIS ON FEATURE DISTRIBUTION
Simple Settings In this section, we provide empirical analysis of visual disconnection and struc- turing on feature representations. We extract them from a trained vanilla ViT (Dosovitskiy et al.) 2020) described at Section 4 In Figure 4 each dot represents feature corresponding to an image patch assigned to an latent object index, and the four figures illustrated via t-SNE (Van der Maaten & Hinton 2008) show visual or non-vin non-visual feature distributions for the internal or final layer. Check Appendix Section B.I closer look the figure.
we observe that Impact of Visual Dependency Disconnection Comparing Figure 4a with 4b
visual features for objects tend cluster by both sample and class. In contrast, non-visual features exhibit a distribution closer to randomness. This disparity indicates the existence of the dominance of visual input-domain focused inductive bias.
Impact of Non- Visual Feature Structuring Figure 4b and 4d and show the change in the final features caused by the structuring non-visual features. The non-visual features are largely over- lapped across samples and classes, showing their ambiguity for classification. However, the final features are more concentrated and separated by their class and sample. This implies that sufficient structuring is required for learning label-focused inductive bias useful in practical classification.

---

Published as a conference paper at ICLR 2024
Impact of Label-focused Bias In comparison of Figure 4c with 4d of input-domain focused bias and label-focused bias. For most samples, the visual features are split into two clusters, with exceptions of sample 2 and 6. Notably, samples of different classes overlap in the cases of samples 2, 3, 4, 6, and 7. Furthermore, the sample area of two classes (0 and 1) overlapped. In contrast, the label-focused inductive bias does not show any of the negative phenomena. This suggests that leveraging the feature concentration from the label-focused bias can further enhance generalization that was not achieved by input-domain focused inductive bias.
can observe the difference
4 EXPERIMENT SETTING
Our LLB can be applied to any networks based on Transformer architectures. We evaluate the effectiveness of non-visual feature by comparing it to vanilla networks in standard benchmarks. With pre-trainee transformer networks, we first train our LLB in supervised manner, and measure its performance via an integration method (Section 3.2) that does not require any additional training. We also provide series of analyses of the impact of each component.
Training Details. Our LLB is built upon pre-trained classical visual feature backbones. We ex- tract hidden vectors from backbone while keeping the backbone parameter freezed. For backbone, we use ViT Dosovviskkiy et al. 2020) networks. We use ly-th layer and consider them as visual features V By oi : ofy We found that, extraction from lv = Ly 1 layer showed best performance (Figure 7a Appendix). L.B. takes Vi, and cluster them into O latent objects. Based on experiments (Figure 7blin Appendix),
use O=2048. We report the results of differ- ent a in Figure 7 in Appendix, and selected best one among them. Additional model settings are We train model with CrossEntropy loss with additional object diversity regularization term in Equation (2). Look at Table in Appendix for detailed hyper- hyper-parameterr parameters we used. Our experiments are on 8 A100 with additional 4 x A6000 GPUs for both reproduce baselines and training LLB.
summarized in Table 3
image classification task. We show the effectiveness of the non-visual feature through the performance gained by adding our LLB to base- lines. We use standard ImageNet (Deng) al. 2009), which consist 1.28M training images with 1000 classes, We also use additional benchmarks including IN reassessed labels ImageNet- Real (IN- (IN-Real) (Beyer et al. 2020), scene recognition dataset Maces356-Standdee
Image Classification. We perform the evaluation
(Places) (López-)
Cifuentes et al. 2020), -eniigr and long-tailed INaturalist2018 8 (iNat18) 8) (Van Horn et al.) 20188)
dataset. For baselines, we first followed (Dosovitskiy et al.] 2020 Steiner et al.J 2021) to get vanilla VIT pre-trained using ImageNet21K (IN21K) (Ridnik et al. 2021). Also, to evaluate robustness of
our method, we evaluate our method with diverse pre-training schemes. SWAG (Singh et al. 2022) is wuakly-supervised VIT pre-trained via weakly supervision with hashtag labels (IG3.6B). Addi- tionally, we use self-supervised method MAE (He et al., 2022) trained on IN1K. MAE is trained to reconstruct masked portion of image. For some models, we used parameter weights from
open source MFF For others, we followed the training details described on (Singh et al. 2022 He
al. 2022 Singh et al. 2023) with 5 runs All reproduced and official performance are reported in
Table
RESULT AND DISCUSSION
5.1 QUANTITATIVE ANALYSIS
Performance In Table [ the performance before and after applying LLBB is shown. Overall per- formance significantly improved compared to the pre-trained models. Additionally, LLB remains effective even when applied to larger base models.
We also examine the robustness to different pre-training schemes. these experiments, we replace supervised pre-trained VIT with weakly (SWAG) and self (MAE) supervised ViT. The results in Table show that the different pre-training schemes benefit from label-focused inductive bias on
VTT http://ghhbbcouhnnggggeeeeoooooe MAE: http::/ggihna.ccm/ffcchhoeeessss SWAG: http://gthhh.commaaccccooccccsss

---

Published as a conference paper at ICLR 2024
Model
Resolution
Image Classification (Top) acc.)
Pre.
Fine.
INIK
IN-Real
Places365
INar18
VIT B/16"
INIK
234
224
79.00p.00 (77.91)
83. 760.00 (83.57)
LLB (Ours)
224
79.43o.00
84.250.c
VTT B/16"
IN2IK
234
224
84.400.00 (83.97)
88.550.0 (88.35)
LLB (Ours)
224
84.80g.01
88,90p.02
VIT L/16"
IN2IK
224
224
85.680.00 (85.15)
89.05o.co (88.40)
LLB (Ours)
224
86, 90.922...
89, 20,26pp..u
MAE B/16
INIK
224
224
83.63g.o (53.60)
88,29b.oo (-)
37 84y (57.9990)
71. 21.200..s (75.40)
LLB (Ours)
224
33.780.02
88,400.44
57 77.990n.ud
71.320.08
MAE MAEL/16 L/16
INIK
224
224
86. 86.088.00 (85.90)
89.636.c0 (-)
59 60 (69.40)
80.066.0s (80.10)
LLD (Ours)
224
86.12p.01
89.65p
59.70g 70,
80.00p
SWAG B/16
1B3.6B
224
384
85. 280.28.. (85.30)
89,00p.co
(89.10)
58 90 (09.10)
79.78 (79. (79.90p ue)
LLLB (Ours)
224
85.356.04
89.10p.co
59.17
79.86p.08
Table 1: Top-1 accuracy (Acesta on four image classification benchmarks (red: positive, blue: negative). We report the results from the paper in parentheses(*: the result of (Steiner et al., (02T), t: the result of (He et al.] 2022). f: the result of (Singh et al. (022)). The baseline results for INIK and IN-Real are reproduced from the fixed open-source fine-trained models. For models that reproduced our-self, we state standard deviation well. For both reproduce and proposal, We report the results of 5 runs.
most of the evaluation benchmarks. Note that LLB does not additional pre-training with large data and regularization such as (Zhang et al. 2017 Yun et al. 2019 Szegedy et al. 2016).
Ablation Study Table reports ablation study results. We configure LLB with ViT B/16 trained on INIK as the main model, and assess performance by excluding each attribute. First, we study the impact of the disconnecting visual dependency (Visual Disc.). We do not use the disconnection method while remaining the structuring method. Results show that disconnecting visual depen- dency significantly improves performance than maintaining it. Structuring visual features shows worse results than the baseline, indicating that LLB does not benefit from simply adding additional parameters. Then, we evaluate the impact of implementation component. We ablate the diversity loss for latent object extraction (Diversity), and also set the transformer to not positional encoding (w/o Pos.). The results show that each configuration is required to obtain the best perfor- mance. Finally, the performance without our integration module with visual features (Integration) shows the worst performance.
Model
Visual Disc.
Diversity
wlo Pos.
Integration
INIK (Topl-Aoc,1a
L.B (Ours)
84.80p.01
/
/
81 05
/
/
81.74o.01
/
/
84.77o.02
82.5 12.588..12
Baseline
84.400.00
Table 2: Ablation study results of 5 runs with random seeds. We study the impact of visual depen- dency disconnection (Visual Disc.), diversity loss (Diversity), not using positional encoding (w/o Pos.), and integration module with visual features (Integration).
5.2 QUALITATIVE ANALYSIS
In this section, we provide an in-depth qualitative analysis on visual examples where LLB success- fully predicts.
Final Feature Distribution Figure 5 visualizes the final feature distribution generated from visual features (left) and non-visual features (right). In the left figure, we can find some visually similar features that are classified under semantically different labels. In the right figure, we trace the location of the sample (c.g. the screwdriver, quill, dough images with circle marker), but these samples are adjacent to the sample in the correct class. This outcome demonstrates that the label- focused inductive bias can effectively refine features toward their correct classes and rectify incorrect decisions made visually ambiguous samples.
Object Map on All Patches Figure 6 shows the examples of latent object indices mapped to each patch of an image via visual-featur based latent object extraction. We selected the examples that
8

---

Published as a conference paper at ICLR 2024
Input-Domain Focused Inductive blas
Label Focused Inductive blas
Dough
Puzzle
Quill
Paperknife
Screwdriver
Paddlewhee!
Figure 5: Final visual (left) and non-visual (right) feature for each sample. (color: a pair of two confused classes in visual feature based prediction, marker: one of the classes). Note that the circle sample is confused in visual feature map, but not in non-visual feature map.
1173
Paperkmife
Frequency
1373
1613
Latent Feeture Extraction
(a) Latent object map extracted from its overlapping image
(b) latent object frequency for each class
Figure 6: An example of extracted latent object distribution sample and over samples in each class. In (a), each tile shows assigned latent object index of O(=2048) to image patch. Red and green patches are dominating latent objects (1173 and 1813) in each image. (b) shows patch samples for the dominating objects and the frequuecy of the objects over all samples in each class.
visual-feature based models are confused. In Figure 6a
we observe the object 1173 over the feather and some background the quill image, while the image patches on the paper knife assigned to the different object 1813. As shown in (b) the two objects are distinct in the comparison of the other assigned image patches, and they are also identified as the most frequently used objects across all samples of the classes, serving clear markers to distinguish between the classes. This finding suggests that latent objects derived from visual features inherently possess the ability to differentiate their respective classes. However, the input-domain focused inductive bias locates patches of the objects close together, confounding the prediction. In contrast, LLB disconnects the bias, and can leave the two objects effective identifiers for correct classification. Additional object map results are in Figure 9 in Appendix.
CONCLUSION AND FUTURE WORK
Conclusion In this paper, we highlighted the conflict of input-domain focused inductive bias and undescribed world knowledge on this issue, we introduced Label-focused Latent-object Biasing (LLB) method that simply learns a separate classification model from intermediate object features disconnected from an input, and then integrates with the original visual feature based classification model. Implementing the method on Vision transformer architecture, we can confirm its positive impact model generalization through qualitative and quantitative analysis of its results in image classification tasks.
latent objects in human labeling. To advance regularization
Future Work Beyond using the plug-in model, effectively harmonizing the undescribed world knowledge with input-domain focused bias on a simple network still remains an open question. Also, we hope that approach will evoke further research on diverse visual domains.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
We demonstrate the reproducibility statement for this paper as follows.
For reprodueibility of our Label-focused Latent-object Biasing (LLB), we first demonstrate our method in Section Also provide details of our implementation in Section 4 and Appendix A.I
To reproduce our implementation, we provide codes for training our LLB using various backbone models. Implemented codes are included in the supplementary material. We also provide README.md file for detailed description in the supplementary material.
8 ACKNOWLEDGMENTS
This work was supported by National Research Foundation of Korea (NRF) grant funded by the Ko- government (MSIT) (No. NNo.2022R1. 10221122221120 Development of AT for Canonicalized Expres- sion of Trained Hypotheses by Resolving Ambiguity in Various Relation Levels of Representation Learning), Electronics and Telecommunications Research Institute(ETRI) grant funded by the Ko- rean government (No.201 1001--2022 1-2023-00046 Development of Large Korean Language Model Technol- ogy for Efficient Pre-training). if you intend utilize the contents of this report, you must disclose that the research was funded by Electromics and Telecommunications Research Institute(ETRI), and Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2022 (Development of service robot and contents supporting children's 's reading activities based on artificial intelligence No. 20.22222000001. contribution rate 25%) and appreciate the high-performance GPU computing support of HPC-AI Open Infrastructure via GIST SCENT.
REFERENCES
Houda Alberts, Teresa Huang, Yash Deshpande, Yibo Liu, Kyunghyun Cho, Clara Vania, and Iacer Calixto. Visualsem: a high-quality knowledge graph for vision and language. arXiv preprint arXiv:2008.09/50, 2020.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? arXiv preprint 2rXiv:2006607/599 2020.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint S0Kd:::0003042997 2020b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations.(211)) arXiv preprint arXiv: 1909. 1 11740, 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- crarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18 2018.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 06), volume 2, pp- 1735-1742. IEFF, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. Proceedings of the European conference on computer vision (ECCV), pp. 771-787, 2018.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang. Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11487-11496 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6980, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Alex Krizhevsky, lly Sutskever. and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Yoann Lemesle, Masataka Sawayama, Guillermo Valle-Perez, Maxime Adolphe, Hélène Sauzéon, and Pierre-Yvee Oudeyer. Language-biased image classification: evaluation based semantic representations. arXiv preprint 0rXiv:2201.11014. 2022.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
Alejandro López-Cifuentes, Marcos Escudero- Ascuuero-VVnolo, Jesús Bescós, and Álvaro Gareía-Marde Semantic-aware scene recognition. Pattern Recognition, 102:107256, 2020.
Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. Ei-elip: Entity- interventional contrastive learning for e- commerce cross-modal retrieval. In Proceedings of the IEEE/CVE Conference on Computer Vi- sion and Pattern Recognition, pp. 1805 1-0561 2022.
Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2673-2681. 2017.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
11

---

Published as a conference paper at ICLR 2024
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang. Zheng-Jun Zha, and Yongdong Zhang. nomain-awar visual bias eliminating for generalized zero-shot learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12664-12673, 2020.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-image mixture of experts. arXiv preprint arXiv:2206.02777, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. Xrviv preprint 1807.0:/00.03310 2018
Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, and Gao Huang. Contrastive language-image pre-training with knowledge graphs. Advances in Neural Information Processing Systems, 35: 22895-22910, 2022.
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference Learning Representations.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10977 10972, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035 2018.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateck Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 804 814, 2022.
Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Ad- cock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross Girshick, et al. The effec- tiveness of mae pre-pretraining for billion-scale pretraining. arXiv preprint arXiv:2303.1 2023.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv: 106. 10270, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Hao Tan and Mohit Bansal. Lxmert: Learning modality encoder representations from trans- formers. arXiv preprint 1rXiv:1908.0740 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(11), 2008
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings. Part X, pp. 268- 285. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
Grant Van Horn Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6857-6866, 2018.
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9138, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171000422 2017.
Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu, and Yueting Zhuang. Consensus graph representation learning for better grounded image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3394-3402, 2021.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded video description. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6578-6587, 2019.
Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 8782-8791, 2021.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 IMPLEMENTATION DETAILS
VTT architecture Our LLB is built upon pre-trained VIT backbones. We use ViT-Base and it's scaled version ViT-Large for LLB. T detailed information about model variants. We follow the settings from (Dosovitskiy al. 2020 for ViT parameters.
LLB stacks LN layers of transformer layers to structure non-visual features. We report the impact of the number of layers the LLB in Figure [cc and selected values for LN based on the results. Our LLB adds additional MLP layers for latent feature extraction and stacks transformer layers for non-visual feature structuring.
VII
LLB
Size
Lv
FF
H
LN
D
FF
H
lv
VTT Base
768
3072
12
6
768
3072
12
11
2048
ViT-Large
1024
4096
1024
4096
23
2048
Table 3: Details of model variants
Hyper-patte pprrrppa selection Depending the input-domain and UWK in task, the conflict may be caused by different numbers of objects in different layers. So we set the layer to extract the objects and their number as byper-parameter for tuning by tasks. We also set the number of layers to structure non-visual and the value of a for integration as a hyper-parameter, and measured their influence the INIK classification task. The effective range of the haper-parameters are shown in Figure 7a
24
COR
1N22 Neforoorrmmrcaaoommm
IN3X Performenceeeeeee
013
063
(a) Visual Feature Extraction Layer
(b) Latent Object Number
IN23 www.noooomrrraaaooomm
04.
300
(c) Number of Structure Layer
(d) Integration Alpha
Figure 7: Impact of each hyper-parameter on INIK image classification.
Training details We report our default training settings for INIK image classification task in Table 4 For other evaluation benchmarks, only normalization values are changed. Table 5 reports the image classification performance on INIK.
B ADDITIONAL QUALITATIVE ANALYSIS RESULTS
Object Clusters Figure shows successful examples our latent object extraction. Each grid represents individual object cluster. We randomly sample clusters and clustered patches, and map them to the original image. For example, the second image in the first row has patterns like animal prints, and the second image in the second row has parts of fruit
14

---

Published as a conference paper at ICLR 2024
Setring
Value
Epochs
Batch size
1024
Oplamizer
Adam
Kingma & Ba 2014
Optimizer Momentum
01 0.9, D2 = 0.999
Learning rate:
Schedule
Cosine Ic-4
Peak
Weight decay
5e-d CrossEntropy
Loss
Augmentations:
224pa 384px
RandAugment
Cabuk
2020
Magnitude
Normalize
mean
[0.485, 0.456, 0.406] [0.229, 0.224, 0.225)
std
Table LLB training setting
Model
Pre.
Params
Resolution
Topl (acc.)
Fine.
INIK
B/16
INIK
86.57
224
224
79.00 (77.91)
LLB (Ours)
+46.45
224
79.434..03
VIT B/16
IN2IK
86.57
224
224
84.40 (83.97)
LLB (Ours)
+46.45
224
84.78:..11
VIT L/16
IN21K
304.33
224
224
85.68 (85.15)
LLB (Ours)
+80.80
224
85,924.02
MAE B/16
INIK
86.37
224
224
83.63 (83.60)
+45,92
83.78:5.02
MAET/I6
(85.90)
(Ours)
+80.80
16.12:0.11
SWAG B/16
IB3.6B
86.37
224
384
85.28 (85.30)
LLB (Ours)
+45.92
224
85.35:.04
Table 5: Detailed top-1 accuracy INIK (accuracy in parenthesis: reference performance, red:
positive, blue: negative).
Object Map on All Patches with Other Images Figure shows additional examples of object indices mapped to each patch of an image. the mapped image in the top row, we found that the patches of the screwdriver are mapped to object 391 and the patches of the metal body mapped to object 1736. From the frequency results on the right side, we can see that both features are distinctive features for each class.
B. EMPIRICAL ANALYSIS RESULTS
We provide larger version of the visualization in Section 3.3
C ADDITIONAL PROBLEM CONFIRMATION AND COMPARISON WITH LLB
Figure Iaa from clearly shows the problem of the dominance of the visual-domain focused bias over the undescribed world knowledge over latent object in human labeling. The dots in the leftside figure represent the centroids of all features each class of ImageNet, extracted from the ViT network trained the data. When zoomed in region of closed centroids, we found five adjacent but semantically unrelated class labels, shown on the rightside.
We also confirm this problem with the Convolutional Neural Network (CNN). We follow the same procedure that is described in Section 3.1 but replace VIT with the well-known CNN network ResNet50 (Krizhevsky et al. 2017). We used two versions of ResNet50 pre-trained with ImageNet training data. First, we used the pre-trained ResNet50 (Krizhevsky et al.] 2017) a supervised manner. For supervised pre-trained ResNet50, we followed the details of (Krizhevsky et al. 2017)
15

---

Published as a conference paper at ICLR 2024
Bam
POR
2007
Figure 8: Positive examples of object latent clusters.
Paddlewheel
Frequency
Feature
Plastic bag
Joy
Prequency
Latent Feature Extraction
Figure 9: Additional example of object map patches. In left, each tile shows assigned object index to an image patch. right shows patch samples for the dominating objects and the frequency of the objects over all samples in each class.
and used parameters from open source to reproduce a top-1 accuracy 75.86% for INIK (the reported performance from the open source is 73%1 We also ResNet50 pre-trained with self-supervised contrastive learning framework (Hadsell et al. 2006 Oord et al. 2018). Momentum Contrast. (MoCo) (He al. 2020) interpreted contrastive learning dictionary look-up and built dy- namic dictionaries with momentum based moving average updates. MoCo v2 (Chen et al.) 2020b) improved MoCo with the successes in (Chen al. (2020a). We collected pre-trained ResNet50 weights using MoCo v2 from open source") We then fine-tuned it using INIK with the details described in (Chen al. 2020b), and reproduced 77.01 % top-1 accuracy IN1K
C. INPUT-DOMAIN FOCUSED BIAS IN CNN
Figure Il shows the results of CNN in the classification benchmarks. In comparison with the bias in ViT 11a semantically distinct classes ('840: Mop', '462: Broom', '764: Puck', and '523: Crutch") are still closely located, which is the common input-focused inductive bias of the dataset. This observation is an evidence for the conflict of the input-domain focused bias even CNN.
*ResNet50: http://wwwwhhhh..oooooooooo/nnnn
'MoCo v2: http://htiicaaaacccccarrreeeeeee
16

---

Published as a conference paper at ICLR 2024
(a) Visual feature (Internal)
(b) Non-visual feature (Internal)
(c) Visual feature (Final)
(d) Non-visudl feature (Final)
Figure 10: Feature distribution results.
C.2 INPUT-DOMAIN FOCUSED BIAS IN CNN WITH CONTRASTIVE LEARNING
Figure 11d shows the results of the CNN trained with contrastive learning. Using contrastive learn- ing, the centroids of some classes (e.g. '523:Crutch' against '840: Mop', '462: Broom', '764: Puck') are slightly decoupled compared to supervised learning. However, this approach still fails to widen the gap between '462: Broom' and '746: Puck', where two class labels are visually sim- ilar in stick parts, but semantically distinguished by other objects. This observation shows that the domut-domain focused bias is still strongly used in determining the features.
C.3 COMPARISON WITH LABEL-FOCUSEE LATENT-OBJECT BIASING
Figure Iib shows the results of LLB using the same classes in Figure lle Compared to ViT (Fig- l where the centroids of all features of five classes closed located, LLB shows distant gaps between classes. Also, while other networks fail to widen the gap between '462: Broom and 7746 Puck', LLBB placed them in distant location.
17

---

Published as a conference paper at ICLR 2024
Additionally, we can see that 8840: Mop" and '462: Broom' closed located in LLB. We hypoth- esize that, the way of structuring over components of mop and broom are similar, making LLB to generate their features in close location. In contrast, the other methods placed '840: Mop' and 446:: Broom' relatively more distant locations. This observation implies that LLB can diminish the dominance of the visual input-domain focused bias, and introduce distinct bias, considered as the label-focused inductive bias.
523
840
462
746.
462 523
600
600
228
400
400
746
200
200
228
(a) VIT
(b) LLB (Ours)
1000
1000
228
228
600
774
746
462
462
523
400
523
400
840
840
200
200
(c) CNN with supervised pre-training
(d) CNN with contrastive learning
228 Komondor
840
Mop
462 Broom
746
Puck
523
Crutch
(e) Image Samples Adjacent Classes
Figure 11: Comparison of the distribution of centroids of all features in each class of ImageNet. Centroids of all output features from VIT: (a), LLB (Ours): (b), CNN with supervised pre- mre-training: (c), CNN with contrastive learning: (d). We highlighted the dots of five classes in (e) with sky-blue color.
18


Reviews:
[{"review_id": "xoCUMeJGtz", "reviewer": "Reviewer_9BG9", "summary": "This paper analyzes the inductive bias problem in existing methods and proposes Output-Domain focused Biasing (ODB) training strategy to overcome this limitation without external resources. The authors implemented ODB on  a vision transformer architecture and achieved improvements on image classification benchmarks.", "questions": "1.Is the ODB method equally effective in convolutional networks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper raises the interesting issue of inductive bias in existing methods.", "weaknesses": "1.Humans benefit from world information being able to avoid input domain bias. This actually benefits from humans having more prior information and being able to build better semantic relationships between classes. While Output-Domain focused Biasing (ODB) is more like a feature enhancement method, which improves performance through decoupling and enhancement of features.\n\n2.Figure 2 shows that the class centroids between some semantically unrelated classes are close to each other. Using triplet loss or contrast loss can also achieve the effect of widening the class centroids distance. It is recommended to add comparative experiments with this type of method.\n\n3.The paper is not easy to follow, especially the descriptions of Visual Dependency Disconnection and Non-visual Feature Structuring need more details.\n\n4.The proposed method has limited improvement in performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q9jq3ctI5a", "reviewer": "Reviewer_CuBT", "summary": "This paper addresses the issue where vision neural networks learn latent features which are (naturally) too focused on the exact pixels that are part of the training dataset -- since this is all they see, ignorant of the real-world relationships between objects (as illustrated by how ignorant a neural network can be of the distance between a mop and a komondor dog). To remedy this, the authors propose a novel method (ODB) where an auxiliary loss is introduced in order to enforce diversity in the latent vectors across classes. This loss term is disconnected from the regular discriminative visual features, so as to not pollute it. The authors show across 5 random seeds that they achieve better results on ImageNet1K than well-known baselines.", "questions": "### Detailed comments\n\n* In A.1, you refer to Tables 7a and 7c although I believe you mean Figure 7 and 7c.\n* Conclusion: the phrasing \"harbors unseen knowledge from human labeling\" was difficult to parse and ambiguous.\n* Conclusion: \"on vision transformer architecture\" >> \"on the Vision transformer architecture\"\n* Conclusion: \"in qualitative and quantitative analysis on its results\" >> \"through qualitative and quantitative analysis of its results\"\n* Table 2: Why not show the standard deviation if in effect these results were run using 5 random seeds? It seems relevant in your comparison to the baseline which is quite close (84.80 vs. 84.40). You cannot use the term 'significant' in the Ablation study section of Section 5.1 if you do not show these standard deviations.\n* Table 2: explain what w/o Pos. stands for in table caption, even if you also say it in the text (if somebody glances quickly at the table.)\n* Formatting of references generally needed, use {} around text which should be case-sensitive in the .bib-file. (e.g., \"mixup\" >> \"MixUp\" for Zhang et al.)\n* In the end of Section 5.2, it would be great to specify either in the text or in the figure caption which classes numbers 1173 and 1813 correspond to. Are they the quill and paperknife or other visually similar classes?\n* 5.1 title: quantiTative*\n* Section 4: \"doesn’t\" >> \"does not\" (too informal)\n* Section 4: \"provide series of analysis the impact\" >> \"provide a series of analysEs OF the impact\"\n* The second paragraph of the related work contains a duplicate sentence \"Lemesle et al...\". Should be removed.\n* Fig. 3 is a nice figure. Howeve,r it currently says \"L_diveristy\"  whereas I think you want to say \"L_diversity\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The ODB approach does not use additional pre-training with large datasets and/or regularization such as MixUp or CutMix\n* The performance of the approach is inspected thoroughly both quantitatively and qualitatively.\n* The authors include transparent results on how they selected their hyperparameters in the appendix.", "weaknesses": "* The term of output-domain knowledge is very vague/difficult to interpret. At one point in the introduction you instead write 'undescribed world knowledge', which I think refers to the same thing, and is more clear to me. Would it make sense to use another name than ODB? Specifically to change the OD to something else. This would make the paper clearer. Output domain is the domain where you test, which I think does not really do the job here.\n\n* In the related work, a knowledge graph is mentioned briefly while describing the method of one work. However, I miss a small section about the general idea of including hierarchical information or knowledge-graph representations into visual representations, which these authors are not the first to explore. For that reason, it becomes a bit difficult to buy when the authors claim they are the first to raise the issue of the implicit 'output-domain' knowledge missing during training, and a more complete related work section here would make the paper stronger (see for example the related work of Pan et al.)\n\n* The conclusion and future work section does not really contain recommendations for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QremWk6V53", "reviewer": "Reviewer_7b4C", "summary": "The paper shows that neural networks tend to focus on input-domain-related information, such as visual similarities, which can conflict with unseen relations determined by human labeling in the output domain. This conflict limits the generalization of models. To address this problem, the authors propose a training strategy called Output-Domain focused Biasing (ODB), which emphasizes inductive biases based on output labels. ODB consists of four steps: learning intermediate latent object features, decoupling visual dependencies, capturing structured features optimized for classification, and integrating these features for prediction.", "questions": "Is it possible for the authors to validate the efficacy of ODB in the context of domain adaptation? Is there a meaningful application of ODB in addressing input-domain bias when dealing with diverse visual domains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper aims to create inductive biases based on output labels, in order to avoid the dominance of input-domain focused bias. The motivation is relatively novel.\n2. The paper is well-structured and written in an accessible manner.", "weaknesses": "1. The experimental improvements over the baselines are relatively modest, suggesting that the significance of Output-Domain focused Biasing (ODB) may be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)