DIVINE: Diverse-Inconspicuous Feature Learning to Mitigate Abridge Learning
Authors: Saheb Chhabra
Kartik Thakral
Surbhi Mittal
Mayank Vatsa
Richa Singh
Venue: Accepted by TMLR ()

Abstract: Deep learning algorithms aim to minimize overall error and exhibit impressive performance on test datasets across various domains. However, they often struggle with out-of-distribution (OOD) data samples. We posit that deep models primarily capture prominent features beneficial for the task while neglecting subtle yet discriminative features, a phenomenon we refer to as Abridge Learning. To address this issue and encourage more comprehensive feature utilization, we introduce DIVINE (DIVerse and INconspicuous FEature Learning), a novel approach that leverages iterative feature suppression guided by dominance maps to ensure that models engage with a diverse and complementary set of discriminative features. Through extensive experiments on multiple datasets, including MNIST, CIFAR-10, CIFAR-100, TinyImageNet, and their corrupted and perturbed variants (CIFAR-10-C/P, CIFAR-100-C/P, TinyImageNet-C/P), we demonstrate that DIVINE significantly improves model robustness and generalization. On perturbation benchmarks, DIVINE achieves mean Flip Rates (mFR) of 5.36%, 3.10%, and 21.85% on CIFAR-10-P, CIFAR-100-P, and TinyImageNet-P respectively, compared to 6.53%, 11.75%, and 31.90% for standard training methods exhibiting Abridge Learning. Moreover, DIVINE attains state-of-the-art results on CIFAR-100-P, demonstrating that addressing Abridge Learning leads to more robust models against real-world distribution variations.

Published in Transactions on Machine Learning Research (07/2025)
DIVINE: Diverse-Inconspicuous Feature Learning to Miti- gate Abridge Learning
Saheb Chhabra Deportment of Computer Science IIT--ellis New Delhi, India 110020
sahebo@tiit..cc.
Kartik Thakral
thakral. 1 1diitj. aeaitt.c...a. ac.in
Deportment Computer Science and Engineering II Jodhpur, Rajasthan, India 342037
Surbhi Mittal
mittal 5.iitt in
Department of Computer Science and Engineering IIT Jodhpur, Rajasthan, India 342037
Mayank Vatsa
meatsa
Department Computer Science and Engineering IIT Jodipur, Rajasthan, India 342037
Richa Singh
richa Wattj.ac...... oc.
Deportment of Computer Science and Engineering IIT Jodhpur, Rajasthan, India 342037
Reviewed OpenReview: https: /ttps://openrevieee net/ ttr/ffrmm? id= 8NGXGTAD6
Abstract
Deep learning algorithms aim to minimize overall error and exhibit impressive perfor- mance on test datasets across various domains. However, they often struggle with out- of-distribution (OOD) data samples. We posit that deep models primarily capture promi- nent features beneficial for the task while neglecting subtle yet. discriminative features, a phenomenon we refer to as Abridge Learning. To address this issue and encourage more introduce DIVINE (DIVerse and INconspicuons comprehensive feature utilization, FEature Learning), novel approach that leverages iterative feature suppression guided by dominance maps to ensure that models engage with a diverse and complementary set of discriminative features. Through extensive experiments on multiple datasets, including MNIST, CIFAR-10, CIFAR-100, TinyImageNet, and their corrupted and perturbed vari- ants (CIFAR-10-C/P, CIFAR-I00-C/P, Tiny ManyImageNNeeC/P.. we demonst that DIVINE significantly improves model robustness and generalization, On perturbation benchmarks, DIVINE achieves mean Flip Rates (mFR) of 5.36%, 3.10%, and 21.85% on CIFAR-1O-P, CIFAR-100-P, and TinylmageNet-P respectively, compared to 6.53%, 11.75%, and 31.90% for standard training methods exhibiting Abridge Learning. Moreover, DIVINE. attains state-of-the-ate results on CIFAR-100-P, demonstrating that addressing Abridge Learning leads to more robust. models against real-world distribution variations.
1
Introduction
Deep learning algorithms have demonstrated remarkable success various domains, including image classifi-
cation TTassiri et al. 2024 Thakral al. 2024 Dehghani et al. 2023 Su et al. (2023), object detection (Wang
et al. 2023b Zong al. 2022: Tan et al. 2020)), and segmentation (Fang al. 2023 Dosi al.) 2021 Xie
et al.
2020 Despite these ad advancements, ensuring robustness and generalizability in real-workd scenarios

---

Published in Transactions on Machine Learning Research (07 2025)
Abridge Learning
Model learns input feature during training
Misclassification by the model during testing
Class
Class
Class
Learned Discriminative Feature: Paw
Missing Paw
Ignored Discriminative Feature: Ears
Diverse and Inconspicuous Feature Learning
Correct classification
Model learns input feature during training
by the model during testing
Class
Class
Class
Learned Discriminative Feature: Paw, Ears
Missing Paw,
Ear Present
Illustration of Abridge Learning (AL) and the proposed approach. model trained conventionally using AL learns only the 'paw' feature, neglecting other informative cues, causing failure when this feature is missing. Our proposed method, DIVINE, learns additional inconspicuous features such the 'ear', ensuring correct classification even when dominant features are absent.
Figure 1:
remains significant open challenge. Current supervised deep learning paradigms primarily aim to maximize accuracy by identifying and exploiting the simplest and most prominent patterns in datasets (Geirhos et al. 2020: Keshari et. al. 2019 2018). Such an approach often results models taking shorlculs, selectively
focusing on dominant input features sufficient for confident classification (Dogra al. 2024 Li et al. 2023
[lyas et al., 2019 Pezeshki al. 2021). Unfortunately, this leads to poor generalization performance on
out-of-distributtin data, as illustrated in Figure I
In practical scenarios, relying exclusively on these dominant features can lead models to become brittle under even slight variations in input data. For example, in medical image analysis, a classifier might rely excessively on the presence medical equipment artifacts rather than subtle pathological indicators. Similarly, in autonomous driving, models may overly depend on clear road markings, neglecting less prominent yet crucial signals under adverse weather conditions. Such overreliance emphasizes the critical necessity of methods designed explicitly to encourage comprehensive feature learning approach.
Defining Abridge Learning: In this research, we term this shorteut-seeking behavior Abridge Learning (AL), describing phenomenon where model overly relies on limited set of dominant features while ignoring other crucial yet subtle cues. Typically, these overlooked features fall into two distinct categories:
Inconspicuous features: Subtle yet cousal cues (e.g., detailed textures such as an animal's paw) that genuinely aid classification but reccive small magnitude gradient signals during training.
Spurious features: Non-causal correlations (e.g., background grass in images labeled as 'cow") exploited merely due to frequent co-occurrence with class labels in training data.
Conflating these two types negatively impacts generalization: amplifying spurious correlations diminishes robustness, whereas neglecting inconspicuous features leaves crucial discriminative information underutilized. The implications of such negligence become evident in varied real-world scenarios, highlighting the necessity for explicit mechanisms to balance feature reliance effectively.
2

---

Published in Transactions on Machine Learning Research (07/2025)
Formally, consider an observed input image R #xWxC generated via Structural Causal Model (SCM):
I x= xx(x( y y= fr(s),
where E Rhdh represents latent generative factors tied semantically meaningful features, and Zs C Z indicates the subset causally influencing the label e V. Given predictive model f(z; 0) parameterized by 8, trained via empirical risk minimization using cross-entropy loss:
min Ex. -yy log (x;
we formally characterize Abridge Learning as the condition when the learned predictor f(z;0") predomi- nantly exploits dominant features Zd S Zs, identified by strong gradient signals and high mutual informa- tion with labels, while neglecting inconspicuous yet predictive features Z, = Zs \ Zp. Mathematically, this phenomenon can be stated as:
(1(x;0");; > I(f(x;0);;;) with 1(2r; Y Zp) > 0.
In other words, the model's prediction far more informed by dominant features (Zp) than by inconspicuous
ones (Zz), even though the inconspicuous features still hold predictive information. Consequently, the reliance on dominant features results in significantly degraded performance under distributional shifts or feature perturbations:
E af (C(I('') > (
where s' denotes inputs with corrupted or missing dominant features. To address this challenge, we propose novel method that explicitly suppresses overly exploited dominant signals, allowing diverse inconspicuous features emerge and contribute meaningfully to classification. By enriching the learned representations with these subtle yet causal cues, our method significantly enhances robustness and generalization across varied data distributions.
While existing methods aim to enhance robustness and generalizability, they typically do not explicitly focus on identifying and learning subtle yet discriminative inconspicuous features. This limitation often results in models developing shortcut-bard based biases, causing them to falter under real- world conditions. To bridge this gap, we introduce DIVINE (Diverse and Inconspicuous Feature Learning), an approach explicitly designed to mitigate shortcut reliance and facilitate the learning of diverse set of discriminative features. Specifically, DIVINE accomplishes two primary objectives:
Research Contributions:
1. Identifying minimal yet diverse set of inconspicuous discriminative input features (see Figure !
Training unified model that robustly incorporates these identified inconspicuous discriminative features to enhance generalizability.
The diverse features discovered via DIVINE are explicitly disjoint, thereby maximizing representational diversity. Extensive experiments on benchmark datasots such as CIFAR10 (Krizhvssky & Hinton 2009). MNIST (LeCun et al. 1998), CIFAR10-C (Hendryeks & Dietterich, 2019b), and CIFAR100-C (Hendrycks & Dietterich] (2019b) highlight DIVINE's effectiveness and confirm broad applicability across multiple machine learning tasks.
2
Related Work
The phenomenon of Abridge Learning (AL), specific form of shorteut learning where deep neural networks overly rely dominant input features while neglecting subtle yet causal cues, poses a significant. challenge
3

---

Published in Transactions on Machine Learning Research (07/202)
to achieving robust and generalizable models (Geirhos et al. 2020 T This section reviews prior work on shortcut learning, highlighting its identification, miligation strategies, data augmentation techniques, and causal representation learning, emphasizing their relevance to AL. We emphasize limitations in existing approaches, thereby motivating the necessity of the proposed DIVINE method, explicitly designed to address diverse, inconspicuous feature learning for enhanced robustness.
Identifying Shortcut Learning: Shortcut learning occurs when models exploit simplistic, often non- causal patterns to minimize training loss, compromising generalization on out-of-distribution data. [Geirhos et al. (020)) formally conceptualized this issue, demonstrating that models trained on biased datasets tend to disproportionately focus on casily exploitable features, such as background textures, over semantically meaningful ones. For example, Carter et al. (2021) found that deep networks achieve high confidence predictions by relying only on a small subset of spurious pixels, reinforcing how standard training paradigms can inadvertently amplify non-causal correlations.
In critical domains such as medical imaging, shorteut learning has particularly severe implications. In med- et al. (2020) showed that models trained on medical datasets often exploit
ical contexts, irrelevant features like imaging artifacts, inflating performance metrics artificially. Such shortcuts frequently remain undetected due to evaluation metrics focused primarily on in-distribution accuracy, masking vulnera- bilities in real-world deployment. Similarly, Lapuschkin et al. (019)) discussed "Clever Hans" phenomena in computer vision and gaming contexts, employing attribution heatmaps to highlight model reliance on sim- plistic patterns. Central to these identification issues Gradient Starvation (Pezeslii et al. 2021); , where training gradients disproportionately favor dominant features, thereby neglecting subtle yet informative cues.
Mitigation Strategics for Shortcut Learning: Numerous methods have addressed shortcut learning, although few explicilly (arget AL's unique challenges. Du et al. (2020) proposed CREX, utilizing expert- guided regularization to limit model reliance on annotated features. However, the dependency on costly annotations restricts scalability, particularly for complex datasets. DFM-X (Wang et al. 2023a) leverages prior model knowledge to guide robust feature learning, yet it requires high-quality prior models which may not always be available. Methods like SADA (Zhang et al. (2023) and DDA (Gao et al., 2023) adopt adaptive frameworks, either augmenting data strategically or using diffusion models for improved robustness, albeit indirectly addressing AL by promoting data diversity without explicitly emphasizing inconspicuous features.
Recent ensemble and gradient-based techniques offer promising directions. Carter et al. (2021) reveal that a classifier can achieve high confidence while attending to as little as 5% of spurious pixel subsets, many of which are semantically meaningless to humans. Their result substantiates our contral assumption that abridge learning arises during standard training because dominant. but non-causal cues monopolize the gradient signal. The Spare mothod (Yang et al., 2024) identifics spurious correlations via gradient analysis and importance sampling, though it may prioritize dominant signals in imbalanced scenarios. COMI (Zhao) et al. 2024) integrates shorteut mitigation into empirical risk minimization, yet its efficacy can diminish with highly skewed data distributions. Addressing such imbalances, PDE (Deng et al. 2024) leverages group annotations to balance fcature learning but depends critically on accurate group labels. Additionally, specialized methods largeling niche contexts, such as privacy preservation (Ling et al.. 2024) and semantic segmentation (Kwon et al. 2024), often lack broader applicability.
Gradient- regularization provides another approach to robustness enhancement. Input gradient training methods (Ross& & Doshi Velez 2018) and Jacobian-based regularizers, including those by Varga et al. (2017)
and Jacobian Adversarially Regularized Networks (JARN) (Chan et al. 2020), optimize model gradients
to mitigate shorteut reliance. However, such techniques often require careful tuning and do not explicitly
prioritize inconspicuous features. Moreover, existing robustness evaluations (Hoffman et al. 2019 Rusak
et al. 2020; Taori et al. 2020) primarily assess performance under distributional shifts without directly addressing AL-induced feature neglect.
Data Augmentation for Robustness Data augmentation methods indirectly address shorteut learning by enhancing training diversity (Jin et al. 2024). CutOut (DeVries) 2017) randomly masks image regions, com-
pelling models to utilize broader feature sets. Extensions like CutMix Yun et al. 2019 and Mixup (Zhang
IIn the literature, Shortcut Learning broadly refers to models learning unintended strategies to minimize loss. Abridge Learning is sub--roblem where models prioritize dominant cues, ignoring other relevant features.
4

---

Published in Transactions on Machine Learning Research (07/)025)
et al. 2018 further diversify imputs through hybrid image-label constructions, albeit potentially introducing label ambiguities. AugMix (Hendrycks et al. 2019) mitigates such ambiguities by applying diverse, stochas- tic augmentations. Recent approaches like IPMix (Huang et al., 2023 NoisyMix (Erichson et al.) 2024),
DIFFUSEMIX (Islam et al., 2024a), and GenMix (Islam et al., 2024b) leverage sophisticated generative
models to further enhance data diversity. While effective, these a augmentatiins generally do not explicitly encourage learning inconspicuous features critical to addressing AL.
Causal Representation Learning: Causal representation learning seeks to identify and leverage se- mantically meaningful causal features to bolster robustness. Classical Independent Component Analysis (ICA) theory illustrates that fcature identifiability typically requires stringent assumptions (Xi & Bloem- Reddy 2023). Recent generative model studies highlight identifiability challenges in VAEs and GANs, while Lachapelle et al. (2023) proposed additive decoders to recover causal latent factors. Despite theoretical ad- vancements, practical utility remains limited without strong priors. In contrast, our discriminative approach, DIVINE, explicitly suppresses dominant signals, facilitating the emergence and integration of previously ne- glected inconspicuous causal features to enhance robustness.
The Last Layer Ensemble Li et al. 2023 tackles multiple known shortcuts by replacing the final classifier with a committee of heads and an auxiliary shift detector. In contrast, DIVINE pursues a fundamentally different route: it rebalances the gradient flow so that dominant and subtle features are learned simultaneously, thereby retaining accuracy even under severe corruptions and perturbations. Empirically, LLE is validated on two bespoke datasets- UrbanCars, where background and co-occurring objects act as controlled spurions cues, and ImageNet- W, which injects watermarks into images while DIVTNE is stress-tested across far broader spectrum that includes MNIST, CIFAR10/100, Nmnge,mageNet, and their corruption (-C) and perturbation (-P) suites.
Relationship to Last- Ensemble (LLE):
Limitations and the Need for DIVINE: While existing methods improve robustness, they often fall short in addressing AL comprehensively. Identification techniques (e.g., Lapuschkin et al. (2019) reveal shortcuts but do not provide actionable mitigation strategies. Mitigation methods like CREX (Du et al., 2020), LLE (Li et al. 2023), and PDE (Deng et al., 2024) rely on expert annotations, prior knowledge, or group information, which are often impractical. Data augmentation methods (e.g., AugMix, DIFFUSEMIX) enhance diversity but rarcly target inconspicuous features explicitly. Causal representation learning, while theoretically promising, is constrained by identifiability challenges. In contrast, DIVINE explicitly suppresses dominant signals to learn diverse, inconspicuous features, offering a scalable and generalizable solution to AL without requiring prior knowledge or extensive annotations.
3 Diverse and Inconspicuous Feature Learning
As illustrated in Figure 2 the proposed DIVINE algorithm comprises two-stage learning strategy designed explicitly to counteract the effects of Abridge Learning. In the first stage, DIVINE systematically identifies dominant input features using dominance feature map, which quantifies the relative importance of each input feature for classification. Once these dominant features are identified, the algorithm suppresses them, com- pelling the model to discover previously overlooked yet informative inconspicuous features. Subsequently, in the second stage, a unified model leverages both dominant and newly identified inconspicuous features to enhance the overall generalization performance. Tt. crucial to acknowledge that the exhaustive identifica- tion of all inconspicuous fcatures is computationally intractable. Hence, DIVINE strategically identifies a diverse subset of inconspicuous features that collectively maximize model learning elficacy, thereby effectively samples.
mitigating Abridge Learning and significantly improving robustness to
We conducted a synthetic experiment involving two informative features, 2 and 22, to explicitly demonstrate Abridge Learning. Figure a() shows that under conventional training using the cross-entropy loss, the model pre- dominantly relies on feature 22, creating a horizontal decision boundary, thus clearly ignoring the predictive capability of feature 21. Consequently, suppressing feature 22 severely reduces model accuracy unless re- trained, highlighting the pitfalls of Abridge Learning.
Illustaation of Abridge Learning and Proposed Algorithm with Synthetic Data:
5

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
Feature Suppressed Dataset
Compute
Minimite
Suppressed Dataset
Cross entropy
less
Prediction
Input Dataset
Input Dataset
Block
Fyyeyer
Block
Block
Dominance
Matrix
Consolational Neural Network
Convo uttonal Neurs Network
Model Training
Identification of Input Features
Trained Model
(a)
Unified Model Training
ceatur Maps
Dominance Matrix
Dominamee
Computt Dominance
Block
Dataset
Compute Crois-enropy
Compute Total Loss
lozs
LaSs
Convolutioal Noural Network
(b)
Figure 2: Pipeline of the proposed method for learning diverse and inconspicuous features. (a) Illustrates the process identifying inconspicuous input features. Steps and involve training the model with original images using cross-cntropy loss function. Steps 3 and 4 depict the computation of the dominance matrix for each image. step 5, a feature-suppressed dataset derived from the dominance matrix and dominance feature maps. The final step involves training the model with the feature-suppressed dataset to identify inconspicuous input features. (b) Demonstrates the unified model training process with original images.
As illustrated in Figure 3(b,, explicitly masking the dominant feature 22 during training enables the model to learn the previously neglected feature 21. This scenario confirms that 21 indeed carries predictive information that was overlooked during standard training due optimization biases favoring easily learnable dominant features. Ultimately, depicted in Figure 3(c), the unified DIVINE model successfully integrates both ž1 and 22 features, yielding diagonal decision boundary reflective of balanced feature utilization. This demonstrates DIVINE's effectiveness in overcoming the gradient bias of cross-entropy training, fostering a more comprehensive and robust feature representation.
DIVINE aims to tystematically identify diverse inconspicuous input features through dominance feature maps and subsequently train a unified model that robustly incorporates these features enhance generalization performance. The remainder of this section details the feature identification method, followed by the training methodology for the unified model.
3.1 Identification of Dominant and Inconspicuous Input Features
As illustrated in Figure 2 the proposed DIVINE algorithm is designed explicitly to counteract Abridge Learning, a phenomenon where models overly rely on dominant input features while neglecting subtle yet predictive inconspicuous features. DIVINE systematically identifies, suppresses, and subsequently leverages these inconspicuous features through comprehensive, two-stage learning approach, thereby significantly enhancing robustness and generalizability out-of-distribution data. In this section, we provide the formu- lation of each algorithmic component.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Abridge Learning: Dominant feature
Suppressed feature 2, and learn feature 2;
Unified Learning: Used both features z,and
22
-2
21
[b]
[d
Figure Illustration of Abridge Learning and the effectiveness of the proposed DIVINE on synthetic data for binary classification. (a) Under standard training, the model heavily relies the dominant feature 22: ignoring the informative yet subtle feature ø1. (b) Suppressing the dominant feature 22 forces the model to recognize and leverage the previously ignored predictive feature ž1. (c) The unified learning approach of DIVINE enables the model to utilize both 21 and 22, resulting in improved decision decision-making and robustness.
We start by defining the set of input features F, where each Fi(x) F is a masked variant of an input image x E #xxxx with identical dimensionality H W C. Each (z selectively highlights pixel regions representing features learned by the model. Typically, these regions correspond to subtle, inconspicuous features that standard training procedures overshadow in favor of dominant, highly salient cues. For instance, Figure illustrates pixels associated with an animal's paw region, exemplifying informative yet often overlooked feature.
Previous research Geirhos et al. 2020: Pezeshki et al. 2021 indicates that traditional training methods bias models toward identifying the most dominant input features. To systematically quantify pixel dominance, DIVINE utilizes the input-output Jacobian method (Chan al. 2020 Hoffman et al.] 2019). which captures each pixel's influence the model's predictive decisions. Formally, given input image and small perturbation e, the model output f(x 0) is approximated using the Taylor series expansion around x:
f(x c() + df(x;) 0((2)
(1)
dx
By neglecting higher-order terms for small perturbations, we simplify the expression to:
f(++00) f(x;0) + df(r;;)
(2)
dr
where the input-output Jacobian matrix is represented by -(x:) Since specifically compute Jacobians concerning the true class prediction, denote this the Dominance matrix (1 l'xxxx
D1(x) (I; dr
(3)
In the dominance matrix (x(x,, larger magnitude values (positive or negative) indicate pixels with substantial influence over the model's prediction, thus marking them as dominant pixels.
To identify the first dominant input feature, we select the top p% of pixels with the highest absolute dom- inance values. A binary mask M (x) XC is defined based on threshold t, computed by sorting dominance values:
(1 = { 1, if Di())
(4)
0, otherwise

---

Published in Transactions on Machine Learning Research (07 (07/2025)
X
X51
X52
X53
q
Feature suppressed Images
Original Images
Figure 4: Sample original images and the corresponding intermediate feature-suppressed images from the MNIST dataset obtained using the proposed method.
The first identified feature Fi(x) is then obtained by element-wise multiplication of the mask with the original image:
Fi(x) M1 Øz.
(5)
Subsequently, a corresponding dominance feature map Dmn (x) explicitly capturing the dominance values of Fi (x) is computed as:
Dm, (α) - (1 (1)))
(6)
3.2 Iterative Suppression and Discovery of Diverse Inconspicuous Features
To systematically uncover additional inconspicuous features and ensure feature diversity, DIVINE explicitly enforces that identified features are mutually disjoint, formally represented as:
F((z) = Vi
(7)
Achieving disjointness requires suppressing the previously identified dominant fratures. We thus suppress the initially identified dominant feature Fi(x) by setting the corresponding pixels to zero. This suppression is applied pixcl-wise follows:
I, if Dm
m, 0
if Dmn (x)
+
resulting in suppressed dataset Xs, created by applying this suppression step to all images in the original dataset X. By training a new model on X,,, recalculating dominance values, and repeating suppression iter- atively, DIVINE progressively identifies further inconspicuous features. Figure g visually illustrates original and suppressed MNIST images, clearly demonstrating how suppressing dominant features reveals inconspic- nous yet informative regions. This iterative process continues until a predefined stopping criterion is met usually involving diminishing returns in feature novelty or computational limits.
(see Subsection 4.3
3.3 Unified Model Training Incorporating Identified Features
Upon identifying a comprehensive set of diverse inconspicuous features, DIVINE advances to training a unified model explicitly designed leverage all discovered features simultaneously. Suppose k inconspic- hous input features are identified for input image v, each associated with its dominance feature map
8

---

Published in Transactions on Machine Learning Research (07/2025)
Dmn (z), Dmz (z): (z)...., De (z). These individual feature maps combined into unified dominance map
(x) through summation:
Dm (2)) = Σ Dm, (2).
(9)
1=1
The unified model [2; 0), parameterized by 0, is then trained using the original datasct X to comprehen- sively leverage all the identified features. Training employs composite loss function composed of standard cross-entropy and dominance loss to promote balanced feature utilization:
min (x 1-y" log (=; + [1)m(z) - Du (=)
(10)
where Da(x) denotes the dominance matrix computed by the unified model fa(x; The dominance loss specifically aligns the unified dominance feature map (m) with the learned dominance values (D)) of the model:
(D(x) = 11 (z) (2)
(11)
Optimizing this composite loss explicitly counteracts the bias introduced by traditional cross-entropy train- ing, which predominantly favors dominant features. Consequently, DIVINE's unified model robustly in- tegrates both previously dominant and newly identified inconspicuous features, effectively overcoming the limitations posed by Abridge Learning. As a result, DIVINE significantly enhances model robustness and generalization, especially under challenging out-of-distribution conditions.
4 Experimental Setup
The primary objective of this paper is to remove the shortcuts learned by the model via learming incon- spicuous and diverse input features. Our hypothesis is based on the observation that existing algorithms learn dominant features while ignoring other relevant features from the dataset within distribution. The performance of the models suffers when dominant features are distorted /spppeesee.. This is because the distorted/ssuppressed.
inductive bias of the model is based on the dominant feature only. To address this, the proposed DIVINE algorithm is designed to learn the dominant features along with inconspicuous features reducing the depen- dence on the dominant feature only. Since the proposed learning process introduces the model to suppressed Teatures as well, making the training diverse nature. This ensures the generalized inductive bias of the final trained model. In order to validate this hypothesis, the experiments are performed for Abridged Learn- ing the foaturo-suppress (Krizhevsky & Hinton| 2009); and Tiny ImagenaeNee. (Le & Yang 2015) datasets. These lealure-suppreseed datasets are obtained by suppressing the identified input features (described in Section 3.1
datasets corresponding to MNIST (LeCun et al. 1998), CIFAR10, CIFAR100
To further analyze the applicability of the proposed algorithm in real-world scenarios, we perform the ex- periment by evaluating the unified model on introf-distribution samples. These samples are taken from the corrupted datasets CIFARIO-C, CIFAR100-C, TinyImageNeta and perturbed datasots CIFARIO-P, CIFAR100-P, and TinyInageNet-P. The corrupted datasets contain 15 different corruptions correspond- ing to the CIFAR10, CIFAR100, and TinyImageNet datasets, respectively. We discuss the details of the corruption and perturbed datasets employed for evaluation below:
MNIST (LeCun et al., 1998) consists of 60,000 training images and 10,000 testing images, of handwritten digits from 10 different classes (0-9). Each image is a grayscale image of 28 x 28 resolution. The standard pro-defined protocol is used for ovaluation.
CIFAR10 (Krizhevsky & Hinton, 2009) contains 60,000 32 x 32 color images of 10 different classes with 50,000 images in training set and 10,000 images in testing set. The standard pre-defined protocol is used for evaluation.
CIFARI00 (Krizhevsky & Hinton 2009) contains 60,000 32 32 color images across 100 fine- grained classes grouped into 20 superclasses with 50,000 images in training set and 10,000 images in testing set. The standard pre-defined protocol is used for evaluation.
9

---

Published in Transactions on Machine Learning Research (07/2025)
TinyImageNet (Le & Yang 2015) is a large-scale dataset consisting of 100,000 images of 200 classes (500 for each class). The dataset consists of color images of 64 x 64 resolution. The standard pre-defined protocol is used for evaluation.
CIFARIO-C (Hendrycks & Dietterich, 2019b) contains 15 corruptions on the CIFARIO dataset defocus blur, contrast, pixelate, snow, fog, glass blur, brightness, elastic transform, frost, jpeg compression, shot noise, impulse noise, zoom blur, Gaussian noise, and motion blur. The images are corrupted five different levels with increasing severity.
CIFARI00-C (Hendrycks & Dictterich, 2019b) contains 15 corruptions on the CIFAR100 dataset. The corruptions used the CIFARTO-C datasel. also used this dataset. The images corrupted at five different levels with increasing severity.
TinyImageNet-C (Hendrycks & Dictterich, 2019a) contains 15 corruptions the TinyIm- ageNel dataset. The corruplions used in CIFAR10 and CIFARI00 datasets and also used in this dataset. The images are corrupted at five different levels with increasing severity. This results in a total of 75 distinct corruptions.
CIFARIO-P, CIFAR100-P, and TinyImageNec-P (Hendrycks et al., 2019): The perturbed datasets CIFARIO-P, CIFAR100-P, and TinylmageNet-P modify the original CIFAR and ImageNet datasots. These datasets have smaller perturbations compared to corruption datasets and are used measure the model's prediction stability. Each example these datasets video and we measure the model's prediction consistency. Ideally, the model should not change the prediction with the increase in perturbation intensity.
4.1 Comparison with Existing Approaches
To thoroughly evaluate the effectiveness of DIVINE, provide comparisons against relevant alternative approaches that involve input pixel manipulation or gradient-based regularization. Since the DIVINE al- gorithm systematically suppresses dominant input pixels based on dominance maps, we include Random Suppression (RS) bascline. In RS, a random subset comprising p% of imput pixels is suppressed (set to zero) during training, allowing us to determine whether the systematic selection via dominance maps provides a significant advantage over random pixel suppression.
Additionally, we compare DIVINE with the established Jacobian Regularization method from prior litera- ture (Chan et al. 2020). Jacobian Regularization is widely utilized approach for enhancing model robust- ness by reducing sensitivity to small input perturbations. Specifically, this method applies regularization to the input-output Jacobian, encouraging smoother decision boundaries and mitigating model sensitivity to specific input pixels. We implement Jacobian Regularization combined with the standard cross-entrope loss to provide a meaningful baseline for assessing the benefits of DIVINE's structured approach to dominance- based feature suppression.
Rationale for Not Comparing DIVINE with Data Augmentation Methods on Corrupted Datasets: In existing literature, various data augmentation techniques such as AugMix (Hendrycks al., 2019), CutMix (Yun al. 2019), and Mixup (Zhang al. 2018) have been proposed to enhance model ro- bustness against distribution shifts, particularly when evaluated corrupted datascts such as CIFAR10-C, CIFAR100-C, and TinyImageNet-C: Although DIVINE is evaluated using these corrupted datasets, direct comparison to these data augmentation methods is not appropriate for several fundamental reasons:
1. Different Underlying Objectives: DIVINE explicitly addresses the model's inherent bias to- ward dominant input feat by iteratively identifying, suppressing, and promoting the learning of inconspicuous features. modifies the model's intrinsic learning dynamics based on structured su- pervision via dominance matrioes. Conversoly, data augmentation methods primarily modify input data increase diversity without explicitly correcting model biases or systematically influencing the internal feature-learning processes.
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 1: Comparison of classification accuracy (%) of existing algorithms and the DIVINE on the original and feature-suppressed datascts.
Abridge Learning
Jacobian Regularization
Random Suppression
DIVINE
MNIST
Original
99.21
85.80
98.67
97.31
X,1
66.53
76.52
85.56
91.41
X,,
50.98
70.18
68.10
84.98
Xsg
40.87
62.97
54.24
75.20
Average
64.40
73.87
71.64
87.22
CIFAR10
Original
82.38
81.63
81.73
80.34
X,1
64.16
65.90
63.97
67.41
X 52.63
54.88
52.01
58.18
Xey
44.45
46.87
45.19
51.83
Average
60.98
62.32
60.72
64.44
CIFAR100
Original
74.63
75.95
75.46
74.37
Xx1
37.44
48.31
45.21
56.31
X
27.84
39.61
36,86
49.79
Xag
15.22
26.25
14.71
33.49
Average
39.23
44.6
43.06
53.54
Finy-ImageNett
Original
60.59
49.06
51.66
54.50
X
40.46
39.29
43.71
43.42
X
27.35
31.91
33.92
32.67
X
21.67
27.28
27.94
26.66
Average
37.51
36.88
39.30
39.33
2. Distinct Methodological Paradigms: DIVINE directly intervenes in the feature learning process to mitigate biases and enhance robustness. Data augmentation approaches, however, assume that generalization improvements arise indirectly through increased data variability and do not directly address model biases rolated to feature dominance or noglect.
Complementarity Rather than Competition: Positioning DIVINE directly against data aug- mentation methods could lead to misinterpretation, suggesting competition rather than complemen- tarity. DIVINE and data augmentation serve fundamentally distinct but complementary purposes. DIVINE focuses explicitly correcting featuro-learning biases, while data augmentation aims to enrich dataset diversity. Notably, inlegrating data augmentation with DIVINE may further amplify robustness, underscoring the potential synergies rather than competition between these approaches.
Hence, the scope of our evaluations and comparisons intentionally excludes direct benchmarking against data augmentation techniques, focusing instead on methods that directly influence the feature-learning process or pixel-level manipulation through explicit mechanisms.
4.2 Evaluation Metrics
We compute the classification accuracy to evaluate model performance the MNIST and CIFAR10 datascts. To evaluate the performance the corrupted images, report the Relative Corruption Error (Relative CE) and Relative Mean Corruption Error (Relative mCF) (Hendrycks & Dietterich, 2018
ES Relative CE Σ. S.C. Prrigina Σ. Eb F5 Frrgginal
(12)
where, f denotes the model to be evaluated, b denotes the bascline model obtained using Abridge Learning (AL), E Edenn and Eb Elean denote the error obtained corresponding to the model be evaluated and the AL model on original images. Ef and E denote error rates corruption at severity level corresponding
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
to the model to be evaluated and the AL model, respectively. A lower Relative CE indicates higher performance over the baseline.
To evaluate the performance on the perturbed datasets, we measure the probability that two consecutive frames with different intensity of perturbations, have "flipedd" or mismatched predictions. This is termed as mean Flip Rate (mFR) (Hendrycks et al. 2019).
4.3 Selection Criteria for Number of Feature Maps
In order to decide the number of features, we have computed the running average of the classification accuracy obtained on the original and feature suppressed dataset using AL method. We decided to iterate computing feature maps at most 3 times given the average running classification should not be below 50% of the classification accuracy obtained on the original dataset.
4.4 Implementation Details
For the MNIST dataset, a Convolutional Neural Network (CNN) architecture is used. The network consists of 5 convolutional layers, each followed by ReLU activation and maxpool. Two fully connected layers of dimensions 512 and 64 are added after the convolutional block followed by softmax. ResNet50, XceptionNet, and DenseNet121 architectures are used for the CIFARIO, CIFAR100, and TinyImageNet datascts. Models are trained on the original and feature-suppressed datasets with learning rate of 0.0001. For the CIFAR10 and CIFAR100 datasets, models trained using Adam optimizer with batch size of 32 whereas, models are trained using RMSProp optimizer with batch size of 64 for the InngemeeNeet dataset. For the MNIST dataset, the models are trained for 10 epochs and the features are suppressed with 3%. The models on the CIFARIO and CIFAR100 datasets are trained for 20 epochs and 3% is used.
For both datasets, the architectures used to train the AL models are used for training the unified model. The unified models are trained using Adam optimizer with learning rate of 0.0001 for 20 epochs and batch size of 32. Code is implemented in Tensorflow 2.3.1. The model trainings are performed
DGX station with Intel Xeon CPU, 256 GB RAM, and four 32 GB Nvidia V100 GPU cards. The source code is available at
5 Results and Analysis
The proposed DIVINE algorithm evaluated on two types of datasets: (i) feature-suppressed datasets and (ii) corruptions. In the first set of evaluations, we validate assertion of "Abridge Learning" using the MNIST, CIFAR10, CIFAR100, and TinylmageNet datasets. For corruptions, we showcase results on CIFAR10-C, CIFAR100-C, and TinyImageNet-CC datasets, We further compare the DIVINE algorithm with an algorithm proposed by Carter et al. 2021
5.1 Evaluation on Feature-suppressedd Datasets
This experiment validates that conventionally trained models rely dominant features, causing performance to degrade when those features are absent. We evaluate the performance of the AL model trained on original images using both original and feature-suppresser test sets, i.e., X, X X., and X Dataset X,, has
Feature Suppressed
Original Digtt
Original Digi-
Fexture Suppoesued
2
Bigit
2 (o]
Z [bi
4
[c]
Figure 5: Visualizations of icalnt relevant features learned by the model. (a) shows the strokes learned by the model, which distinguishes digit from 2, and (b) shows the feature-suppreseed image. Similarly, (c) and (d) show the distinguishing strokes learned by the model and the featuro-suppressed image.
12

---

Published in Transactions on Machine Learning Research (07 (025))
Table 2: Comparison of relative corruption error obtained using the existing and proposed methods for each corruption type the CIFARIO-C and CIFAR100-C datasets.
CIFAR10-C
CIFAR100-C
Corruptions
Jacobian Regularization
Random
Jacobian Regularization
Random
DIVINE
DIVINE
Suppression
Suppression
Defocus Blur
102.35
108.77
92.90
105.88
108,38
102.08
Contrast
100.19
103.55
99.64
103.88
114.32
122.66
Pixelate
95.09
113.08
75.58
108.59
106.95
74.61
Snow
91.57
81.15
97.17
99.32
Fog
105.04
111.62
99.17
111.01
110.61
110.52
Glass Blur
96. 96.11
87 62.22
73.60
98.51
98.49
82,90
Brightness
106. 40
133. 70
101.00
129.75
136.48
125 60
Flastic
103.84
109.48
97.95
107.35
106.55
96.89
Frost
100.47
113.05
88.12
104.53
111.80
83.74 74
JPEG
93.95
70.86
93.23
91.82
73.24
Shot Noise
95.65
111.77
72.62
101.47
107.27
76.46
Impulse Noise
91.70
92.17
73.02
105. 16
96.99
78.15
Zoom Blur
99.16
111.06
95.85
97.47
98.42
89.50
Gaussian Noise
95.49
110.70
73.32
101.74
107.97
79.18
Motion Blur
105.37
116.31
99.40
107.86
113.26
98.38
Relative mCE
98.32
106.29
86.30
104.90
107.24
91.53
Table 3: Classification accuracy (%) obtained using Jacobian Regularization (JR), Random Suppression
(RS), and DIVINE algorithm the Tiny TinylmageNet-CC dataset for different corruptions.
Jacobian Regularization
Random Supression
Corruptions
DIVINE
Defocus Blur
21.07
19.97
29.43
Contrast
12.37
12.51
18.70
Pixelate
27.98
31.79
39.53
Snow
25.16
26.83
32.80
Fog
21.46
23.37
33.09
Glass Blur
28.33
30.36
32.14
Brightness
27.02
28.33
36.14
Elastic
21.23
21.47
30.89
JPEG
26.30
29.07
37.25
Shot Noise
35.77
37.68
43.09
Impluse Noise
29.71
34.09
39. 14
Zoom Blur
20.07
18.83
28.20
Gaussian Noise
34.75
36.99
41.84
Motion Blur
21.47
22.11
30.87
Mean
25.19
26.67
33.79
images with suppressod dominant input feature in each image. Similarly, datascts X,, and X,, have images with two and three suppressed dominant input features in each image, respectively. Table [ shows the performance of the AL models corresponding to the MNIST, CIFAR10, CIFAR100, and TinyImageNet datasets. It is observed that the performance of the AL models degrades significantly on the feature- suppressed datasets. For instance, the performance of the AL model trained on the MNIST dataset drops from 99.21% to 66.53% on feature-suppressed dataset X,, (32.68% drop), which further degrades 50.98% on featurc-suppressed dataset X,,- This shows that the performance of the models trained using conventional methods is heavily dependent the dominant input features. On the other hand, the performance of the unified model drops by only 5.90% when evaluated on X,, As seen in Table I the unified model trained using the proposed DIVINE algorithm performs well on the featuro-suppresser datasots.
Results of the unified model in Table are compared with random suppression, and the Jacobian regular- ization method (Chan et al.) 2020). Both approaches are used to enhance the robustness the models. It is observed that existing approaches perform better than the AL model, especially on the MNIST datasct. However, the performance is not as good as that obtained using the unified model. In the random suppression method, there is supervision to the model for learning diverse set of features. While in the Jacobian
13

---

Published in Transactions on Machine Learning Research (07/2025)
regularization method, the model reduces its dependency the dominant features during training, it is not able to learn the inconspicuous features.
Figure ) & (b), show that the discriminative stroke of digit '8' (highlighted in red) is the most dominant feature differentiating it from digit '2', and is therefore suppressed. Similar observations can be made for digits '4' and '1' in Figure ((c) & (d)). Figure ( effectively illustrates the core principle of the DIVINE algorithm: iterative feature suppression forces the model to learn inconspicuous, diverse features that enhance generalization and robustness. The key insights from the Figure 6 are follows:
1. The original images contain all features, including dominant and inconspicuous ones, which the model initially uses for classification.
Feature-suppressed images represent the results of successive iterations of feature suppression. Con- sidering the first row representing the sample of class horse:
(a) The image Ts, represents the suppression of the dominant feature identified by the model specifically the horse's body.
(b) After suppressing the first dominant feature, the model identifies and suppresses the second dominant fcature i.c., parts of the horse's legs, as shown in Ezz.
(c) In the third iteration, the next set of dominant features is suppressed, which are other structural elements like part of the tail or mane, as shown in Tay
The features learned by the model during suppression are semantically meaningful. For example: The model's attention transitions from the horse's body and legs (dominant features) to the tail, mane, and even background details (inconspicuous features). This process ensures that the model does not overly depend any single set of dominant, features for classification.
The visualization of the dominance matrix computed corresponding to both the AL and unified models is shown in Figure We have made the following obscrvations from Figure 7
1. The dominance matrices D₂ (1x) corresponding to AL models are highly concentrated specific dominant input features. This confirms that AL models primarily rely on dominant features for making predictions, neglecting other potentially useful inconspicuous features.
The matrices D2(x) and Dz(#), which correspond to models trained on feature-suppresseed datasets, show a shift in focus from dominant features to inconspicuous features.
The unified dominance matrix (a(x), representing the fully trained DIVINE model, shows bal- anced focus across all identified input. features. This includes both dominant. and inconspicuous features, ensuring robustness and generalization.
X5.
x
X
Feature suppressed Images
Original Images
Figure 6: A few sample original images and the corresponding intermediate feature-suppressed images on the CIFAR10 dataset obtained using the proposed method.
14

---

Published in Transactions on Machine Learning Research (07 (225))
Table 4: mean Flip Rate (mFR) obtained using Abridged Learning and DIVINE algorithm on the C1FAR10- P and CIEAR100-P datasets for different perturbations. A lower mFR indicates better performance.
CIFARIO
CIFAR100
mFR %
Abridged Learning
DIVINE
Abridged Learning
DIVINE
Brightness
1.33
1.22
2.98
1.03
Gaussian Noise
5.21
3.86
15.30
2.21
Motion Blur
11.26
9.90
14.74
2.10
Rotate
8.29
6.24
11.12
3.09
Scale
9.55
7.99
13.20
4.87
Shot Noise
6.40
4.82
17.96
2.90
Snow
3.75
2.96
6.59
1.15
Tilt
3.13
2.49
5.52
1.55
Translate
15.63
13.53
28.34
11.79
Zaom Blur
0.79
0.67
1.78
0.32
Overall mFR
5.36
11.75
3.10
(x)
D2(x)
(x)
Du (x)
x
(x)
D2(x)
D2(x)
Du(x)
O
5
6
2
Figure 7: Illustration of the dominant and incouspicuous features obtained in the MNIST dataset. Sample original images x and their corresponding dominance matrices are shown. (x) and Da(x) are obtained from the Abridge Learning (AL) and unified DIVINE models, respectively, trained on the original dataset. In contrast, (2(x)) and Dz(z) are obtained from models trained on feature-suppressed datasets. Note the progression from the highly concentrated features in Dj (œ) to the more distributed and balanced feature focus in (u(x), which visually confirms that the DIVINE model learns comprehensive set of features.
4. The broader focus in (x) aligns with the improved performance of DIVINE feature-suppressedd datasets (as highlighted in Table I and dutto--iisti datasets.
Overall, Figure provides clear visualization of the progression from narrow feature focus (AL) to compre- hensive feature learning (DIVINE). The shift in dominance matrices underscores the success of DIVINE in mitigating Abridge Learning by promoting the learning of diverse and inconspicuous features. This balanced learning enhances robustness and generalization, as demonstrated in the experimental results.
We have also validated the above learning dynamics on the CIFAR10 dataset using Grad-CAM visualizations in Figure g The visualizations in Figure S qualitatively demonstrate that the proposed DIVINE model attends more semantically meaningful and diverse regions of the input compared the Abridge Learning (AL.) model. While AL often fixates on narrow or spurious cues such textures or localized patches, DIVINE distributes attention over broader, object-centric regions. This indicates that AL models tend to over-rely on dominant features and overlook other predictive cues, leading to limited generalization. In contrast, DIVINE effectively suppresses such dominance and recovers inconspicuous yet informative features, resulting in improved interpretability and robustness across examples.
Ablation Study for parameter pt On updating the values of p from 3% to 10%, the performance of the unified model degrades on festure-suppressed datasets. Since, the model prediction is dependent only on a few input pixels, sctting higher value of p results in suppressing of dominant as well as other input features, which in turn decreases the performance of the unified model on the feature-suppressed datasets.
5.2 Evaluation on Corruptions and Perturbations
This experiment is performed to evaluate the generalizability and robustness of the proposed unified model out-of-distributiin images. The performance on corruptions and perturbations are discussed below:
15

---

Published in Transactions on Machine Learning Research (07/2025)
Original
AL
DIVINE
Original
AL
DIVINE
Original
AL
DIVINE
Figure 8: Grad-CAM visualizations comparing attention regions of models trained using standard Abridge Learning (AL) and proposed DIVINE approach on the CIFAR10 dataset. The leftmost column in each triplet shows the original imput image, the middle column shows Grad-CAr maps for the AL model, and the rightmost column shows Grad-CAM maps for the DIVINE model. DIVINE attends to more semantically diverse, object-centric regions, while AL focuses narrowly on dominant or spurions cues.
Performance Corruption We have reported the relative corruption relative mean corruption error (Relative mCE) on the CIFARIO-C and CIFAR100-C datasets. The results are shown in Table 2 The proposed unified model outperforms random suppression and Jacobian Regularization methods on the corruption datasets. The proposed unified model gives a Relative mCE 86.30 and 91.53 corresponding to the CIFARIO-C and CIFAR100-C datasets, respectively. On comparing the performance of CIFAR10-C and CIFAR100-C datasets for individual corruptions, the proposed unified model trained with the DIVINE method out perforrmms other methods 14 corruptions (excluding snow corruption the CIFAR1O-C and contrast corruption the CIFAR100-C) and gives comparable performance on the snow and contrast corruption corresponding to CIFARIO-C dataset, respectively.
(Relative CE) and the
Table 1 shows that the performance of existing methods is comparable to the original images. However, random suppression and Jacobian regularization methods fail to generalize well on out-of-distribution images as shown in Table due the following reasons:
1. In case of random suppression, the model lacks supervision to focus on meaningful inconspicuous features. This approach does not ensure that suppressed features are either dominant or relevant, potentially leading to suboptimal learning.
In case of Jacobian Regularization (JR), it effectively reduces dependency on dominant features, however, does not actively guide the model learn inconspicuous features, leaving gap in robust feature learning.
To test our method on large-scale dataset, we computed the performance on the TinyImageNet-C dataset and report the results obtained in Table 3 On the IinyImageNet-C dataset, DIVINE yields an absolute improvement 25.54% and 19.02% over the random suppression and Jacobian regularization methods, re- spectively as shown Table 3 We observe the robust of the proposed DIVINE, algorithm against. a variety of corruptions. These results are illustrated for three different learning methods namely- Jacobian Regularization, Random Suppression, and the proposed DIVINE algorithm. From the table, it is clearly visible that the proposed algorithm outperforms other algorithms on all corruptions. We also achieve sig- nificantly better mean classification accuracy using the proposed DIVINE algorithm. This clearly describes the applicability of DIVINE algorithm on large-scale datasets as well.
Performance on Perturbations We have reported the Flip Rate (FR) and the Overall mean flip rate (Overall mFR) on the CIFARIO-P, CIFAR100-P, and TinelmageNet-P datasets. The results are shown in Tables | and 5 The proposed unified model outperforms abridge learning on the perturbed datasets. The proposed unified model gives an overall mFR of 5.36%. 3.10%, and 21.85% corresponding to the CIFAR10- P. CIFAR100-P, and TinylmageNet-P datasets, respectively. On comparing the performance of individual perturbations all datasets, the proposed unified model trained with the DIVINE method outperforms
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table mean Flip Rate (mFR) obtained using Abridged Learning and DIVINE algorithm on the TinyImageNet-P dataset for different perturbations. A lower mFR indicates better performance. TinyImageNet
mFR %
Abridged Learning
DIVINE
Brightness
12.09
9.53
Gaussian Noise
28.56
18.58
Gaussian Noise V3
56.82
42.36
Rotate
45.49
30.19
Shear
37.83
24.19
Shot Noise V2
41.11
26.99
Snow
15.66
11,26
Speckle Noise
26.40
15.26
Spockle Noise V3
38.08
Translate
41.92
31.51
Gaussian Blur
5.17
3.91
Gaussian Noise V2
47.06
32.98
Motion blur
7.66
5.36
Scale
29.87
Shot Noise
32.09
20.14
Shot Noise V3
50.29
34.85
Spatter
17.50
12.22
Speckle Noise V2
44.10
28.65
Tilt
25.26
15.45
Zoom Blur
5.74
Overall mFR
31.90
21.85
Original Images
Jacobian Regularization
Random Suppression
Proposed Unified
Sample images corrupted with impulse noise and the corresponding dominance matrices obtained using Jacobian regularization, random suppression, and the proposed unified model (Best viewed in color).
Figure 9:
Table 6: Comparison of Mean Flip Rate (mFR) of DIVINE with existing algorithms on the CIFAR100-P dataset. lower mFR indicates better performance.
Method
mFR %
MixUp
Zhang et al.
2018
8.9
CutOut
DeVries
2017
11.9
CutMix
Yun et all
2019
12.0
AugMix
(Hendrycks et al.
2019)
5.6
PixMix
[Hnndrycks et al.
2022
5.6
IpMix (Huang al.
2023
4.3
DIVINE
3.1
abridge learning. We have also compared the performance DIVINE with existing methods and the results are shown Table 0 The proposed algorithm DIVINE outperforms the second-best algorithm IpMix (Huang
et al. 2023 by mF mFR 1.2% and achieves the state-of-the-art. (sota) results.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Baseline
DIVINE
Avg Accuracy (%)
100
90
80
70
60
0.5
1.5
10
p (N)
Figure 10: Plot representing the trend line of the average classification accuracy corresponding to baseline and proposed algorithms on different values of p on the MNIST dataset.
Table 7: Asymptotic time and space complexity of DIVINE.
Time complexity
Space complexity (GPU)
DIVINE
(K E + E2) (F- 2(K E1 + E2) Thase
O(P BBWC)
The visualization of the dominance matrix obtained corresponding to the unified model and existing ap- proaches on the corrupted images of the CIFAR10 dalaset are shown in Figure g We can see that the unified model focuses on multiple imput features/regions while the existing approaches fail to focus diversely. On observing the relative corruption error corresponding to impulse noise corruption in Table 2 it is found that the error for the proposed method 73.02, which is 18.68 and 19.15 less than Jacobian regularization and random suppression methods, respectively. This shows the applicability of the proposed method real- world scenarios where external corruptions are common. Additionally, the proposed method can be used in combination with existing approaches for improving robustness.
5.3 Ablation Experiment to Visualize the Trend of Parameter
We conduct a series of experiments for varying values of p, specifically at 0.5%, 1.5%, 3%, 6%, and 10%. The trend observed average classification accuracy for these values is depicted in Figure IO and detailed results are presented in Table 1. From this representation, we notice a slight docrcase in accuracy at p=0.5% and p=1.5% for the baseline model, indicating that dominant features are still present in the datasets with feature suppression at these lower p values. However, at p=3%, there is noticeable decline in the baseline model's performance, highlighting the successful elimination of dominant features in the dataset. Consequently, we have selected p=3% as the optimal value for conducting our experiments.
5.4 Computational Complexity and Runtime
In Table we illustrate the time and space complexity of the proposed DIVINE algorithm. We let denote the number of training images, B the mini-batch size, and P' the total number of model parameters; the quantities F, G, and J are the per-batch costs of a forward pass, paramoter-gradient backward pass, and additional Jacobian- correspond to the number of suppression rounds and the epochs spent in Stages 1 and 2, while H x W x C' is the spatial resolution of an input image. Using these notations, the table shows that DIVINE's overall training time scales as 2 (KE₁ E + E2) times the baseline cost, with no change in the leading-order memory footprint.
backward pass, respectively. The hyper-parameters K, Fi, and F2
Despite this increase in training-ting time complexity, the inference-time cost of the unified model trained using DIVINE remains identical to that of standard models, which is a significant practical advantage. This addi- tional computational investment during training enables the model to learn both dominant and inconspicuous features, resulting in improved robustness and generalization. Thus, the added training cost represents a strategic trade-off for greater reliability without affecting deployment efficiency.
Further, we have calculated the temporal cost associated with the MNIST, CIFAR1O, CIFAR100, and TinyImageNet datasets during the first phase. In this phase, the model undergoes training on the dataset with suppressed features, and this process is repeated three times. The MNIST dataset requires 10 epochs
18

---

Published in Transactions on Machine Learning Research (07/2025)
Table Training time of Abridged Learning and the proposed DIVINE method on CIFAR1O dataset.
Abridged Learning
DIVINE
Time per Epoch (in seconds)
73
177
Total Training Time (in minutes)
25
60
for training, with each epoch taking approximately 14 seconds to complete. Therefore, the entire training process for MNIST takes approximately 420 seconds (7 minutes). In the case of CIFARIO and CIFAR100 datasets, each epoch lasts about 61 seconds, while for the TinylmageNet dataset, it is around 935 seconds per epoch. These time durations present opportunities for optimization, possibly through the application of methods like the one proposed by (Wang et al. 2020), which involves pruning the network at the initial stage, prior to training. It is important to note that the time required for inference (testing) remains consistent between the proposed unified model and the AL model. Further, we compute the time complexity of Abridge Learning and the proposed DIVINE algorithm for each epoch as well as the total training time. From Table we observe that additional overhead per epoch is 104 seconds, which is mostly spent on the computation of Jacobians. Since the proposed DIVINE method increases the computation time over Abridge Learning, consider this limitation of the proposed method the minimization of this overhead can be explored in the future work.
6 Limitations
We acknowledge the following limitations of the proposed DIVINE algorithm:
Computational Complexity: Although DIVINE effectively mitigates Abridge Learning, its iter- ative feature suppression and retraining mechanism significantly increases computational demands. The repeated identification and suppression of dominant features, coupled with subsequent retrain- ing, inherently requires more computational resources and time. However, this iterative approach crucial for DIVINE's ability to comprehensively learn both dominant and inconspicuous features, substantially improving robustness and generalization. To address this computational limitation, we plan to explore optimization strategies such gradient approximation methods, selective iteration approaches, adaptive early stopping criteria, and model compression pruning techniques.
Modality and Task Generalizability: The current validation of DIVINE has been conducted solely within imago-based classification tasks. While the underlying principles of DIVINE could be Uheoretically applicable across diverse modalities and lasks, its practical effectiveness in contexts such as text, audio, video remains to be empirically verified.
Applicability to Non-Semantic Dominant Features: DIVINE specifically targets semantic dominant features; thus, is not suitable for addressing Abridge Learning problems driven by non- semantic features, such color biases observed in datasets like ColoredMNIST. In these scenarios, the spurious correlation arises from attributes lacking semantic meaning, making them difficult to capture using DIVINE's feature dominance approach. Future efforts should focus on integrating complementary bias mitigation techniques, such as specialized feature attribution methods, targeted data augmentation strategies, or domain adaptation frameworks, to effectively handle non-semantic feature biases.
7 Conclusion and Future Work
Conventional deep learning methods typically prioritize optimizing classification accuracy, which often lcads to overreliance on dominant features and consequently poor generalization to out-of-distributian scenarios. In this paper, we introduced Diverse and Inconspicuous Feature Learning (DIVINE), a novel framework explic- itly designed to mitigate Abridge Learning by systematically identifying and leveraging diverse inconspicuous yet discriminative features. Extensive empirical evaluations across multiple benchmark datasets, including MNIST, CIFAR10, CIFAR100, TinyImageNet, and their corrupted variants, demonstrate DIVINE's superior
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
robustness and generalization compared to state-of-the-art basclines. Our dominance maps effectively guide the model in learning comprehensive feature representations, significantly enhancing robustness and ensur- ing reliable predictions under data corruption and distribution shifts. Future work will focus on extending DIVINE to additional modalitics and diverse machine learning tasks, addressing scenarios involving non- semanntic dominant features, and optimizing computational efficiency to broaden its applicability in practical, real-world contexts.
8
Acknowledgements
This rescarch supported by grant from IndiaAI and Mcta via the Srijan: Centre of Excellence for Generative AI. Thakral was partially supported by PMRF Fellowship, Mittal was partially supported by IBM Fellowship, and M. Vatsa was partially supported through the Swarnajayanti Fellowship.
References
Brandon Carter, Siddhartha Jain, Jonas W Mueller, and David Gifford. Overinterpretation reveals image classification model pathologies. Advances in Newral Information Processing Systems, 34, 2021.
Alvin Chan, Yi Tay, Yew Soon Ong, and Jie Fu. Jacobian adversarially regularized networks for robustness. In International Conference on Learning Representations, 2020. URL http::////penrreeeeenett///oooot id-HkeOVIrKPS
Mostafa Dchghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heck, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Yihe Deng, Yu Yang, Baharan Mirzasolciman, and Quanquan Gu. Robust learning with progressive data expansion against spurious correlation. Advances in neural information processing systems, 36, 2024.
Terrance DeVries. Improved regularization of convolutional neural networks with cutout. orXiv preprint arXiv:1708. WvXi::108.04552,, 2017.
Varun Dogra, Sahil Verma, Marcin Wožniak, Jana Shafi, Muhammad Fazal Ijaz, et al. Shortcut learning explanations for deep natural language processing: A survey on dataset biases. IEEE Access, 2024.
Muskan Dosi, Kartik Thakral, Surbhi Mittal, Mayank Vatsa, and Richa Singh. Aecnet: Attentive efficient- net for crowd counting. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1-8. IEFE, 2021.
Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. Learning credible duns via incorporating prior knowledge and model local explanation. Knowledge and Information Systems, pp. 1-28, 2020.
Benjamin Erichson, Soon Hoe Lim, Winnie Xu, Francisco Utrera, Ziang Cao, and Michael Mahoney, Noisymix: Boosting model robustness to common corruptions. In International Conference Artifi- cial Intelligence and Slalislics, pp. 4033 4041. PMLR, 2024.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1855-11969,, 2023.
Jin Gao, Jialing Zhang, Xihui Läu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11786-11796, 2023.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
20

---

Published in Transactions on Machine Learning Research (07/2025)
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019a. URI. https://
spenreview.net/foruee wtt.://ooorriidd-//ttii0oo0
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learving Represenlations, 2019b.
Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint orXiv:1807.01699 2018
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ar preprinl 1912. 02781, 2019.
Dan Hendrycks, Andy Zou, Mantas Mazcika, Leonard Tang, Bo Li, Dawn Song, and Jacob Stcinhardt. Pixmix: Dreamlike pictures comprehensively improve safely measures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16783 16792, 2022.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. arxiv preprint Xix::19908. 02729, 2019.
Zhenglin Huang, Xiaoan Bao, Na Zhang, Qingqi Zhang, Xiao Tu, Biao Wu, and Xi Yang. Ipmix: Label- preserving data augmentation method for training robust classifiers. Advances in Neural Information Processing Systems, 36:63660 63673, 2023.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alcksander Madry. Adversarial examples are not bugs, they are features. In Advances in Newrol Information Processing Systems, pp. 125 136, 2019.
Khawar Islam, Muhammad Zaigham Zahoor, Arif Mahmood, and Karthik Nandakumar. Diffusemix: Label- preserving data augmentation with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 27621 -2660, 2024a.
Khawar Islam, Muhammad Zaigham Zaheer, Arif Mahmood, Karthik Nandakumar, and Naveed Akhtar. Genmix: Effective data augmentation with generative diffusion model image editing. arXiv preprint 0rXiv:2/12.02366 2024b.
Xin Jin, Hongyu Zhu, Siyuan Li, Zedong Wang, Zicheng Läu, Chang Yu, Huafeng Qin, and Stan Z Li. A survey mixup augmentations and beyond. arXiv preprint arXic:2409.0523 052002, 2024.
Rohit Keshari, Mayank Vatsa, Richa Singh, and Afzel Noore. Learning structure and strength of cnn filters for small sample size training. In proceedings of the IEFE conference compuler vision and pattern recognition, pp. 9349 9358, 2018
Rohit Keshari, Richa Singh, and Mayank Vatsa. Guided dropout. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 33, pp. 4065-4072, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
JuncHyoung Kwon, Eunju Lee, Yunsung Cho, and YoungBin Kim. Learning to detour: Shortcut mitigating augmentation for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 819 828, 2024.
Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36:25112-255 2-25100, 2023.
21

---

Published in Transactions on Machine Learning Research (07 (025))
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1 8, 2019.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015. URL https: wwww.kaggle. http://oooottiiioooooodeeeeeeeae
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2222223,227 1998.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassmer, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one am- plifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20071 20082, 2023.
Tao Ling, SHI Siping, Hao Wang, Chuang Hu, and Dan Wang. Federated morozov regularization for shortcut learning in privacy preserving learning with watermarked image data. In ACM Multimedia 2024, 2024.
Luke Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM Con- ference on Health, Inference, and Learning, pp. 151-159, 2020.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Andrew Ross and Finale Doshi-Veler, Improving the adversarial robustness and interprelability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pp. 53 33-69. Springer, 2020.
Weijie Su, Xizhon Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai. Towards all-in-onc pre-training via maximizing multi-modal mutual information. In Proceedings of the TEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 15888-15899, 2023.
Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor-Paul Bezemer, and Anh Nguyen. Imagenel- hard: The hardest images remaining from a study of the power of zoom and spatial biases in image classification. Advances in Neural Information Processing Systems, 36, 2024.
Mingxing Tan, Ruoming Pang, and Quoe V. Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mea- suring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33, 2020.
Kartik Thakral, Harsh Agarwal, Kartik Naraya, Surbhi Mittal, Mayank Vatsa, and Richa Singh. Deephynet: Towards detecting phylogeny in deepfakes. IEEE Transactions Biometrics, Behavior, and Identity Science, 2024.
Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient rogularization improves accuracy of discrimi- native models. arXiv preprint ar arxi:: 27 09936, 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Shunxin Wang, Christoph Brune, Raymond Veldhuis, and Nicola Striscinglio. Dfinxx: Augmentation by leveraging prior knowledge of shorteut learning. arv preprint arXi::2308.000 066222 2023a.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 14408 14419, 2023b.
Quanhan Xi and Benjamin Bloem-Redee Indeterminacy in generative models: Characterization and strong identifiability. In International Conference on Artificial Intelligence and Stalislies, pp. 6912 1911-999.. PML.R, 2023.
Yutong Xie, Jianpeng Zhang, Hao Lu, Chunhua Shen, and Yong Xia. Sesv: Accurate medical image seg- mentation by prodicting and corrocting errors. IEEE Transactions on Medical Imaging, 40(1):286 296, 2020.
Yu Yang, Eric Gan, Gintare Karolina Dziugaite, and Baharan Mirzasoleiman. Identifying spurious biases early in training through the lens of simplicity bias. In International Conference on Artificial Intelligence and Statistics, pp. 2953-2961. PMLR, 2024.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the LEEE/CVP international conference on computer vision, pp.
2019.
Hongyi Zhang, Moustapha. Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In International Conference on Learning Representations, 2018. URL https:
Jiajin Zhang, Hanqing Chao, Amit Dhurandhar, Pin-Yu Chen, Ali Tajer, Yangyang Xu, and Pingkun Yan. When neural networks fail to generalize? model sensitivity perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11219-11227, 2023.
Lili Zhao, Qi Liu, Linan Yue, Wei Chen, Liyi Chen, Ruijun Sun, and Chao Song. Comi: Correct and mitigate shorteut learning behavior in deep neural networks. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 218 228, 2024.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. arXio preprint ar 2ix: 12860, 2022.
23


Reviews:
[{"review_id": "6tRBdUTQ6R", "reviewer": "Reviewer_nDyE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength: \n- the motivation of work is clearly stated.\n- the methodology is fairly simple and intuitive to understand.\n\n## Weakness:\n- This work lack a careful mathematical formulation on the Abridge Learning phenomenon.  I would recommend having a precise mathematical definition of the AL. For example, assume the data distribution follows a structure causal model (SCM), can you formulate AL learning precisely using rigorous identifiability-type argument? \n\n- The author discusseed a few prior works related to the Abridge Learning phenomenon, but to my knowledge, I don't think there is a consensus in literature that the cross entropy loss will \"identify\" the most dominant feature, particularly given that there is no precise identifiability argument supporting this. I would like to see rigorous theoretical discussion on this phenomenon, supported with an illustrating simulation example, given that the AL phenomenon serves as the main motivation of this work.\n\n-  Sec 3.1 - 3.2 deserves some clarification on the notations. Specifically, \"feature learned using the loss function Eq.1\", what does this mean? There is no $F$ in Eq.1. Also the input feature $F_i$ (first sentence in Sec 3.1) is not defined (does the definitnion of $F_i$ follows Eq(5)?) Also, for preciseness, I recommend the authors specify explicitly the input and output spaces/dimensions of the feature map, mask, etc. \n\n-  I find the notion \"the identified feature\" (right above Eq(6)) very confusing. Does $F_1(x)$  correspond to a subset of pixels? From Eq(5) it looks like the feature just represents a subspace of the original input data? (this is very different from the typical language used in representation learning).\n\n- From the cross entropy loss (Eq1.), it seems to be more natural to perform Tayloe expansion on $\\log f$ instead of $f$, and define the dominance matrix around the Jacobian of $\\log f$. I'm curious whether it's a viable option.\n\n-  The Jacobian matrix is in some sense affine equivarient, meaning that I can alter the relative scale of the Jacobian matrix values by apply affine transformations to the raw input X. With different pre-processing applied to the same original dataset, the dominance map may identify completely different features. Can the author elaborate on this?\n\n- I'm not convinced that the proposed methodology achieved the authors original goal---leaning more diverse features instead of just the dominance one. Most of the experimental results focus on looking the classification quality of the methods on corrupted test set, but not examining the learned features/latent representations. I'd like to see both a mathematical justification of this methodology in the setting of simple multinomial logistric regression (let's assume the true model is going to be linear) and an empirical validation. I think this work will benefit from having a proper identification results (even in the linear setting, mirroring the standard (non)linear ICA-type of argument, e.g., **Hyvarinen, A. and Pajunen, P. Nonlinear independent component analysis: Existence and uniqueness results.**)\n\n- Finally, I think this paper lacks a careful discussion on literature around causal representation learning, disentanglement. For example, the set of identifibility analysis on ICA mentioned above, some recent advancedment in identifiabilty on generative models (**Indeterminacy in Generative Models: Characterization and Strong Identifiability**) , and additive identifiability  (**Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation**).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the phenomenon called Abridge Learning or shortcuts, when supervised learning methods primary pick out a dominant feature that impact the training loss, and overlook other important features that are semantically relavant, hence resulting poor out-of-distribution generalization. The author proposed a pipline that enforces the model to also identify other inconspicuous features to enhance generalizability.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the discussion on weakness.", "broader_impact_concerns": "This work focuses on improving learning more diverse features of supervised learning. The work mostly focusing on methodological design. I do not forsee potential concerns on misuse.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CbnhLJKprK", "reviewer": "Reviewer_rBiM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper overall has lot of strong aspects, and some minor weaknesses which I feel are outweighed by the strengths.\n\nStrengths:\n\n* The conceptualization of \"Abridge Learning\" is clear and made distinct from the more common and general shortcut learning\n* The two-stage methodology seems sound and innovative with good technical basis\n* The paper also effectively makes use of Jacobian-based computations to identify feature importances. \n* Paper nicely evaluates multiple datasets and corruption variations.\n* The paper helps us get informative visualizations using features maps, dominance matrices. \n* Gives best performance compared to existing methods on robustness to standard perturbations\n\nWeaknesses:\n\n* I feel this method would incur high computation requirements due to multiple runs and Jacobian computations.\n* It seems limited to images, and we are not yet sure about how this method performs and translates to other modalities, but this is a minor one as it still provides a very useful technique and basis which can be expanded in the future by others. \n* I'm still unclear about how to optimally select parameters like 'p' which seem to be important. \n* How does this work apply to non-semantic dominant features like color biases etc, would this still be meaningful? \n* The work somewhat lacks theoretical analysis of when and why the method outperforms alternatives.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces novel approach DIVINE (DIVerse-INconspicuous feature lEarning) to address \"Abridge Learning,\" a phenomenon where deep learning models focus more on dominant features in data and neglect other subtle discriminative features. DIVINE employs a two-stage methodology: first identifying inconspicuous features through iterative dominant feature suppression, then training a unified model incorporating both feature types. Extensive experiments on multiple datasets demonstrate improved robustness to corruptions and perturbations, achieving state-of-the-art results on CIFAR100-P.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No major revision needed, some minor ones which the authors can consider to improve paper quality I feel is:\n\n* Providing the code is critical for reproducibility. Maybe I missed a github link in this paper, but can the authors more clearly point to code and how to implement this method?\n* A more thorough analysis of computational complexity analysis across different datasets would help. \n* A more detailed explanation on the limitation regarding non-semantic features using concrete examples can throw more light on the method and its inherent weak points.\n* some more strategies could be discussed to help reduce computational overhead. \n* a more detailed description of selecting dominance threshold 't' will help improve reproducibility", "broader_impact_concerns": "The paper doesn't present any significant ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HPnYsfl2gg", "reviewer": "Reviewer_gg3a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- S1: The problem of abridged learning is important, and addressing it would align models with human perception.\n- S2: Identifying dominant features via gradient and using them for learning inconspicuous features is intriguing.\n\n## Weaknesses\n- W1: The definitions of abridged learning and inconspicuous features are ambiguous. They are also known as shortcut learning and spurious features. Although the difference between shortcut and abridged learning is described in footnote 1, the difference remains vague, particularly due to the unclear definition of inconspicuous features.\n- W2: In the experiment, more baselines should be compared.\n- W3: Checking if abridged learning is allviated is necessary. While the robustness to image corruptions is examined, the efficacy of learning inconspicuous features is not evaluated. Shortcut learning often stems from correlations between objects and their backgrounds; evaluating robustness to distribution shifts involving such semantics would provide stronger evidence.\n- W4: A deeper analysis of the trained models is needed to better understand what features are being learned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses an abridged learning problem, in which a model primarily learns prominent features while subtle yet discriminative (inconspicuous) features are ignored, resulting in vulnerability to distribution shifts.\nTo leverage inconspicuous features, the proposed method DIVINE first identifies dominant features learned by a model and then suppresses them to focus on inconspicuous features.\nExperimental results show that DIVINE improved classification accuracy and robustness to image corruption.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major changes\n- For W1: Clarifying the definition of inconspicuous features and the difference from spurious features would be persuasive. Discussing related works of shortcut learning focusing on objects, e.g., [a] and [b], and why the existing works on spurious features cannot handle inconspicuous features would also be convincing.\n- For W2: Adding methods for suppressing spurious features discussed in the related work section would be helpful.\n- For W3: Experimenting on datasets containing known shortcuts, such as waterbirds and CelebA, or datasets used in other related works, would support the efficacy of learning inconspicuous features. Testing distribution shifts other than image corruption, e.g., DomainNet, PACS, or ImageNet-A/R, would also demonstrate the robustness.\n- For W4: Visualizing learned features using explanation methods, such as Grad-CAM, would be beneficial.\n\n\n[a] Singh et al., Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias, CVPR2020.   \n[b] Choi et al, Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition, NeurIPS2019.\n\n## Minor changes\n- The use of the term abridged learning is inconsistent. In the introduction, it is described as a problem arising during training, but in experimental sections (e.g., Tables 1 and 4), it is framed as a training strategy. This inconsistency should be addressed for clarity.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)