Title: H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hilbert-Schmidt Independence Criterion (HSIC)

Salient Feature Learning

Latent Space Separation
Abstract: We introduce H-SPLID, a novel algorithm for learning salient feature representations through the explicit decomposition of salient and non-salient features into separate spaces. We show that H-SPLID promotes learning low-dimensional, task-relevant features. We prove that the expected prediction deviation under input perturbations is upper-bounded by the dimension of the salient subspace and the Hilbert-Schmidt Independence Criterion (HSIC) between inputs and representations. This  establishes a link between robustness and latent representation compression in terms of the dimensionality and information preserved. Empirical evaluations on image classification tasks show that models trained with H-SPLID primarily rely on salient input components, as indicated by reduced sensitivity to perturbations affecting non-salient features, such as image backgrounds.

H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition

Abstract

We introduce H-SPLID, a novel algorithm for learning salient feature representations through the explicit decomposition of salient and non-salient features into separate spaces. We show that H-SPLID promotes learning low-dimensional, task-relevant features. We prove that the expected prediction deviation under input perturbations is upper-bounded by the dimension of the salient subspace and the Hilbert-Schmidt Independence Criterion (HSIC) between inputs and representations. This establishes a link between robustness and latent representation compression in terms of the dimensionality and information preserved. Empirical evaluations on image classification tasks show that models trained with H-SPLID primarily rely on salient input components, as indicated by reduced sensitivity to perturbations affecting non-salient features, such as image backgrounds.

1 Introduction

The acquisition of salient, task-relevant features from high-dimensional inputs constitutes a fundamental challenge in representation learning. Such features offer multiple advantages, including reduced dimensionality [2], enhanced generalization and transferability [42, 30], and improved robustness [22, 12]. Nevertheless, learning true salient features remains challenging, as many neural networks operate within a single, entangled latent space that mixes task-relevant signals with redundant information [6, 38]. We illustrate this sensitivity using a simple diagnostic test in Figure 1: a classifier trained to predict the left digit in an image of double digits should ignore perturbations to the right digit, which is irrelevant to the label. However, in practice, we observe that neural networks with high test classification accuracy on the left digits exhibit a significant performance drop when subjected to a high-magnitude adversarial PGD [34] attack ( ϵ = 1 . 0 ) on the right digits, revealing their dependence on irrelevant, non-salient features. This corroborates several empirical and theoretical studies [3, 36, 19] showing that redundant dimensions enhance vulnerability to attacks. Driven by these findings, we introduce H-SPLID (HSIC-based Saliency-Preserving Latent Information Decomposition), a new method that learns salient features by explicitly decomposing the latent space into two subspaces coupled with information compression regularization: a low-dimensional salient space , which contains features essential for classification, and a non-salient space , which captures the remaining input variability. Training the same neural network as above with H-SPLID alleviates the dependence on irrelevant features to a large degree, as shown on the right side of Figure 1. Crucially, H-SPLID is significantly less sensitive to right digit perturbations, without any prior knowledge of the redundant region or adversarial training.

Non-Salient Space

2. Attack right digit

after training

1. Train to

classify left digit

H-SPLID

Cross-Entropy

ACC: 96.86 %

ACC: 8.57 %

ACC: 97.14 %

ACC: 87.46 %

ACC: 9.41 %

ACC: 31.76 %

Cor

Cla

We extend this analysis in our experiments by applying adversarial attacks to the background of COCO [28] images, applying image corruptions to medical images of skin lesions [9] and further show that H-SPLID improves the transfer accuracy of ResNet-based [20] classifiers trained on ImageNet [11] under real-world perturbations [58, 55]. In addition to our empirical evidence, we theoretically prove that the expected change in predictions under input perturbations is bounded by the dimension of the learned salient subspace and the Hilbert-Schmidt Independence Criterion (HSIC) [16] between inputs and salient representations. This establishes a formal link between robustness and the salient representation. Our main contributions are:

We propose H-SPLID, a novel algorithm that promotes the learning of salient features by decomposing the network's latent space into salient and non-salient subspaces.

We prove that the two key design components of H-SPLID, namely, dimensionality reduction of the salient subspace, together with the HSIC between inputs and salient latent representations, upper bounds the expected change in predictions under input perturbations. Moreover, we show that the above HSIC and reduced salient space dimensionality bounds the volume of the input domain that is vulnerable to perturbations.

We empirically demonstrate that H-SPLID learns salient features by leveraging attacks and other perturbations against non-salient regions of an image, such as its background.

2 Related Work

Salient Feature Learning. Saliency methods in interpretability aim to identify input features that influence a model's prediction the most. Traditional post hoc approaches include gradient-based methods [44, 48], Class Activation Maps (CAMs) [43, 54], and perturbation-based methods [31, 41]. Unlike post hoc interpretability methods, however, H-SPLID aims to learn latent salient features for a given task, such as image classification. Existing works on salient feature learning include saliencyguided training for interpretability [22] and saliency-based data augmentation [8, 5, 52] methods that can complement our approach. However, H-SPLID does not use saliency maps as an auxiliary signal to improve training [8, 52], or pretrained models [5] to generate them. Moreover, the division of the latent space into 'salient' and 'non-salient' spaces, as in H-SPLID, is comparatively unexplored in literature. Contrastive Analysis (CA) methods [2, 1, 57] leverage this concept by learning explicit 'common' and 'salient' latent spaces with separate encoders via external supervision. While they

share the idea of learning separate spaces with H-SPLID, they rely on a dedicated target dataset containing the salient class, and a background dataset with samples exhibiting non-salient features. In contrast, our method does not rely on external data, and learns an initial unified latent space, before partitioning it into salient and non-salient dimensions.

Feature Decomposition and Selection. Feature selection methods aim to identify a subset of input features that is most predictive of the target variable [61, 18], with popular approaches including L 1 regularization [49] and Group-Lasso regularization [62]. Similar to these methods, H-SPLID transforms and selects features during training, but diverges in its approach through its decomposition of the latent space. For the task of clustering, Miklautz et al. [37] recently introduced the idea of latent space partitioning, whereas H-SPLID embeds this split directly into a classifier's training loop, using labels to shape the salient vs. non-salient space. Moreover, H-SPLID incorporates the HSIC penalty [32] to regularize the statistical dependence between the inputs and salient features from each subspace, ensuring that the salient subspace retains only task-relevant information while the non-salient subspace absorbs redundant variability, thereby reducing feature dimensions.

Adversarial Robustness and Saliency. We use adversarial attacks to evaluate the quality of the learned salient features. Several studies have begun exploring the interplay between saliency and robustness [12, 51, 17, 29]. Among these methods, many require adversarial training [34], which is not only computationally demanding, but also tailored to specific attacks. An alternative line of research seeks to enhance robustness without adversarial training. Multiple works [19, 3, 56, 13] have attributed adversarial vulnerability to the network's reliance on high-dimensional, task-irrelevant features. For instance, Alemi et al. [3] hypothesize that neural networks falsely rely on task-irrelevant features from the training data, negatively impacting robust generalization. Melamed et al. [36] show, under a simplified two-layer model, that when data is confined to a low-dimensional manifold, there exists an off-manifold space in which weights remain mostly unchanged and can be exploited by adversarial perturbations. Haldar et al. [19] demonstrate that when there are redundant latent dimensions, off-manifold attacks can lead to decision boundaries that rely on task-irrelevant feature dimensions. Fischer [13] introduced an information bottleneck [50] to compress input information and preserve task-relevant features without adversarial training. Wang et al. [56] extends this framework to the HSIC bottleneck [32], upon which H-SPLID is built. Importantly, H-SPLID departs from the HSIC bottleneck penalty, and introduces separate terms for salient and non-salient features. Our method also improves upon the guarantees of Wang et al. [56], tightening their bounds to account for the impact of the dimensionality reduction induced by H-SPLID.

3 Methodology

3.1 Problem Setup

We consider k -class classification over dataset D = { ( x i , y i ) } n i =1 ⊆ X × Y , where X ⊆ R d is a compact input space, Y ⊆ R k is the label space, and each input x i ∈ X is associated with the corresponding one-hot class label y i ∈ Y . The neural network h θ : R d → R k consists of an encoder followed by a linear layer. The encoder, denoted by f ψ : X → R with parameters ψ ∈ R p , maps an input x to a latent representation z ∈ R ⊆ R m , i.e., z = f ψ ( x ) . The linear output layer, g W : R → R k , computes the k logits using parameters W ∈ R k × m , i.e., g W ( z ) = Wz . Thus, we can express the neural network h θ = g W ◦ f ψ with parameters θ = { ψ , W } . Hence, the prediction of sample i is ˆ y i = W f ψ ( x i ) . Lastly, parameters θ = { ψ , W } are trained by minimizing the cross-entropy loss with a softmax layer, i.e., L ce ( D ; θ ) = 1 n ∑ n i =1 ℓ ( x i , y i , θ ) , where ℓ ( x , y ; θ ) = -∑ k j =1 y j log( σ j (ˆ y )) and σ i ( y ) = e y i / ∑ j e y j is the softmax function σ : R k → R k .

3.2 Saliency-Aware Latent Decomposition

We introduce a representation learning framework that separates latent features into salient (i.e., task-relevant) and non-salient (i.e., task-irrelevant) components. The model is trained with a structured objective that integrates classification, geometric regularization, and statistical independence constraints, producing representations that improve both predictive performance and robustness. An overview of our method is shown in Figure 2 .

Given the encoder f ψ , we introduce a learnable diagonal mask matrix M s = diag { β } ∈ { 0 , 1 } m × m , where β ∈ { 0 , 1 } m selects salient (task-relevant) features. The complementary non-salient mask is defined as M n = I -M s . The latent representation z = f ψ ( x ) ∈ R is then decomposed into the following salient and non-salient representations: z s = M s z = β ⊙ z and z n = M n z = ( 1 -β ) ⊙ z , with 1 being a vector of 1 .

Define ( y i ) j as the j -th element of the y i label. Then, classification is performed using only the salient component of the latent space z s ∈ Z , and the corresponding cross-entropy loss is given by:

3.3 Regularizing and Preserving the Separated Features

While the saliency masks enable latent space partitioning, the quality and stability of this separation depend on additional constraints that encourage discriminative utility. We achieve this via two regularization mechanisms: masked clustering losses and Hilbert-Schmidt Independence Criterion (HSIC) [16] penalties.

Let X = [ x 1 , . . . , x n ] ∈ R d × n be the matrix of input vectors and Z ≡ f ψ ( X ) ∈ R m × n be the matrix of latent vectors for n samples, with masked variants z s = M s Z and Z n = M n Z . For each class k , let C k denote the set of indices with label k , and define µ k as the class centroid and µ as the global centroid of latent vectors. We define the following masked norm-based losses [37]:

Loss L s encourages the clustering of class-specific representations in the salient subspace, strengthening its discriminative capacity. Each class has a simple uni-modal form which further removes redundant information. The loss L n aligns the other features globally across samples in the nonsalient space and captures shared variation. Moreover, by capturing task-irrelevant variations-such as background information-non-salient features help isolate predictive factors in the salient space, enhancing robustness and disentanglement. To further promote robust but accurate decompositions, we incorporate two additional HSIC terms:

As HSIC is a measure of similarity (see Appendix A), the term ̂ HSIC( X , Z s ) reduces the dependence between the input features and the salient subspace, thereby removing redundant information. Similarly, ̂ HSIC( Y , Z n ) reduces the dependence between label information in the non-salient subspace. We use the unbiased empirical estimator [16] of HSIC:

where K x and K z have elements K x ( i,j ) = k x ( x i , x j ) and K z ( i,j ) = k z ( M s z i , M s z j ) , and H = I -1 n 11 ⊤ is the centering matrix.

Putting everything together, we define H-SPLID's overall training objective as:

where Z s , Z n ∈ R m × n are the concatenated salient and non-salient latent representation, and λ ce , λ s , λ n , ρ s , ρ n ≥ 0 are scalar weights. Training amounts to solving the following constrained optimization problem:

3.4 The H-SPLID Algorithm

We solve Problem (6) using an alternating optimization procedure over the neural network parameters θ and the diagonal mask matrix M s ∈ R m × m (see Algorithm 1 in Appendix B). At each outer epoch t , the procedure consists of two alternating steps:

(a) Latent Representation Update (Fix M s , optimize θ ) : Given a fixed mask M ( t -1) s , we update the encoder parameters θ = { ψ , W } by minimizing the loss L as in Eq. (5) using minibatch stochastic gradient descent with B ⊂ D for an epoch:

where the class means µ k and global means µ are computed based on the minibatch via Eq. (6c).

(b) Mask Update (Fix θ , optimize M s ) : With updated latent representations z ( t ) i = f ψ ( t ) ( x i ) , we optimize the following optimization problem to learn the masks M ( t ) s , M ( t ) n :

Miklautz et al. [37] show that Prob. 7 has a closed-form solution:

where the global mean µ and class means µ k are computed from the full dataset via Eq. (6c). We use a moving average when updating the masks to improve convergence (See Algorithm 1 line 11 in Appendix B). The above optimization yields a continuous mask β ∈ [0 , 1] m . After convergence, we obtain a binary version by thresholding each entry at 0 . 5 . In practice, using the continuous mask directly gives similar results, as the learned values typically concentrate near 0 or 1 .

3.5 Theoretical Guarantees

In our experiments, we study whether H-SPLID relies on salient vs. non-salient features by examining the trained network's response to perturbations to task-irrelevant portions of the input (e.g., the right digit in Fig. 1, an image background in the COCO dataset in Section 4, etc.). Wang et al. [56] showed that HSIC regularization terms promote feature invariance and improve robustness even without adversarial training; we extend their analysis by integrating HSIC regularization (Eq. (3)) with salient space isolation (Eq. (2)), which structurally separates class-discriminative and redundant information in the representation space. To do so, we make the following assumptions on the kernel families used in the regularization terms Eq. (3):

Assumption 3.1 (Kernel Function Boundedness and Universality) . Let K x : X × X → R d × d and K z : Z × Z → R k × k be continuous positive-definite kernels defined on compact metric spaces X and Z , respectively. Let F and G denote their associated RKHSs. We assume that:

The kernels are universal on X and Z , i.e., F is dense in C ( X , R d ) and G is dense in C ( Z , R k ) under the supremum norm topology;

All functions in these RKHSs are uniformly bounded in the pointwise 2 -norm, that is,

Many widely used kernels are known to be universal on compact subsets of R d , including the Gaussian (RBF), Laplacian, and Matérn kernels [46, 45]. Universality ensures that the RKHS is rich enough to approximate any continuous function on the domain, while boundedness holds automatically on compact input spaces when the kernel is continuous. These properties collectively justify the use of kernel-based function classes for comparing or aligning with neural network outputs, particularly when both are assumed to operate over compact, bounded input spaces.

As is common when modeling bounded inputs [33, 40], we use the truncated multivariate normal (tMVN) distribution: x ∼ N R (0 , σ 2 I d ) , with density ˜ p ( x ) = 1 C exp ( -∥ x ∥ 2 2 σ 2 ) · 1 ∥ x ∥≤ R , where

R > 0 denotes the truncation radius, σ 2 > 0 is the variance parameter, and C is the normalization constant. Our main robustness guarantee shows that the sensitivity of the model prediction is controlled by (a) the dimensionality of the salient space and (b) the HSIC between the inputs and salient representations.

Theorem 3.2 (HSIC-Based Robustness Bound) . Let x be sampled from a tMVN distribution x ∼ N R (0 , σ 2 I d ) , and let the neural network h θ : R d → R k be differentiable almost everywhere with an L -Lipschitz encoder f ψ and a bounded linear output layer with ∥ W ∥ ∞ ≤ B . Suppose the RKHSs associated with K x and K z satisfy Assumption 3.1, with kernel sup-norm bounds K F , K G . Let s := ∥ M s ∥ 0 be the count of non-zero entries of the salient mask. Then, for all perturbation maps δ : R d → R d such that ∥ δ ( x ) ∥ 2 ≤ r for all x ∈ X , the expected output deviation satisfies

where z s := M s f ψ ( x ) is the salient representation and θ = { W , ψ } denotes the collection of neural network parameters.

The proof of Theorem 3.2 can be found in Appendix C. Intuitively, with stronger information compression imposed by minimizing both the HSIC term (thereby reducing HSIC( x , z s )) and the masks (thereby reducing the salient mask support s := ∥ M s ∥ 0 ), the model is forced to rely only on salient features: Theorem 3.2 suggests that, in this case, the perturbation is more likely to end up in the non-salient space, and the majority of the attack does not contribute to the change of the output of the neural network, as ∥ h θ ( x + δ ( x )) -h θ ( x ) ∥ stays small for any perturbation map δ ( · ) with ∥ δ ( x ) ∥ 2 ≤ r . From a technical perspective, our theorem differs from Wang et al. [56] in two aspects. First, we sharpen the dependence of the upper bound on the power of the perturbation; this allows us to explicitly link it to the dimension of the salient mask s . Second, we extend their binary classification framework (i.e., k = 1 with one output value) to multi-class classification (with arbitrary k ) to cover a wider range of classification models.

Additionally, we can quantify how perturbations will trigger prediction changes from the volume of the entire input domain. In particular, we define the salient-active region as X s ( ϵ ) := { x ∈ X R : ∥∇ x h θ ( x ) ∥ F > ϵ } . Under the tMVN distribution, the probability that an input falls into this region equals its measure: µ ( X s ( ϵ )) = P ( ∥∇ x h θ ( x ) ∥ F > ϵ ) . We can bound this probability as follows:

Corollary 3.3 (Salient Region Volume Bound via HSIC) . Under the same assumptions of Theorem 3.2, for any threshold ϵ > 0 , the probability that a random input falls into the salient-active input region is upper bounded by

(a)

(b)

1

Thus, Corollary 3.3 implies that the volume of the salient-active region is tightly controlled by the dimensionality of the salient space and the HSIC between inputs and the salient representation. The proof is in Appendix E.

4 Experiments

In this section, we describe our datasets, comparison methods, experiment setup, and performance metrics. Additional runtime experiments, implementation details, and hyperparameter tuning and configuration protocols provided in the Appendix F. Our code is publicly available at https:// github.com/neu-spiral/H-SPLID .

Datasets and Encoder Models We evaluate H-SPLID on synthetic and natural image benchmarks, spanning five datasets and three architectures (Table 1). We create a synthetic Concatenated-MNIST (C-MNIST) dataset (see Fig. 1) by concatenating two MNIST digits with the left digit as the class label. We use LeNet-3 [27, 26] as an encoder. We construct a four-class subset of COCO [28] ( bear , elephant , giraffe , zebra ), coupled with a ResNet-18 [20] encoder. ISIC-2017 is a medical imaging dataset [9]. ImageNet-9 (IN-9) [58], encompasses 368 classes from ImageNet-1K instantiated in three variants: Original images, a MixedRand variant in which object foregrounds are put onto randomclass backgrounds, and an Only-FG variant with backgrounds entirely removed (See Figure 4). CounterAnimal (CA) [55], splits iNaturalist wildlife photos into a Common set (exhibiting typical backgrounds) and a Counter set (featuring atypical yet plausible backgrounds) (see Fig. 3a). We use ResNet-50 as an encoder for ISIC-2017 and ImageNet derived datasets.

Comparison Methods. To demonstrate how H-SPLID focuses on task-relevant characteristics, we compare it with several methods for feature selection and weight regularization, including weight decay ( L 2 regularization) [25], L 1 regularization [49], Group-Lasso regularization [62], and two activation-sparsity variants - one applying L 1 penalty to the penultimate layer's activations and another applying a Group-Lasso penalty across those activations to promote instance-level sparsity. Finally, we compare against HBaR [56] under non-adversarial training, and a vanilla baseline trained only with cross-entropy loss. Appendix F provides further details on training and implementation.

We train H-SPLID and all comparison methods exclusively on clean data without employing adversarial attacks or having access to saliency masks. In all datasets, we employ a 80-20 validation split for tuning, and use held-out test sets for final evaluation. Following prior art [32, 60, 59, 56], we use the Normalized Cross Covariance Operator [14] to get a scale-insensitive HSIC penalty. All methods are evaluated using clean test accuracy (over three seeds) and robust test accuracy under different attacks, described below.

Testset Attacks. Methods are evaluated w.r.t. a broad array of attacks on non-salient features at test-time. On C-MNIST, we evaluate predictive performance against a PGD attack on the (non-salient) right digit. For COCO experiments, we pretrain a ResNet-18 [20] from random initialization for 100 epochs with cross-entropy, followed by 200 epochs of method-specific training before evaluating on the held-out test set. We test PGD and AA in three ways: random blocks of pixels in the background, pixel perturbations in the background, and full-image attacks. On ISIC-2017, we use a ResNet-50 pretrained on ImageNet for feature extraction, train a three-class head for 50 epochs, and then run 50 epochs of method-specific training. We test robustness under real-world corruptions (brightness, defocus blur, and snow/occlusion from the corruptions benchmark [21]) applied to non-salient regions (non-lesion pixels). We use IN-9 and CA for transfer learning experiments as follows. First, we train a ResNet-50 [20] initialized from ImageNet-1K pretrained weights (TorchVision [35]) for 20 epochs of method-specific training on ImageNet-1K. Then, we test the model on the IN-9 (the original IN-9 and its MixedRand and Only-FG variants) and also on CA (CA-Common and CA-Counter) evaluation sets (see Table 4).

PGD is implemented via 10 iterations with a step size α = 0 . 0156 and AutoAttack [10] is implemented using the rand ensemble. Attacks per block (Block Atk., see Fig. 3b) are confined to a single randomly placed block in the background, with size 1 4 × 1 4 of the image dimensions. Attacks restricted to background pixels (Block Atk., Background Atk., see Fig. 3b) use saliency masks, which are available for COCO and ISIC-2017. Full attacks (Full Atk.) are across the entire image. PGD and AA Attacks are conducted over a range of ϵ levels, with each configuration repeated across five random seeds. Additional implementation details and hyperparameter settings are provided in Appendix F.

No Perturb.

Brightness

Defocus

Occlusion

Va.

75.45

±

0.986 66.43

±

2.527 63.77

±

2.388 62.87

±

3.081

WD

75.63

±

1.545 67.53

±

2.980 64.57

±

2.295 63.55

±

3.851

LSA

75.62

±

1.211 66.50

±

2.171 64.33

±

1.653 62.27

±

4.256

LSW 75.30

±

1.040 66.13

±

2.432 63.22

±

2.506 62.70

±

3.810

GLA 75.38

±

1.383 66.50

±

3.136 61.32

±

4.501 62.68

±

2.969

GLW 70.65

±

4.118 60.23

±

6.698 58.63

±

9.564 60.62

±

5.697

HBaR75.90

±

0.844 68.70

±

1.942 65.62

±

2.058 66.18

±

3.013

Ours

76.78

±

0.778 70.00

±

1.619 68.38

±

1.376 69.50

±

1.716

4.1 Results

Controlled attack Benchmark COCO. We quantitatively demonstrate the ability of H-SPLID to learn salient features on the four-class COCO benchmark by evaluating adversarial robustness under block, background, and full-image perturbations. As shown in Table 2, H-SPLID achieves 57.5% under background-only PGD attacks at ϵ = 3 / 255 , with the closest competitor attaining 43.0%. Even when attacks span the entire image, H-SPLID sustains 58.9% accuracy under a PGD attack with ϵ = 2 / 255 , surpassing the 34.2% of the vanilla network and 41.9% of the best performing competitor. Against the stronger AutoAttack ensemble, H-SPLID consistently outperforms all baselines in robustness to adversarial perturbations.

These results show that explicitly decomposing latent features into salient and non-salient subspaces delivers substantial robustness gains, with the most pronounced improvements occurring under background-only perturbations, validating that H-SPLID effectively isolates redundant information. Moreover, robustness gains are achieved without any adversarial training, demonstrating that HSPLID's latent decomposition strategy yields inherently saliency preserving representations.

Medical imaging Benchmark ISIC-2017. To further assess domain generality and robustness, we evaluate H-SPLID on the ISIC-2017 skin lesion classification dataset [9] (three classes: nevus, melanoma, seborrheic keratosis). We perturb only non-salient regions (e.g., non-lesion pixels) and adopt three real-world corruptions from the corruptions benchmark [21]: brightness (lighting), defocus blur (blur), and snow (which effectively occludes small patches; we report it as occlusion ). Results are averaged over 10 random seeds.

These medical imaging results mirror our COCO findings: explicitly separating salient from nonsalient latents confers consistent robustness gains under realistic, non-adversarial corruptions, especially when perturbations target only non-salient regions. Together with Table 2, this strengthens the evidence that H-SPLID learns saliency-preserving representations that generalize beyond natural images to specialized clinical domains.

Saliency Benchmarks. We measure the saliency of our model on the ImageNet-9 and CounterAnimal benchmarks. In Table 4, H-SPLID attains the highest accuracy on the IN-9 test set (76.7%), outperforming the vanilla baseline by 2.7% and exceeding the next best regularization method by over 1%. When the backgrounds are entirely removed (Only-FG), H-SPLID once again surpasses all methods with a 64.5% test accuracy, demonstrating its ability to distill object-centric features. On the more challenging MixedRand variant, it achieves a 59.5% test accuracy, a substantial 3.1% gain over the strongest baseline. On the CA Common set, which preserves typical contextual correlations, H-SPLID matches the top performing method (80.3% vs. 80.7%). Finally, on the CA Counter set of atypical contexts, it surpasses all competitors with a 60.6% test accuracy, a 2.1% improvement over the HBaR model. The consistent performance across original, background-altered and contextually shifted datasets demonstrates that the explicit separation of salient and non-salient subspaces in H-SPLID yields representations that transfer more robustly to new tasks and real-world perturbations.

Loss Term Ablations. We ablate the loss components according to their conceptual grouping, namely cross-entropy ( L ce ), cross-entropy loss with space separation ( L ce + L s + L n ), cross-entropy loss with HSIC components ( L ce +HSIC( X , Z s ) + HSIC( Y , Z n ) ) and the full H-SPLID loss ( L ). The mask computation (Section 3.4) remains unchanged across all ablations (the difference is whether the clustering loss terms L s + L n are optimized). All ablations are performed starting from the best-performing COCO model (Appendix F.4) by removing individual components of the full L objective. Each loss combination was independently tuned to achieve its best performance. As shown in Table 5, simply using the cross-entropy loss yields poor background robustness (33.75%). Adding the L s and L n terms or the HSIC penalties improves robustness to approximately 43-46%, while maintaining clean accuracy above 96%. The complete objective results in the best robust performance (57.12%) while maintaining competitive clean accuracy (97.59%). We further assess the sensitivity of H-SPLID to its hyperparameters in Appendix G.2.

5 Limitations & Conclusion

Limitations. H-SPLID assumes the presence of irrelevant information in the input, as well as a sufficiently diverse dataset in which class-specific features occur across varying contexts. If a particular feature always co-occurs with the same context, H-SPLID cannot separate salient from non-salient information, since both appear inseparably - a challenge that would require external knowledge to resolve. Further, we restricted our analysis to image data, where the distinction between salient and non-salient regions is intuitive to humans. Investigating whether similar decompositions apply to other data modalities remains an exciting direction for future work.

Conclusion. We introduce H-SPLID, a novel method for salient feature learning that decomposes the latent space of a neural network into task-relevant and task-irrelevant components during training. Unlike prior work, H-SPLID performs supervised feature selection in an end-to-end manner, without relying on external saliency annotations. Our theoretical analysis provides formal insight into how this decomposition promotes compact and informative representations. Empirically, we show that H-SPLID learns class-discriminative features and naturally reduces reliance on irrelevant input variations. In future work, we would like to combine H-SPLID with self-supervised models such as I-JEPA [4], with the goal of learning features that generalize better to downstream tasks. Additionally, we plan to explore the decomposition of salient and non-salient spaces in other data modalities, including graphs, text, and multi-modal data.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge support from the National Science Foundation through grant CNS2414652. Further, this work is supported in part by NIH 5U24CA264369-03. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the MareNostrum supercomputer (hosted at BSC, Spain), MeluXina (operated by LuxProvide, Luxembourg), Deucalion (hosted at the Minho Advanced Computing Center, Portugal), and Discoverer (hosted at Sofia Tech, Bulgaria) through EuroHPC Access allocations.

References

A. Abid and J. Y. Zou. Contrastive variational autoencoder enhances salient features. ArXiv , abs/1902.04601, 2019. URL https://api.semanticscholar.org/CorpusID:61153677 .

A. Abid, M. J. Zhang, V. K. Bagaria, and J. Zou. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature Communications , 9(1):2134, May 2018. ISSN 2041-1723. doi: 10.1038/s41467-018-04608-8. URL https://doi.org/10.1038/ s41467-018-04608-8 .

A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR , pages 15619-15629. IEEE, 2023.

B. Aydemir, D. Bhattacharjee, T. Zhang, M. Salzmann, and S. Süsstrunk. Data augmentation via latent diffusion for saliency prediction, 2024. URL https://arxiv.org/abs/2409.07307 .

Y. Bengio, A. C. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35:1798-1828, 2012. URL https://api.semanticscholar.org/CorpusID:393948 .

C. Carmeli, E. D. Vito, A. Toigo, and V. Umanità. Vector valued reproducing kernel hilbert spaces and universality, 2008. URL https://arxiv.org/abs/0807.1659 .

M. Chen, Y. Wang, S. Tang, F. Zhu, H. Yang, L. Bai, R. Zhao, D. Qi, and W. Ouyang. Saliency guided contrastive learning on scene images, 2023. URL https://arxiv.org/abs/2302. 11461 .

N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018) , pages 168-172. IEEE, 2018.

F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

C. Etmann, S. Lunz, P. Maass, and C. Schoenlieb. On the connection between adversarial robustness and saliency map interpretability. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1823-1832. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/etmann19a.html .

I. Fischer. The conditional entropy bottleneck. Entropy , 22(9):999, 2020.

K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. Kernel measures of conditional dependence. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/file/ 3a0772443a0739141292a5429b952fe6-Paper.pdf .

D. Greenfeld and U. Shalit. Robust learning with the Hilbert-schmidt independence criterion. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3759-3768. PMLR, 13-18 Jul 2020.

A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77. Springer, 2005.

A. Guesmi, N. S. Aswani, and M. Shafique. Exploring the interplay of interpretability and robustness in deep neural networks: A saliency-guided approach, 2024. URL https://arxiv. org/abs/2405.06278 .

I. Guyon and A. Elisseeff. An introduction to variable and feature selection. J. Mach. Learn. Res. , 3:1157-1182, 2003.

R. Haldar, Y. Xing, and Q. Song. Effect of ambient-intrinsic dimension gap on adversarial vulnerability. In International Conference on Artificial Intelligence and Statistics , pages 10901098. PMLR, 2024.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=HJz6tiCqYm .

A. A. Ismail, H. C. Bravo, and S. Feizi. Improving deep learning interpretability by saliency guided training. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum? id=x4zs7eC-BsI .

H. Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950 , 2020.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. doi: 10.48550/arXiv. 1412.6980.

A. Krogh and J. Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems , 4, 1991.

Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4): 541-551, 1989.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13 , pages 740-755. Springer, 2014.

C. Liu and J. JaJa. Feature prioritization and regularization improve standard accuracy and adversarial robustness. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 2994-3000. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijcai.2019/415. URL https: //doi.org/10.24963/ijcai.2019/415 .

M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 97105, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/ long15.html .

S. M. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 4768-4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

W.-D. K. Ma, J. P. Lewis, and W. B. Kleijn. The hsic bottleneck: Deep learning without back-propagation. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):50855092, Apr. 2020. doi: 10.1609/aaai.v34i04.5950. URL https://ojs.aaai.org/index. php/AAAI/article/view/5950 .

H. Maatouk, D. Rullière, and X. Bay. Truncated multivariate normal distribution under nonlinear constraints. 2024.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018.

T. maintainers and contributors. Torchvision: Pytorch's computer vision library. https: //github.com/pytorch/vision , 2016.

O. Melamed, G. Yehudai, and G. Vardi. Adversarial examples exist in two-layer relu networks for low dimensional linear subspaces. Advances in Neural Information Processing Systems , 36: 5028-5049, 2023.

L. Miklautz, L. G. M. Bauer, D. Mautz, S. Tschiatschek, C. Böhm, and C. Plant. Details (don't) matter: Isolating cluster information in deep embedded spaces. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2826-2832. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai.2021/389. URL https://doi.org/10.24963/ijcai.2021/389 . Main Track.

M. L. Montero, J. Bowers, R. P. Costa, C. J. Ludwig, and G. Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=7yUxTNWyQGf .

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html .

P. Ray, D. Pati, and A. Bhattacharya. Efficient bayesian shape-restricted function estimation with constrained gaussian process priors. Statistics and Computing , 30:839-853, 2020.

M. T. Ribeiro, S. Singh, and C. Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/2939672.2939778. URL https://doi.org/10.1145/2939672.2939778 .

A. S. Ros and F. Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 618-626, 2017. doi: 10.1109/ICCV.2017.74.

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6034 .

B. K. Sriperumbudur, K. Fukumizu, and G. R. Lanckriet. Universality, characteristic kernels and rkhs embedding of measures. Journal of Machine Learning Research , 12(7), 2011.

I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journal of machine learning research , 2(Nov):67-93, 2001.

I. Steinwart and A. Christmann. Support Vector Machines . Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387772413.

M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 3319-3328. JMLR.org, 2017.

R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW) , pages 1-5, 2015. doi: 10.1109/ITW.2015.7133169.

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=SyxAb30cY7 .

A. F. M. S. Uddin, M. S. Monira, W. Shin, T. Chung, and S.-H. Bae. Saliencymix: A saliency guided data augmentation strategy for better regularization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= -M0QkvBGTTq .

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 24-25, 2020.

Q. Wang, Y. Lin, Y. Chen, L. Schmidt, B. Han, and T. Zhang. A sober look at the robustness of clips to spurious features. In NeurIPS , 2024.

Z. Wang, T. Jian, A. Masoomi, S. Ioannidis, and J. G. Dy. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. In NeurIPS , pages 586-597, 2021.

E. Weinberger, N. Beebe-Wang, and S.-I. Lee. Moment matching deep contrastive latent variable models. In AISTATS , pages 2354-2371, 2022. URL https://proceedings.mlr.press/ v151/weinberger22a.html .

K. Y. Xiao, L. Engstrom, A. Ilyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR . OpenReview.net, 2021.

J. Xu, W. Lu, J. Li, and H. Yuan. Dependency maximization forward feature selection algorithms based on normalized cross-covariance operator and its approximated form for high-dimensional data. Inf. Sci. , 617(C):416-434, Dec. 2022. ISSN 0020-0255. doi: 10.1016/j.ins.2022.10.093. URL https://doi.org/10.1016/j.ins.2022.10.093 .

M. Yamada, W. Jitkrittum, L. Sigal, E. P. Xing, and M. Sugiyama. High-dimensional feature selection by feature-wise kernelized lasso. Neural Computation , 26:185-207, 2012. URL https://api.semanticscholar.org/CorpusID:2742785 .

L. Yu and H. Liu. Efficient feature selection via analysis of relevance and redundancy. Journal of machine learning research , 5:1205-1224, Dec. 2004. ISSN 1532-4435.

M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology , 68(1):49-67, 2006.

A Technical Preliminary

Hilbert Space. A Hilbert space is a complete inner product space. More formally, a real or complex vector space H is called a Hilbert space if it is equipped with an inner product ⟨· , ·⟩ H that induces a norm ∥ f ∥ H := √ ⟨ f, f ⟩ H , under which H is complete; that is, every Cauchy sequence in H converges to a limit in H . The inner product structure generalizes the geometric notions of angle and length, while completeness ensures that limits of convergent sequences remain in the space. Examples include R n , L 2 spaces of square-integrable functions, and reproducing kernel Hilbert spaces (RKHSs).

Reproducing Kernel Hilbert Space (RKHS). Let X be a nonempty set. A Hilbert space H ⊆ R X is called a reproducing kernel Hilbert space if there exists a positive definite kernel k : X × X → R such that, for every x ∈ X , the function k ( x, · ) ∈ H and the reproducing property holds: that is, for all f ∈ H and x ∈ X ,

Hilbert-Schmidt Operator [16]. Let F and G be separable Hilbert spaces, and let A : F → G be a bounded linear operator. 2 Then A is called a Hilbert-Schmidt operator if, for any orthonormal bases { f i } ∞ i =1 ⊂ F and { g j } ∞ j =1 ⊂ G , the following Hilbert-Schmidt norm is finite:

Cross-Covariance Operator. Let x ∈ X , z ∈ Z be random variables and let F and G be RKHSs over X and Z . Then, the cross-covariance operator C XZ : G → F is the unique linear operator such that

for all f ∈ F , g ∈ G .

Proposition A.1 (Covariance Bounded by HSIC [16, 15]) . Let X ∈ X and z ∈ Z be random variables, and let F and G be RKHSs on X and Z , respectively. Then the scalar covariance is bounded by the Hilbert-Schmidt Information Criterion, i.e., the Hilbert-Schmidt norms of the cross-covariance operators:

The above proposition is the HSIC for the scalar value RKHS defined in Gretton et al. [16]. To connect the above scalar value function spaces to vector-value spaces, we use the external direct sum as below.

External Direct Sum of Hilbert Spaces. Let H 1 , . . . , H k be Hilbert spaces. We can then denote the vector-valued Hilbert space H via the external direct sum as

which is equipped with the inner product ⟨ ( f 1 , . . . , f k ) , ( g 1 , . . . , g k ) ⟩ H := ∑ k j =1 ⟨ f j , g j ⟩ H j . The corresponding norm is given by ∥ f ∥ H := ( ∑ k j =1 ∥ f j ∥ 2 H j ) 1 / 2 , which makes H itself a Hilbert space.

Moreover, if we construct the RKHS by the direct sum, i.e., H := ⊕ k j =1 H j , the resulting space H is a vector-valued RKHS [7].

Next, we define the corresponding covariance matrix for a vector-valued RKHS. Let f : X → R k and g : Z → R k be vector-valued functions, and ( x , z ) be random variables jointly distributed over X × Z . The covariance matrix between f and g is defined as:

Algorithm 1 Alternating Optimization of θ and M s

1: Input: Dataset D = { ( x i , y i ) } n i =1 ; initial θ (0) ; M (0) s = I ; β step ∈ [0 , 1] ;

2: for epoch t = 1 to T do Step 1: Update θ via SGD (minibatches)

3: for each minibatch B ⊂ D do

4: Compute latent codes z i = f ψ ( x i ) for i ∈ B

5: Compute minibatch means: µ , { µ k } K k =1

6: Compute loss L ( B ; θ , M s , I -M s )

7: Update θ ← θ -η ∇ θ L

8: end for

Step 2: Update M s via closed-form solution

9: Compute z i = f ψ ( x i ) for all i ∈ D

10: Compute dataset means: µ , { µ k } K k =1

11: for each feature dimension i = 1 to m do

12: end for

13: end for

14: Return: θ ( T ) , M ( T ) s

Hilbert-Schmidt Independence Criterion (HSIC) The HSIC [16] is a kernel-based measure of dependence. Let x ∈ X ⊆ R d and z ∈ Z ⊆ R m be random variables. Let also G = ⊕ k i =1 G i and F = ⊕ k j =1 F j be vector-valued RKHSs over the input and representation domains with k values, respectively (i.e., direct sums of k scalar RKHSs). The cross-covariance operator C XZ : G → F is the unique linear operator such that ⟨ f, C XZ g ⟩ F := Cov[ f ( x ) , g ( z )] = E [( f ( x ) -E [ f ( x )])( g ( z ) -E [ g ( z )])] , for all f ∈ F , g ∈ G . The vector-valued HSIC between x and z is defined as

where C ( i,j ) XZ is the cross-covariance operator between G i and F j , and ∥ · ∥ HS denotes the HilbertSchmidt norm. This quantity upper-bounds the scalar covariances [15]:

We can empirically estimate HSIC within O ( n -1 ) accuracy [16] given n i.i.d. samples { ( x i , y i ) } n i =1 via:

where K x and K z have elements K x ( i,j ) = k x ( x i , x j ) and K z ( i,j ) = k z ( z i , z j ) , while H = I -1 n 11 ⊤ is the centering matrix.

Notation Summary. Table 6 summarizes the notation used in the main paper and Appendix.

B H-SPLID Pseudocode

Algorithm 1 contains pseudo-code for H-SPLID, i.e., the alternating optimization algorithm presented in Section 3.4 to solve Problem (6).

C Proof of Theorem 3.2

We next show that the output of a masked neural network is uniformly bounded in sup-norm under standard Lipschitz and compactness conditions. This provides the foundation for connecting the model class to the kernel-bounded spaces introduced above.

Lemma C.1 (Bounded NN with Saliency Space) . Let X ⊆ { x ∈ R d : ∥ x ∥ 2 ≤ R } be a bounded input space, and let f ψ : X → R m be an L -Lipschitz encoder. Consider a network h θ ( x ) := WM f ψ ( x ) where W ∈ R k × m is a linear weight matrix satisfying ∥ W ∥ ∞ ≤ B , and M ∈ R m × m is a diagonal binary mask with at most s nonzero entries. Then, the network output is bounded in sup-norm:

√

The proof is deferred to Appendix D.1. This bound shows that the sparsity level s of the mask plays a direct role in constraining the model's output magnitude, which is essential for robustness.

To connect neural network outputs to kernel-based function spaces, we reparameterize the neural network h θ ( · ) := WM s f ψ ( · ) by g W ( z s ) ≡ Wz s . Then, we show how the g W belongs to a bounded subset of C ( Z , R k ) .

Corollary C.2 (Bounded Function Spaces) . By redefining the neural network in Lemma C.1, g W belongs to the following closed ball:

where ∥ g ∥ ∞ , 2 := sup z ∈Z ∥ g ( z ) ∥ 2 denotes the sup-2-norm.

This corollary imposes uniform boundedness of the neural network output values via g W on representations over the compact domain X , ensuring that the function belongs to a bounded subset of continuous function spaces C N Z b . See Appendix D.2 for the proof.

Given the RKHSs F and G in Assumption 3.1, we define the rescaled RKHS spaces ˆ F and ˆ G as

Thus, we establish the equivalence between the rescaled RKHSs and the bounded continuous function spaces.

Lemma C.3 (Rescaled RKHS Equals C b ( X , R d ) ) . Given Assumption 3.1 and the continuous universal kernel k x therein, its corresponding RKHS F , and a bounded continuous function space C b ( X , R d ) such that

then we have the rescaled RKHS space

where F := ˆ F ∥·∥ ∞ , 2 denotes the closure of ˆ F w.r.t. the || · || ∞ , 2 .

Similarly, we can show that the rescaled C N Z b = ˆ G ∥·∥ ∞ , 2 . See Appendix D.3 for the proof.

Moreover, as we have two spaces containing functions that are different by a scalar, we are interested in how the supremums of the covariance relate between spaces.

Lemma C.4 (Scaling of Supremum Covariance - Sum Version) . Let F and G be vector-valued RKHSs over X and Z , respectively. Then, for all M F , M G > 0 and x ∈ X , z s ∈ Z , the following holds:

where ˆ F := { ˜ f = f/M F : f ∈ F} and likewise for ˆ G .

Lastly, given the supremum of covariance of the function space containing the neural network, we need to use the variant of Stein's Lemma to bound the gradient of the neural network.

Lemma C.5 (Stein's Lemma for Scalar-Valued Functions on a Bounded Domain) . Let x be sampled from a truncated multivariate normal (tMVN) distribution, i.e., with density ˜ p ( x ) = 1 C exp ( -∥ x ∥ 2 2 σ 2 ) 1 ∥ x ∥≤ R , supported on the compact domain X R := { x ∈ R d : ∥ x ∥ ≤ R } , where C is the normalization constant. Let h : R d → R be a differentiable almost everywhere such that E | ∂h ( x ) /∂x i | < ∞ and | h ( x ) | ≤ N X for all x ∈ X R . Then for all i ∈ { 1 , . . . , d } ,

satisfying

where the truncation error term satisfies

and C d is the surface area of the unit sphere in R d .

See Appendix D.5 for the proof.

Then, we formally state the proof of Theorem 3.2.

Proof. Step 1. Continuous function spaces C N X X and C N Z Z .

Let π : X → R d denote the identity map, defined by π ( x ) := x . This vector-valued function can be decomposed into scalar coordinate projections:

Since the input domain X ⊆ R d is contained within a Euclidean ball of radius R , we have ∥ π ( x ) ∥ 2 ≤ R for all x ∈ X . Therefore, the identity function satisfies:

and lies in the vector-valued bounded continuous function space π ∈ C N X b ( X , R d ) . Correspondingly, each coordinate function belongs to the subspace π i ∈ C N X b,i .

Now consider the function g W : Z → R k on the representation domain Z . From Corollary C.2, the composed function g W ◦ M s f ψ over X satisfies:

Similarly, over the representation space Z , we have g W ∈ C N Z b ( Z , R k ) , and each scalar component g ( j ) W ∈ C N Z b,j .

Step 2. Equivalence between RKHS and continous function spaces.

By Lemma C.3, we can rescale the RKHS F and G in Assumption 3.1 as

so that their closure are equivalent to the bounded continuous function space C N X X and C N Z Z as in step 1.

According to Lemma C.4, if we set M F := K F N X , M G := K G N Z we relate covariance bounds between the RKHSs ( F and G ) and the rescaled RKHSs ( ˆ F and ˆ G ) through rescaling.

where the second last line applies closure under the sup-norm (preserving the supremum), and the last line substitutes the equivalent bounded continuous function space by Lemma C.3.

Step 3. Bound covariance in continuous function spaces with HSIC.

Then, based on Eq. (16), we obtain:

Combining eq. (26) and eq. (27), we have

As shown in Step 1, we have π i ∈ C N X b,i , h ( j ) θ ∈ C N Z b,j , and the following holds

Step 4. Bound the gradient with covariance.

By Lemma C.5, the following holds

Combining (29) and (30) gives:

Consider the first-order Taylor expansion in a Euclidian ball of radius r around x ∈ X : that is,

for all x ∈ X and all δ ∈ R d s.t. ∥ δ ∥ 2 ≤ r .

Consider now a measurable perturbation function on X as δ : R d → R d , such that

As Eq. (32) holds for all x , δ pairs, for all x ∈ X , we have that:

We thus have that, for all x ∈ X :

Hence, it follows that:

where the last inequality follows from Eq. (31) and the fact ϵ R σ 2 = o (1) , due to the exponentially decaying term.

D Proof of Lemmas used in Theorem 3.2

D.1 Proof of Lemma C.1

Proof. Let W n ∈ R k × s and M s ∈ R s × m be the pruned matrices selecting the active coordinates corresponding to the s nonzero entries of the mask. Then the function can equivalently be rewritten as

Since f ψ is L -Lipschitz and ∥ x ∥ 2 ≤ R , it follows that

The masking operation M selects s coordinates from f ψ ( x ) , and can be equivalently represented via M s ∈ R s × m as a selector matrix with exactly one nonzero entry per row and no more than one nonzero per column. Then

The corresponding reduced weight matrix W s ∈ R k × s selects the columns of W associated with the active coordinates. Since ∥ W ij ∥ ∞ ≤ B , it follows that

and thus where ∥·∥ F is the Frobenius norm. Hence, for any x ∈ X ,

Taking the supremum over x ∈ X concludes the proof.

D.2 Proof of Corollary C.2

Proof. As we have shown in Lemma C.1, we have:

Moreover, since the NN can be expressed as h θ ( · ) = WM f ψ ( · ) , we have:

Therefore, we can upper bound g w as:

Moreover, as g W is a continuous function, we finish the proof.

D.3 Proof of Lemma C.3

Proof. Step 1. F ⊆ C N X b

Since k x is continuous and X is compact, it follows from Lemma 4.28 of Steinwart and Christmann [47] that all f ∈ F are bounded and continuous. Hence, for any f ∈ F , we have

thus

This implies that every function in ˆ F belongs to C N X b . Since C N X b is closed in the ∥ · ∥ ∞ , 2 norm, it follows that

Step 2. C N X b ⊆ F

Let g ∈ C N X b . Define h := K F N X g . Then,

so h ∈ C b ( X , R k ) and is bounded in sup-norm. Since F is universal by Assumption. 3.1, it is dense in C b ( X , R k ) under the ∥ · ∥ ∞ , 2 norm. Therefore, there exists a sequence { f n } ⊂ F such that

Define the corresponding rescaled sequence ˆ f n := N X K F f n ∈ ˆ F , and set ˆ g := N X K F h = g . Computing the limit,

Thus, g = ˆ g ∈ F .

D.4 Proof of Lemma C.4

Proof. We show that

For each pair ( j, i ) , let { f ( j ) n } ⊂ F j and { g ( i ) n } ⊂ G i be sequences converging to the limit

Define the rescaled sequences:

Then, by the bilinearity of the covariance operator, we have

and taking the limit:

Summing over all i, j yields the result.

Furthermore, the reverse inequality follows from the same argument.

D.5 Proof of Lemma C.5

Proof. Step 1 (Integration by Parts). Let ϕ ( x ) = exp ( -∥ x ∥ 2 2 σ 2 ) , and define f ( x ) = h ( x ) ϕ ( x ) . Applying the product rule:

Integrating over X R and applying the divergence theorem gives:

where ν ( x ) = x ∥ x ∥ is the outward unit normal and ν i ( x ) = x i R .

Thus:

Step 2 (Pass to Expectation Form). Dividing through by C = ∫ ∥ x ∥≤ R ϕ ( x ) dx , the normalization constant, gives:

Step 3 (Bounding the Boundary Term). Since | ν i ( x ) | ≤ 1 and | h ( x ) | ≤ N X √ k on X R , we have:

The surface area of the sphere is:

Rearranging:

Then, as R →∞ , we have:

where the constant C d > 0 is the surface area of a d-dimensional unit sphere, depending only on d . Thus, we can bound the error term as

since the exponential decay dominates polynomial growth.

E Proof of Lemma 3.3

Proof. Under the same assumptions as in Theorem 3.2, recall the HSIC bound on gradients in eq. (31) as

Thus, we can bound the Frobenius norm of the gradient as

By Markov inequality, we have

Thus, plugging eq. (67) to eq. (68), we have

As the error term ϵ R = O ( e -R 2 2 σ 2 ) , we have

Substituting the N X , N Z , we finish the proof.

F Reproducibility Details

F.1 Datasets

COCO is a segmentation dataset consisting of labeled images of various species of animals (See Figure 3b). For our experiments, we utilize a subset of the dataset composed of images drawn from one of four labels. The four species were carefully selected to ensure the largest possible dataset containing images without overlapping labels. Since we use the dataset for image classification, each sample should belong to one class and thus include animals from one and only one of the four selected classes. During pre-processing, the dataset is resized to 224x224 pixels. Finally, segmentation information is used to construct 224x224 masks, where the 0 entries denote the pixels occupied by the animal (salient object) in the original image. These masks specify the portion of the image shielded from adversarial perturbations. The splits are created from the public training data of COCO by splitting them into train (4509 samples), validation (1127 samples) and test (1411 samples).

C-MNIST is a synthetically constructed variant of the original MNIST dataset [27]. To generate it, we first load the standard 28x28 single-channel digit images. Subsequently, each sample is randomly paired with another digit using a fixed seed for reproducibility. The two images are concatenated along the width to form a 56x28 composite, then symmetrically zero-padded to a uniform 64x64 resolution. During training and evaluation, we treat only the left-hand digit as the classification target, ensuring each composite image belongs to exactly one class. We use 80% of the original train split of MNIST as training data and 20% as validation data. For testing we use the test set of MNIST, where we also create image pairs as described above.

To assess whether H-SPLID attends preferentially to salient objects rather than background cues, we evaluate it on two complementary benchmarks: (1) CounterAnimal (CA) [55], which splits iNaturalist

IN-9

Only-FG

Mixed-Rand

wildlife photos into a Common set (exhibiting typical backgrounds) and a Counter set (featuring atypical yet plausible backgrounds, see Figure 3a), and (2) ImageNet-9 (IN-9) [58], defined as a subset of ImageNet-1K consisting of 368 categories, instantiated in three distinct variants: Original images, a MixedRand variant in which object foregrounds are transposed onto random-class backgrounds, and an Only-FG variant with backgrounds entirely removed (See Figure 4).

Table 7 provides each dataset's source URL and applicable licensing terms.

F.2 Implementation

The HBaR code was adapted from the original codebase, which is publicly available at GitHub (under MIT License). For weight decay, we reuse the PyTorch [39] implementation and pass it directly to the optimizer. We re-implemented the following regularization methods: Group Lasso Weights, Group Lasso Activations, L1 Sparse Activations, and L1 Sparse Weights. For both the Projected Gradient Descent (PGD) [34] and AutoAttack (AA) [10] adversaries, we utilize our version of the TorchAttacks [23] library, that is adapted for masked attacks.

F.3 Software and Hardware Setup

We built our pipeline in Python, leveraging the PyTorch [39] library. To conduct our experiments, we use two identical internal servers running Ubuntu 22.04.3 LTS ('Jammy Jellyfish') on a 5.15.084 x86_64 kernel. Each server is equipped with two Intel Xeon Gold 6326 processors (16 cores each, hyper-threaded for a total of 64 logical CPUs), 512 GiB of RAM, and a single NVIDIA A100 80 GB GPU. For the ablation studies and experiments conducted on the ISIC-2017 dataset, we additionally made use of EuroHPC compute resources, including MareNostrum (BSC, Spain), MeluXina (LuxProvide, Luxembourg), Deucalion (MACC, Portugal), and Discoverer (Sofia Tech, Bulgaria).

F.4 Hyperparameters

We divide our hyperparameters into two groups: those shared by all models, and those tuned or adapted per method and dataset.

Shared parameters. All ImageNet-1K ResNet-50 and COCO ResNet-18 experiments use the Adam optimizer [24] with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 10 -8 . We perform an initial grid search on a vanilla ResNet-18, sweeping the learning rate over { 10 -3 , 10 -4 , 10 -5 , 10 -6 } in logarithmic steps, and select LR = 5 × 10 -4 for all subsequent runs. The batch size is set to 256, and weight decay is 0 by default (except in weight-decay experiments). For the C-MNIST experiments we use LeNet-3 [26] with a 1024 embedding space, (as in Figure 1), we use the learning rate of LR = 1 × 10 -5 and train for 50 epochs from random initialization.

For both COCO and ImageNet-1K we use TorchVision [35] augmentations. Training augmentations include (1) Random Resized Crop to a 224 × 224 patch (scaling and cropping with a random area and aspect ratio), (2) Color Jitter applied with probability p = 0 . 8 (brightness ± 40%, contrast ± 40%, saturation ± 20%, hue ± 10%), (3) Random Grayscale with p = 0 . 2 , (4) Random Horizontal Flip with

p = 0 . 5 , (5) Random Solarize with threshold 0.5 and p = 0 . 2 , followed by (6) ToTensor and (7) Normalization using per-channel means and standard deviations (ImageNet defaults [0 . 485 , 0 . 456 , 0 . 406] , [0 . 229 , 0 . 224 , 0 . 225] or COCO-computed statistics). At test time, inputs are first resized so that the shorter side is 256 px, then center-cropped to 224 × 224 , and finally passed through ToTensor and the same Normalization.

Tuning strategy and per-method tuning ranges. We employ identical hyperparameter tuning strategies for H-SPLID and all comparison methods. 30% of the training corpus is randomly sampled for ImageNet, while the complete training set is used for all other datasets. In either case, 20% of the samples are used to constitute the validation set. Hyperparameters are optimized via grid search by selecting the model configuration exhibiting the highest robust validation accuracy at the end of training, in which robustness is measured with respect to Projected Gradient Descent (PGD) [34] attacks applied to the entire image, so no knowledge of salient or non-salient regions is used. We use dataset-specific perturbation budgets of ϵ = 1 255 for ImageNet and ϵ = 2 255 for COCO. These values were chosen to be strong enough to select for more robust models, while at the same time being not too strong to induce model collapse to random accuracy, so we can use the metric for model selection. Based on this selection criterion we trained each method on COCO three times and selected the run with highest robust accuracy for validation. For ImageNet we only trained one run. Importantly, no information pertaining to the salient or non-salient regions is leveraged during the tuning. Table 8 summarizes the grid ranges we search for each method on ImageNet-1K and COCO.

H-SPLID selected settings. After tuning as above, the final hyperparameters chosen for H-SPLID on each dataset are listed in Table 9. In all H-SPLID runs we also set λ ce = 10 to balance the cross-entropy scale. Due to the scale of ImageNet-1K, we introduce two scheduling parameters: (i) β init _ fraction , the fraction of training data used to compute the initial mask values (20% for ImageNet1K, 100% for COCO), and (ii) β update _ fraction , which determines the amount of training data that must be processed before updating the masks (5% of the dataset for ImageNet-1K, corresponding to multiple updates per epoch; 100% for COCO, corresponding to one update per epoch).

Baseline feature selection/regularisation methods. Finally, Table 10 reports the single best regularization strength found for each of the comparison methods.

G Additional Experimental Results

G.1 Training Time Comparison

COCO Dataset. The COCO experiments presented in the previous sections were executed for a total of 300 epochs on different machines with varying background workloads. To compare and report the computational intensity of the different training methods, the experiments were repeated on a single machine, albeit for a reduced number of epochs. Table 11 provides the average training time per epoch for the various training methods employed on the coco dataset. Those methods were executed for only 20 epochs on the same A100 GPU with 100 GB of memory, whence the average time per epoch was computed.

ImageNet-1K Dataset. The ImageNet-1K experiments were run for 20 epochs on an internal server (see Appendix F.3) that was not exclusively reserved for these trials, leading to varying background workloads. Logging was enabled throughout, with H-SPLID performing evaluations on the validation set every 5 epochs (during which additional metrics were recorded), while all other methods were evaluated every 10 epochs. These factors may contribute to H-SPLID's longer runtime. As a result,

the runtimes reported in Table 12 should be taken only as an overview and rough estimate rather than precise timing measurements. Due to the high computational cost of ImageNet-1K experiments and the lack of exclusive server access, precise timing experiments with exclusive access were conducted only on the COCO dataset (See Table 11).

G.2 Hyperparameter Sensitivity on COCO-Animals

We conduct the sensitivity analysis starting from the best-performing configuration on COCO, varying λ s , λ n , ρ s , ρ n , β step while keeping all other settings fixed. In Tables 13-17, we report clean accuracy (%), PGD robustness at ϵ ∈ { 1 , 2 , 3 } / 255 , and the learned salient dimensionality. Increasing λ n or ρ n generally improves robustness and compresses the salient subspace up to a regime where excessive regularization degrades performance. ρ s yields modest robustness gains with gradual salient-space shrinkage, and λ s exhibits a mild non-monotonic trend around the optimum. For β step, very small values (e.g., 0.1) lead to model collapse, while intermediate values (0.3-0.5) reduce clean accuracy despite moderate robustness gains. Larger values (0.8-0.9) maintain high accuracy, with β step = 0 . 8 achieving the best results. Overall, the best configuration attains a salient subspace of 14 (out of 512) dimensions with strong robustness.

G.3 Effect of Learning Rate

To verify the effect of learning rate, we vary the learning rate (LR) around the best hyperparameter setting by powers of ten (see Table 18). Results indicate a strong influence on both robustness/accuracy and the learned subspace: LR = 5 × 10 -4 yields the best overall performance with a compact salient subspace (14/512), lower LRs underfit and produce diffuse salient representations, and a higher LR collapses training.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide empirical evidence for our claim of learning salient features using adversarial attacks against non-salient features. Our theoretical claims are supported with proofs. We show that the output deviation is bounded by HSIC and the salient space dimensionality and also bound the region that could be vulnerable to attacks with HSIC the the salient space dimensionality.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a dedicated paragraph with limitations in the Conclusion 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present the assumptions in Section 3.5 and the proofs appears in Appendix C, Appendix D, and Appendix E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detail our experimental setup in Section 4. Appendix F describes the implementation details, our hyperparameter-tuning procedure, and the exact settings used to train our models. In Table 7, we provide links to download the datasets used. We are submitting our code alongside the supplementary materials and will open-source it, together with the best-performing models, upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In Table 7, we provide links to download the datasets used. The C-MNIST dataset can be constructed using our code. We are submitting our code alongside the supplementary materials and will open-source it, together with the best-performing models, upon acceptance. We include a 'README' file that details the environment setup, folder and script structure, and execution instructions for our method and all comparison methods.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We detail our experimental setup in Section 4. Appendix F describes implementation details, our hyperparameter-tuning procedure and lists the exact settings used to train our models. Data splits are described in Section F.1 and in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide standard deviations over attacks for COCO, where our method, H-SPLID outperforms other methods by more than two standard deviations (see Section 4 and Table 2), but with no dedicated statistical significance testing. The experiments on ImageNet were performed using a single seed due to their computational cost. We used train, val and test splits for all experiments and provide details on data splits in Section F.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The timing experiments (Appendix G) provide the relevant information on compute resources. Details on the hardware used are given in Appendix F.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We read and followed the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential scientific impact of producing salient latent representations in the introduction and related work. We do not see a direct relation between our work with any other societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not have high-risk of misuse and we do not release data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provide licenses and citations of used datasets in Table 7. We cite original code packages and repositories in Appendix F.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We detail model training in Appendix F and Section 4. Appendix F.1 describes how the C-MNIST dataset was constructed. Licenses for all assets are specified in Table 7 and F.2. The submission and all supplementary materials have been anonymized.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No use of human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No use of human subjects or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were only used for writing assistance.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
comments: SUMMARY:
H-SPLID presents a method to decompose a network’s latent space into a low-dimensional salient subspace (task-relevant) and a non-salient subspace (task-irrelevant), using learnable binary masks and regularization via norm-based clustering losses and HSIC penalties. The authors prove (Theorem 3.2) that the expected prediction change under small input perturbations is upper-bounded by the HSIC between inputs and the salient representation, and the dimension of that subspace. Empirically, H-SPLID improves robustness to adversarial attacks targeted at backgrounds (COCO animals) and transfer performance on saliency benchmarks (ImageNet-9, CounterAnimal) without adversarial training

STRENGTHS AND WEAKNESSES:
I am not an expert on this domain. I have some questions as follow
1. Experiments focus on image classification with fixed ResNets; no tests on other modalities (text, graphs) or real robotics pipelines. The synthetic MNIST concatenation and filtered COCO subsets do not capture the complexity of real-world feature entanglement.
2. The method introduces five new parameters (λₛ, λₙ, ρₛ, ρₙ, mask-update rates) and mask-update schedules (β-fractions, α-smoothing). Yet no study quantifies sensitivity or provides guidelines.
3. HSIC computation is O(n²) per minibatch—how does this scale to larger batch sizes or high-dimensional embeddings? Have the authors considered linearized/approximate HSIC estimators to reduce compute?

QUESTIONS:
1. More experiments on diverse datasets and modalities should be presented.
2. More studies about the parameters should be presented.

LIMITATIONS:
See weakness and questions.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces H-SPLID, an algorithm for learning salient feature representations through explicit decomposition of latent spaces. The H-SPLID approach leverages information compression regularization and the Hilbert-Schmidt Independence Criterion to promote compact and informative representations within a task-relevant subspace. Some experimental results are provided which demonstrate H-SPLID’s efficacy in mitigating the vulnerability to irrelevant input variations, linking representation dimensionality and robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Figure 1 is compelling and informative, demonstrating the strengths of the proposed approach in a visual manner with a clear t-SNE visualization.\n+ The problem is of significance to the community - the ability to learn representations robust to irrelevant input is a foundational requirement for deploying deep neural networks in real-world scenarios.\n\nWeaknesses:\n- Relying heavily on adversarial attacks to demonstrate robustness is a somewhat narrow approach. While it’s effective to some extent, exploring the method's performance under more natural, real-world perturbations would be more valuable. (Suggestions: lighting, blur, occlusion, etc)\n- There is a limitation in the experimentation on COCO and ImageNet datasets - extended analysis beyond these and ResNet models would make for a more compelling demonstration of the capabilities of the proposed H-SPLID. Even evaluations on a medical imaging dataset for example would convey better the breadth of applicability of the approach as well as robustness to “real world” corruptions.\n\nQUESTIONS:\n- Does the learning rate influence the final outcome, e.g. for the salience subspace versus the non-salient subspace?\n- Why PGD over any other attack?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to learn salient features for supervised learning tasks, separating out non-salient features into a subset of the feature space dimensions. Unlike previous methods, the proposed H-SPLID algorithm learns to separate these salient and non-salient features without additional data that goes beyond the data used for supervised learning. The loss function contains 3 types of terms. The cross entropy term is the cross entropy loss when predicting from only the salient features. The masked clustering terms encourage features to match mean features, with class-specific means for salient features and a global mean for non-salient features. Lastly, the HSIC terms minimize the dependence between the data and the salient features as well as the labels and the non-salient features. The H-SPLID algorithm minimizes the loss function through alternating optimization of the neural network and the saliency mask. H-SPLID is supported by a theoretical result that bounds the change in model prediction by the HSIC of the data and the salient features. Experiments show that the method improves robustness to adversarial attacks and transfer accuracy for saliency benchmarks. An ablation additionally shows that performance degrades when only a subset of the loss terms are used.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The design of the experiments is well documented, including a detailed description of the datasets, methods, experiment process, and performance metrics. It seems like the experiment could be reproduced even without the code that is additionally provided.\n\nS2: The proof in Appendix C appears to be a significant improvement over previous work and represent a lot of effort on the authors’ part. Although I am not familiar enough with some of the mathematical concepts to give a rigorous evaluation of the proof, I do not see any errors. The outline of the proof provided by some of the guiding text was very useful for understanding the logic of the proof.\n\nW1: The mathematical notation has various issues throughout the paper (Q1-4). These issues reduced clarity and necessitated educated guesses as to some specifics of the method.\n\nW2: It’s unclear to me how much benefit there is in learning non-salient features. Are these features expected to frequently be turned into salient features throughout training? As I understand it, the non-salient features do not affect the final prediction.\n\nQUESTIONS:\nQ1: I think the end of line 115 should use the identity matrix as in line 114.\n\nQ2: Is there a difference between $L_{ST}$ and Equation 5? It’s ambiguous why the different notation for the loss differs from the left side of Equation 5. Did you introduce the additional notation for when a minibatch is used instead of the full dataset? Are $L_s$ and $L_n$ used in both steps of H-SPLID or just the mask update?\n\nQ3: The equation after line 152 made me confused about the definition of $\\beta$. My initial assumption was that $\\beta$ was a vector of 0 or 1 values. But instead line 113 says that $\\beta$ is exactly one 0 or 1 value, which doesn’t make sense in the context of the mask definition in the previous line. Line 112 seems to best correlate with the equation after line 152, which seems to imply that each of the elements of $\\beta$ lie in the range $[0, 1]$, although that definition is a bit overly broad since it doesn’t capture the diagonal nature of the mask matrix.\n\nQ4: What variable is the expectation in Equation 9 taken with respect to? Looking at Appendix C suggests to me that it is with respect to $\\delta$, but I would like that to be more explicitly stated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper presents H‑SPLID, an algorithm that explicitly separates salient from non‑salient features by learning two disjoint latent sub‑spaces. The authors demonstrate its effectiveness on two image‑classification benchmarks and under two adversarial attack settings.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is technically sound and its claims are supported by a thorough theoretical analysis in both the main text and the appendix.
- The empirical evaluation is comprehensive and well explained.
- The exposition is generally clear, and the authors provide code and full hyperparameter details, ensuring reproducibility. 
- Explicitly disentangling salient and non‑salient latent dimensions is a novel contribution that remains underexplored in existing work.

Weaknesses:
- Lack of tests with alternative backbones (e.g., ViT, Inception, VGG).
- Parts of the methodology appear unnecessarily intricate and could be streamlined (e.g. Sec. 3.1)

If the approach works on a broader set of architectures and tasks, it could meaningfully influence saliency‑based robustness research.

QUESTIONS:
1. What was the selection criteria for the COCO subset?
2. Why didn't the authors provide more experiments on several classifiers?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02