Learning Low-Rank Feature for Thorax Disease Classification
Authors: Yancheng Wang
Rajeev Goel
Utkarsh Nath
Alvin C Silva
Teresa Wu
Yingzhen Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Low-Rank Feature Learning
Low Frequency Property
Thorax Disease Classification
Primary area: machine_learning_for_healthcare
TL;DR: We propose a novel Low-Rank Feature Learning (LRFL) method for thorax disease classification, which learns low-rank features of a neural network so as to effectively reduce the adverse effect of background or non-disease areas.

Abstract: Deep neural networks, including Convolutional Neural Networks (CNNs) and Visual Transformers (ViT), have achieved stunning success in the medical image domain. We study thorax disease classification in this paper. Effective extraction of features for the disease areas is crucial for disease classification on radiographic images. While various neural architectures and training techniques, such as self-supervised learning with contrastive/restorative learning, have been employed for disease classification on radiographic images, there are no principled methods that can effectively reduce the adverse effect of noise and background or non-disease areas on the radiographic images for disease classification. To address this challenge, we propose a novel Low-Rank Feature Learning (LRFL) method in this paper, which is universally applicable to the training of all neural networks. The LRFL method is both empirically motivated by a Low Frequency Property (LFP) and theoretically motivated by our sharp generalization bound for neural networks with low-rank features. LFP not only widely exists in deep neural networks for generic machine learning but also exists in all the thorax medical datasets studied in this paper. In the empirical study, using a neural network such as a ViT or a CNN pre-trained on unlabeled chest X-rays by Masked Autoencoders (MAE), our novel LRFL method is applied on the pre-trained neural network and demonstrates better classification results in terms of both multi-class area under the receiver operating curve (mAUC) and classification accuracy than the current state-of-the-art. The code of LRFL is available at \url{https://github.com/Statistical-Deep-Learning/LRFL}.

Learning Low-Rank Feature for Thorax Disease Classification
Yancheng Wang'
Rajeev Goel'
Utkarsh Nath' Yingzhen Yangb
Alvin C. Silva? Teresa Wu
School of Computing and Augmented Intelligence, Arizona State University [ywan1053, rgoel15, unath, teresa. wu, yingzhen srngzhen.yange. yang Masu. edu
Mayo Clinic Arizona
silva.alvin alvemmmyoole
Abstract
Deep neural networks, including Convolutional Neural Networks (CNNs) and Visual Transformers (ViT), have achieved stunning success in the medical image domain. We study thorax discase classification in this paper. Effective extraction of features for the disease areas is crucial for disease classification on radiographic images. While various neural architectures and training techniques, such as self- supervised learning with contrastive/restoratit learning, have been employed for disease classification on radiographic images, there are no principled methods that can effectively reduce the adverse effect of noise and background or non- disease areas on the radiographic images for disease classification. To address this challenge, we propose novel Low-Rank Feature Learning (LRFL.) method in this paper, which universally applicable the training of all neural networks. The LRFL method is both empirically motivated by Low Frequency Property (LFP) and theoretically motivated by our sharp generalization bound for neural networks with low-rank features. LFP not only widely exists in deep neural networks for generic machine learning but also exists all the thorax medical datasets studied in this paper. In the empirical study, using neural network such as a ViT or CNN pre-trained on unlabeled chest X-rays by Masked Autoencoders (MAE), novel LRFL method is applied the pre-trained neural network and demonstrates better classification results in terms of both multi-class area under the receiver operating curve (mAUC) and classification accuracy than the current ofate-of-the-art. The code of LRFL is available at tps:::/// //g:: //ghbbb.. com/ Statistical-Dest Leeep-Learning/LRFF
Introduction
Following the huge success of deep Icarning, recent studies have developed deep neural networks (DNNs) for various tasks in medical imaging, such as disease classification and abnormalities detection in anatomy in chest X-rays [ 2] Accurate clinical decision-making with DNNs heavily relies on learning informative medical feature representation. Early works adopt convolutional neural networks (CNNs) such as U-Net 3 for representation learning on radiography images. Recently, Visual Transformers (ViTs) are also adopted to learn informative medical representations from radiography images 12). utilizing their capabilities in capturing long-range feature dependencies. Albeit the success of CNNs and ViTs in analyzing radiography images, their accuracy heavily relies on the quality and quantity of data and annotations
[5]. However, the collection of large amounts of training data and high-quality annotations in the medical imaging domain are extremely hard (2. To
Indicates equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tackle this problem, self self-supervised learning (SSL) has been employed as solution for acquiring representations from unlabeled data. Given the greater availability of unlabeled medical images [6], SSL proves to be an efficient approach for obtaining discriminative representations. SSL employs range of pretext tasks to acquire transferable representations without manual annotations. Over recent years, numerous variations of self-supervised learning have surfaced using contrastive learning 7 and restorative learning [2].
Challenges in the Current Literature for Disease Classification. We study thorax disease clas- sification in this paper. Clinical studies show that the disease areas on radiographic images are subtle and exhibit localized variations. Such conditions are further complicated by the inevitable noise that ubiquitous in radiographic images, as detailed in Section [2.] Effective and robust extraction of features for the disease areas is crucial for disease classification on radiographic images. Although various neural architectures, such as CNNs and ViTs, and different training techniques, such as self-supprvised learning with contrastive/restorati learning, have been employed for disease classification on radiographic images, there have been no principled methods that can effectively reduce the adverse effect of noise and background, or non-disease areas, for disease classification on radiographic images.
The contributions of this paper are presented as follows. First, in order to address the aforementioned challenge, we propose a novel Low-Rank Feature Learning (LRFL) method in this paper, which is universally applicable to the training of all neural networks with the application for thorax disease classification. Our LRFL method employs low-rank features for disease classification. The usage of low-rank features is empirically motivated by Low Frequency Property (LFP) illustrated in Figure [ That is, the low-rank projection of the ground truth training class labels possesses the majority of the information of the training class labels. In fact, LFP widely holds for a broad range of classification problems using deep neural networks, such as [& 9. Inspired by LFP, our LRFL method adds the truncated nuclear norm low-rank regularization term to the training loss of a neural network so as to perform classification using low-rank features. Because the actual features used for classification are approximately low-rank and the high-rank features are significantly truncated, all the noise and the information about the background or the non- -on-diseese areas on radiographic images in the high-rank features are largely discarded and not learned in neural network. Importantly and significantly different from existing low-rank learning methods reviewed in Section 2.3] we introduce novel separable approximation for the TNN, enabling the optimization of the LRFL training loss using standard SGD. The appropriate feature ranks retained in the LRFL method across various datasets are determined through efficient validaaaiootionn process, and the optimal ranks are detailed in Table g Extensive experimental results demonstrate that our LRFL method renders new record mAUC on three standard thorax disease datasets, NIH- ChestX-ray [0) COVIDx I. and Cherp 12 surpassing the current state-of-the-art 2 with the same pre-training traininng setup.
Our Contributions.
Second, we provide a theoretical analysis showing a sharp generalization bound for the LRFL method, underscoring the substantial benefits of employing low-rank regularization within LRFL. Given these theoretical insights and the versatility of LRFL. across various neural networks, we anticipate broader applications of LRFL in the classification of other diseases beyond thoracic ones, potentially enhanc- ing classification tasks across different radiographic imaging contexts. It is worthwhile to mention that the literature has studied low-rank learning using TNN resembling LRFL, as be reviewed in Section 2.3 Our LRFL method builds upon these foundational principles by incorporating low-rank regularization into the training of neural networks, aiming to improve thorax disease classification by reducing the adverse effects of noise and irrelevant background information. Different from the conventional low-rank learning methods, our approach introduces separable approximation to the TNN, facilitating the optimization process and enhancing the generalization ability of the model. Such improved generalization is evidenced by the improved prediction accuracy of LRFL compared to the current
(SOTA) methods in medical image analysis.
Moreover, we have employed conditional diffusion model trained on COVIDx and Che datasets to generate synthetic images. These synthetic images are then added to their respective training sets to form the augmented training data on which our LRFL models are trained. This approach has further elevated the state-of-the-art MAUC scores achieved by LRFL on both COVIDx and CheXpert datasets.
2

---

Projecttion Value
Projectiion Value
0.6
Projecttion Value
0.3
0.4
0.2
0.2
0.1
0.2
0.1-
600
200
400
600
800
200
400
600
800
Rank
Rank
Rank
1.0
1.0
Concentration
0.8
Concentration
0.8-
Concentration
0.8
0.6
0.6
0.6-
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
200
400
600
800
200
400
600
800
Rank
Rank
Rank
(a) NiH-ChestXray-14
(b) COVIDx
(c) CheXpert
Figure Eigen-projection (first row) and signal concentration ratio (second row) Vit-Base on WiH-ChestXray-14, 14, COVIDx, and CheXpert. To compute the eigen-pos vigen-poojection
first calculate the eigenvectors of the kernel gram matrix K E Rax" computed by a feature matrix RR*d
then the projection value is computed by p E Y 1 \ IR", where C' is the number classes, and Y {0, 1)mxC is the one-hot labels of all the training data, Y(e) is the e-th column of Y. The eigen-projection p, for E [min(n, d)] reflects the amount of the signal projected onto the r-th eigenvector of K, and the signal concentration ratio of rank r reflects the proportion of signal projected onto the top eigenvectors of K. The signal concentration ratio for rank is computed by |Ip() (1=))] where (1:r) contains the first elements of p. For example, by the rank r = 38, the signal concentration ratio of Y on NIH ChestX-ray144 COVIDx, and CheXpert are 0.959, 0.964, and 0.962 respectively.
Motivation for using synthetic images to boost the accuracy for thorax disease classification. The computer vision literature [4314 [5] has extensively studied the usage of the generated synthetic images which augment the training data and improve the prediction accuracy of image classification. Inspired and motivated by this observation, we propose to generate synthetic images and use them to form the augmented training data and improve the performance of thorax disease classification. The augmented training data comprise the original training images and the synthetic images. However, too many synthetic images tend to introduce more noise to the augmented training data so excessive synthetic images can hurt the prediction accuracy of DNNs trained on the augmented training data [16]. As evidenced in the ablation study in Section C33 our proposed LRFL method, coupled with the selection the number of synthetic images, effectively mitigates this issue. The proposed low-rank learning method only learns the low rank part of the features learned by deep learning model so that noise in the high-rank part would not affect the learned model. Also, cross cross-validane validation is used to select proper number of synthetic images, which will boost the prediction accuracy while not introducing too much noise to the augmented training data.
We also present ablation study results evidencing our contributions. We compare the eigenvalues of the kernels and the kernel complexity associated with the LRFL models and the corresponding base models in Section B.4. of the appendix, and the lower kernel complexity of the LRFL models suggests their lower generalization error [17 19].
Notations. We use bold letters to denote matrices or vectors. [A], stands for the i-th row of matrix A. Il denotes the p-norm of a vector or a matrix. I1- is the Frobenius norm of matrix. We use (m... ... n) to indicate numbers between m and n inclusively, and [n]] denotes the natural numbers between 1 and inclusively.
2 Related Works
2.1 Radiographic Imaging
Radiographic imaging 20 a cornerstone in medical image analysis. Unlike photographic images 21 radiography images have consistent backgrounds due to fixed imaging proto-
cols 22
24 2. Clinical details are spread across the images, while areas indicating illness
show localized variations
2 25 26), making analysis challenging. Noise is unavoidable in ra-
3

---

diography images, stemming from quantum fluctuations, electronic interference, scatter radiation, motion blur, and overlapping structures [27 28
Quantum noise, originating from statisti- is often the primary source. Quantum noise introduces graininess, obscuring details and diminishing contrast [31]. Modeled as a Poisson process [25] 30), it can be approximated by Gaussian distribution under high photon flux (3,04] enabling noise reduction techniques [34].
29
30
cal fluctuations in detected X-ray photons 31
32
25
30
2.2 Medical Image Analysis with Deep Learning
Deep learning has made remarkable progress in photographic image analysis 35 36 37,, spark- ing interest in applying it to medical imaging due to the ability to Iearn complex representations. Convolutional neural networks (CNNs) like U-Net [ 38
39] pioneered this field, achieving state-
of-the-art performance across various tasks such as image classification [40. 41 42 object detec-
tion 43 39 44 and semantic segmentation 44 45
39
46
47]. More recently, visual transformers,
inspired by the success of transformers in natural language processing 48 have outperformed
state-of-the-art CNNs on various computer vision benchmarks [49
50
4
52
53 Despite
debates around transformers vs CNNs in terms of generalization [54]
data require- ments 4 [59 660, and computational costs [61), transformers have shown great potential in medical
56
57
58
image analysis 22 62, 63). Given the scarcity of high-quality annotations, self-supervised con- trastive learning techniques 7 64 65 66 2 have gained traction for pre-training networks in this domain [22 2 62). However, the high similarity between radiographic images due to standardized
protocols 67 68 poses challenges compared photographic images 669 77. To address this, recent
works utilize restorative strategies like masked autoencoders (MAE) 770 71 73 74 75 for
pre-training 22. Similarly, we adopt MAE 2 to pre-train our networks before learning low-rank features.
2.3 Low-Rank Learning
Low-rank learning has garnered significant attention across various fields for its capacity to reduce dimensionality, suppress noise, and enhance feature extraction. Robust Principal Component Analysis (RPCA) (76) serves as a cornerstone in this realm, efficiently separating data matrices into low-rank and sparse components. This technique proves invaluable for vision-related tasks such as image denoising and background subtraction. Building on this foundation, [77] introduced singular value pruning, a method to impose low-rank constraints computational efficiency and performance. The concept of TNN regularization (TNNR) has been further refined by researchers like 778) who noted that TNNR more accurately approximates the rank function by selectively minimizing singular values, essential for precise low-rank matrix recovery in noisy conditions. Following that, some existing works (79 30 81 propose perform low-rank feature learning by minimizing the TNN of the feature matrix. Additionally, the use of TNNR in tensor completion has markedly improved the restoration of incomplete visual data, utilizing More contemporary learning-basee methods, such those developed by (84) have optimized low-rank approximations through targeted training, enhancing practical application outcomes. Some works [85 86 87| also demonstrate that learning low-rank features can significantly enhance the robustness of deep neural networks against noise in input images. In addition, recent works [88 89 |00 find that the good generalization capabilities of deep neural networks are attributed to the fact that deeper networks are inductively biased to find solutions with lower effective rank embeddings.
neural network layers, thereby boosting both
tensor singular value decomposition (I-SVD) 82 83
3 Formulation
3.1 Pipeline for Thorax Disease Classification
We utilize the masked MAE technique (75) for the initial pre-training of both CNNs and ViTs following (2), and subsequently fine-tune the pre-trained networks with our Low-Rank Feature Learning (LRFL). The full training pipeline of learning low-rank features for disease classification can be described in three steps. In the first step, which is the pre-training step. we pre-train the networks using the self-supe restorative learning method, masked MAE (75), on diverse pre- tre-training dataset that includes ImageNet- 1k (91] and a collection of X- X-rays (0.5M) (2. During this phase, we randomly mask patches on input images and drive the networks to optimize pixel-wise image reconstruction for the obscured patches. In the second step, which is the regular fine-tuning step, we fine-tune the pre-trained networks employing cross-entropy loss aimed at image classification on specific target datasets, namely NIH-ChestX-rai [10), COVIDx (.. and CheXpert [12]. In the

---

last step, which is the low-rank feature learning step, we fix the backbones of the networks and fine-tune the linear classifier utilizing our novel LRFL method.
3.2 Problem Setup for LREL
We now introduce the problem setup for LRFL with training details. Suppose the training data are given as (x, where x; and y; E R.C are the i-th training data point and its corresponding class label vector respectively, and C' is the number of classes. Each element y; is binary with y; = 1 indicating the i-th disease is present in x; otherwise y, = O. Suppose that the neural network trained by step two of our pipeline in Section 3.1 generates feature vector /w,(o) (x) E I (the output of the layer preceding the final linear/softmax layer of the network) for any input x, and fw- (.) is the feature extraction function with W' being the weights of the feature extraction backbone of the network. Wi(O) denotes the denotes the weights of feature extraction backbone by step two of the pipeline. We can train neural network by optimizing
min L(W) = 1 Σ
KL (yi, (2/
E=1
where Wi is initialized by (1()), W2 E Rcdd and W (W,, W2). Here o is an element-
wise sigmoid function, (a) E EC with [o(a)], = 1/(1 + exp(-a.)) for a E RC and E [C]. KI. stands for the element-wise binary cross-entropy function. Given two nonnegative vectors
[vi....,ud Vd] E Rd where u, E {0, 1) for all i E [d] and || S 1, KL(u, v) := - log 0; (1 u,) log(1 - 22). We use Y = [y :
= [M1,.... Ud] E Rd
E R=*x to denote the training label matrix by stacking the label vectors of all the training data. Let the mapping function of the neural network used in the loss function L(W) be NNw(x) = W2fw, (x).
Motivation for Low-Rank Regularization The Low Frequency Property is illustrated in Figure that is, the low-rank projection of the ground truth class labels possesses the majority of the infor- mation of the class labels. Inspired by this observation, our LRFL encourages the low-rank part of the feature to participate in the classification process. In this way, the noise and non-disease areas in the high-rank part of the feature are mostly not learned by LRFL so as to improve the classification accuracy. Using notations in Section 3.2. the truncated nuclear norm of F is || = Σ E 0 where T e [0, d]. can be observed by the generalization error bound discussed in Section 3.2 that a smaller ||| y renders a tighter upper bound for the generalization error of the linear neural network used for LRFI.. This observation gives a strong theoretical motivation for us add the truncated to the training loss (WW.
nuclear norm FF IT
3.3 Generalization Bound for Rank Feature Learning
We define the loss function ((NNw(x) y)) == |NNN((x) - ylا and the generalization error of the network NN is the expected risk of the loss l, which is denoted by LD (NNw) = E(x,))= [e(NNw w yx), with D being the distribution of the data and its class label y. The
network NNw generates a feature F Raa of all the training data with F, fw, (x:) for i [n]. X > 0 where F S min (n, d} is the rank of K,, Suppose the Singular Value Decomposition of F is F UEVT. where U E Raxd has orthogonal columns, Σ E Rdd is diagonal matrix with diagonal elements being the singular values of F, and V E ædxd is an orthogonal matrix. The columns of U and V are also called the left eigenvectors and the right eigenvectors of F. respectively. Let 01 2 02 ..... ... 2 oa be the singular values of F, and = U+)) Y be the projection of the training label matrix Y onto the subspace spanned by the top-r left eigenvectors of F. where U E g=xf is formed by the top F eigenvectors in U. Then, we have the following theorem giving the sharp generalization error bound for the linear neural network in (T].
The kernel gram matrix for the feature F is K, _ FFF. We let X 2 X2 2 2
n
Theorem 3.1. For every & > O, with probability at least exp(-x), after the t-th iteration of gradient descent for all 2 1, we have
LD(NNw) < Y || 61 (1-
|
+02 min be(o.r]
h
1 r
X
+
C3
n
12
n
(2)
where C12 C2, C3 are positive constants.
5

---

Remark 3.2. The RHS of 2 is the generalization error bound for the linear neural network used in LRFL as step three of the pipeline Section 3.1 Moreover, let 01 02 ..... oa be the singular
values of F. Due to the fact that
Σ i=++1
X
S
Σ oi it follows by (2 that
i=++1
LD(NNw) sa (1-1111))
nd.)
h
C3
+
02
n
T=T41
which holds for all T E [0,d]. d). 33) motivates the reduction of the truncated nuclear norm of the feature F, as detailed in the next subsection.
3.4 Optimization of the Truncated Nuclear Norm in SGD
The truncated nuclear norm F 1| is not separable, so the training loss with || cannot be directly optimized by the standard SGD, To address this problem, we propose approximation f to || which separable so that | can be optimized by standard SGD.
First, note that if U, V are known, then Σ F.... If we have approximation U to U and an approximation V to V, then E can be approximated by E = U FV.
As result, the approximation F Ir to the truncated nuclear norm is | = Σ Σ U 57 FuFFV
Due
+
to the above discussions, the loss function of LRFL with the approximate truncated nuclear norm
F is L.RFL(WW - Em KL (y:, [ar (FW(in)) ][) + +n/l +n][Fllrr which is separable, so that it can be trained by the standard SGD. 7 > the weighting parameter for the truncated nuclear norm. Because LLRFL(W) (W) is to be optimized by the standard SGD, we have the loss function of LRFL for the j-th minibatch B, S [m] as
(W)
1 B3,
Σ IEB
KL (y:, [o (FW(ll))) +
7 B,
Σ IEB
Σ T ==
The approximation U and V can be computed the left and right eigenvectors of the feature F computed at earlier epochs. In order to save computation and avoiding performing SVD for F at every epoch, we propose to update and only after certain epochs. Algorithm I describes the training algorithm for the neural network trained with LRFL,, which uses the standard SGD to optimize the loss function L (W), as step three of our pipeline in Section 3.1 Before the first epoch, we compute and as the left and right eigenvectors of the feature F at the initialization of the neural network. After every to epoch with ty being constant integer, we update U and V the left and right eigenvectors of the feature F produced by the neural network right after lo-th epoch, with to being a constant integer.
Algorithm Training Algorithm with the Approximate Truncated Nuclear Norm by SGD
1: Initialize the weights W, by W, (O),), and initialize Wa randomly Compute feature by the neural network, and SVD as UEV
3: Update U- = U, V
4: for 1,2 do
5: ift = (mod to) then
6: Compute feature F of the neural network, and its SVD F = UEV.
7: Update U
U =V
end if
9:
for b = 1, 1,,...., B do
10: Update W by applying gradient descent on batch Bj C [m] using the gradient of the loss C3 in Eq.
11: end for
12:
end for
13:
return trained weights W the network
4 Experimental Results
In this section, we conduct experiments on medical datasets to demonstrate the effectiveness of
the proposed LRFL method. The experiments section is organized as follows: In Section 4.1

---

discuss our experimental setup and implementation details. In Sections 4.2 and we evaluate the LRFL models for thorax disease classification on CheXpert and COVIDx. Evaluation results on NIH ChestX-ray 14 are deferred to Section B. of the appendix. In Section 4.4 we evaluate synthetic data augmentation on LRFL. models, with additional details and results deferred to Section of the appendix. Comprehensive ablation studies on LRFL. are performed in Section 4.5 In Section 4.5.1 we study the effectiveness of the LRFL models in reducing the adverse effect of the background for disease classification. In Section 4.5.2, we study the performance of LRFL models for disease localization. Grad-CAM visualization results of LRFL models and baseline models are illustrated Additional ablation studies are deferred to Section B.4 of the appendix. In Section B 1.4.1| we compare the kernel eigenvalues and kernel complexity between the LRFL models and their corresponding base models to show that LRFL improves the generalization capability of the base models by reducing their kernel complexity. In Section B.4.2 we evaluate the performance of the LRFL. models with limited data availability. In Section B.4.3 compare the performance of the LRFL with other fine-tuning strategies. In Section |B4.5 we present the training time of the LRFL. models compared with the corresponding base models.
in Section 4.5.3 3
4.1 Implementation Details
In this section, we evaluate the proposed LRFL for thorax disease classification. We utilize networks pre-trained on ImageNet [92 or chest X-rays in 2 with MAE, a self-suuer learning strategy that reconstructs missing pixels from patches of input images. We fine-tune these pre-trained networks
with low-rank regularization on three public X-ray datasets: (1) NIH ChestX-ray14 [10), (2) Stanford
CheXpert [12), and (3) COVIDx [II. The ADAM optimizer is used with a batch size of 1024
for all datasets. Initially, we fine-tune the entire networks for 75 epochs following the settings in 2], then fine-tune with low-rank regularization for another 75 epochs. We use cosine learning rate schedule, and the initial learning rate, which is denoted as μ., is selected by tross-validation for each model and each dataset. The default values for momentum and weight decay are set to 0.9 and 0, respectively. We use standard data augmentation techniques, including random-resize cropping, random rotation, and random horizontal flipping. For fair comparison, all baselines are also fine-tuned for an additional 150 epochs, showing almost no improvement. An exhaustive analysis of this additional fine-tuning is in Section B.4.3 We evaluate our LRFL method on both CNN and visual transformer architectures, including ResNet-50, DenseNet, ViT-S, and ViT-B. Our model is referred as 'X-LR', where X is the base model (e.g.. ResNet-50-LR for ResNet-50 with low-rank features).
Tuning the T, 1) and μ by Cross- Validation. We search for the optimal values of feature rank T. the weighting parameter for the truncated nuclear norm 7, and the learning rate j on each dataset. Let T [> min(n, d)], where y is the rank ratio. We select the values of y and 17 by performing 5-fold cross-validation on 20% of the training data in each dataset. The value of y is selected from {0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.15, 0.2). The value of 1) is selected from {5 10- 4, 1 x 10-3, 2.5 10 3 3,5 10 x 10-2) The value of μ is selected form {5 x 10-1 2.5 x 10-4, 1 10-1,5 5 10-5 2.5 10-5, 10-9]. To determine the optimal val- ues of the parameters 1). 7, and fr, we employ a sequential greedy search strategy. We first fix and and find the optimal value of y by cross-validation. Subsequently, using this optimized 7, we proceed to search for the optimal while keeping μ constant. Finally, with optimal y and we search for the optimal μ by cross-validation. The optimal values of 7, 7, and μ selected by cross-valien validation are shown in Table 8 in Section of the appendix. The time spent for the entire cross-validation process presented in Table 9 Section B.3 of the appendix, which demonstrates that the cross-validatto process is efficient and does not significantly increase the computational overhead of the training process.
4.2 Stanford CheXpert
Experimental setup. CheXpert 12 consists of 224,316 chest X-rays collected from 65,240 patients, where 191,028 chest X-rays are used for training. Each X-ray in the dataset has radiology reports indicating the presence of 14 diseases. Following the protocol in (2) all images are resized into 224 x 224. We also report the mean AUC (Area Under the Curve) for the 5 distinct classes and conduct a comprehensive comparison with state-of-the-art baseline methods
Results and analysis. Table presents the performance comparisons between the baseline models and the LRFL models on the CheXpert dataset. Throughout this section, use the postfix "-LR" to
7

---

indicate a neural network trained with our LRFL. For example, we use the VIT-B model pre-traiee on 489, 090 and the ViT-S model pre-trained on 266, 340 chest X-rays with Masked Autoencoders (MAF) [2]. The pre-trained ViT-B network is fine-tuned on the CheXpert dataset and achieves observed that ViT-B-LR achieves state-of-the-art performance of 89.8% in mAUC and improves the performance of ViT-B by 0.5% in mAUC. ViT-S-LR also improves the performance ViT-S by 0.4% in mAUC, which demonstrates the power LRFL. We also show the classification accuracy of the five diseases in Table I where our method exhibits much better performance than baseline methods. For example, ViT-S-LR achieves an mAUC of 86.3% on Cardiomegaly, with 4.5% improvement over ViT-S trained with MAE. Such improvements demonstrate the power of LRFL in detecting distinct diseases. The comparison between LRFL models and more comprehensive list of baseline models deferred to Tf the appendix.
mean AUC of 89.3.
Table 1: Performance comparisons between LRFL models and SOTA baselines on CheXpert. The best result is highlighted in bold, and the second-bos bestt result is underlined. This convention is followed by all the tables in this paper. DN represents DenseNet.
Method Irvin 12
Architecture
Atelectasis
Cardiomegaly
Consolidation
Edema
Effusion
(%)
81.8
82.8
93.8
93.4
92.8
88.9
Pham et al.
DNI21
82,5
85.5
93.7
93.0
92.3
89.4
Kang al.
DN121
85.9
94.4
89.2
93.6
MoCo v2
2
DN121
78.5
77.9
92.5
92.8
92.7
88.7
VTT-S 2
VIT-S716
83,5
81.8
93,5
94.0
93.2
89.2
VI--S-LR (Ours)
V/T-
0.05r
86.3
90.9
93.7
93.1
89.6
VIT-8716
83.5
93.8
VIT-B-LR (Ours)
VIT-B/16
0.05r
81.6
85.4
93.4
94.6
93.9
89.8
4.3
COVIDx
Experimental setup. COVIDx (Version 9A) [1] consists of 30,386 chest X-rays collected from 17,026 unique patients. We follow the previous works [11] 2 in splitting the dataset into 29,986 training images with four different classes and 400 testing images with three classes. For fair comparisons with the previous methods, we report Top-1 accuracy on the test set (3 classes).
Results and Analysis. Table 2 compares the performance of SOTA transformer-basee models and LRFL models on the COVIDx dataset. Similar to Section 4.2, base VITs are first pre-trained on chest X-rays using Masked Autoencoders (MAE), and then the pre-trained model is fine-tuned on the COVIDx dataset. It can be observed from Table 2 that both ViT-S-LR and ViT-B-LR outperform their corresponding base models ViT-S and ViT-B, achieving
increase in accuracy of 1.6% and 1.7%, respectively. Table 2 also compares the performance of our LRFL models against the state-of- the-art models the COVIDx dataset. LRFL models achieve much higher accuracy compared to CNN-based models such as DenseNet-121 ViT-B-LR achieves the new SOTA performance of 97% top-1 accuracy with input resolution set 224x224, which exceeds the previous SOTA performance 2 by 1.7% in top-1 accuracy.
Table 2: Performance comparisons between LRFL models and SOTA baselines on COVIDx (in
accuracy). DN represents DenseNet.
Method COVIDNet- Small COVIDNet-CXX Large
Architecture
Rank
Covid 19 Sensitivity
Accuracy
87.1
92.6
96.8
McCo v2
DN121
94.5
94.0
DN121
DN121
97.0
93.5
VIT-S
ViT-S/16
VIT-S-LR (Ours)
ViT-S/16
0.01r
97.5
VIT-B 2
VIT-B/16
ViT-B-LR (Ours)
ViT-B/16
0.003r
4.4 Improved Results using Diffusion Model
Experimental Setup. In this section, we aim further improve the performance of LRFL models by adding labeled synthetic radiographic images of thorax diseases to the training sets of COVIDx and CheXpert. The synthetic radiographic images are generated by a conditional diffusion model, Diffusion Transformer (DiT) [94), trained on the training set of the corresponding dataset. Details on the training of DIT are deferred Section C.2 of the appendix. co-occurrence, synthetic radiographic images generated based on the labels from the label set of
maintain the same disease

---

each dataset. The number of synthetic images added to the training set of each dataset is determined via vross-validation. We first generate synthetic images of the same size as the training set. The optimal percentage of synthetic images is selected using 5-fold cross-validation on the training data, which is detailed Section C.2 of the appendix. Synthetic images are combined with the original dataset for further fine-tuning with low-rank regularization. Ablation studies on the number of synthetic images incorporated are performed in Scctior[C.3
The results of LRFL models trained after adding synthetic images on CheXpert and COVIDx are shown in Table 3 is observed from the results that adding synthetic data into the training set of LRFL models can further increase their performance. For example, ViT-B-LR L- with synthetic images added in training outperforms the corresponding base model ViT-B by 2.2% on COVIDx.
Results.
Table 3: Performance comparison of baseline models and LRFL models on the CheXpert and COVIDx datasets, with and without synthetic data. denotes the number of training images in the respective dataset.
Method
Architecture
CheXpert
COVIDx
Rank
# Synthetic Images
MAUC (%)
Rank
# Synihetic Images
Accuracy (%)
VIT-S
VIT-S776
89.2
95.2
ViT-S-LR (Ours)
VIT-5/16
0.05r
89.6
0.01r
96.8
VIT-S (Ours)
V/1
0.2g
89.3
VIT-S-LR (Ours)
VIT-S/16
0.05r
0.2m
89.7
0.01r
1.0n
97.3
VIT-B 2
VTT-B716
89.3
95.3
VIT--LRR (Ours)
B/T-
0.025r
89.8
0.003r
ViT-B (Ours)
VIT-B/16
0.25m
89.9
1.0n
97.0
VIT-B-LR (Ours)
VIT-B/16
0.025r
0.25m
90.4
0.003r
1.0n
97.5
4.5
Ablation Study
4.5.1 Study of LREL in Reducing the Adverse Effects of Background
To demonstrate that the LREL models are more robust to the background than the baselines, we perform ablation study the LRFL reduce the adverse effects of the background. In this study, we create a mask for the disease area for each original image, then decompose the original image. which has bounding box for the disease, into disease image and a background image. Both the disease image and the background image are of the same size as the original image. The background image has grayscale in the masked disease area, and the disease image has grayscale O in the non-disease area. We feed the three images, which are the original image, the disease image, and the background image, to an LRFL model and obtain the original features, disease features, and background features for the LRFL model, respectively. We also feed these three images to baseline model and obtain the original features, disease features, and background features for the baseline model. For each original image, we measure the distance between the disease features and original features using KL-divergence on the softmaxed features for the LRFL model and the baseline model. We then compute the average feature distance for each model, which is the average distance between the disease features and original features over the images with ground-truit truth bounding box for the disease in the NIH ChestX-ray 14. The results in Table indicate that the original features are closer to the disease features by the L.FL. models compared to the baseline models, evidencing the effectiveness of the LRFL. models in reducing the adverse effect of the background area. We also remark that since only the low-rank part of the original features participates in the classification process, the noise and non-disease areas in the high-rank part of the features are mostly not learned by LRFL, and in this manner, LRFL is robust to both noise and background.
Table 4: Average feature distance between original features and disease features of images with a ground-truth bounding box for the disease in the NIH ChestX-ray 14.
Methods
mAUC (%)
Average Feature Distance
ViT-S
82.3
0.7030
ViT-S-LR
82.7
0.6352
ViT-B
83.0
0.5642
VIT-B-LR
83.4
0.6628
4.5.2 Disease Localization
To study which part of the X-ray image responsible for the model prediction by the LRFL models, we perform the discase localization experiment following the settings in (2]. We first obtain the Grad-

---

CAM visualization results with the last transformer block of ViT-S. The experiments are performed with all the images with ground-true truth bounding box for disease in ChestX-ray14 The predicted bounding box is generated with the thresholded Grad-CAM heatmap, largest connected component, and box regression. We evaluate the performance of disease localization by Intersection over Union (IoU) between the ground-truth bounding box and the predicated bounding box used for evaluation. The Average Precision (AP) on 25% and 50% IoUs, which are denoted as AP25 and AP for ViT-S and ViT-S-LR are shown in Table 5 It is observed from the results that our LRFL model significantly outperforms the base model in detecting the bounding box thorax disease. For example, ViT-S-LR outperforms ViT-S by 26.9% in AP for detecting the bounding box of Mass. In addition, ViT-S-LR outperforms ViT-S by 21.2% in AP25 for detecting the bounding box Effusion.
Table AP2s and AP scores for different diseases using VIT-S and ViT-S-LR models.
AP2:
APso
(# px)
VIT-S
VIT-S-LR
VIT-S
VIT-S-LR
Mass
156
27.0
53.9
Atelectasis
924
31.5
49.3
8.1
11.3
Pneumothorax
1899
4.7
18.3
0.0
1.5
Infiltrate
2754
Effusion
2925
8.8
30.0
1.0
3.1
Pneumonia
2944
27.8
44.1
9.3
12.5
2300
4.5.3 Grad-CAM Visualization
To study how LRFL improves the performance of base models in disease detection, we use the Grad-CAM (95 to visualize the parts in the input images that are responsible for the predictions of the base models and low-rank models. Robust Grad-CAM [95] visualization results of Low-Rank ViT-Base are illustrated Figure 2 All Grad-CAM visualization results illustrate that our LRFL models usually focus more the areas inside the bounding box associated with the labeled disease. In contrast, the base models also focus on the areas outside the bounding box or even areas in the background. Robust Grad-CAs CAM visualization results of Low-Rank ResNet-50 and additional Grad-CAM visualization results of Low-Rank ViT- are deferred to Figure and Figure in Section of the appendix.
Vil-dtee
Love Lone Vil-bbee
Rarniing
Box
Label Mass
hromottonss
Label: Efisaion
Labet Cartionegal
Inditeaic
Labe Eiflsioo
Label
Label. Cantoreegiyy
Figure Robust Grad-CAM [55] visualization results on NIH ChestX-ray 14. The figures the first row are the visualization results of ViT-Base, and the figures in the second row are the visualization results of Low-Rank ViT-Base.
5 Conclusion
In this paper, we propose a novel Low-Rank Feature Learning (LRFL.) method for thorax disease classification, which can effectively reduce the adverse effect of noise and background, non-disease areas, on the radiographic images for disease classification. Being universally applicable to the training of all neural networks, LRFL is both empirically motivated by the low frequency property and theoretically motivated by our sharp generalization bound for neural networks with low-rank features. Extensive experimental results on thorax disease datasets, including NIH-ChestX-ran COVIDx, and CheXpert, demonstrate the superior performance of LRFL in terms of mAUC and classification accuracy. In addition, the performance LRFL models is further improved by adding synthetic radiographic images into the training set for data augmentation.
10

---

Acknowledgments and Disclosure of Funding
This material is based upon work supported by the U.S. Department Homeland Security under Grant Award Number 17STQAC00001-0722
The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the U.S. Department of Homeland Security. This work is also partially supported by the 2023 Mayo Clinic and Arizona State University Alliance for Health Care Collaborative Research Seed Grant Program under Grant Award Number AWD00038846.
References
[1] Scbastian Guendel, Sasa Grbic, Bogdan Georgescu, Siqi Liu, Andreas Maier, and Dorin Comaniciu. Learning recognize abnormalities in chest x-rays with location-aware dense networks. In Iberoamerican Congress on Pattern Recognition, pages 757-765. Springer, 2018.
[2] Junfei Xiao, Yutong Bai, Alan Yuille, and Zongwei Zhou. Delving into masked autoen- coders for multi-label thorax disease classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3588 1888-660, 2023.
[3] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisse Intervention, pages 234-241. Springer, 2015.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
[5] Ruibin Feng, Zongwei Zhou, Michael B Gotway, and Jianming Liang. Parts2whole: Self- supervised contrastive learning via reconstruction. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, 2020.
[6] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Korbblith, Ting Chen, Patricia MacWilliams, S Sara Mahdavi, Ellery Wulezyn, et al. Robust and efficient medical imaging with self-supervision. arXiv preprint arXi::2205.09723 2022.
[7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[8] Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10(1) 19549,, Nov 2020.
[9] Hieu H Pham, Tung T Le, Dat Q Tran, Dat Ngo, and Ha Q Nguyen. Interpreting chest x-rays via cnns that exploit hierarchical disease dependencies and uncertainty labels. Neurocomputing. 437:186-18 194, 2021.
[10] Xiaosong Wang, Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2097-2106, 2017.
[11] Maya Pavlova, Tia Tuinstra, Hossein Aboutalebi, Andy Zhao, Hayden Gunraj, and Alexander Wong. Covidx exr-3: a large-scale, open-source benchmark dataset of chest x-ray images for computer-aided covid-19 diagnostics. arXiv preprint ar arXiv:2206.038 03677, 2022.
[12] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 590-597, 2019.
11

---

[13] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023.
[14] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip H. S. Torr, Song Bai, and Xiaojuan Qi. synthetic data from generative models ready for image recognition? The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[15] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. In The Twelfth International Conference on Learning Representationss ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
[16] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification. Trans. Mach. Learn. Res., 2023, 2023.
[17] Peter L.. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacherr complexities. Ann. Statist,, 1494-134 08 2005.
[18] Vladimir Koltchinskii. Local rademacher complexities and oracle inequalities in risk mini- mization. Ann. Statist., 34(6):2593-28) 2656, 12 2006.
[19] Shahar Mendelson. Geometric parameters of kernel machines. In Conference on Learning Theory (COLT), 2002.
[20] Jun Li, Junyu Chen, Yucheng Tang, Ce Wang, Bennett A Landman, and S Kevin Zhou Transforming medical imaging with transformers? comparative review of key properties, current progresses, and future perspectives. Medical image analysis, page 102762, 2023.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pages 248-255. IEBE, 2009.
[22] Zongwei Zhou. Towards Annotation- Efficcien Deep Learning for Computer-Aided Diagnosis. PhD thesis, Arizona State University, 2021.
[23] Jun Li, Junyu Chen, Yucheng Tang, Bennett A Landman, and S Kevin Zhou. Transforming medical imaging with transformers? a comparative review of key properties, current progresses, and future perspectives. arXiv preprint arXiv:2206.01 136, 2022.
[24] Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey. arXiv preprint 12Xi::200100987: 2022.
[25] Paul Suetens. Fundamentals of medical imaging. 2017.
[26] Zongwei Zhou, Michael Gotway, and Jianming Liang. Interpreting medical images. In Intelligent Systems in Medicine and Health: The Role of AI. 2022.
[27] JH Siewerdsen, LE Antonuk, Y El-Mohri, J Yorkston, W Huang. JM Boudry, and IA Cunning- ham. Empirical and theoretical investigation of the noise performance of indirect detection, active matrix flat-pancl imagers (amfpis) for diagnostic radiology. Medical physics, 24(1):71- 89, 1997.
[28] JH Siewerdsen, LE Antonuk, Y El-Mohri, J Yorkston, W Huang, and IA Cunningham. Signal, noise power spectrum, and detective quantum efficiency of indirect flat- flat-panel imagers for diagnostic radiology. Medical physics, 25(5):614-62 1998.
[29] EN Manson, V Atuwo Ampoh, E Fiagbedzi, JH Amuasi, JJ Flether, and C Schandorf. Image noise in radiography and tomography: Causes, effects and reduction techniques. Curr. Trends Clin. Med. Imaging, 255):555620, 2019.
[30] Tej Bahadur Chandra and Kesari Verma. Analysis of quantum noise-reducing filters on chest x-ray images: A review. Measurement, 153:107420 2020.
12

---

[31] K Kirk Shung, Michael Smith, and Benjamin MW Tsui. Principles of medical imaging. 2012.
[32] K Kirk Shung, Michael Smith, and Benjamin MW Tsui. Principles of medical imaging. 2012.
[33] Sangyoon Lee, Min Seok Lee, and Moon Gi Kang. Poisson-gaussian noise analysis and estimation for low-dose x-ray images in the nset domain. Sensors, 18(4):1019, 2018.
[34] Qiaoqiao Ding, Yong Long, Xiaoqun Zhang, and Jeffrey A Fessler. Statistical image reconstruc- tion using mixed 2018.
noise model for x-ray ct. arXiv preprint arXiv: 012/8/000099333
[35] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[36] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[37] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2017.
[38] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çięek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, page 2018.
[39] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018.
[40] Yan Shen and Mingehen Gao. Dynamic routing on deep neural network for thoracic disease classification and sensitive area localization. In International Workshop on Machine Learning in Medical Imaging, pages 389-397. Springer, 2018.
[41] Hongyu Wang, Haozhe Jia, Le Lu, and Yong Xia. Thorax-net: attention regularized deep neural network for classification of thoracic diseases on chest radiography. IEEE journal of biomedical and health informatics, 24(2):475-435 2019.
[42] Congbo Ma, Hu Wang, and Steven C. H. Hoi. Multi-label thoracic disease image classification with cross-attention networks, 2020.
[43] Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67-70, 2019.
[44] Ruixin Yang and Yingyan Yu. Artificial convolutional neural network in object detection and semantic segmentation for medical imaging analysis. Frontiers in oncology, 11:638182, 2021.
[45] Yuan Yao, Fengze Liu, Zongwei Zhou, Yan Wang, Wei Shen, Alan Yuille, and Yongyi Lu. Unsupervised domain adaptation through shape modeling for medical image segmentation. In Medical Imaging with Deep Learming. 2021.
[46] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider. Bennett A Landman, Geert Litjens, Bjoern Menze, al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint 1ee:::90200000 2019.
[47] Jamshid Sourati, Ali Gholipour, Jennifer Dy, Xavier and Simon K Warfield. Intelligent labeling based on fisher information for medical image segmentation using deep learning. IEEE transactions on medical imaging, 38(11):642-2653, 2019.
Sila Kurugol,
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint 1rX.::7706033662. 2017.
13

---

[49] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay. Jiashi Feng, and Shuicheng Yan. scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 558-567, 2021.
vit: Training vision transformers from
[50] Yancheng Wang, Ping Li, and Yingzhen Yang. Visual transformer with differentiable channel selection: An information bottleneck inspired approach, 2024.
[51] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[52] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
[53] Han Cai, Chuang Gan, and Song Han. Efficientvit: Enhanced linear attention for high- resolution low-computation visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[54] Xiaohan Ding, Xiangyu Zhang. Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 19933-11775, 2022.
[55] Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie. Are transformers more robust than cnns? Advances in Neural Information Processing Systems, 1-6884,,1,3,, 2021.
[56] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12042-12051, 2022.
[57] Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization of vision transformers under distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7277-7286, 2022.
[58] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 273994 PMLR, 2022.
[59] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:21066 106. 10270, 2021.
[60] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022.
[61] Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2071-2081, 2022.
[62] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEFF/CVF conference on computer vision and patlern recognition, pages 12299-12310, 2021.
[63] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang. Le Lu, Alan L. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102. 04306, 2021.
[64] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations, arXiv preprint 20X7002002.05709, 2020.
14

---

[65] Jean- Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv22006.07333, 2020.
[66] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint wrXiv:2006609888) 2020.
[67] Tiange Xiang, Yongyi Liu, Alan L Yuille, Chaoyi Zhang, Weidong Cai, and Zongwei Zhou. In-painting radiography images for unsupervised anomaly detection. arXiv preprint arXiv:2111. 11.::2/ 2021.
[68] Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Michael B Gotway, and Jianming Liang. Dira: Discriminative, restorative, and adversarial learning for self-supervieed medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 00824-20834, 2022.
[69] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9738, 2020.
[70] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Self-supervised learning for medical image analysis using image context restoration. Medical image analysis, 58:101539, 2019.
[71] Zongwei Zhou, Vatsal Sodha, Md Mahfuzur Rahman Siddiquee, Ruibin Feng, Nima Tajbakhsh, Michael B Gotway, and Jiamming Liang. Models genesis: Generic autodidactic models for 3d medical image analysis. In International conference on medical image computing and computer-assisted intervention, pages 384-393, Springer, 2019.
[72] Jiuwen Zhu, Yuexiang Li, Yifan Hu, Kai Ma, Kevin Zhou, and Yefeng Zheng. Rubik's cubet: A self-supervised feature learning framework for 3d medical image analysis. Medical Image Analysis, 64: 101701,,, 2020.
[73] Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels, Advances in Neural Information Processing Systems, 2020.
[74] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEE.//CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 9653- 9663, June 2022.
[75] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[76] Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
[77] Yiming Yang and William W Cohen. Singular value pruning of deep neural networks with application to dialogue response selection. arXiv preprint wwxivv/509.08800 2015.
[78] Qingsong Hu, Deyu Zhang, Wei Zhang, and Xuelong Li. Truncated nuclear norm regularization for tensor completion. arXiv preprint arXiv: (to::/38..0000 00737 2013.
[79] Chul Lee and Edmund Y Lam. Computationally efficient truncated nuclear norm minimization for high dynamic range imaging. IEEE Transactions on Image Processing, 25(9)4414-4415 2016.
[80] Yao Hu, Zhongming Jin, Yi Shi, Debing Zhang, Deng Cai, and Xiaofei He. Large scale multi- class classification with truncated nuclear norm regularization. Neurocomputing 148:310-317, 2015.
15

---

[81] Fanlong Zhang, Heyou Chang. Guowei Yang. Zhangjing Yang, and Minghua Wan. Truncated nuclear norm based low rank embedding. In Biometric Recognition: 12th Chinese Conference, CCBR 2017, Shenzhen, China, October 28-29, 2017, Proceedings 12, pages 708-715 Springer, 2017.
[82]
Jinyu Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. Low-rank tensor completion by truncated nuclear norm regularization. arXiv preprint arXiv:1712.007000 2017.
[83] Xinyu Zhang, Guanghui Wang, Xiangzhao Li, Xuelin Liu, and Wei Liu. An efficient tensor completion method via truncated nuclear norm. ScienceDirect, 2020.
[84] Piotr Indyk, Ali Vakilian, and Yang Yuan. bearning-basel low-rank approximations. In Advances in Neural Information Processing Systems, 2019.
[85] Ming Gao, Runmin Liu, and Jie Mao. Noise robustness low-rank learning algorithm for electroenceepallogam signal classification. Frontiers in Neuroscience, 15:797378, 2021.
[86] Yuwu Lu, Zhihui Lai, Yong Xu, Xuelong Li, David Zhang, and Chun Yuan. Low-rank preserving projections. IEEE transactions on cybernetics, 1608):190-1193,, 2015.
[87] Jiahuan Ren, Zhao Zhang. Richang Hong, Mingliang Xu, Haijun Zhang, Mingbo Zhao, and Meng Wang. Robust low-rank convolution network for image denoising. In Proceedings of the 30th ACM International Conference Multimedia, pages 121111219, 2022.
[88] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Trans. Mach. Learn. Res., 2023, 2023.
[89] Naday Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In Shipra Agrawal and Francesco Orabona, editors, International Conference on Algorithmic Learning Theory, February 20-23, 2023, Singapore, volume 201 of Proceedings of Machine Learning Research, pages 1429-1459. PMLR, 2023.
[90] Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness- aware minimization leads to low-rank features. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[91] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 105, 2012.
[92] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22) 1522, 2015.
[93] Mintong Kang, Yongyi Lu, Alan Yuille, and Zongwei Zhou. Label-assemble: Leveraging multiple datasets with partial labels. In Submission: Thirty-Sixth Conference on Neural Information Processing Systems, 2021.
[94] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[95] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[96] Zhe Li, Chong Wang, Mci Han, Yuan Xue, Wei Wei, Li-Jia Li, and Li Fei-Fei. Thoracic disease identification and localization with limited supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8290- 0099-229,,980 2018
[97] Li Yao, Jordan Prosky, Eric Poblenz, Ben Covington, and Kevin Lyman. Weakly supervised medical diagnosis and localization from multiple resolutions, arXiv preprint 1003.0070,, 2018.
16

---

[98] Yanbo Ma, Qiuhao Zhou, Xuesong Chen, Haihua Lu, and Yong Zhao. Multi-attention network for thoracic disease classification and localization. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1378-1382, IEFE, 2019.
[99] Yuxing Tang. Xiaosong Wang, Adam P Harrison, Le Lu, Jing Xiao, and Ronald M Summers. Attention-guided curriculum learning for weakly supervised classification and localization of thoracic diseases on chest radiographs. In International Workshop on Machine Learning in Medical Imaging, pages 249-258. Springer, 2018
[100] Ivo M Baltruschat, Hannes Nickisch, Michael Grass, Tobias Knopp, and Axel Saalbach. Comparison of deep learning approaches for multi-label chest x-ray classification. Scientific reports, 9(1):1 10, 2019.
[101] Qingji Guan and Yaping Huang. Multi-label chest x-ray image classification via category-wise residual attention learning. Pultern Recognition Letters, 2018.
[102] Laleh Seyyed- Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pages 232-243. World Scientific, 2020.
[103] Renato Hermoza, Gabriel Maicas, Jacinto C Nascimento, and Gustavo Carneiro. Region proposals for saliency map refinement for veakly-superviser disease localisation and classi- fication. In International Conference on Medical Image Computing and Computer-Assisted Infervention, pages 539-549. Springer, 2020.
[104] Eunji Kim, Siwon Kim, Minji Seo, and Sungroh Yoon. Xprotonet: Diagnosis in chest radiography with global and local explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15719-15728, June 2021.
[105] Fengbei Liu, Yu Tian, Yuanhong Chen, Yuyuan Liu, Vasileios Belagiannis, and Gustavo Carneiro. Acpl: Anti-curriculum pseudo-labelling for semi- supervisd medical image clas- sification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697-20706, 2022.
[106] Sina Taslimi, Soroush Taslimi, Nima Fathi, Mohammadreza Salehi, and Mohammad Hossein Rohban. Swinchex: Multi-label classification on chest x-ray images with transformers. arXiv preprint arXiv:2206.000 04246, 2022.
[107] Imane Allaouzi and Mohamed Ben Ahmed. A novel approach for multi-label chest x-ray classification of common thorax diseases. IEEE Access, 0014279-64288 2019.
[108] Mohammad Reza Hosseinzadeh Taher, Fatemeh Haghighi, Ruibin Feng, Michael B Gotway, and Jianming Liang. A systematic benchmarking analysis of transfer learning for medical image analysis. Domain Adaptation and Representation Transfer, and Affordable Healthcare and Al for Resource Diverse Global Health, pages 3-13. Springer, 2021.
[109] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
[110] Rafacl Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[111] Ross Wightman. Pytorch image models, 2019.
A Proofs
Proof of Theorem 3.1 It can be verified that at the 1-th iteration of gradient descent for /. 2 1, we
have
w( ((--1)
F (FW(L-1)- -
(5)
17

---

It follows by 5 that
FW = FW((-1) - ,,, FFW(-D) - Y
-FW(L-)- (FW - -Y),
(6)
where K, 1/n FF.. Y U
We define F(W, = FW(0, then it follows by 6 that
F(W,t) - Y= (In k,,) (F(W.t) - -Y),
which indicates that
F(W, -
-WW,t--Y Y (In -))K)) (F(W,0) 0) (.,,0)-Y. -Y)
-In (In -n) Y
and
((W.t)- Y 1 Y - 1|1 + (1 -,,, ||
1 + - ;,,,) ||
(7)
As a result of (7). by using the proof of 17 Theorem 3.3, 2.3.Corollary 6.7], for every x 0, with
probability at least 1 exp(-a))
IND(NW) I a
Y
AIE + c (1
m/²r
20
|Y
C3
+ C2 min bE(0,r]
+
A
+
(8)
n
1=h+1
B More Experimental Results
B.1 NIH ChestX-ray14
Experimental setup. NIH ChestX-ray 14 [10] consists of 112, 120 X-rays collected from 30, 805 unique patients. Each X-ray can have up to 14 associated labels, with the possibility of multiple labels per image. Following the official data split I0. we use 75, 312 images for training and 25, 596 images for testing. The raw images from the dataset are sized 1024 x 1024. In our experiments, we scale down the input images to 224 224. We report the mean AUC (Area Under the Curve) for 14 distinct classes and conduct comprehensive comparison with 18 widely recognized and influential baseline methods.
Results and Analysis. presens the performance comparisons between several lop-performing baseline models and their corresponding low-rank models the NIH ChestX-ray 14 dataset. Similar to Section 4.2 the ViTs are first pre-trained chest X-rays using Masked Autoencoders (MAE). Then, the pre-trained VIT-B network is fine-tuned on the NIH ChestX-ran dataset and achieves mean AUC 83.0. Next, we fine-tune ViT-B with low-rank regularization for another 75 epochs. The low-rank model, denoted as ViT-B-LR, achieves the state-of- the-art performance with mean AUC of 83.4. It observed that all low-rank models achieve improvement mean AUC compared the corresponding base models. It is important to highlight that the research community dedicated four years to enhancing the AUC score for CNN-type architectures, advancing it from 74.5% to 82.2%. which was primarily attributed to the challenging nature of the classification with the NIH ChestX-ray 14 dataset.
B.2 Comparison with A Comprehensive List of Baselines
We compare the results of LRFL. models with more comprehensive list of baselines on CheXpert. observed from the results Table 7 that LRFI. models significantly outperform all existing of-th-- methods on CheXpert.
18

---

Table 6: Performance comparisons between LRFL models and SOTA baselines on NIH ChestX-ray14
RN, DN, and Swin'T represent ResNet, DenseNet, and Swin Transformer.
Method
Archifecture
Pre-training
Rank
mAUC
Wang et al. [00
RN50
74.5
Li al. 96
RNS0
Yao
RN&DN
76.
Wang et al. 41
R152
78.8
Ma 98
R101
79.4
Tang et
RN50
80.3
Baltruschat al. 100
RN50
80.6
Guendel et al
DN121
80.7
Guan
ImageNet-
Seyyed al [02
DNI21
81.2
Ma ct ul. 42
(N121(x2)
81.7
DN121
Kim et al. [T0]
DN121
82.2
Haghighi ct [
DN121
81.7
al. 1005
Taslimi al. 106
SwinT
81.0
MoCh v2
DN121
X-rays (0.3M)
80.6
MAE
DN121
81.2
50
RN50
ImageNet- 1K
RN-50-LR (Ours)
RN50
0.05c
82.2
DN-121 [
DNI21
82.0
ImageNet- 1K
DN-12I-LR (Ours)
DNI21
0.05r
82.4
VITES 2
V1T-S/16
82.3
X-rays (0.3M)
VIT-S-LR (Ours)
VIT-S/16
0.05r
82.7
VTF-B 2
V/T
83.0
X-rays (0.5M)
VIT-B-LR (Ours)
VIT-B/16
0.05r
83.4
Table 7: The table shows the performance of various state-of-the-art (SOTA) CNN-based and Transformer- based methods CheXpert.
Method
Architecture
Kank
Atelectasis
Cardiomegaly
Consolidation
Exlema
Eifusion
mALC
Allaouzi
72.0
88.00
77.0
90.0
82.8
Irvin et al. 12
81.8
82.8
93.8
93.4
92.8
88.9
Seyyedkalantari
81.2
90.0
87.3
Pham et al. g
82.5
83.5
93.7
93.0
92.3
89.4
Howseinzaleh
DN121
Haghighi et al.
87.6
Kang
DN121 (MoCo v2)
78.5
77.9
92.5
92.8
92.7
88.7
DN121
77.6
92.3
92.0
88.7
VIT.S 2
VTT
R3.5
81.8
93.5
94.0
93.22
89.2
ViT-S-LR (Ours)
VIT-S/16
83.7
86.3
90.9
93.7
93.1
89.6
VIT-B 2
VIT-B/16
82.7
83.5
92.5
93.8
944,1
89.3
VIT-B-LR (Ours)
VIT-B/16
81.6
85.4
93.4
94.6
93.9
89.8
B.3
Cross-Validatie Results
The optimal values of the rank ratio 7, weighting parameter 17, and learning rate decided by cross-validation for different models on different datasets are shown in Table
Table 8: Optimal values of rank ratio % weighting parameter 7, and learning rate μ decided by cross- volssvvaliotionn for different models on different datasets.
Models
Parameters
NITI-ChestX-nn -ay
COVIDs
CheXperi
0.05
0.01
0.03
VIT-S
5 x 10-4
1x 10-3
1 10-3
5 x 10-5
2.5 10-5
1 10
7
0.00
0.003
0.05
VIT-B
17
x 10-4
x 10-a
10
1 x 10-5
2.5 10-6
2.5 10
In addition, the time for the entire cross-validation process in searching for the optimal values of the rank ratio 7, weighting parameter and learning rate μ are shown in Table The evaluation is performed on 4 Nvidia A A100 GPUs. As we use only 20% of the training data for cross-validation and train the models with each option for only 40% of the entire number of training epochs, the entire
19

---

cross-validation process is efficient and does not largely increase the computation cost of the training process.
Table 9: Time Spent for cross-validation on NIH ChestX-rayle 14, CheXpert, and CovidX. All the results are reported in minutes.
Datasets
NIHI ChestX ray14
CheXpert
CovidX
VIT-S-IR
149
178
37
VII-B-LR
172
285
69
B.4
Additional Ablation Study
B.4.1 Study on the Kernel Eigenvalues and Kernel Complexity
18 19 is a widely-studied complexity measure for the generalization ca- pability of kernel-based learning algorithms. In this section, we compare the eigenvalues of the kernel and kernel complexity of ViT-B-LR and ViT-B ChestX-rayl4, COVIDx, and CheXpert. Given the representations of all the training images F learned by ViT-B or ViT-B-LR, the kernel complexity of the gram matrix K,, FFF which also defined in Section 3.3
Kernel complexity [1
can be computed
by min
h
The eigenvalues of ViT-B-LR and ViT-B ChestX-ray14, COVIDx, and CheXpert are illustrated in Figure The computed kernel complexities ViT-B-LR and ViT-B on ChestX-ray 14, COVIDx, and CheXpert are shown in Table It is observed that LRFL significantly reduces the kernel complexity of the image representations, which suggests that the LRFL models have lower generalization
errors [17 18 [9].
Table 10: Kemel complexity comparison between ViT-B-LR and VIT-B ChestX-ray14, COVIDx, and CheXpert.
ChestX-ruy 14
COVIDx
CheXpert
Method
Kerne Complexity
h
Kernel Complexity
b
Kernl Complexity
h
VIT-B
0.0101
465
0,0207
0.0040
756
B-T
0.0076
262
0.0155
187
0.0038
389
0.3
Eigenvaluss
Figenvalues
0.4
Figennalues
VII-B-LR
VYT-B-LR
ViT-BLIR
0.3
0.2
0.2
0.1
0.0
0.0
200
400 Rank
600
800
400 Rank
600
800
200
400
Rank
(a) ChestX-ray14
(b) COVIDx
(c) CheXpert
Figure 3: Eigenvalues comparison between VIT-B-LR and ViT-B on ChestX-ray14, COVIDx, and
CheXpert.
B.4.2 Experiments in Small Data Regimes
Experimental setup. We explore the effectiveness of low-rank features learned in scenarios with limited data availability, which particularly significant given the challenges acquiring high-quality data annotations in the medical imaging domain. We expect that LRFL models can demonstrate improved performance such situations due to our theoretical guarantee the better generalization capability of L.FL.. We randomly select 5%, 10%, 15%, 20%, 25% and 50% of the training data from the NIH ChestX-rayl 14 dataset and then fine-tune the base model using default training configurations. We then train LRFL models for 20 epochs.
20

---

Results and analysis. As depicted in Table 11 our LRFL models consistently outperform their corresponding base methods across all data subsets, including 5%, 10%. 15%. 20%, 25%, and 50% on the NIH ChestX-rayl+ dataset. Notably, the average improvement in performance is more substantial for the 5% data subset compared to the remaining subsets. For instance, ViT-B-LR exhibits remarkable improvement of 1.05% for the 5% data subset, which significantly surpasses the improvements of O. 1,15% 0.06%, 0.06%, 0.09%, and 0. 11% observed for the 10%, 15%, 20%, 25%, and 50% training data subsets, respectively. These findings are consistent with expectations, showcasing the strong generalization capability LRFL models in mitigating over-fitting issues findings in the low-data regimes demonstrate the superiority
with limited data. In conclusion, of our LRFL in delivering more generalizable and robust representations for tasks with limited data availability, thereby contributing to the reduction of annotation costs.
Table 11: The table evaluates the performance of various models under low data regimes on the NIH ChestX-rays14 dataset. Models trained with low-rank features effectively combat overfitting in scenarios with limited data availability, thereby enhancing the quality of representations for downstream tasks.
Label Fractions
Pre-training Dataset
MAUC
Rank
mAUC
Kank
MAUC
Rank
mACC
Rank
MAUC
Rank
50%
MAUC
X-mgy=0.3M)
OR
VITSS
0.05
61.22
61.81
0.2
73.81
73.19
0.04
76.99
77.21
0.0Hr
78.36
78.65
0,057
79.65
79.57
0.05r
81.35
81.20
70.71
$0.59
82.19
X-mayx(0.MM
VT-BB-RR (Ouns)
0.05
71.76
0..
78.82
0.2
80.05
0.1r
80.65
0.05r
81.22
0.05
82,30
B.4.3 Exploring Fine-tuning Strategies
Our LRFL. method learns low-rank features by leveraging models pre-trained on the target dataset. In this section, we conduct an ablation study to investigate the significance of low-rank regularization in the fine-tuning process. A detailed comparative analysis of low-rank regularization against several verformance-enhancinn techniques, including mix-up [109], label smoothing [100), and EMA [ЛI, is presented Table 12 We performed an experiment by fine-tuning without low-rank regularization and other tricks, which serves baseline for studying the effects of fine-tuning strategies. All models underwent equivalent training epochs to ensure a fair comparison. The results demonstrate that LRFL models achieve the highest performance improvement compared to all other approaches. Notably, unlike natural images, applying mix-up, label smoothing, or EMA to the NIH ChestX-ray dataset leads performance drops (see Table 12). Fine-tuning models pre-trained dataset without low-rank regularization does not lead to performance improvements compared to fine-tuning with low-rank regularization. For example, the original ViT-S [2] achieves 82.27% on NIH Chest Xray-14. Fine-tuning this model for 20 epochs without low-rank regularization leads mean AUC of 82.26%, whereas fine-tuning with low-rank regularization for 75 epochs results in a mean AUC of 83.40%. We observe similar results for all models based on low-rank features, demonstrating the significance of LRFL.
the target
AUC of
Table 12: Comparison of fine-tuning strategies NIH ChestX-ray14
MAUC
Model
Base Model
Fine-tuning
Mix-up 109
Label Smoothing [10
EMA
[1]
LRFL
ViT.S
82,27
82.26
82.09
82.24
82.26
82.70
VIT-B
83.00
83.00
82.37
82.99
82.98
83.40
B.4.4 Additional Grad-CAM Visualization Results
Additional CAM visualization results of the Low-Rank ViT-Base on NIH ChestX-ray 14 are illustrated in Figure Robust Crad-CAM visualization results of the Low-Rank 50 are illustrated in Figure We visualize the parts in the input images that are responsible for the predictions of the ground-truth disease label for base models and low-rank models. The visualization results show that our low-rank models usually focus more on the areas inside the bounding box associated with the labeled disease. contrast, the base models also focus on the areas outside the bounding box areas in the background.
21

---

HonNet-30
Low Kork Hechee-30
Brunding
Box
Label: Phoometions
Cantiomeply
Elfenion
LabdiArziostas
Paeamatherns
Figure 4: Robust Grad-CAM [95] visualization results NIH ChestX-ray 14. The figures in the first row are the visualization results of ViT-Base, and the figures in the second row are the visualization results of Low-Rank ResNet-50.
Lowebakk VTT Atoo
Cardionegaly
Canfomepply
Cacdomegaly
Figure 5: Grad-CAM visualization results on NIH ChestX-ray 14. The figures in the first are the visualization results of ViT-Base, and the figures in the second row are the visualization results of Low-Rank ViT-Base.
B.4.5 Training Time Analysis
We evaluate the training time of LRFL models and compare them with the training time of the baseline models. The evaluation of LRFL. models and baseline models is performed on 4 Nvidia 100 GPUs. It observed from the results Table 13 that the training time of L.RFI. models comparable to the training time of LRFL. models. The main computational overhead of LRFL. models is the computation of the eigenvectors the feature matrix F and the truncated nuclear norm. However, the computation overhead is largely reduced by avoiding performing SVD for the feature matrix at every epoch, benefiting from the approximation algorithm we designed in Algorithm |
Table 13:
Training time comparison between LRFL models and baseline models on NIH ChestX-
14y14, CheXpert, and CovidX. All the results reported in minutes.
Datasets
NIH ChestX-my14
CheXperi
CovidX
ViT-S
90
ViT-S-LR
117
VIT-B
162
VIT-BLLR
113
185
45
C Training with Synthetic Data by Diffusion Models
In this section, we explore generative data augmentation using diffusion models. Section C.1 introduces the preliminaries of diffusion models and outlines the specific modifications made for target task. In Section C.2) we discuss the implementation details training of the diffusion model. Finally, we show some of the generated synthetic images in Figure
C.1 Data Generation with the Diffusion Model
The diffusion model operates through a probabilistic framework, employing a forward noising process that gradually introduces noise to the original data xo. Initially, the model defines distribution
22

---

q(x, Xo) where x; is progressively noised from X over time t. This distribution is governed by predetermined hyperparameters oG, with x; sampled using the reparameterization trick g(x, |xx) N(x;; Vo (1 - oz) I), where e (O( I). As l advances, the noise contribution increases, leading x2 to become progressively noisier until it approximates standard Gaussian distribution.
Following the training of the diffusion model, the focus shifts to the reverse process, aimed at denois- ing a noisy sample x, to recover the original data Xo. Utilizing Gaussian noise XT the starting point, the model iteratively refines the sample using pe (x Xx), where is parameterized to ap- proximate the posterior mean of the forward process pe (xx-1 xx) (xx:1;; ( t), Ex(x,, t)). vI-ai co t)). Training involves minimizing simplified loss function Iipple to ensure accurate prediction of noise, facilitating effective denoising during the reverse process Lsimple xx.. [lle - (x,, 1)
where (Xe, t) fai X
1-as
We adopt class of diffusion model known the Diffusion Transformer (DiT) 194, chosen for its efficiency and token-agnostic conditioning, making it particularly suitable for our task. DiTs efficiently leverage label embedding for guidance and exhibit high compute efficiency, which is crucial for scaling to large datasets.
The Diffusion Transformer (DiT) model is trained on the CheXpert and COVIDx datasets, as described by 194). The DiT model, specifically designed for text-based labels, operates without classifiers and instead relies on label embedding to guide the diffusion process. Modifications are made to the label embedding layer to adapt the model to the multi-label problem. Once the training is finished, images are sampled according to the label distribution of the original dataset to maintain the distribution of co-occurring diseases in the synthetic dataset.
C.2 Implementation Details
Training Settings of the Diffusion Model. Following the protocol in [94] the DiT is trained on 256 256 images for 2800 epochs, employing global batch size of 512 distributed across Nvidia A100 GPUs. Throughout training, constant learning rate of 10-4 is maintained. After the training of the diffusion model is finished, synthetic images are sampled using a CFG scale of 4.0 and 128 sampling steps. To preserve the disease co-occurrence distribution within the synthetic dataset, identical image labels as those from the original dataset are utilized, The number of synthetic images added to the training set of each dataset is determined via volidaaaii. We first generate synthetic images of the same size as the training set. The optimal percentage of synthetic images is selected using 5-fold cross- validation on the training data. Synthetic images combined with the original dataset for further fine-tuning with low-rank regularization. Figure 66 presents examples of the synthetic chest X-rays generated using the aforementioned setting.
Table 14: Selected optimal percentage of images on different datasets and models.
Dataset Models
CheXpent
COVIDS
VTS 0.13
VIES-LR
VITB-LR
VIT-B-LR
VITS
WIS--LR
VIT-B
VIT--LRR
0.2
0.25
0.25
0.7
1.0
0.75
1.0
Tuning the Number of Synthetic Images by Cross- Validataii...... We determine the optimal number of synthetic images for each dataset and the corresponding ViT variant. Let N = [a x n.. where a is the percentage of the images and n denotes the size of the training set of the target dataset. The values of a are selected through 5-fold cross-vast mooss-vlldationn on the training data in each dataset. Specifically, is chosen from the set {0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 1.03. The optimal values of a selected by cross-vars validation for each dataset and VIT variant are presented in Table 14
Training Settings of the LRFL Models with the Synthetic Images. Once we obtain the synthetic images generated by the diffusion model, we add the synthetic images into the training set of the target datasets, including COVIDx and CheXpert. We also leverage networks pre-trained on ImageNet [92 or chest X-rays 2 using Masked Autoencoders (MAE). The MAE pre-trained models are fine-tuned following the same pipeline in Section 3.1 and the same implementation details as in Scction 4.1
23

---

Label: Cardiomegaly and Effusion
Label: Effusion, Infiltration and Pneumonia
Figure 6: Synthetic images generated using the Diffusion Model. The images in the first row labeled Cardiomegaly and Effusion, and the images in the second row are labeled Effusion, Infiltration, and Pneumonia.
91
91
IAAC (%)
mAUC (%)
87
87
86
VIT-S
86
VIT-B
85
0.1 0.2
0.5
0.1 0.2
Number Syuthetic Images (*w)
Number of Synthetic Images (*e)
(a) ViT-S and ViT-S-LR
(b) ViT-B and ViT-B-LR
Figure 7: Performance comparisons between base models and LRFL models trained with different numbers of synthetic images added on CheXpert, the number of original training images in CheXpert.
C.3 Ablation Study on the Number of Synthetic Images
Although the usage of the generated synthetic images can improve the prediction accuracy of DNNs for image classification [3] 15), too many synthetic images tend to introduce more noise to the augmented training data so excessive synthetic images can hurt the prediction accuracy of DNNS trained on the augmented training data [14]. Our proposed LRFL method, coupled with the selection of the amount of synthetic images, effectively mitigates this issue. In this section, compare the performance of LRFL models with base models when different numbers of synthetic images are added to the training set of Che As illustrated Figure| 7 the performance of both the LRFL model and the base model can be initially improved with more synthetic images. However, after a certain point, even more, synthetic images start to hurt the performance due to the noise in synthetic images, and the literature has a similar observation. This is the reason why we need to perform a voss--validtiinn on the size of the synthetic data for the best performance. Importantly, it can be observed that our LRFL models (ViT-S-LR or ViT-B-LR) usually improve the performance the corresponding base models (ViT-S or VIT-B) on different choices of the size of the synthetic data. The improvements of our LRFL models over the corresponding base models tend to be more significant as the size synthetic data increases. This observation justifies the effectiveness of LRFL in reducing the adverse effect of noise in the synthetic images. For example, ViT-B-LR outperforms VIT-B by 0.5% in mAUC when 0.1n synthetic images added into the training set, and the improvement escalates 2.5% with synthetic images added into the training set where n the size of the original training data.
using synthetic data for training classifiers such as 13 also

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction in Section accurately reflect the paper's 's contributions and scope, The contributions of the paper are clearly stated in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
25

---

The theorem and proposition in this paper are clearly stated in Section 3 The proof of the theorem attached in Section Af the appendix of this paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The anonymous link to the code is attached at the end of the abstract. Imple- mentation details for the experiments image classification are stated in Section 4 and training the diffusion model is described in Section Of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The anonymous link to the code is attached at the end the abstract. Imple- mentation details for the experiments image classification stated in Section 4 and training the diffusion model is descrihed in Section Of the appendix. The datasets used in the experiments are public benchmarks. Detailed information the datasets for image classification is stated in Section
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
public/guider
for more details.
While we encourage the release of code and data, we understand that this might be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results, See the NeurIPS code and data submission guidelines (https; ctsppicc http:ppwwwwpggguuiu/eeeeeeeeeeeh wttpe///oooeeeeeeeniniiiieeeeeee onPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings, including data splits, hyperparameters, type of optimizer, etc, for image classification are stated in Section 4 and training the diffusion model is described in Section C of the appendix. The datasets used in the experiments are public benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We follow the manners in previous works (2 for reporting the results image classification, in Section g and for training of the diffusion model in Section the appendix.
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computing resources used are specified in Section B.4.5 of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: btthiisshtta:://euuueee httper/pss.ccpppbb///tcchcccccc
Answer: [Yes]
Justification: We have read NeurIPS Code of Ethics EthicsCuidel ines and confirm that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
28

---

The experiments performed do not have societal impacts as we did not perform any experiments on real-world data collected from the society.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper performs experiments on widely used public benchmarks, which are properly cited in Section 4 of our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
29

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paper swi theode com/datasert has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
30

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "BPTsEasS4D", "reviewer": "Reviewer_qSFW", "summary": "This paper introduces a novel Low-Rank Feature Learning (LRFL) method to effectively reduce noise and non-disease areas in radiographic images, enhancing disease classification. The LRFL method, which is theoretically and empirically motivated, demonstrates superior classification performance compared to state-of-the-art methods when applied to pre-trained neural networks, improving both multi-class AUC and classification accuracy.", "questions": "weakness above are self-descriptive and including questions.", "limitations": "- lack of novelty\n- lack of enough and valid comparisons\n- experimental results are not convincing", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- thorax disease classification is not an easy problem, motivation is high, and significance is solid.\n- low rank feature learning is proposed, which maybe applicable to all kinds of neural networks for disease classification (thorax). \n- three large scale x-ray data are used, and good results were obtained.\n- sharp generalization bound analysis is solid", "weaknesses": "- LRFL is based on LFP, and truncated nuclear norm is added as a regularization term. Nothing more. In this sense, there are so many similar methods with different regularizations. \n- training diffusion algorithms to generate synthetic images (xray) is already done by many..why authors propose this as a novelty ?\n- introduction about radiographic images is odd...too simple and already known\n- section 2.2 stands out of nowhere...very broad without specific information related to work.\n- section 2.3 can be longer, that is the main part and motivation but kept short and simple. Put a picture to highlight.\n- figure 2 is useless. \n- Gradcam is noisy, better to use gification and other methods to show the localizations, or robust version of grad cam. there are many works showing grad cam is not a suitable method.\n- comparisons are weak, there are many advanced versions of algorithms there, even with eye tracking supported classification results for the same data available. SOTA is not updated.\n- discussion is missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. LRFL is based on LFP, and truncated nuclear norm is added as a regularization term. Nothing more. In this sense, there are so many similar methods with different regularizations.**\n\nWe respectfully but strongly disagree with this claim since the significant contributions of this paper are missed in this claim.\n\nWhile this paper uses the truncated nuclear norm (TNNR) for low-rank learning and TNNR has also been used for low-rank learning in the existing machine learning literature, the significance and novelty of the proposed LRFL method lies in the following two aspects, with significant advantages over the existing works.\n\n**First, we propose a novel separable approximation to the TNNR, so that standard SGD can be used to efficiently optimize the training loss of LRFL with the TNNR**. The formulation of the separable approximation to the TNNR is described in Section 3.4 of our paper. The training algorithm with such novel separable approximation to the TNNR by SGD is detailed in Algorithm 1 of our paper. Results in Table 1-3 show that minimizing the training loss with the separable approximation to the TNNR significantly improves the performance of baseline models for disease classification.\n\nTo further verify the efficiency of our training algorithm compared to the existing optimization method for the TNNR, we compare the training time of our LRFL models with an existing method for optimizing the TNNR, TNNM-ALM [1], on NIH ChestX-ray14, CheXpert, COVIDx. The results in the table below show that our LRFL method achieves 7$\\times$-10$\\times$ acceleration in the training process on the three datasets, demonstrating the effectiveness and efficiency of the separable approximation to the TNNR proposed in our paper.\n\n|     Methods      |   NIH ChestX-ray14 (minutes)   |   CheXpert (minutes)   |   COVIDx (minutes)   |\n| :--------------: | :------: | :------: | :------: |\n|    ViT-S    |   54   |   90   |   23   |\n| ViT-S (TNNM-ALM) | 804 | 854 | 342 |\n| ViT-S-LR | 98 | 117 | 38 |\n| ViT-B | 72 | 162 | 32 |\n|     ViT-B (TNNM-ALM)     |   915   |   1461   |   418   |\n|   ViT-B-LR    | 113 | 185 | 45 |\n\n**Second, we provide rigorous theoretical result justifying the proposed low-rank feature learning**. In particular, it is shown in Theorem 3.1 that the upper bound for the generalization error of the linear neural network in our framework involves the TNNR, and a smaller TNNR leads to a smaller generalization bound thus improves the generalization capability of the network.\n\n**2. training diffusion algorithms to generate synthetic images (xray) is already done...**\n\nTo the best of our knowledge, this work is among the first to use synthetic images to boost the performance of DNNs on thorax disease classification tasks coupled with the proposed efficient low-rank feature learning method.\n\n**3. introduction about radiographic images is odd...too simple and already known**\n\nWe will simplify such an introduction to radiographic imaging in the final version of this paper. \n\n**4. section 2.2 stands out of nowhere...very broad without specific information related to work.**\n\nWe respectfully disagree with this factually wrong claim. Section 2.2 covers important works in using DNNs for medical imaging tasks including thorax disease classification. Importantly, the MAE method [1] introduced in Section 2.2 is the important pre-training method used in medical imaging for thorax disease classification, which is also adopted as the pre-training method in this work.\n\n**5. section 2.3 can be longer, that is the main part and motivation but kept short and simple. Put a picture to highlight.**\n\nWe will elaborate existing low-rank learning methods with more details, and add a figure similar to Figure 1 to Section 2.3.\n\n**6. figure 2 is useless.**\n\nWe will change Figure 2 to a text description in Sec. 3.1.\n\n**7. Gradcam is noisy, better to use gification and other methods to show the localizations, or robust version of grad cam. there are many works showing grad cam is not a suitable method.**\n\nPlease refer to the robust GradCAM results in our rebuttal PDF file in our global response. \n\n**8. comparisons are weak, there are many advanced versions of algorithms there, even with eye tracking supported classification results for the same data available. SOTA is not updated. discussion is missing**\n\nWe have already incorporated the most recent SOTA results in [1] published in 2023 for thorax disease classification on the same datasets as that in this paper, and our LRFL models render significantly better results than the current SOTA [1] and other competing baselines. We also provide detailed discussions about our results in Section 4, and the experimental result for each dataset has a paragraph for discussion titled \"Results and Analysis\" or \"Results\". Eye tracking supported classification methods are not in the scope of this work and the relevant rich literature, because this work and the relevant rich literature such as [1] and those reviewed in Section 2.2 are using DNNs for automatic thorax disease classification or general medical imaging tasks without eye tracking information.\n\n**References**\n\n[1] Xiao et al. Delving into masked autoencoders for multi-label thorax disease classification. WACV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfI7XrXXDv", "reviewer": "Reviewer_cPxj", "summary": "The paper introduces LRFL, a method for reducing the effect of noise and background or non-disease areas in radiograph images for Thorax Disease Classification.  LRFL utilizes low-rank regularization to leverage low-rank features during network training.", "questions": "1- Why did the author not use another radiograph to prove his/her claim that their approach can be used broader with any classification problems in radiographic images?\n\n2- Why did the author not add another evaluation method for their approach, likewise the baseline [2] Table 7?", "limitations": "1- We recommend the author test their approach with another radiograph dataset such as knee disease to prove their claim.\n\n2-Table 4 in P.17 illustrates a close performance (0.4%) between the baseline [2] and the proposed method. Therefore, we encourage the author to include another evaluation norm (i.e., disease localization) and compute the Average precision (i.e., AP25 and AP50) between the ground truth and predicted bounding box to improve their work. Although the author compares his/her approach with the baseline in Figure 5 using Grad-CAM visualization, it is better to include a table comparing the approaches rather than pick random images from the Grad-CAM visualization.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1-The motivation for reducing the adverse of the noise and background to learn better features is interesting and reasonable.\n\n2-Extensive experiments have shown that the proposed method improves performance over prior STOA approaches on different thorax disease datasets.\n\n3- The content flow of the paper makes it easy for readers to grasp the presented information.", "weaknesses": "1-The author claims that their approach could be applied to classify other diseases beyond thorax diseases or even general classification problems with radiographic images without conducting any experiments on other datasets. \n\n2-The approach's evaluation is only done using mAUC, while the baseline uses IoU with average precision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the review and the suggestions in this review. The raised issues are addressed below.\n\n**1. Why did the author not use another radiograph to prove his/her claim that their approach can be used broader with any classification problems in radiographic images?**\n\nThank you for your suggestion! Because this paper focuses on the application of the proposed low-rank feature learning framework to thorax disease classification, we do not put the results for classification problems for other diseases such as knee disease due to the page limit. However, we will present such results for more types of diseases in the final version of this paper.\n\n**2. Why did the author not add another evaluation method for their approach, likewise the baseline [2] Table 7?**\n\nWe show the improved Average Precision (AP) results for disease localization in the table below, where the AP for disease localization is computed following the same settings as [2] (reference in the paper). The experiments are done on a subset of ChestX-ray14 which offers 787 cases with bounding-box of a total of eight thorax diseases. It is observed from the results below that our LRFL model improves the $AP_{25}$ and $AP_{50}$ for disease localization by 1.1 and 1.2 respectively.\n\n|   Disease    | Size (# of px) | ViT-S AP$_{25}$ | ViT-S AP$_{50}$ |ViT-S-LR AP$_{25}$ | ViT-S-LR AP$_{50}$ |\n| :----------: | :------------: | :-----------: | :-----------: |:-----------: | :-----------: |\n|    Nodule    |      224       |      9.2      |      3.9      |     11.7     |     5.1      |\n|     Mass     |      756       |     27.0      |     11.1      |    29.3      |     12.2      |\n| Atelectasis  |      924       |     31.5      |      8.1      |    34.2      |      9.6      |\n| Pneumothorax |      1899      |      4.7      |      0.0      |     6.2      |      1.7      |\n|  Infiltrate  |      2754      |     11.4      |      1.3      |    12.9      |      1.9      |\n|   Effusion   |      2925      |      8.8      |      1.0      |     10.2     |      2.0      |\n|  Pneumonia   |      2944      |     27.8      |      9.3      |    29.6      |      10.2      |\n| Cardiomegaly |      8670      |     16.3      |      3.0      |    18.8      |      4.2      |\n|     All      |      2300      |     18.0      |      4.7      |    19.1      |      5.9      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eO99vn6JVa", "reviewer": "Reviewer_td2a", "summary": "The paper is concerned with the problem of thorax disease classification in radiographic images. The authors propose a novel low-rank feature learning (LRFL) method which is applied on pre-trained masked autoencoders (MAE) and evaluated on two datasets (CheXpert and COVIDx). The authors also provide theoretical results on the generalization bound of the proposed approach. The authors show that their method outperforms multiple baselines on the two datasets.", "questions": "- Table 12: Do the authors have an explanation why the optimal $\\alpha$ is 1.0 for COVIDx but 0.2/0.5 for CheXpert? Also, did the authors conduct experiments with $\\alpha > 1.0$?", "limitations": "The authors do not discuss the limitations of their method in the paper and answer the question 2 in the NeurIPS paper checklist with \"[NA]\" indicating that \"the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.\" I do not agree with this assessment and think that the authors should discuss the limitations of their method openly. Some limitations are e.g.,:\n- The method is evaluated on two datasets only, both of which are concerned with thorax disease classification. Whether the method generalizes to other datasets remains unclear.\n- The authors do not evaluate their method on any non-pretrained network and on few architectures only.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, the topic of disease detection in radiographic images is of high importance and particularly improving robustness and generalization capabilities of models is a relevant research direction. The authors provide a good overview of the related work and perform many ablation studies and experiments to evaluate their method.", "weaknesses": "**Limited evaluation**\n\n- The authors motivate their method with an \"adverse effect of noise and background, or non-disease areas, for disease classification on radiographic images.\" (L. 47-49). While it seems, that the method quantitatively outperforms multiple baselines, the authors do not provide any evidence that the proposed method is more robust to noise or background than the baselines. Also, the GradCAM visualizations in Fig. 4 are not convincing and GradCAM visualizations for many CNN-based methods (e.g., [90], Fig. 7; [2], Fig. 4) show much better localization capabilities than what is shown here for both the baseline and the proposed method. I admit that these may be different samples and visualizations are not directly comparable. Can the authors comment on this and also show visualizations using different architectures such as CNNs?\n- \"Unlike traditional methods, our approach introduces a separable approximation to the truncated nuclear norm, facilitating the optimization process and enhancing the generalization ability of the model, thus advancing the state-of-the-art in medical image analysis.\" (L. 134-137). I don't see any experiments that specifically evaluate the generalization ability of the model. \n- L. 9-11: \"To address this challenge, we propose a novel Low-Rank Feature Learning (LRFL) method in this paper, which is universally applicable to the training of all neural networks.\" - The authors apply their LRFL method only to four different architectures from one reference [2]. Whether the method universally improves the training of all neural networks is not shown in the manuscript.\n\n**Unsupported claims**\n\nThe authors state several claims that are not supported by the literature or the experiments in the paper. Some examples:\n- L. 41-43: \"Clinical studies show that the disease areas on radiographic images can be subtle which exhibit localized variations, and such conditions are further complicated by the inevitable noise which is ubiquitously on radiographic images as detailed in Section 2.1\" - Can the authors give examples where noise affects disease detection in the radiographic images of the datasets used in this study or provide references to support this claim?\n- L. 56-57: \"That is, the low-rank projection of the ground truth training class labels possesses the majority of the information of the training class labels. In fact, LFP widely holds for a broad range of classification problems using deep neural networks, such as [1, 8, 9].\" - Can the authors provide a reference for this claim? None of the provided references [1,8,9] mentions low-rank features or a low frequency property at all.\n- L. 14-15: \"LFP not only widely exists in deep neural networks for generic machine learning [...]\" - Similar to the previous point, can the authors provide a reference for this claim?\n- L. 63-64: \"As a result, the adverse effect of such noise and background is considerably reduced in a network trained by our LRFL method.\" - The authors do not perform any experiments which systematically evaluate the effect of noise or background on their method or the baseline methods (see previous point **Limited evaluation**)\n\n**Confusing notation**\n\nI find the mathematical notation in the paper to be inconsistent and confusing. Some examples:\n- \"Suppose the training data are given as $\\\\{\\boldsymbol{x}_i, \\boldsymbol{y}_i\\\\}\\_{i=1}^n$ where $\\boldsymbol{x}_i$ and $\\boldsymbol{y}_i \\in \\mathbb{R}^C$ are the $i$-th training data point and its corresponding class label vector respectively, and $C$ is the number of classes. Each element $\\boldsymbol{y}_i$ is binary with $\\boldsymbol{y}_i = 1$ indicating the $i$-th disease is present in $\\boldsymbol{x}_i$, otherwise $\\boldsymbol{y}_i = 0$. The authors first denote with $\\boldsymbol{y}_i$ a \"$C$-dimensional class label vector\" but then say that \"$\\boldsymbol{y}_i$ is binary with $\\boldsymbol{y}_i = 1$ indicating the $i$-th disease present in $\\boldsymbol{x}_i$\". I would denote samples as $\\boldsymbol{x}^{(i)}$ and class labels as $\\boldsymbol{y}^{(i)}$ and then $\\boldsymbol{y}^{(i)}_j$ indicates whether disease $j$ is present in sample $i$. Or use a matrix $\\boldsymbol{Y}$ as introduced later in the manuscript.\n- Eq. (1): If $\\boldsymbol{W}\\_1$ is also optimized during training, then it should be $f\\_{\\boldsymbol{W}\\_1}(\\boldsymbol{x})$ instead of $f\\_{\\boldsymbol{W}\\_1(0)}(\\boldsymbol{x})$.\n- L. 171: \"Using notations in Section 3.2, the truncated nuclear norm of $\\boldsymbol{F}$ is $\\lVert F\\rVert := \\sum_{i=T+1}^{d} \\sigma_i$ where $T ∈ [0, d]$.\" In section 3.2, the authors introduce with $\\sigma$ the element-wise sigmoid function, which is of course not what is used to compute the truncated nuclear form. Instead the authors use the singular values of the matrix $\\boldsymbol{F}$ without introducing them. Also, the sum is indexed from $T+1$ to $d$ which doesn't make sense for $T=d$. The same index error is also made in Eq. 3.\n\nGenerally, these inconsistencies make it hard to follow the manuscript and mathematical derivations. I would recommend the authors to carefully revise the notation and make it consistent throughout the manuscript.\n\n\n**Typos and other**\n\n- L. 2, 29, 105, 245, ... \"Visual Transformer (ViT)\" -> \"Vision Transformer (ViT)\"\n- L. 25-26: \"[...] abnormalities detection in anatomy in chest X-rays\" -> \"[...] abnormalities detection in chest X-rays\" \n- L. 28: \"Early works adopt convolutional neural networks (CNNs) such as U-Net [3] for representation learning on radiography images.\" - I am not sure how one would use a U-net for representation learning. U-nets are typically used for segmentation tasks or image-to-image mappings. Please clarify.\n- L. 157: \"denotes the denotes the weights\" -> \"denotes the weights\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the review and the suggestions in this review. The raised issues are addressed below.\n\n**1. ...the authors do not provide any evidence that ...more robust to noise or background than the baselines...**\n\nPlease refer to the robust GradCAM results in our rebuttal PDF file using the suggested architecture (CNNs). Furthermore, we performed an additional ablation study showing that the proposed method is more robust to background than the baselines. In this study, we created a mask for the disease area for each original image, then decomposed the original image (with a bounding box for the disease) to a disease image and a background image. Both the disease image and the background image are of the same size as the original image, the background image has greyscale 0 in the masked disease area, and the disease image has greyscale 0 in the non-disease area. We fed the three images, which are the original image, the disease image, and the background image, to a LRFL model, and obtained the original features, disease features, and background features for the LRFL model respectively. We also fed these three images to a baseline model, and obtained the original features, disease features, and background features for the baseline model respectively. For each original image, we measure the distance between the disease features and original features using KL-divergence on the softmaxed features for the LRFL model and the baseline model. We then compute the average feature distance for each model, which is the average distance between the disease features and original features over the images with a ground-truth bounding box for the disease in the NIH dataset. **The average feature distance for the LRFL model is 0.5642, which is smaller than the average feature distance for the baseline model, 0.6628**. Such results indicate that the original features are closer to the disease features by the LRFL model compared to the baseline model, evidencing the effectiveness of the LRFL model in reducing the adverse effect of the background area. We also remark that since only the low-rank part of the original features participates in the classification process, the noise and non-disease areas in the high-rank part of the features are mostly not learned by LRFL, and in this manner, LRFL is robust to both noise and background. \n\n**2. I don't see any experiments that specifically evaluate the generalization ability of the model.**\n\nThe improved generalization ability of a model in this work and the machine learning literature refers to its better prediction accuracy on the unseen test data, which has been shown in Table 1-3 of this paper.\n\n**3. ...Whether the method universally improves the training of all neural networks...**\n\nTo show that LRFL universally improves the performance of different neural networks, we apply LRFL to four other networks for disease classification on NIH ChestX-ray14, namely Dira [5], Acpl [6], XProtoNet [7], and Swinchex [8]. We trained the LRFL models using the settings described in Section 4.1 of our paper. Results in the table below show that LRFL universally improves the performance of all the baseline models.\n\n|     Methods      |   mAUC   |\n| :--------------: | :------: |\n|     Dira [5]     |   81.7   |\n|   **LR-Dira**    | **82.5** |\n|     Acpl [6]     |   81.8   |\n|   **LR-Acpl**    | **82.3** |\n|  XProtoNet [7]   |   82.2   |\n| **LR-XProtoNet** | **82.7** |\n|   Swinchex [8]   |   81.0   |\n| **LR-Swinchex**  | **81.8** |\n\n**4. ...Can the authors give examples where noise affects...?**\n\nStudies in the literature [9, 10] show that inevitable noise exists in radiographic images and can affect disease detection on them. We will add the references to our paper to support the claim.\n\n**5. ...Can the authors provide a reference for this claim?  ... LFP...can the authors provide a reference for this claim?**\n\nLFP is commonly observed in various classification scenarios utilizing deep neural networks, and please refer to [1-4] for the claim about LFP.\n\n**6. ...The authors do not perform any experiments...effect of noise or background...**\n\nPlease refer to point 1 of this rebuttal for the experiment showing that LRFL models reduce the adverse effect of background on the radiographic images for disease classification.\n\n**Confusing notations/typos**\n\nWe will fix the confusing notations and typos following your suggestions. Moreover, U-Net can be used for representation learning [11, 12, 13]. In fact, segmentation tasks or image-to-image mappings are achieved by using the features/representations learned by U-Nets [11, 12, 13].\n\n**Table 12...with  $\\alpha > 1$?**\n\nBecause the synthetic data contains noise as they are generated from a diffusion model using random noise as the input, adding more synthetic data does not always improve the prediction accuracy of our models and general deep neural networks [14, 15]. For example, \nthe experiments in [14] show that adding excessive synthetic images to the training set hurts the accuracy of image classification. \nIn our work, we use cross-validation to select the amount of synthetic data for training the model and find the corresponding $\\alpha$ for each dataset. We performed an additional experiment and extended the candidate values of $\\alpha$ which include $(1.5, 2,\\ldots, 5)$ with an increment of 0.5, and obtained the same $\\alpha$ values as those in Sec. C.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXsRH6htJw", "reviewer": "Reviewer_JazK", "summary": "The authors proposed to use Low-rank Feature Learning (LRFL) to improve model performance specifically for thorax diseases classification. Ideas come from the assumption that the low-rank features capture the majority of the information. They implemented the LRFL by adding a self-modified regularization term, and provided theoretical results on the boundaries of the LRFL methods.", "questions": ". “truncated nuclear norm as” how did the truncate nuclear norm work? May need references.\n\t2. “Because the actual features used for classification are approximately low-rank and the high-rank features are significantly truncated, all the noise and the information about the background, or the non-disease areas on radiographic images in the high-rank features are largely discarded and not learned in a neural network.” Any reference to support your statement?\n\t3. How does the image dimensions (HW dim & C dim) matched when you use both datasets “ImageNet-1k and X-rays (0.5M)” for pretraining? \n\t4. Table 2, how the decision cutting threshold was chosen? Is it the same across different models/datasets?\n\t5. I have a different thought against the authors. I understand that the low-rank features can maintain the majority of the information. However, some of the nodules or lesions or abnormalities that used for diagnosis are actually small and tiny. Then, by using the low-rank feature approach, the small region information might be lost but will not negatively impact the over-all information that much. How does the method handle this situation?\n\t6. It seems like the authors proposed a very large and general topic on improving model performance by learning low-rank features for some specific. There is no specific design related to thorax disease, the X-ray image, or even the medical image. I would assume this method could be applied to any imaging with some background noise. Therefore, since there are existing studies doing low-rank features-related experiments, I would see this is a similar implementation of the low-rank feature learning but very hard to observe valuable novelties.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Rigorous proof, nice writing and relative comprehensive experimentations.", "weaknesses": "It seems like the authors was tackling a very large and general topic on improving model performance by learning low-rank features for some specific. There is no specific design related to thorax disease, the X-ray image, or even the medical image. I would assume this method could be applied to any imaging with some background noise. Therefore, since there are existing studies doing low-rank features-related experiments, I would see this is a similar implementation of the low-rank feature learning but very hard to observe valuable novelties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the review and the suggestions in this review. The raised issues are addressed below.\n\n**1. ...how did the truncated nuclear norm work? May need references.**\n\nThe truncated nuclear norm is defined in line 171 of our paper. Existing works [1,3, 4] perform low-rank learning by minimizing the truncated nuclear norm of the feature matrix. \n\n**2. “Because the actual features used for classification are approximately low-rank and the high-rank features are significantly truncated, all the noise and the information about the background, or the non-disease areas on radiographic images in the high-rank features are largely discarded and not learned in a neural network.” Any reference to support your statement?**\n\nStudies in the literature [5,6,7] show that learning low-rank features can enhance robustness to noise in the input images. We will add these references to our paper to support the claim.\n\n**3. How does the image dimensions (HW dim & C dim) matched when you use both datasets “ImageNet-1k and X-rays (0.5M)” for pretraining?**\n\nWhen ImageNet-1k and X-rays (0.5M) are used for the pre-training of models in our paper, all the images will be reshaped to $224 \\times 224 \\times 3$ following the settings in [8].\n\n**4. Table 2, how the decision cutting threshold was chosen?...**\n\nWe performed cross-validation described in line 248-261 to decide the cutting threshold for the rank in Table 2. Yes, we performed the same cross-validation process across different models/datasets.\n\n**5. I have a different thought against the authors...**\n\nWe respectfully disagree with your thought that “by using the low-rank feature approach, the small region information might be lost”. We now demonstrate the evidence that the features of small disease regions are still captured well by our LRFL model. In particular, because ViT is known to be robust to diseases in a small size such as nodules [2], LRFL on a ViT model can still capture such small-sized disease areas. This is evidenced by the results in Average Precision (AP) about disease localization in point two of our response to Reviewer cPxj or point 3 of our global response. In these results, it is shown that LRFL model renders higher AP for small disease areas such as nodules for disease localization.\n\n**6. ...I would see this is a similar implementation of the low-rank feature learning but very hard to observe valuable novelties.***\n\nWe respectfully but strongly disagree with the claim that “I would see this is a similar implementation of the low-rank feature learning”. The significant contributions of this paper have been missed in this claim.\n\nWhile this paper uses the truncated nuclear norm (TNNR) for low-rank learning and TNNR has also been used for low-rank learning in the existing machine learning literature, the significance and novelty of the proposed LRFL method lies in the following two aspects, with significant advantages over the existing works.\n\n**First, we propose a novel separable approximation to the TNNR, so that standard SGD can be used to efficiently optimize the training loss of LRFL with the TNNR**. The formulation of the separable approximation to the TNNR is described in Section 3.4 of our paper. The training algorithm with such novel separable approximation to the TNNR by SGD is detailed in Algorithm 1 of our paper. Results in Table 1-3 show that minimizing the training loss with the separable approximation to the TNNR significantly improves the performance of baseline models for disease classification.\n\nTo further verify the efficiency of our training algorithm compared to the existing optimization method for the TNNR, we compare the training time of our LRFL models with an existing method for optimizing the TNNR, TNNM-ALM [1], on NIH ChestX-ray14, CheXpert, COVIDx. The results in the table below show that our LRFL method achieves 7$\\times$-10$\\times$ acceleration in the training process on the three datasets, demonstrating the effectiveness and efficiency of the separable approximation to the TNNR proposed in our paper.\n\n|     Methods      |   NIH ChestX-ray14 (minutes)   |   CheXpert (minutes)   |   COVIDx (minutes)   |\n| :--------------: | :------: | :------: | :------: |\n|    ViT-S    |   54   |   90   |   23   |\n| ViT-S (TNNM-ALM) | 804 | 854 | 342 |\n| ViT-S-LR | 98 | 117 | 38 |\n| ViT-B | 72 | 162 | 32 |\n|     ViT-B (TNNM-ALM)     |   915   |   1461   |   418   |\n|   ViT-B-LR    | 113 | 185 | 45 |\n\nSecond, **we provide rigorous theoretical result justifying the proposed low-rank feature learning**. In particular, it is shown in Theorem 3.1 that the upper bound for the generalization error of the linear neural network in our framework involves the TNNR, and a smaller TNNR leads to a smaller generalization bound thus improves the generalization capability of the network.\n\n**References**\n\n[1] Lee et al. Computationally Efficient Truncated Nuclear Norm Minimization for High Dynamic Range Imaging. IEEE Transactions on Image Processing 2016.\n\n[2] Xiao et al. Delving into masked autoencoders for multi-label thorax disease classification. WACV 2023.\n\n[3] Hu, Yao, et al. \"Large scale multi-class classification with truncated nuclear norm regularization.\" Neurocomputing 2015.\n\n[4] Zhang, Fanlong, et al. \"Truncated nuclear norm based low Rank Embedding.\" Biometric Recognition 2017\n\n[5] Gao, Ming, et al. \"Noise robustness low-rank learning algorithm for electroencephalogram signal classification.\" Frontiers in Neuroscience 2021.\n\n[6] Lu, Yuwu, et al. \"Low-rank preserving projections.\" IEEE transactions on cybernetics 2015.\n\n[7] Ren, Jiahuan, et al. \"Robust low-rank convolution network for image denoising.\" ACM MM 2022.\n\n[8] Xiao, Junfei, et al. \"Delving into masked autoencoders for multi-label thorax disease classification.\" WACV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the review and the suggestions in this review. We have posted our response to individual reviews addressing all the raised concerns. Here we provide global responses itemized below.

**1. Significance and novelty of this paper**

While this paper uses the truncated nuclear norm (TNNR) for low-rank learning and TNNR has also been used for low-rank learning in the existing machine learning literature, the significance and novelty of the proposed LRFL method lies in the following two aspects, with significant advantages over the existing works.

**First, we propose a novel separable approximation to the TNNR, so that standard SGD can be used to efficiently optimize the training loss of LRFL with the TNNR**. The formulation of the separable approximation to the TNNR is described in Section 3.4 of our paper. The training algorithm with such novel separable approximation to the TNNR by SGD is detailed in Algorithm 1 of our paper. Results in Table 1-3 show that minimizing the training loss with the separable approximation to the TNNR significantly improves the performance of baseline models for disease classification.

To further verify the efficiency of our training algorithm compared to the existing optimization method for the TNNR, we compare the training time of our LRFL models with an existing method for optimizing the TNNR, TNNM-ALM [1], on NIH ChestX-ray14, CheXpert, COVIDx. The results in the table below show that our LRFL method achieves 7$\times$-10$\times$ acceleration in the training process on the three datasets, demonstrating the effectiveness and efficiency of the separable approximation to the TNNR proposed in our paper.

|     Methods      |   NIH ChestX-ray14 (minutes)   |   CheXpert (minutes)   |   COVIDx (minutes)   |
| :--------------: | :------: | :------: | :------: |
|    ViT-S    |   54   |   90   |   23   |
| ViT-S (TNNM-ALM) | 804 | 854 | 342 |
| ViT-S-LR | 98 | 117 | 38 |
| ViT-B | 72 | 162 | 32 |
|     ViT-B (TNNM-ALM)     |   915   |   1461   |   418   |
|   ViT-B-LR    | 113 | 185 | 45 |

**Second, we provide rigorous theoretical result justifying the proposed low-rank feature learning**. In particular, it is shown in Theorem 3.1 that the upper bound for the generalization error of the linear neural network in our framework involves the TNNR, and a smaller TNNR leads to a smaller generalization bound thus improves the generalization capability of the network.

We would also like to remind the reviewers that this work is among the first to effectively use synthetic data generated by a diffusion model and a low-rank feature learning model to achieve the state-of-the-art accuracy for thorax disease classification, which is an important research problem in the medical imaging domain. 

**2. Robust Grad-CAM visualization results**

Following the suggestions in the reviews, we illustrate the robust Grad-CAM [1] visualization results in the attached rebuttal PDF file.

**3. LRFL for disease localization**

We show the improved Average Precision (AP) results for disease localization in the table below, where the AP for disease localization is computed following the same settings as [2].  The experiments are done on a subset of ChestX-ray14 which offers 787 cases with
bounding-box of a total of eight thorax diseases. It is observed from the results below that our LRFL model improves the $AP_{25}$ and $AP_{50}$ for disease localization by 1.1 and 1.2 respectively.

|   Disease    | Size (# of px) | ViT-S AP$_{25}$ | ViT-S AP$_{50}$ |ViT-S-LR AP$_{25}$ | ViT-S-LR AP$_{50}$ |
| :----------: | :------------: | :-----------: | :-----------: |:-----------: | :-----------: |
|    Nodule    |      224       |      9.2      |      3.9      |     11.7     |     5.1      |
|     Mass     |      756       |     27.0      |     11.1      |    29.3      |     12.2      |
| Atelectasis  |      924       |     31.5      |      8.1      |    34.2      |      9.6      |
| Pneumothorax |      1899      |      4.7      |      0.0      |     6.2      |      1.7      |
|  Infiltrate  |      2754      |     11.4      |      1.3      |    12.9      |      1.9      |
|   Effusion   |      2925      |      8.8      |      1.0      |     10.2     |      2.0      |
|  Pneumonia   |      2944      |     27.8      |      9.3      |    29.6      |      10.2      |
| Cardiomegaly |      8670      |     16.3      |      3.0      |    18.8      |      4.2      |
|     All      |      2300      |     18.0      |      4.7      |    19.1      |      5.9      |


**References**

[1] Selvaraju, Ramprasaath R., et al. "Grad-cam: Visual explanations from deep networks via gradient-based localization." ICCV 2017.

[2] Xiao et al. Delving into masked autoencoders for multi-label thorax disease classification. WACV 2023.

Meta-review: The paper received mixed reviews with two reviewers leaning towards acceptance and two towards rejection. The main concerns that supported rejecting the paper related to lack of novelty, limited applicability and generalisation on other datasets. Firstly I would like to highlight that the paper has selected a primary area related to medical applications/healthcare . Therefore it is expected that the paper will be addressing a medical problem and the novelty claim should factor that.
Secondly, the authors have presented a mathematical formulation for their modified truncated nuclear norm for low rank learning that is a separable approximation that allows SGD to be used to optimise the training loss of the low rank feature learning method with TNNR. They show competitive results in terms of universally improving the training of other neural networks and also new results on disease localisation (bounding boxes).
The use of diffusion to generate synthetic images is a good addition perhaps but I agree with the reviewer that in and of itself is not novel in the context of a conference like NeurIPS. However this is not the main claim of this paper.

Overall the authors provided comprehensive responses and new results that enhance the message of the paper. The method has both theoretical merit and relevance to the specific medical domain but I also appreciate that the scope as presented is a bit narrow. The authors could have added another medical imaging dataset but at least they slightly expanded the scope by providing the experiments in the rebuttal.

I think all new results must be added to the camera ready version.

Factoring all reviews and the great discussions that took place I believe the paper has just passed the threshold to be accepted in NeurIPS.
Decision: Accept (poster)