Informative Data Selection for Thorax Disease Classification

Authors: Yancheng Wang, Rajeev Goel, Marko Jojic, Alvin C Silva, Teresa Wu, Yingzhen Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Informative Data Selection, Generative Data Augmentation, Thorax Disease Classification

Abstract:
Although Deep Neural Networks (DNNs) such as Vision Transformers (ViTs) have demonstrated superior performance in medical imaging tasks, the training of DNNs usually requires large amounts of high-quality labeled training data, which is usually difficult or even impractical to collect in the medical domain. To address this issue, Generative Data Augmentation (GDA) has been employed to improve the performance of DNNs trained on augmented training data comprising both original training data in the standard benchmark datasets and synthetic training data generated by generative models such as Diffusion Models (DMs). However, the synthetic data generated by GDA universally suffer from noise, and such synthetic data can severely hurt the performance of classifiers trained on the augmented training data. Existing works, such as data selection and data re-weighting methods aiming to mitigate this issue, usually depend on a given clean metadata or external classifier.
In this work, we propose a principled sample re-weighting method, Informative Data Selection (IDS), based on an established information theoretic measure, the Information Bottleneck (IB), to improve the performance of DNNs trained for thorax disease classification with GDA. Extensive experiments demonstrate that IDS successfully assigns higher weights to more informative synthetic images and significantly outperforms existing data selection and data re-weighting methods in GDA for thorax disease classification.
The code of IDS is available at \url{https://anonymous.4open.science/r/IDS-20D1}.

Decision: reject
Reviewer scores: 3, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1729952327863

thread_type:
official_review

<thread_object>
creator: reviewer_y6vf

<content>
summary:
This paper proposes an Informative Data Selection (IDS) method to re-weight synthetic images generated by generative models based on an information-theoretic measure, the Information Bottleneck (IB). IDS trains a sample re-weighting network to minimize the IB loss on the synthetic data. This approach better adheres to the IB principle by encouraging the learning of features that are more correlated with the outputs and less correlated with the inputs. The authors conducted experiments on three thorax disease classification datasets (CheXpert, COVIDx, and NIH ChestX-ray14). They compared their method against other data selection and sample re-weighting techniques, showing superior results.

strengths:
+ The use of the IB principle for sample re-weighting sounds interesting.
+ The paper shows promising results based on the evaluation using three significant thorax disease classification benchmarks compared to state-of-the-art methods.
+ The paper is well-organized and easy to follow.

weaknesses:
1- Regarding methodology, my main comments are lack of sufficient details:

- The computation of gradients for parameters of the classifier and sample re-weighting in Steps 4 and 5 is not detailed. For example, how are the gradients of the VIB loss with respect to θ computed? Are there any approximations to handle the expectations in the IB loss?
- Step 3 refers to computing the class centroids using Equation (3), which involves both real and synthetic data. However, the notation is dense, and the practical implementation of this step, especially with large datasets, could be challenging. Can you elaborate on details of how centroids are updated during training?
- Also, in practice, Bi-level optimization can be sensitive to hyperparameters. Maybe providing discussion (e.g., choice of learning rates, use of momentum) for a better understanding

2- Regarding computational complexity: 

- Maybe I missed it, but I could not find a discussion on the computational overhead introduced by the sample re-weighting network and the IB loss minimization, especially when scaling to larger datasets or higher-resolution images.
- Generating synthetic data using diffusion models can be time-consuming. The authors should elaborate on the efficiency of the data generation process and its impact on the overall training time.

3- About Practical Constraints

- The method assumes that the diffusion models can generate sufficiently realistic and diverse synthetic images. However, the limitations of the synthetic data, such as potential biases introduced during generation, have been limitedly analyzed.
- Also, since IDS relies on pre-trained VAEs and diffusion models, its performance may be sensitive to the quality of these models; I think analysis is required to see their impact.

4- Regarding experiments

- The paper would benefit from more extensive ablation studies to isolate the effects of different components of IDS. For example, analyzing the impact of the VIB term separately from the re-weighting network. In addition, the results are presented without discussion of statistical significance. 

Last minor comment about clinical relevance: I suggest authors include a discussion of how IDS might impact the clinical routine of decision-making or how these weights correlate with specific data characteristics.

questions:
- While the authors reference the standard bi-level optimization approach, I suggest including a brief explanation or highlighting any adaptations made that would improve clarity. 
- Additional ablation studies are suggested (see above). Discuss the statistical significance to substantiate performance claims.
- Address the computational costs associated with IDS, including training and inference times
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537906707

thread_type:
official_review

<thread_object>
creator: reviewer_iqbz

<content>
summary:
This paper introduces a novel method called Informative Data Selection (IDS) to improve thorax disease classification by addressing challenges inherent in Generative Data Augmentation (GDA). While GDA leverages generative models like diffusion models to create synthetic training data—particularly valuable in medical fields with limited high-quality annotated data—it often introduces noise that can degrade model performance. IDS mitigates this issue by applying the Information Bottleneck (IB) principle to re-weight synthetic samples, emphasizing informative data to enhance the classifier’s accuracy.

strengths:
The paper presents an innovative method, IDS, which effectively addresses the scarcity of annotated training data for lung images by assigning higher weights to more informative synthetic images. This approach shows promising potential for practical clinical applications.
The authors have provided the code, offering valuable resources for the community to learn from and build upon, which enhances reproducibility and fosters collaborative advancement in the field.
Comprehensive Related Work: The related work section is thorough, detailing the challenges of obtaining high-quality annotations in the medical field and providing an in-depth analysis of the noise introduced by synthetic data from generative models.
The paper includes extensive experiments on three datasets—CheXpert, COVIDx, and NIH-ChestXray-14—that validate the effectiveness of IDS, supported by a robust amount of experimental data.

weaknesses:
The metric mAUC used in the experiments lacks a detailed formula or explanation, which may hinder readers’ understanding of the evaluation process.
Section 3.2 contains numerous embedded formulas within the main text, making it less intuitive to read. Reorganizing this section by presenting formulas separately and providing explanatory text could enhance readability.
Placement of Algorithm 1: While Algorithm 1 is clearly described, placing it in the appendix could streamline the main text and allow readers to focus on the conceptual framework before diving into algorithmic details.
Discussion on Noise Handling: The method effectively handles noisy and imbalanced datasets; however, the paper lacks a specific discussion on the generation and nature of the noise, as well as how IDS mitigates its effects.
Comparison of Computational Efficiency: The authors focus on accuracy comparisons but do not address efficiency and computational costs. Including this analysis would provide a more comprehensive understanding of IDS’s practicality in real-world applications.

questions:
1.	Why were ViT-S, ViT-B, and DN121 selected as the base classification networks? Is there a particular reason for not evaluating the performance on larger models like ViT-L?
	2.	While the paper focuses on lung classification tasks, could the IDS method be extended to other lung-related tasks such as segmentation or lesion detection? Exploring this could demonstrate the versatility of the approach.
	3.	The experiments utilize classification baseline models based on Transformer and CNN architectures. Have you considered including recent State Space Models, such as Vision Mamba, in your experiments to assess IDS’s effectiveness across different model architectures?
	4.	In practical applications, how would you recommend users balance data selection efficiency with computational costs? Providing guidelines or strategies based on your findings could be beneficial for practitioners.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629969648

thread_type:
official_review

<thread_object>
creator: reviewer_q4pt

<content>
summary:
The authors proposed a data augmentation method by assigning weights to synthetic data. The synthetic data are generated based on a diffusion model, while the data re-weighting is computed based on the information bottleneck. Public chest X-ray datasets are adopted for the experiments and evaluation, where prior data selection and reweighting methods are included in the comparison study. Only marginal performance improvements of the proposed method are reported. The manuscript is overall easy to follow. However, it also suffers from several flaws, which are detailed below.

strengths:
+ Important data selection and reweighting strategy for model training is investigated.
+ The manuscript is easy to follow overall.

weaknesses:
- The comparison study is conducted between the proposed method and previous data selection and sample reweighting methods. However, the proposed method is indeed a data augmentation method with add-on synthetic data as extra training data. Previous and common data augmentation methods (especially those using synthetic data) should be included in the comparison.

- The current sample reweighting is only applied to the synthetic data. How about applying the reweighting strategy to the real data as well? How will that affect the training speed and performance?

- The reported performance gain of the proposed method is really marginal (often less than 1-2% in classification metrics). Such performance gain further raise the questions about the performance gap of the proposed method to common data augmentation methods. 

- The contribution section could be better summarized. 

- The presented results for localization on cardiomegaly are not necessary and are not good example showing relation between weights and localization results. Cardiomegaly is about the heart, and all the bounding boxes will be around the heart. I did not how that could reveal the benefit of reweighting. 

- The proposed method is also strongly related to the active learning methods (using selective synthetic data instead of real ones). The authors could consider prior arts in this field for the related work and comparison study.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696070886

thread_type:
official_review

<thread_object>
creator: reviewer_ntdn

<content>
summary:
This work presents a weighting scheme for samples while training a model with synthetic data to take into account more informative samples.

They train an external network called the "sample re-weighting network". I believe this external network is trained (to state information bottleneck it as bluntly as possible) such that it minimizes the mutual information between embeddings of synthetic images and the respective images and then maximizes mutual information between embeddings of synthetic images and those image labels. This is performed to improve the diversity of generated samples.

I'm not very clear if the synthetic samples are generated dynamically during training or if the samples are already generated and have a weight applied. Maybe having a figure would make this more clear.

strengths:
.

weaknesses:
The comparisons are not conclusive. Specifically, there is no significance testing or estimate of variance so nothing can be concluded from these results. The datasets are not so big that this is intractable. This is specifically important to have for this work because the numbers are so close.

I don't share the intuition that this method works, so it should be asserted by the authors using a significance test that I am wrong. I do not believe sufficient evidence is presented to support the claims of significance in this work.

You can achieve this by randomizing train/val/test splits and reporting the mean test performance and the stdev.

questions:
Why are the experiments limited to chest X-ray data? Has this already been done on other image tasks? Perhaps something common like faces or animals?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper’s introduction of the Informative Data Selection (IDS) method for re-weighting synthetic images in thorax disease classification was met with mixed reactions. While one of the reviewers found the method promising, the other reviewers identified key flaws in the paper's presentation, statistical rigour, computational efficiency, and practical relevance. The authors made substantial efforts to address these issues, conducting new experiments, improving clarity, and adding statistical significance tests. However, this was not enough to overcome the reviewers' criticisms. Some other issues:

-- The distinction between re-weighting and data augmentation was seen as inadequate by some reviewers. One reviewer suggested that IDS is, in essence, a data augmentation method.
-- Reviewers asked for a deeper discussion on how IDS could impact clinical practice and called for comparisons with methods focused on interpretability and uncertainty-based sampling.
-- Reviewers criticised the focus on thorax disease datasets and suggested extending IDS to other datasets. While the authors justified the focus, they did not demonstrate its generalisability.
-- Some reviewers felt that the performance improvements were too small to justify the complexity of IDS, despite the statistical significance

I would like to encourage the authors to improve their paper based on the feedback provided and resubmit to another venue.