Title: A case for reframing automated medical image classification as segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Medical imaging

segmentation

classification
Abstract: Image classification and segmentation are common applications of deep learning to radiology. While many tasks can be framed using either classification or segmentation, classification has historically been cheaper to label and more widely used. However, recent work has drastically reduced the cost of training segmentation networks. In light of this recent work, we reexamine the choice of training classification vs. segmentation models. First, we use an information theoretic approach to analyze why segmentation vs. classification models may achieve different performance on the same dataset and overarching task. We then implement multiple methods for using segmentation models to classify medical images, which we call *segmentation-for-classification*, and compare these methods against traditional classification on three retrospective datasets. We use our analysis and experiments to summarize the benefits of switching from segmentation to classification, including: improved sample efficiency, enabling improved performance with fewer labeled images (up to an order of magnitude lower), on low-prevalence classes, and on certain rare subgroups (up to 161.1\% improved recall); improved robustness to spurious correlations (up to 44.8\% improved robust AUROC); and improved model interpretability, evaluation, and error analysis.

A case for reframing automated medical image classification as segmentation

Abstract

Image classification and segmentation are common applications of deep learning to radiology. While many tasks can be framed using either classification or segmentation, classification has historically been cheaper to label and more widely used. However, recent work has drastically reduced the cost of training segmentation networks. In light of this recent work, we reexamine the choice of training classification vs. segmentation models. First, we use an information theoretic approach to analyze why segmentation vs. classification models may achieve different performances on the same dataset and task. We then implement methods for using segmentation models to classify medical images, which we call segmentation-for-classification , and compare these methods against traditional classification on three retrospective datasets (n=2,018-19,237). We use our analysis and experiments to summarize the benefits of using segmentation-for-classification, including: improved sample efficiency, enabling improved performance with fewer labeled images (up to an order of magnitude lower), on low-prevalence classes, and on certain rare subgroups (up to 161.1% improved recall); improved robustness to spurious correlations (up to 44.8% improved robust AUROC); and improved model interpretability, evaluation, and error analysis.

1 Introduction

Classification and segmentation are two popular applications of deep learning to radiology [1]. Classification produces coarse-grained, image-level predictions, while segmentation produces finegrained, pixel-level maps. While many tasks can be framed using either classification or segmentation (Figure 1), classification is often the default framing. For one, classification has been cheaper to label, requiring only one label per image compared to segmentation's tedious, pixel-wise labels. Second, training classification networks to produce image-level outputs mirrors radiologists' standard workflows, where radiologists summarize findings at the image-level and only routinely perform segmentation in a few sub-disciplines that rely on quantitative measurements over the segmentation masks. These factors have led to an abundance of medical image classification models [2, 3]. In contrast, because segmentation has been so cumbersome to label and is performed relatively infrequently by radiologists, segmentation networks are often only trained when the downstream application requires it (e.g., for computing the volumes or diameters of structures in an image).

Recent work in label-efficient training enables us to reexamine this paradigm. Self-supervised learning, in-context learning, weakly-supervised learning, and semi-supervised learning can substantially reduce labeling burden [4, 5, 6, 7, 8, 9, 10]. Additionally, more public datasets and broad-use pre-

1

Classification

Segmentation

Classification

Segmentation

Classification

Segmentation

Does this chestx-ray contain apneumothorax?

DoesthisCTcontain avertebralfracture?

Is thislesionmelanoma?

trained networks are coming online [11, 12, 13, 14, 15]. Together, these new methods, datasets, and pretrained backbones are enabling users to develop networks for new segmentation tasks with less and less annotation burden-a trend that will likely continue.

In light of this progress, we reexamine the convention around when to employ segmentation networks. Specifically, we investigate using segmentation networks for medical image classification, exploring if segmentation's fine-grained annotation and output can lead to benefits compared to classification's coarse annotation and output. Our contributions include:

Exposition. We analyze why classification and segmentation networks may perform differently on the same datasets. We show that segmentation leads to more separable and robust embedding spaces, guiding what benefits we should expect to see (Section 2).

Best practices. We implement multiple methods to obtain classification labels from segmentation networks, which we call segmentation-for-classification (Section 3). We empirically confirm our analysis (Section 4.1) and compare methods across datasets, tasks, and training conditions to build up best practices for training segmentation-for-classification networks (Section 4.2). We show segmentation-for-classification can improve aggregate performance compared to traditional classification models by up to 16.2% (from 0.74 AUROC to 0.86 AUROC). Finally, we explore semi-supervised segmentation-for-classification, showing existing label-efficient training methods for segmentation can directly benefit segmentationfor-classification.

Trade-offs. We pull together our analysis and experiments to provide consolidated and expanded evidence of the benefits of using segmentation-for-classification, including: improved sample efficiency, enabling improved performance on small datasets, on lowprevalence classes, and on certain rare subgroups (up to 161.1% improved recall); improved robustness to spurious correlations (up to 44.8% improved robust AUROC); and improved model interpretability, evaluation, and error analysis (Section 5).

In summary, conventional wisdom driving the choice between classification and segmentation was formed when labeling burden was a major bottleneck to building neural networks. With recent progress in label-efficient training, we should reconsider conventions. Ultimately, we show that the choice of training a classification vs. segmentation network not only impacts how the model can be used but changes the properties of the model itself, and we show that leveraging segmentation models can lead to higher quality classifiers in common settings. These results indicate segmentation may be a more natural way to train neural networks to interpret and classify images, even though it's not the conventional way humans classify images.

Related work. Our study is motivated by past works that employ segmentation networks in classification problems and report various benefits of doing so [16, 17, 18, 19, 20, 21]. A full discussion of related work is included in Appendix A1.

2 Analysis

In this section we develop an understanding of why segmentation and classification networks may perform differently given the same dataset and overarching task (i.e., to find a class-of-interest). We cover task specification and notation in Section 2.1 then build up intuition for the benefits of segmentation in Section 2.2.

00000000000000

00000000000000

Positive class

Negativeclass

Positiveclass

Negativeclass

Classification

Segmentation

2.1 Preliminaries

Task specification. We first specify the classification tasks we consider replacing with segmentation. We study classification tasks where the following properties are true: the class-of-interest can be localized in the foreground and does not occupy the entire image; the class-of-interest appears if and only if the classification label is positive; the background should not inform the classification label. Example tasks that meet these specifications are shown in Figure 1. Examples of tasks that don't meet these specifications include classifying the imaging modality or radiographic bone age, where global features of an image inform the class label and there is no clear segmentation target.

Notation. We consider a dataset of N images, class labels, and segmentation masks D = { ( X i , y i , S i ) } N i =1 . Each image X i is made up of L pixels, { X i 1 , X i 2 , . . . , X i L } . For simplicity of notation, we assume binary classification, so the image-level classification label is y i ∈ { 0 , 1 } where 1 is the positive label. The segmentation mask S i contains L pixel-level labels { S i 1 , S i 2 , . . . , S i L } indicating which pixels contain the class-of-interest. The image-level class label and pixel-level segmentation mask are related by y i = ✶ { ∑ L k =1 S i k ≥ 1 } . Put simply, this means if the class label is positive, there is at least one pixel labeled as positive in the segmentation mask and vice versa. We train a segmentation network f ( · ) to produce a predicted segmentation mask f ( X i ) = ˆ S i by minimizing the cross entropy loss CE ( ˆ S i , S i ) . For notation's sake, we will drop the sample index i , thus referring to the i th image X i simply as X and the k th pixel X i k simply as X k .

Data generating process. To study segmentation vs. classification, we take an information theoretic approach and use the following simplified data generating process. First, the pixel-level segmentation labels { S 1 , . . . , S L } are drawn from a joint distribution P S . Then, each pixel X k is drawn from the distribution P X ( ·| S k ) . As above, the image-level classification label y can be defined in terms of the segmentation mask, y = ✶ { ∑ L k =1 S k ≥ 1 } . 2.2 Analyzing segmentation vs. classification

To understand why segmentation and classification may perform differently on the same dataset, we consider the distributions that classification vs. segmentation networks are trained to discriminate. As an example, we'll use chest x-ray pneumothorax classification. In classification, supervision is applied at the image-level-the network seeks to discriminate images with a pneumothorax vs. images without a pneumothorax (Figure 2, left). In segmentation, supervision is applied at the pixellevel-the network seeks to identify pixels that contain a pneumothorax (Figure 2, right). Looking at these distributions, differences emerge.

Sample complexity. First, we see that segmentation has more annotated input-output pairs, as each pixel is annotated. Moreover, we see that classification's positive and negative classes share more features than segmentation's-implying segmentation may have greater class separability. We capture this notion of separability using the Kullback-Leibler (KL) divergence and show the following:

Proposition 1 It holds that

define

g(·)

1

VS.

Classificationworkflow

Segmentation-for-classification workflow

In other words, the KL divergence between data given pixel-level labels (as is done in segmentation) is greater than or equal to the scaled KL divergence given image-level labels (as is done in classification). We note that the scaling factor Pr( y =1) Pr( S k =1) is always greater than or equal to one. We provide the proof for Proposition 1 in Appendix A3.1, confirming our intuition that segmentation supervision results in more separable data distributions. Because of the greater quantity of input-output pairs and the greater KL divergence, we expect segmentation's discrimination function to be easier to learn, requiring fewer images or a simpler function class [22, 23]. Since we study the setting in which the segmentation mask directly indicates the classification label ( y = ✶ { ∑ L k =1 S k ≥ 1 } ), we similarly expect segmentation-for-classification to have higher performance than standard classification in the limited data regime. Finally, from Proposition 1, we expect segmentation-for-classification to impart more benefit for tasks with small targets; for large targets, the terms in Proposition 1 approach one another.

Robustness to spurious correlations. The second observation we make from Figure 2 is that, because background features appear in classification's positive class, it is possible for the classification network to rely on background features to identify positive samples if those background features spuriously correlate with the class label. Such failure modes have been observed in prior work. For example, classifiers relying on metal L/R tokens to identify pneumonia [24]; classifiers looking for chest drains to classify pneumothorax [25]; and classifiers looking to image edges (which can change with patient positioning and radiographic projection) to classify COVID-19 [26]. Segmentation should be more robust to such background features, since background features are less correlated with the target class. This notion has been formalized in past work [21], which we recall here.

Specifically, for spurious features that do not overlap with the segmentation target, the mutual information between positive features and spurious features decreases as we transition from supervising with image-level labels to pixel-level labels. Since the spurious feature is less informative of the pixel-level labels than the classification label, we expect segmentation to be more robust to the presence or absence of spuriously correlated background features.

3 Methods

To use segmentation for classification tasks, we need to determine how to use segmentation networks to obtain classification labels. We'll do this by defining a summarizing function , g ( · ) , which takes as input the image and segmentation network and outputs a class label g ( X , f ( · )) = ˆ y . The segmentation-for-classification workflow is shown in Figure 3. In this section, we describe the summarizing functions that we evaluate. We note these methods can be used for both binary class problems and multiclass problems, as long as the classes of interest follow the task specification set above. We provide multiclass results in Appendix A5.6.

3.1 Rule-based summarizing functions

We first consider a simple, rule-based summarizing function. This method is intuitive: if there is a positive region in the predicted segmentation mask, we infer a positive classification label.

Synthetic

CANDID

ISIC

SPINE

Specifically, at inference, we binarize the probabilistic mask output from the segmentation network using threshold t . If the binarized mask contains over threshold τ total positive pixels, we return a positive classification label. To compute the class probability, we average the pixel-wise probabilities of all pixels in the class. Algorithms for this method are provided in Appendix A2.1.

3.2 Trained summarizing functions

Next we consider summarizing functions trained to transform the segmentation information into classification labels. The intuition behind these methods is that trained functions may learn more nuanced differences between true positive and true negative predicted segmentation masks. We train these summarizing functions using the same splits used to train the segmentation network, and freeze the segmentation networks before training the summarizing functions.

We first consider trained summarizing functions that operate on the predicted segmentation masks. We consider three architectures for the summarizing function, each with increasing complexity: a fully connected layer, a global average pooling layer followed by a fully connected layer, and existing image classification architectures such as SqueezeNet [27] and ResNet50 [28].

We also consider summarizing functions that operate on segmentation network embeddings, similar to traditional transfer learning. We consider both shallow and deep embeddings and two different summarizing function architectures: a simple architecture consisting of pooling and fully connected layers, and a more complex classification head proposed in past work [20].

We fully describe these summarizing functions and training processes in Appendix A2.2.

4 Experiments

In this section, we first confirm the takeaways from our analysis in Section 2 on a synthetic dataset (Section 4.1). Then, we evaluate the summarizing functions we described in Section 3 on medical imaging datasets (Section 4.2). We briefly describe the datasets and training below; we provide full details of the synthetic dataset and training in Appendix A4 and the medical datasets and training in Appendix A5.

Datasets. The synthetic dataset consists of many colored shapes on a gray background; the task is to determine if a navy blue circle is present. We also evaluate three medical imaging datasets: CANDID, in which we aim to classify pneumothorax in chest x-rays (n=19,237) [29]; ISIC, in which we aim to classify melanoma from skin lesion photographs (n=2,750) [30]; and SPINE, in which we aim to classify cervical spine fractures in CT scans (n=2,018, RSNA 2022 Cervical Spine Fracture Detection Challenge). These datasets are visualized in Figure 4. The SPINE dataset is an example of where the growing number of off-the-shelf networks enables segmentation-for-classification: we use a pretrained CT segmentation network to generate segmentation targets for this task, as detailed in Appendix A5.

Training. To establish baseline classification performance, we train a standard supervised classification network. To establish baseline segmentation+classification performance, we train a multitask learning network where the classification and segmentation networks are trained concurrently and

Balanced accuracy

1.0

Balanced accuracy

1.0

Balanced accuracy

1.0

Balanced accuracy

1.0

0.9

0.9

0.9

0.9

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0

15,000 30,000 45,000

0.010.020.030.040.05

0.90 0.92 0.94 0.96 0.98 1.00

0.00

0.040.080.12

(a)

No.training images

(b)

Class prevalence (low) (c)

Class prevalence (high)(d)

Smalltargetprevalence

during training

Balanced accuracy

1.0

Balanced accuracy

1.0

Balanced accuracy

1.0

0.9

0.9

0.9

0.8

0.8

0.8

Seg-for-class

0.7

0.7

0.7

Classification

0.6

0.6

0.6

0.5

0.5

0.5

10

20

30

40

0.800.850.900.951.00

0.0 0.2 0.4 0.6 0.8 1.0

(e)

Target radius

(f)

Spurious correlation (g)

Prevalenceof targetsin

strength

image centerduring

training

share the same network backbone using the Y-Net structure [31]. To obtain segmentation-forclassification performance, we implement each summarizing function as described in Section 3. We train classification, multitask, and segmentation networks by minimizing the cross entropy loss between the predicted and ground truth labels using the Adam optimizer. We provide additional training details and information on hyperparameter tuning in Appendix A4.2 and A5.2, and we provide ablation experiments controlling for segmentation vs. classification model capacity in Appendix A5.3.

4.1 Experiments on synthetic data

We first perform experiments on synthetic data to confirm the takeaways from our analysis. By using a synthetic dataset, we have precise control over dataset and task characteristics, allowing us to isolate and adjust these variables to assess model performance. We provide highlights of our synthetic experiments below; a full description of the experimental procedure on synthetic data can be found in Appendix A4. In these synthetic experiments, we use the rule-based summarizing function.

The first takeaway from Section 2 is that segmentation-for-classification should require fewer images to achieve high performance, particularly for tasks with small targets. We examine this with our synthetic dataset in the following experiments.

We sweep the number of training images from 1,000 to 100,000 and plot classification and segmentation-for-classification balanced accuracy in Figure 5a, where we see segmentationfor-classification has improved performance in the limited data regime.

Similar to learning with small datasets, we also expect segmentation to perform better with high or low class prevalence, which we see in Figure 5b and Figure 5c.

Segmentation's ability to learn with fewer examples should also extend to rare subtypes-we expect higher performance on class subtypes that appear infrequently during training. To evaluate this, we train on small, medium, and large navy blue circles and evaluate network performance on the small circles as they are made increasingly rare during training (Figure 5d). We see segmentation-for-classification achieve higher performance on this data slice with few training examples.

We vary the object size in the train and test dataset and find segmentation-for-classification provides more boost with smaller targets (Figure 5e).

The second takeaway from our analysis is that segmentation-for-classification should be more robust to background features that spuriously correlate with the class of interest. We evaluate this with two synthetic experiments.

We first evaluate an obvious case of spurious correlation: when a background object correlates with the class label. We make a large, pink square increasingly correlated with

CANDID

ISIC

0.72

0.95

0.70

0.90

0.68

0.85

0.65

0.60

0.70

0.57

0.65

0.55

0.60

0.53

Limited training data

All training data

Limited training data

All training data

SPINE

0.72

0.70

Classification

Multitask

0.68

Seg output →threshold

0.66

Seg output →FC layer

0.64

Seg output → pool + FC layers

Seg output→ classification network

0.62

Deep seg embeddings → simple class head

0.60

Deep seg embeddings → complex class head

Shallow seg embeddings → simple class head

0.58

....... Shallow seg embeddings → complex class head

0.56

Limited training data

All training data

the target (a small navy blue circle) in our synthetic dataset-then remove this correlation in the test dataset-and show that segmentation is robust to this correlation (Figure 5f).

Spurious background correlations can also include less obvious features, like where the target object occurs; in other words, we expect segmentation to be more robust to target location. We explore this by changing where the target navy blue circle appears during training, but allowing the navy blue circle to appear anywhere in the test dataset. As the target is increasingly restricted to a particular location during training, classification is less able to generalize to finding the target in other regions of the image (Figure 5g).

4.2 Experiments on medical imaging data

Given the promising results on synthetic data, we next evaluate performance on the medical datasets and evaluate segmentation-for-classification training methods. In Section 4.2.1 we compare the performance of different summarizing functions, classification networks, and multitask networks. In Section 4.2.2 we evaluate semi-supervised segmentation-for-classification.

4.2.1 Comparing summarizing functions, classification, and multitask training

We evaluate each summarizing function and model on the three medical datasets in two regimes: limited labeled data, which consists of 10% of randomly sampled training data, and all labeled training data. We report mean AUROC on the test set in Figure 6. We observe that the simple, rulebased segmentation-for-classification method achieves higher mean AUROC than both the traditional classification and multitask networks for all tasks and amounts of training data. Specifically, we observe a 10.0%, 6.9%, and 2.8% improvement in mean test AUROC in the limited training data regime for CANDID, ISIC, and SPINE respectively over traditional classification, and a 2.6%, 0.6%, and 6.1% performance improvement in the all training data setting.

Comparing summarizing functions, we find that summarizing functions which operate over segmentation network outputs generally outperform those that operate over embeddings. Among summarizing functions that operate over segmentation outputs, three perform similarly well: thresholding the segmentation output, learning a fully connected layer, and training a full classification network on top of the segmentation output. The threshold-based summarizing function has additional advantages: it does not require training, thus saving training effort and avoiding potential overfitting to training data; it is fast to compute at inference; and the results are intuitive-when there is a segmentation mask, there is an associated positive class label. For these reasons, we recommend simple rule-based summarizing functions for converting a segmentation mask into a classification label. We compare the error modes of these methods in Section 5.

0.96

0.94

AUROC

0.92

Seg-for-class

Classification,alllabels(reference)

Seg-for-class,semi-supervision

0.90

Seg-for-class,boosted semi-supervision

0.88

0.86

5%

10%

35%

100%

Percentoftrainingdatawithsegmentationlabels

4.2.2 Semi-supervised segmentation-for-classification

As discussed in the Introduction, a historical obstacle to widespread application of segmentation is the difficulty of curating labeled training datasets. However, recent work has led to more public datasets, pretrained networks, and methods to reduce the labeling burden of training segmentation models. We posit that this progress in segmentation will also benefit segmentation-for-classification. In the previous section, we showed that an off-the-shelf spine segmentation model could be used to generate segmentation labels for training the SPINE segmentation-for-classification models. In this section, we verify that a semi-supervised segmentation method can improve performance of segmentation-for-classification models. We consider two common settings:

Setting 1, additional unlabeled data. Because unlabeled data is often abundant, practitioners may have a subset of their dataset labeled with segmentation masks and a larger set of unlabeled images.

Setting 2, additional classification data. Other practitioners may have a subset of their dataset labeled with segmentation masks and a larger set of images with classification labels.

For Setting 1, we apply a previously-published semi-supervised segmentation method which takes a small set of labeled data and a large set of unlabeled data as input and trains a segmentation model using a combination of data augmentation, consistency regularization, and pseudo labeling [32].

For Setting 2, we update the previously-proposed methods by using the classification labels to improve the pseudo segmentation labels, which we call 'boosted' semi-supervision. The original semi-supervised approach uses pseudo labeling, in which a small amount of labeled segmentation data is used to train an initial segmentation network; the initial network predicts pseudo segmentation masks for all unlabeled data; and a final network is trained on the pseudo segmentation masks. In the 'boosted' algorithm, we use the available classification labels to modify the pseudo segmentation masks. Specifically, for images that have negative classification labels, we zero out the pseudo segmentation masks: because we know the classification label is negative, we know that there should not be any positive region in the segmentation mask. For images that have a positive classification label, we check to see if the pseudo segmentation mask has a positive segmentation region; if it does not have a positive segmentation region, we know that the pseudo segmentation mask is wrong and we exclude the sample from training of the final segmentation network.

These semi-supervised approaches learn from the unlabeled data to train a better segmentation model, which we hypothesize will also improve segmentation-for-classification performance. In these experiments, we use the threshold-based summarizing function and train models using a balanced training dataset. We report model performance for varying amounts of labeled training data in Figure 7. We report additional performance metrics and the number of labels for each method at the most limited and abundant data settings in Table A8.

From these results, we observe that all of the segmentation-for-classification models trained with 10% of the training data labeled exceed the performance of the classification model trained with 100% of the training data labeled. These results confirm one of the takeaways from our analysis, which is that segmentation-for-classification can achieve higher performance with fewer images. Additionally, we see that the semi-supervised training further improves segmentation-for-classification performance,

confirming that semi-supervised segmentation methods can directly be used for segmentation-forclassification. Finally, we see that the boosted semi-supervised approach indeed improves performance beyond standard semi-supervision in the limited data regime.

5 Benefits and drawbacks of segmentation-for-classification

In this section, we tie together our analysis, synthetics, and experiments on real data to summarize the benefits and drawbacks of using segmentation-for-classification.

Segmentation can improve aggregate performance, particularly with small datasets. Using segmentation-for-classification can lead to higher-performing models compared to traditional classification. Our analysis in Section 2 shows this benefit is due in part to greater divergence between segmentation's positive and negative classes and higher quantity of annotation (we evaluate the contributions of each of these factors with an additional experiment in Appendix A4.3). We observe this takeaway empirically in Figure 6, showing segmentation-for-classification improves mean performance on all datasets and training settings. Our analysis suggests greater performance differences occur in the limited data regime, which can include:

Small datasets. We expect greater boosts from segmentation-for-classification with limited dataset sizes. We observe this with synthetics in Figure 5a and on two of the three medical datasets in Figure 6. Small datasets often occur when developing models on custom data or when training models for rare diseases.

Lowclass prevalence. Segmentation can also achieve higher performance than classification when the class balance is very low or high (shown with synthetic datasets in Figure 5b,c), since segmentation is able to better learn from the few examples of the low-prevalence class.

Rare subtypes. We expect segmentation-for-classification models to have improved ability to identify rare subtypes, which appear infrequently in the training dataset. We observe this on the synthetics (Figure 5d) and see this property revealed in the medical datasets as well: on small lesions that are positive for melanoma, which are less commonly seen during training, traditional classification achieves a recall of 0.18 and segmentation-forclassification achieves a recall of 0.47.

Finally, the analysis in Section 2 suggests segmentation provides greater benefit for tasks with small targets. We observe this with synthetics in Figure 5e. On the real data experiments, models trained with the full datasets reflect this property as well: the SPINE dataset has the lowest average target-to-background size ratio (with the target taking up an average 0.6% of the image) and achieves the greatest performance boost with segmentation-for-classification, while the ISIC dataset has the highest target-to-background size ratio (with the target taking up an average 26.3% of the image) and sees the lowest performance boost.

Segmentation reduces susceptibility to spurious correlations. A worrying failure mode of classification networks is that instead of learning to identify the pathology of interest, the model instead looks for easier-to-identify features that spuriously correlate with the true target-such as the examples given in Section 2. Segmentation-for-classification should be more robust to background features spuriously correlated with the target task.

Empirically, we observe this in our synthetic data (Figure 5f) and evaluate a spurious correlation on the medical datasets here. Specifically, in the CANDID dataset, we evaluate performance under a known spurious correlation condition: most pneumothoraces in chest x-ray datasets co-occur with a chest tube, which are used to treat pneumothorax [25]. Thus, it is important that the pneumothorax classification algorithms are able to find pneumothoraces without chest tubes, as those are the patients who have not yet been identified and treated. To evaluate each model's robustness to this spurious correlation, we evaluate model performance on patients who do not exhibit the spurious correlation (i.e., patients with pneumothorax but no chest tube or patients without pneumothorax but do have a chest tube) and observe that segmentation-for-classification achieves an AUROC of 0.84 while the classifier's AUROC drops to 0.58.

We also expect segmentation-for-classification to be more robust to the target location, which is another type of spurious correlation. We observe this in the synthetics (Figure 5g) and on the medical data: recall of the CANDID classifier drops up to 22.5% conditioned on the lung region in which the pneumothorax occurred (dropping from 0.92 recall in the right lower hemithorax to 0.71 recall on the

left upper hemithorax) while segmentation-for-classification recall drops at most 7.5% (from 0.94 in the left lower hemithorax to 0.87 in the right upper hemithorax).

Segmentation facilitates human assessment. Segmentation-for-classification inherently delivers location information. While there are methods to probe classification models for which areas of an image contribute to the model's prediction (e.g., saliency maps [33], class activation maps [34, 35]) these methods are often unreliable; shortcomings are further discussed in prior work [36, 37]. By providing location information as well as classification information, segmentation-for-classification models enable the user to more effectively adjudicate model findings. This location information is more pertinent for some applications where the abnormality may be hard to find (e.g., pneumothorax, fracture) versus use cases where the visual features used by the model are obvious (e.g., skin lesion).

Segmentation enables more precise model evaluation. We can perform more detailed model evaluation and error analysis using segmentation-for-classification's location and size information. For example, we can evaluate models with a stricter definition of recall: instead of identifying a true positive as when the image-level label is positive and correct, we can also require that the correct region of the image is identified. When we compute the stricter recall metric for models trained with all labeled data, we see reductions in recall: CANDID drops from 0.87 to 0.85 and SPINE drops from 0.74 to 0.55. We can't similarly assess the classification algorithms with this stricter definition of recall because of the lack of reliable methods to acquire accurate location information from classifiers. However, precise model evaluation is important for low-failure-tolerance applications such as medicine, and segmentation-for-classification provides an obvious means of doing so. Finally, we can perform additional error analysis using segmentation-for-classification's size and location information, such as assessing the average abnormality size and region.

Segmentation has a higher per-image labeling cost. The burden of generating segmentation training labels is a drawback of training segmentation networks and can change for different tasks (e.g., 3D images have a higher labeling burden). In comparison, classification is cheap to generate labels forparticularly for cases where classification labels can be pulled from electronic health records [38]. In this work, we've explored two ways of reducing segmentation's labeling burden while improving classification performance: using an off-the-shelf segmentation model and using semi-supervised training methods. As new tools emerge and make it easier to generate segmentation training labels [13, 14, 15], we expect the labeling burden gap between classification and segmentation to lessen. Still, the increased annotation cost of segmentation given available tools should be weighed against the expected benefits (detailed above) of using segmentation-for-classification on a given dataset.

6 Conclusion

Historically, segmentation networks have been employed only when required due to their high annotation cost. Given the increasing ease of training segmentation networks, we aim to better flesh out the decision space between training a segmentation vs. classification network. In Section 2, we develop intuition and formally analyze why segmentation and classification networks can perform differently. In Section 3 and 4, we explore methods to implement segmentation-for-classification and empirically evaluate segmentation-for-classification over many datasets and training conditions. Finally, in Section 5, we discuss the benefits and drawbacks of segmentation-for-classification. We show that segmentation-for-classification can lead to more performant, robust, and interpretable models. In clinical settings, one may prefer more these upsides despite the added annotation cost.

Broadly, these results underline the potential of deep learning-based segmentation as a general tool for automating image analysis. Segmentation data provides an information-rich environment for deep neural networks to learn from, which we've shown results in separable, robust embedding spaces. In turn, segmentation network outputs capture detailed information about images, which can support many downstream analysis tasks.

In future work, we are interested in how to leverage segmentation for targets that don't fit the task specification we study here, perhaps utilizing anatomical or coarser forms of segmentation to still reap some of the benefits of training segmentation models. Additionally, we are interested in reader studies focused on quantifying the value of the location information contained in segmentation masks for radiologists working with AI algorithms. Finally, we are interested in how to leverage segmentation to benefit other machine learning applications in radiology, extending our study of how task framing impacts a model's properties.

References

Phillip M Cheng, Emmanuel Montagnon, Rikiya Yamashita, Ian Pan, Alexandre CadrinChênevert, Francisco Perdigón Romero, Gabriel Chartrand, Samuel Kadoury, and An Tang. Deep learning: an update for radiologists. Radiographics , 41(5):1427-1445, 2021.

Simeng Zhu, Marissa Gilbert, Indrin Chetty, and Farzan Siddiqui. The 2021 landscape of fda-approved artificial intelligence/machine learning-enabled medical devices: an analysis of the characteristics and intended use. International journal of medical informatics , 165:104828, 2022.

S Niyas, SJ Pawan, M Anand Kumar, and Jeny Rajan. Medical image segmentation with 3d convolutional neural networks: A survey. Neurocomputing , 493:397-413, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414 , 2022.

Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems , 29, 2016.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Jakob Wasserthal, Manfred Meyer, Hanns-Christian Breit, Joshy Cyriac, Shan Yang, and Martin Segeroth. Totalsegmentator: robust segmentation of 104 anatomical structures in ct images. arXiv preprint arXiv:2208.05868 , 2022.

Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, et al. Vindr-cxr: An open dataset of chest x-rays with radiologist's annotations. arXiv preprint arXiv:2012.15029 , 2020.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.06131 , 2023.

Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 , 2023.

Gautam Rajendrakumar Gare, Andrew Schoenling, Vipin Philip, Hai V Tran, P deBoisblanc Bennett, Ricardo Luis Rodriguez, and John Michael Galeotti. Dense pixel-labeling for reversetransfer and diagnostic learning on lung ultrasound for covid-19 and pneumonia detection. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI) , pages 1406-1410. IEEE, 2021.

Gautam Rajendrakumar Gare, Jiayuan Li, Rohan Joshi, Rishikesh Magar, Mrunal Prashant Vaze, Michael Yousefpour, Ricardo Luis Rodriguez, and John Michael Galeotti. W-net: Dense and diagnostic semantic segmentation of subcutaneous and breast tissue in ultrasound images by incorporating ultrasound rf waveform data. Medical Image Analysis , 76:102326, 2022.

Michal Heker and Hayit Greenspan. Joint liver lesion segmentation and classification via transfer learning. arXiv preprint arXiv:2004.12352 , 2020.

Nooshin Mojab, Philip S Yu, Joelle A Hallak, and Darvin Yi. Cvs: Classification via segmentation for small datasets. arXiv preprint arXiv:2111.00042 , 2021.

Ken CL Wong, Tanveer Syeda-Mahmood, and Mehdi Moradi. Building medical image classifiers with very limited data using segmentation networks. Medical image analysis , 49:105-116, 2018.

Khaled Saab, Sarah Hooper, Mayee Chen, Michael Zhang, Daniel Rubin, and Christopher Ré. Reducing reliance on spurious features in medical image classification with spatial specificity. 2022.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, 1999.

John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine , 15(11):e1002683, 2018.

Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning , pages 151-159, 2020.

Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence , 3(7):610-619, 2021.

Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360 , 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Sijing Feng, Damian Azzollini, Ji Soo Kim, Cheng-Kai Jin, Simon P Gordon, Jason Yeoh, Eve Kim, Mina Han, Andrew Lee, Aakash Patel, et al. Curation of the candid-ptx dataset with free-text reports. Radiology: Artificial Intelligence , 3(6), 2021.

Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018) , pages 168-172. IEEE, 2018.

Sachin Mehta, Ezgi Mercan, Jamen Bartlett, Donald Weaver, Joann G Elmore, and Linda Shapiro. Y-net: joint segmentation and classification for diagnosis of breast biopsy images. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 893-901. Springer, 2018.

Sarah M. Hooper, Sen Wu, Rhodri H. Davies, Anish Bhuva, Erik B. Schelbert, James C. Moon, Peter Kellman, Hui Xue, Curtis Langlotz, and Christopher Ré. Evaluating semi-supervision methods for medical image segmentation: applications in cardiac magnetic resonance imaging. Journal of Medical Imaging , 10:024007, 2023.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, and Pranav Rajpurkar. Benchmarking saliency methods for chest x-ray interpretation. Nat Mach Intell , 2022.

Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al. Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiology: Artificial Intelligence , 3(6):e200267, 2021.

Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L Rubin, et al. Cross-modal data programming enables rapid medical machine learning. Patterns , 1(2):100019, 2020.

Kim-Han Thung and Chong-Yaw Wee. A brief review on multi-task learning. Multimedia Tools and Applications , 77(22):29705-29725, 2018.

Puyang Wang, Vishal M Patel, and Ilker Hacihaliloglu. Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided cnn. In International conference on medical image computing and computer-assisted intervention , pages 134-142. Springer, 2018.

Xiaofei Wang, Lai Jiang, Liu Li, Mai Xu, Xin Deng, Lisong Dai, Xiangyang Xu, Tianyi Li, Yichen Guo, Zulin Wang, et al. Joint learning of 3d lesion segmentation and classification for explainable covid-19 diagnosis. IEEE transactions on medical imaging , 40(9):2463-2476, 2021.

Ozan Ciga and Anne L Martel. Learning to segment images with classification labels. Medical Image Analysis , 68:101912, 2021.

Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell , 172(5):1122-1131, 2018.

Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Attention gated networks: Learning to leverage salient regions in medical images. Medical image analysis , 53:197-207, 2019.

Yanming Guo, Yu Liu, Theodoros Georgiou, and Michael S Lew. A review of semantic segmentation using deep neural networks. International journal of multimedia information retrieval , 7(2):87-93, 2018.

Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review , 54(1):137-178, 2021.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3942-3951, 2021.

Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747 , 2020.

Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems , 33:12546-12558, 2020.

Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv preprint arXiv:1904.00625 , 2019.

Yang Wen, Leiting Chen, Yu Deng, and Chuan Zhou. Rethinking pre-training on medical imaging. Journal of Visual Communication and Image Representation , 78:103145, 2021.

Mohammad Amin Morid, Alireza Borjali, and Guilherme Del Fiol. A scoping review of transfer learning research on medical image analysis using imagenet. Computers in biology and medicine , 128:104115, 2021.

Zhuo Chen, Ruizhou Ding, Ting-Wu Chin, and Diana Marculescu. Understanding the impact of label granularity on cnn-based image classification. In 2018 IEEE international conference on data mining workshops (ICDMW) , pages 895-904. IEEE, 2018.

Tanveer Syeda-Mahmood, Ken CL Wong, Joy T Wu, Ashutosh Jadhav, and Orest Boyko. Extracting and learning fine-grained labels from chest radiographs. In AMIA Annual Symposium Proceedings , volume 2020, page 1190. American Medical Informatics Association, 2020.

Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3478-3488, 2021.

Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems , 33:19339-19352, 2020.

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning , pages 6781-6792. PMLR, 2021.

Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods , 18(2):203-211, 2021.

William Falcon et al. Pytorch lightning. https://github.com/PyTorchLightning/pytorch-lightning , 3, 2019.

GitHub. Note:

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Gautam Machiraju, Sylvia Plevritis, and Parag Mallick. A dataset generation framework for evaluating megapixel image classifiers and their explanations. In European Conference on Computer Vision , pages 422-442. Springer, 2022.

Appendices

A1 Related work

Segmentation to benefit classification. Our study is inspired by past works that employ segmentation networks in classification problems and report various benefits of doing so. Improved overall accuracy. For example, [16] uses a segmentation network to generate segmentation masks of different tissue types in lung ultrasound images, then trains a sequence of global average pooling and fully connected layers to convert the tissue masks into classification labels. The authors report higher aggregate performance compared to training a classification model from scratch for their lung ultrasound classification task. In [17], the authors expand this approach by including diagnostic tissue segmentation targets (i.e., segmenting benign vs. malignant tissue as different segmentation classes), then converting the segmentation masks into class labels via a likelihood score over the segmented tissue types. Similarly, [18] jointly segments and classifies liver lesions in CT scans by segmenting the lesion as one of multiple possible lesion types and finds that the segmentation approach results in higher classification performance than traditional classification or multitask learning. Training with fewer labels. Other reports focus on segmentation networks' ability to learn from limited labeled data. For instance, [19] converts segmentation outputs into classification labels via a likelihood score and compares the aggregate performance of the segmentation network against classification and multitask learning networks when trained with varying dataset sizes. The authors find that the segmentation network achieves higher classification scores in the limited data regime. Similarly, [20] uses features from a UNet pretrained on a segmentation task to fine tune a classifier and again finds higher classification performance using the pretrained segmentation weights in the limited data regime. Improved robustness. Finally, [21] shows that training a segmentation network instead of a classification network results in predictors more robust to spurious correlations.

Multitask learning. Other works have combined segmentation and classification via multitask learning, in which the classification and segmentation networks share the same backbone but different task heads trained concurrently [39, 40, 41, 31, 42]. The multitask approach differs from the approaches discussed above, which each train a segmentation network, freeze it, then use the segmentation network's embeddings or outputs to produce a classification label. In multitask frameworks, the classification task can impact the weights learned for the segmentation task via the shared backbone, and training schedules and task weighting must be managed.

Classification to benefit segmentation. Other studies focus on the reverse of what we are interested in by first training a classification network then using that classifier to generate segmentation information. These approaches typically aim either to generate segmentation labels for 'free' by extracting location information from the classification models or to use this location information to improve a classification network's interpretability. Examples of this class of related work include using saliency maps [33], class activation maps [34, 35], occlusion maps [43], and classifiers with built-in attention mechanisms [44].

Semantic segmentation. A great deal of work has focused on improving semantic segmentation methods [45, 46], which underlie the segmentation-for-classification methods we explore here. Segmentation-for-classification frameworks contain an additional step converting a semantic segmentation network or output into an image-level classification label, particularly for tasks which are not typically framed as segmentation tasks. Additionally, many recent works have advanced our ability to train semantic segmentation networks with limited labeled data, including using selfsupervised networks, semi-supervision, and foundation models [4, 5, 6, 7, 47, 8], many of which have been adapted specifically to medical imaging [48, 49, 50]. These methods make training segmentation-for-classification models less labor-intensive.

Transfer learning. The segmentation-for-classification framing can be viewed as a type of transfer learning. While many transfer learning studies have focused on settings where the task type stays consistent (e.g., transferring one classification network to a different classification task), some studies have looked at transferring classification weights to segmentation tasks or vice versa [51, 52, 53]. Some segmentation-for-classification methods can be considered transfer learning, where

the segmentation network is trained first and much of the segmentation network is frozen before transferring the learned weights to a classification task.

Label granularity. A related line of work explores the right label granularity for training classification models (e.g., training 'herbivore vs. carnivore' compared to training 'horse vs. deer vs. cat vs. dog vs...') [54, 55]. Similar to our findings, these works find that more granular labels can lead to improved network optimization and generalization [54].

Modified classification networks. Finally, many works have modified straightforward supervised classification training schemes. For example, self- and semi-supervised learning aim to improve performance with limited labeled data [56], and past work has proposed various modifications to classification training to improve robustness [57, 58].

In this manuscript, we use straightforward classification as we are interested in exploring fundamental differences between classification and segmentation. We observed that many performance benefits can be achieved simply by using a segmentation model, improving performance along many axes without specialized tooling. Conversely, classification modifications are typically different for different objectives (e.g., methods for reducing reliance on spurious correlations are different from methods that handle class imbalance are different from methods that improve interpretability). We believe the fact that segmentation results in many benefits without specialized tooling for each is a major strength of the proposed approach.

Our contributions. We first analyze why we see performance differences, supporting and helping to explain the empirical results that we and others have observed as well as the conditions under which we can expect to see benefits. Next, we implement both previously proposed and new methods for using segmentation-for-classification and compare these approaches across multiple datasets using varying amounts of training data. We perform this head-to-head comparison to build up a recommendation of best practices for using segmentation networks in classification problems, including: recommending a summarizing function; showing an off-the-shelf segmentor can achieve improved classification performance; showing an existing semi-supervised training method can improve performance; and adapting that semi-supervised method specifically to the segmentation-forclassification setting. Finally, we aggregate and expand on the benefits of switching from classification to segmentation. We expand on benefits by showing improved performance with low class prevalence, rare subtypes, and location spurious correlations, and we discuss these quantitative benefits in conjunction with qualitative benefits like improved interpretability and model evaluation. We use this consolidated evidence to make the broader case that many classification problems would benefit from a segmentation framing.

A2 Summarizing functions

Below we provide additional details on the summarizing functions we evaluate in the main text.

A2.1 Rule-based summarizing functions

In Section 3.1, we introduce a rule-based summarizing function using thresholds to transform the segmentation output into a classification label. Here we give further details of that summarizing function. Specifically, in Algorithm 1, we give the algorithm we use to compute a binary, image-level label from a probabilistic segmentation mask. Given a probabilistic segmentation mask-which

contains the segmentation network output, the pixelwise probabilities indicating where the class of interest was found-we first binarize the segmentation mask using threshold t . At default, t = 0 . 5 . We then sum all pixel values in the binary segmentation mask; if there are more than threshold τ positive pixels in the segmentation mask, we return a positive binary image-level label. In this work, we set τ = 100 .

In Algorithm 2 we provide the method we use to compute a probabilistic image-level label from a probabilistic segmentation mask. We use these probabilistic labels to compute the AUROC. Given a probabilistic segmentation mask, we first compute the binary label using Algorithm 1. If the binary label is positive, we compute the probabilistic image-level label as the mean of pixel values that are greater than threshold t . Similarly, if the binary label is negative, the probabilistic image-level label is the mean of pixel values less than t . At default, t takes value 0.5.

A2.2 Trained summarizing functions

In Section 3.2, we introduce trained summarizing functions to transform the segmentation output into a classification label. We give further details of those summarizing functions here.

Summarizing functions that operate over segmentation outputs. After first training the segmentation network, we generate predicted segmentation masks ˆ S for all training and validation images. Note that we do not binarize these masks, but retain the pixel-wise probabilities. Then, using the same dataset splits used for training the segmentation network, we train the summarizing function g ( · ) to take the predicted segmentation mask ˆ S as input and produce a class label vector g ( ˆ S ) = ˆ y by minimizing the cross entropy loss CE (ˆ y, y ) . At inference, the segmentation network first generates a predicted segmentation mask for the test image, then the predicted mask is fed into the summarizing function which produces class probabilities.

We consider three architectures for trained summarizing functions that operate over segmentation network outputs. The first and simplest architecture is a fully connected layer on the flattened predicted segmentation mask ˆ S , which is used to produce class scores that are normalized with a softmax function.

The next summarizing function we consider is a global average pooling layer followed by a fully connected layer. We take the output segmentation mask ˆ S and produce a vector with values representing the average of each class's segmentation mask. The fully connected layer then transforms these class averages into scores for each class, which are normalized with a softmax function. This method is similar in spirit to thresholding as described above but with a threshold learned from the values observed in the train set. This global average pooling to fully connected approach was previously proposed in [16].

Finally, we consider known image classification architectures as summarizing functions. At a high level, the task of transforming a segmentation mask into a classification label is exactly an image classification task: does the 'image' represented by the predicted segmentation mask correspond to a positive or negative class label? So, we investigate if existing image classification architectures are suited for learning mappings from predicted segmentation masks to class labels. In this work, we consider SqueezeNet [27] and ResNet50 [28] as summarizing functions on top of the segmentation output.

Summarizing functions on pretrained segmentation embeddings. Wenext consider summarizing functions that operate over segmentation network embeddings. We first train the segmentation network f ( · ) , then store embeddings for all training and validation images. We train the summarizing function g ( · ) to take the segmentation embeddings as input and produce a class label vector ˆ y as output by minimizing the cross entropy loss CE (ˆ y, y ) . At inference, the image is passed through the pretrained segmentation network and embeddings from the segmentation network are stored. These embeddings are then processed with the summarizing function to produce class probabilities.

We consider two types of embeddings, shallow and deep. The intuition for exploring different embedding depths is that different depths contain different information about the input image, which may vary in their usefulness for the classification task. We give the name of the layer in the Pytorch model that we pull embeddings from in Table A1. We also consider two different architectures to classify the segmentation network's embeddings. First, we define a simple network architecture consisting of a global average pooling layer followed by three fully connected layers, which produces class scores that we normalize with a softmax function. Second, we consider a more complex classification head proposed previously in [20]. The classification head takes the segmentation embeddings as input, then processes the embeddings with three blocks of convolutionbatch normalization-ReLU-maximum pooling operations. The convolutional kernels are size 3x3 and each produce 16 channels; the maximum pooling operation operates over a 2x2 window. The resulting embedding is flattened. Then, we process the flattened embedding with two blocks of linear transform-batch normalization-ReLU-dropout. The linear layer produces 100 features and the dropout layer has a probability of dropout set to 0.25. Finally, the resulting feature map is processed with a single linear layer to produce the final output, a vector equal in length to the number of classes, which can be normalized with the softmax function.

A3 Supporting analysis

A3.1 Proof of Proposition 1

Proof. We can write Pr( X k | y = 1) as

where λ = Pr( S k =1) Pr( ∃ j s.t. S j =1) . Therefore, we have written the probability of a pixel value given the image-level label, Pr( X k | y = 1) , as a mixture of the probability of the pixel value given the pixel-level label, Pr( X k | S k = 1) and Pr( X k | S k = 0) . Next, we can write Pr( X k | y = 0) as Pr( X k | S 1 , . . . , S L = 0) = Pr( X k | S k = 0) by our data generating process.

Putting together Pr( X k | y = 1) and Pr( X k | y = 0) , the right hand side of the inequality in Proposition 1 is

where we have obtained the desired inequality due to convexity of the KL divergence. Note that by keeping λ in the inequality, we can make the stronger claim

A4 Synthetic dataset and experiments

We conduct experiments on a synthetic dataset to evaluate how dataset and task characteristics impact the performance of classification and segmentation-for-classification models. Here we provide additional details on our synthetic dataset and experiments. The synthetic dataset is visualized in Figure 4.

A4.1 Synthetic dataset

The synthetic dataset is made up of images containing a gray background and many objects of varying shapes and colors. The task is to classify if a given image has a certain colored shape present. Each synthetic dataset is defined by the set of parameters given in Table A2.

To construct a synthetic image, we initialize a gray background of size (224, 224, 3). We then draw the class label y ∼ Bernoulli ( p pos ) . We sample the spurious label s according to the strength of the spurious correlation p spur . Specifically, we define and sample the random variable match ∼ Bernoulli ( p spur ) . If match = 1 , we set the spurious label s = y ; else, we set s = 1 -y . If s = 1 , we create the spurious object specified by ( c spurious , r spurious , s spurious ) -the object's color, radius, and shape-and place the spurious object at the coordinates drawn from the distributions ( l spurious x , l spurious y ). Wefollow similar processes to generate and place the n obj -y -s background objects and target object. We place the target object last to ensure it is placed in the foreground.

To run experiments on the synthetic dataset, we vary one dataset variable, train the classification and segmentation-for-classification models, and evaluate performance. The variable we adjust for each experiment is described in Table A3. Additionally, we visualize the training dataset for each experiment in Figure A1.

A4.2 Training details for synthetic datasets

We train a ResNet50 to classify the synthetic data and a ResNet50 with a fully convolutional segmentation head to segment the synthetic data. We train the classifier and segmentation networks by minimizing the cross entropy loss between the predicted and ground truth labels. We do not use data augmentation on the synthetic data. We train each network with an Adam optimizer and a learning rate of 1e-4, tuned from [1e-6, 1e-5, 1e-4, 1e-3]. We checkpoint the networks on the maximum balanced accuracy on the validation set. We use the threshold-based summarizing function.

Experiment

Experiment

Experiment

Experiment

Number of samples

Low class balance in

High class balance in

Strong size bias in the

in the training set

training set

training set

training set

Synthetic dataset

visualization

Positives contain a

navy blue circle

Experiment

Experiment

Experiment

Experiment

Spurious correlation

Strong locationbias

Increasing target size

Increasing background

in the training set

in the training set

variance

Classification

Seg. supervision,

Seg. supervision,

Inputimage

supervision

with subsampling

withoutsubsampling

0

0

0

0

0

0

0

0

1

0

1

0

0

0

0

0

0

Balancedaccuracy

1.0

0.9

0.8

Class

Seg-for-class, with subsampling

0.7

Seg-for-class, without subsampling

0.6

0.5

010002000300040005000

No.trainingimages

A4.3 Additional synthetic experiments

In the main text, we focus on the divergence between positive and negative classes as to why classification and segmentation perform differently. We do not explicitly model the benefit from denser annotation during training, instead taking it as a given that the denser annotation should help segmentation learn with fewer images. To show that annotation quantity is not the only component contributing to segmentation's improved performance, we run the following experiment.

We study a simplified setting visualized in Figure A2. Specifically, we divide a 112 × 112 grid into sixteen isotropic squares. The task is to identify if any of the squares are navy blue. Class balance is set to 0.5, meaning an image is as likely to have the target class as not. The remaining square colors are drawn randomly. In this simplified setting, we still expect divergence between segmentation classes to be greater than divergence between classification classes, following our analysis in Section 2.

Classification supervision is applied at the image-level, per usual. To understand the influence of annotation quantity, we evaluate segmentation-for-classification performance in two settings. First, we provide segmentation supervision with only one pixel per square. This level of supervision results in 16 annotated pixels per image, but an equal number of positive class annotations as the classification supervision. Second, we provide segmentation supervision over all pixels (this is the usual segmentation-for-classification setting), in effect providing 12,544 annotated pixels per image and 784 × as many positive class annotations. The supervision for each of these models is also visualized in Figure A2.

In effect, this experiment separates the impact of just divergence vs. divergence+higher quantity of (positive) annotations. We sweep the number of training images and report results in Figure A3.

We see that segmentation without subsampling outperforms segmentation with subsampling outperforms classification. This result confirms that both the denser annotation and greater divergence during training benefit segmentation-for-classification performance.

A greater number of annotated samples should improve most ML tasks, as has been extensively studied before. We focus on characterising the divergence as it is specific to segmentation-forclassification and less studied in prior work. Future work may study the relative benefits of these two aspects of supervision.

A5 Real datasets and experiments

We run experiments on three public medical imaging datasets, described below and fully detailed in their original studies. The medical datasets are visualized in Figure 4.

A5.1 Medical imaging datasets

CANDID: classifying pneumothorax in chest x-rays [29]

Classification task. Our aim is to classify whether a chest x-ray contains a pneumothorax; ground truth classification labels are provided with the original dataset.

Segmentation target. Our aim is to segment pneumothorax when it appears in a chest x-ray; ground truth segmentation masks are provided with the original dataset.

Dataset. This dataset contains 19,237 chest x-rays. The images are two-dimensional, one-channel images of size 1024 x 1024, and we normalized each image to the range [0, 1]. We split this dataset randomly into 60% training images, 20% validation images, and 20% test images. In this dataset, 16.6% of images are positive for pneumothorax.

ISIC: classifying melanoma from photographs of skin lesions [30]

Classification task. Our aim is to classify whether a skin lesion is melanoma or not; ground truth classification labels are provided with the original dataset.

Segmentation target. Our aim is to segment the skin lesion when it is melanoma; ground truth segmentation masks are provided with the original dataset.

Dataset. This dataset contains 2750 images of skin lesions. The images are two-dimensional, three-channel images that we resized to 224 x 224. We use the splits provided by the ISIC challenge organizers, resulting in 2000 training images, 150 validation images, and 600 test images. 18.9% of images are positive for melanoma.

SPINE: classifying cervical spine fractures in CT scans (RSNA 2022 Cervical Spine Fracture Detection Challenge)

Classification task. Our aim is to classify whether there is a fracture in the cervical spine; ground truth classification labels are provided with the original dataset.

Segmentation target. Our aim is to segment the vertabrae in which the fracture occurs. Unlike the previous two datasets, the ground truth segmentation masks were not available in this dataset for all training images. Instead, we used an off-the-shelf CT segmentation network [11] to generate segmentation masks of the vertebra. For each positive classification label, we create a corresponding segmentation mask which delineates the vertebrea in which the fracture occurs. This use case shows how off-the-shelf models can be used to convert some classification problems into segmentation problems without any additional labeling.

Dataset. This dataset contains 2018 CT scans of the spine. The images are three-dimensional volumes with one channel. We applied a CT bone window then normalized each scan to the range [0, 1]. We resized each axial slice to 128 x 128 voxels using bilinear interpolation and resampled the axial dimension to 128 slices around the spine, resulting in 128 x 128 x 128 image volumes. 47.6% of these scans contain a cervical spine fracture. We randomly split this dataset into 60% training, 20% validation, and 20% test.

A5.2 Training details for medical imaging datasets

In Table A4 we list each hyperparameter we evaluated for the backbone networks trained on the medical datasets. We chose hyperparameters that maximized the validation AUROC. Chosen hyperparameters are in bold.

For the trained summarizing functions, we only tuned the learning rate. We evaluated learning rates [1e-6, 1e-5, 1e-4, 1e-3, 1e-2] for each summarizing function and chose the learning rate that maximized the validation AUROC. In Table A5 we list the chosen learning rate for each summarizing function and dataset.

To evaluate performance, we report mean AUROC over the test split of each dataset. We binarize the probabilistic labels to report additional performance metrics by setting the operating point on the ROC curve that maximizes the Younden Index. We compute 95% confidence intervals using bootstrapping (1000 iterations). We conduct our experiments using Pytorch Lightning (Pytorch version 1.9.0, Lightning version 1.5.10) [60, 61].

A5.3 Ablations on model capacity

To report results in the main text, we tested multiple architectures for each task and dataset and selected the best-performing architecture for both segmentation and classification (Table A4). We did this to give each task and dataset the best performance, as it's not clear the best architecture for segmentation on a given dataset is the same that is best for classification.

However, segmentation networks are often higher capacity than classification networks due to their dense, pixel-wise outputs and corresponding decoders. To show that it is not simply a difference in model capacity that is leading to the observed performance differences between segmentation and classification, we perform the following ablations.

First, we use the same backbone for both segmentation and classification (Table A6). We use Resnet50 as the 2D backbone as it is a common backbone known to be useful for classification and segmentation. The Resnet50 classification head is a linear layer, while the Resnet50 segmentation head is the head described in the Fully Convolutional Network paper [62]. We use the UNet from the nnU-Net paper [59] for the 3D backbone. We see that segmentation shows the same or greater performance enhancement than what appeared in the original paper when using these standardized backbones. We further note that all of our synthetic experiments were run with a standardized backbone between classification and segmentation.

However, even when using the same backbone, segmentation models still typically have more parameters due to the convolutional decoder head. We next provide results comparing performance using

a higher capacity classification model-Resnet101, which has more parameters than the Resnet50 segmentation model-in Table A7. Again, we see improved performance with segmentation.

Finally, we emphasize that we matched training procedure for classification and segmentation networks, including matching the input data, augmentations, and codebase (which standardizes model checkpointing, loss function, hyperparameter tuning, etc.). Together, these results show it is not just model capacity or training procedure that lead to segmentation-for-classification's improved performance.

A5.4 Additional performance metrics

In Table A8 we report additional performance metrics using standard classification and segmentationfor-classification (including semi-supervised methods) on the CANDID dataset.

A5.5 Results on a natural image dataset

To show the proposed method extends to a natural image dataset, we include results classifying dog and cat breeds in the Oxford Pets dataset in the limited data regime (50 training images per class). We trained these models using ResNet50 architectures and the rule-based summarizing function. We observe segmentation-for-classification achieves an AUROC of 0.96 while standard classification achieves an AUROC of 0.81. These results show segmentation-for-classification improves average AUROC by 18.5% on a natural image dataset, mirroring our results on medical imaging datasets.

A5.6 Results on multiclass datasets

While we focus on single-class tasks in the main text, we show here that segmentation-forclassification can directly be used for multiclass tasks. Instead of having a binary mask output from the segmentation network, a user simply needs to specify multiple output channels. First, we note the Oxford Pets dataset contains 37 classes of dog and cat breeds; as shown in the previous subsection, we see segmentation outperforms classification by 18.5% on this multiclass dataset. Further, we extend one of our medical imaging datasets to the multiclass setting and perform 3class classification on ISIC, classifying lesions as benign nevi, melanoma, or seborrheic keratosis. Again, we see improved performance with segmentation-for-classification, which achieves a balanced AUROC of 0.61 compared to classification's 0.50 in the limited data regime.

A6 Limitations

A limitation of this work includes the maximum sizes of the datasets. With much larger datasets, we expect overall classification performance to improve. However, the dataset sizes we use in this manuscript (2,000-20,000 for medical datasets; 100,000 for synthetic dataset) are not uncommon medical imaging dataset sizes, particularly for new applications or datasets specific to an institution. Further, we expect many benefits to hold in the very large data regime (e.g., robustness to spurious correlations, performance on very rare subsets, human assessment, and model evaluation). As an additional limitation, the segmentation-for-classification methods explored here are most applicable to abnormalities that can be segmented. Images for which even experts do not know which regions directly correspond to the class labels (e.g., some histopathology images [63]) have less obvious segmentation targets. In these situations, classification in combination with methods like GradCAM can help with feature discovery, for which segmentation-for-classification may be less well-suited. Similarly, classification labels which are more global in nature (e.g., age prediction from radiographs, atrophy of the brain in a head CT) do not have clear segmentation targets and may not benefit as much from the segmentation signal.

A7 Broader impacts

This work focuses on automated medical image analysis. Automated image analysis algorithms have the potential to support radiologists by providing decision support, improving analysis efficiency, and improving analysis reproducibility. The work presented here helps to make progress towards these benefits by improving the performance of automated medical image analysis algorithms; improving the quality of model evaluation; and providing location information to the radiologist to facilitate human assessment. However, medical image analysis is a low failure tolerance application. The work presented here is meant as a contribution to shift the way we consider segmentation vs. classification algorithms in building medical image analysis tools; the specific neural networks trained here are not meant for clinical use.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the benefits and drawbacks of using segmentation-based methods for classification tasks, particularly in medical imaging. This approach, known as segmentation-for-classification, has been shown to outperform traditional classification models, especially when the available dataset is small or when the classes are imbalanced. It can also handle rare subtypes better than conventional models.\n\nSTRENGTHS:\n- Improve aggregate performance: by enhancing the divergence between segmentation's positive and negative classes and increasing the quantity of annotation, segmentation-for-classification provides greater performance differences in situations with limited data, such as in small datasets, scenarios with low class prevalence, and rare subtypes.\n- Reduced susceptibility to spurious correlations: Segmentation-for-classification is generally more robust to background features that are spuriously correlated with the target task, thus making it more reliable in classification tasks.\n- Location information: Segmentation-for-classification inherently delivers location information, facilitating human assessment of the model's predictions. This location information is particularly useful in medical imaging, where abnormalities can be challenging to find.\n- Comprehensive experiment: It is impressive to see the breadth of datasets used, covering synthetic data as well as three distinct medical datasets. The thorough exploration of different training regimes, including fully labeled, semi-supervised, and a boosted semi-supervised approach, greatly contributes to the depth and robustness of this investigation.\n\nWEAKNESSES:\n- Lack of Baseline Comparisons: While the study establishes that segmentation-for-classification performs better than traditional classification in certain scenarios, it does not benchmark these findings against other state-of-the-art methods for handling imbalanced datasets or for performing classification with limited data. Without these comparisons, it's hard to evaluate the true value of the proposed approach.\n- Handling of Spurious Correlations: While the authors claim that segmentation-for-classification is less susceptible to spurious correlations, the empirical evidence provided seems limited. The specific mechanisms through which the proposed method avoids or mitigates these correlations could be elaborated on more clearly.\n\nQUESTIONS:\nPlease address the questions mentioned in weakness.\nAlso, I am wondering how the segmentation-for-classification performs in 3D segmentation scenario.\n\nLIMITATIONS:\nAdd more baseline comparison. Specific mechanism for mitigating spurious correlations. 3d data performance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an intriguing and somewhat disruptive approach to medical image classification tasks. It implies that due to advancements in weakly-supervised, self-supervised, and semi-supervised segmentation techniques, the historical inclination towards image classification due to ease of training and label acquisition might be reconsidered. The authors argue for a shift towards segmentation, traditionally a more complex task, and demonstrate this by implementing \"segmentation-for-classification\" models. Using three retrospective datasets, they show these models outperforming traditional classification techniques in sample efficiency, performance on low-prevalence classes and rare subgroups, robustness to spurious correlations, and interpretability. However, the authors limit their consideration to tasks where the class of interest can be localized.\n\nSTRENGTHS:\n* The paper presents an innovative idea that can potentially change the current paradigm. If the results withstand, this reviewer sees a strong argument in saying that all existing segmentation models can easily be turned into powerful classification models by adding a small g() function at the end.\n* The simplicity of the language used in the paper aids understanding.\n* Figure 2 does an excellent job of visualizing the difference between classification and segmentation problems in the medical domain.\n\nWEAKNESSES:\n1. The most important issue this reviewer has discovered is how readily the authors dismissed the cost of providing pixel-level annotation. The difference between pixel-level and image-level (i.e. classes) annotation can be orders of magnitude. For example, in the study, this reviewer has been part of, an expert spent 279 s on average to produce pixel-level annotation for a single image, while it took only 2 s to generate a class label. Therefore, even a need to produce pixel-level annotations for even an absolute minority of images may be devastating. So far, from the literature and experience, it seems that such data is still needed, albeit in smaller quantities.\n2. It is also important to point out a worry related to the study's design. In this work, the performance of augmented segmentation models is compared to the classification models, but no apparent effort is taken to ensure that the results obtained are not due to segmentation models being more powerful (which is often the case). Even the simplest segmentation architecture with the most naïve thresholding may have more parameters than the corresponding classification model. This is important to comment on in the main paper.\n3. This brings another important observation - a lot of critical content, such as related works and algorithm details for summarising the probability maps, how multi-layer networks were trained and what architectures were used, the results for 4.2.2 etc., are relegated to the appendix, while arguably less crucial content occupies the main body of the text. This hampers the logical flow and comprehensiveness of the paper. For example, the discussion on the inherent differences between segmentation and classification, though interesting, feels redundant as it seems obvious that different methods would yield different results. This space could be better utilized to expand on related work and other methodological bits mentioned above. Section 5 can be greatly compressed, as it seems repetitive and less important.\n4. Most figure captions are insufficiently informative, requiring the reader to refer back to the main text. Figure captions should be independent of the rest of the text.\n5. The presentation of the results can be expended, with key information (e.g., number of true positives and false negatives) at this point missing.\n6. The last two sections (4 and 5) feel more rushed and less systematic, reducing the overall readability of the paper.\n\nQUESTIONS:\n1. How do the authors justify the much higher cost and time commitment associated with pixel-level annotation as opposed to class-level annotation? Given the substantial difference, could the authors provide insights on the practicality of their approach in real-world scenarios?\n2. Is it possible that the improved performance of the segmentation models over the classification models is simply due to them being inherently more powerful, possibly because they have more parameters? Could the authors elaborate on this aspect and discuss how they ensured a fair comparison between the two models?\n3. Could the authors elaborate on the decision to relegate critical content to the appendix? Would it not improve the logical flow and comprehensiveness of the paper if key aspects such as related works, algorithm details, and results were included in the main text? Conversely, could parts of the discussion on the differences between segmentation and classification be moved to the appendix to make space for the aforementioned content in the main text?\n4. Have the authors considered re-running their results multiple times to ensure they are reproducible?\n\nLIMITATIONS:\nThe authors acknowledge their consideration only for classification/segmentation tasks where the class of interest can be localized. They do not address situations where the entire image must be evaluated, which is a significant limitation. Additionally, while the authors advocate for segmentation, they fail to adequately address the issue of pixel-level annotation's increased complexity relative to image-level annotation.\n\nThe paper does not discuss any potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a set of insights obtained when using segmentation networks for a classification task. Classification was the task of choice due to issues with obtaining appropriate segmentation labels. However, with wider availability of datasets with appropriate labels, this is no longer the case. To facilitate consideration of using segmentation or classification networks, the paper presents a set of best practices and tradeoffs.\n\nSTRENGTHS:\nThe manuscript is very thorough.\n\nAn important problem and best practices to solve it are discussed. In particular, the insights benefit the study of smaller datasets, rare subgroups, and robustness, all of which are important concerns in medical image analysis. \n\nThe supplementary material is very thorough and includes analysis of a synthetic dataset and three real datasets (X-ray, CT, skin). \n\nFinally, some theoretical analysis evaluating separability of positive and negative segmentation and classification classes is provided. \n\nI believe the analysis provided in the paper would be helpful to those trying to design an analysis task on medical image data.\n\nWEAKNESSES:\nAfter reading through the manuscript and supplementary material, I believe that some of the contents of the supplementary material (e.g., visualizations, choice of datasets) and the description of the pipeline can be summarized in a more detailed overview Figure (instead of Figure 3). While a lot of work (e.g., comparisons across networks) has been put into the project, not all of it is clearly explained in the main manuscript.\n\nQUESTIONS:\nHave the authors evaluated the effect of network size (# of parameters) vs performance? It seems like a smaller classification network and a larger segmentation network should in theory learn the same information. \n\nSeveral weaknesses of classification networks and associated strengths of segmentation networks are discussed (e.g., in Figure 4, segmentation nearly always outperforms classification). When is the reverse true? Would there be cases where segmentation networks more easily overfit?\n\nLIMITATIONS:\nThe paper presents several interesting insights into performance of classification via segmentation vs classification networks. However, I am not sure if the presented conclusions will generalize to other datasets/modalities. The paper could do a better job in clarifying the extent of its contributions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors explored the applications of deep learning in radiology, specifically focusing on image classification and segmentation tasks. The authors investigated the performance differences between classification and segmentation models on the same dataset and task using an information theoretic approach. They proposed a method called \"segmentation-for-classification\" that utilizes segmentation models for medical image classification. The authors compared this approach with traditional classification on three retrospective datasets, consisting of 2,018 to 19,237 samples. Through their analysis and experiments, they highlighted the benefits of segmentation-for-classification, including improved sample efficiency, better performance with fewer labeled images (up to 10 times fewer), particularly for low-prevalence classes and rare subgroups (up to 161.1% improved recall). They also demonstrated improved robustness to spurious correlations (up to 44.8% improved robust AUROC), as well as enhanced model interpretability, evaluation, and error analysis.\n\nSTRENGTHS:\nThrough theoretical analysis, the benefits of using segmentation for learning classification datasets are validated, and extensive analysis and validation are conducted on small datasets.\n\nWEAKNESSES:\n1.The proposed method is not specifically designed for medical images; it is a general method but lacks validation in some common settings.\n2.The method proposed in the paper is limited to binary classification requirements, which limits its scalability.\n3.The method does not introduce a new module specifically designed to extend segmentation for classification tasks, lacking novelty.\n\nQUESTIONS:\nAs shown in Figure 4, why is there such a significant difference between segmentation and classification on small datasets? Are the model parameters and training processes for segmentation and classification strictly controlled to be consistent?\n\nLIMITATIONS:\nJust as mentioned in the weakness section, the method proposed in the paper is limited to binary classification.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}