A case for reframing automated medical image classification as segmentation
Authors: Sarah Hooper
Mayee F Chen
Khaled Kamal Saab
Kush Bhatia
Curtis Langlotz
Christopher Re
Venue: NeurIPS 2023 poster (2023)
Keywords: Medical imaging
segmentation
classification
TL;DR: We show that segmentation models can lead to higher quality medical image classifiers in common settings.

Abstract: Image classification and segmentation are common applications of deep learning to radiology. While many tasks can be framed using either classification or segmentation, classification has historically been cheaper to label and more widely used. However, recent work has drastically reduced the cost of training segmentation networks. In light of this recent work, we reexamine the choice of training classification vs. segmentation models. First, we use an information theoretic approach to analyze why segmentation vs. classification models may achieve different performance on the same dataset and overarching task. We then implement multiple methods for using segmentation models to classify medical images, which we call *segmentation-for-classification*, and compare these methods against traditional classification on three retrospective datasets. We use our analysis and experiments to summarize the benefits of switching from segmentation to classification, including: improved sample efficiency, enabling improved performance with fewer labeled images (up to an order of magnitude lower), on low-prevalence classes, and on certain rare subgroups (up to 161.1\% improved recall); improved robustness to spurious correlations (up to 44.8\% improved robust AUROC); and improved model interpretability, evaluation, and error analysis.

A case for reframing automated medical image classification as segmentation
Sarah M. Hooper Electrical Engineering
Mayee F. Chen Computer Science Stanford University
Khaled Saab Electrical Engineering
Stanford University
Stanford University
Kush Bhatia
Curtis Langlotz
Christopher Ré
Computer Science
Radiology and Biomedical Data Science Stanford University
Computer Science
Stanford University
Stanford University
Abstract
Image classification and segmentation are common applications of deep learn- ing to radiology. While many tasks can be framed using either classification or segmentation, classification has historically been cheaper label and more widely used. However, recent work has drastically reduced the cost of training segmentation networks. light of this recent work, we reexamine the choice of training classification vs. segmentation models. First, we use an information theoretic approach to analyze why segmentation vs. classification models may achieve different performances the same dataset and task. We then implement methods for using segmentation models to classify medical images, which we call segmentation-fore forraionfooorclssffccatoooooo and compare these methods against traditional classification on three retrospective datasets (0=2,018-19,237) We use our analysis and experiments to summarize the benefits of using segmentation segmentation-forcclssifi for-classification including: improved sample efficiency, enabling improved performance with fewer labeled images (up to order of magnitude lower), on cow-prevalence classes, and on certain rare subgroups (up to 161.1% % improved recall); improved robustness spurious correlations (up 44.8% improved robust AUROC); and improved model interpretability, evaluation, and error analysis.
Introduction
Classification and segmentation are two popular applications of deep learning to radiology [ Classification produces coarse-grained, image-level predictions, while segmentation produces fine- grained, pixel-level maps. While many tasks can be framed using either classification or segmentation (Figure D. classification is often the default framing. For one, classification has been cheaper to label, requiring only one label per image compared to segmentation's tedious, pixel-wise labels. Second, training classification networks produce image-level outputs mirrors radiologists' standard workflows, where radiologists summarize findings at the image-level and only routinely perform segmentation in few sub-diseiplines that rely quantitative measurements over the segmentation masks. These factors have led to an abundance of medical image classification models 2 33.
In contrast, because segmentation has been so cumbersome label and is performed relatively infrequently by radiologists, segmentation networks are often only trained when the downstream application requires (e.g., for computing the volumes diameters of structures in an image).
Recent work in label-efficient training enables us to reexamine this paradigm. Self-supervised learn- ing, in-context learning, weakly-supervised learning, and semi-supervised learning can substantially reduce labeling burden [4, 6 [0]. Additionally, more public datasets and broad-use pre-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Classificatio
Segmentation
Classification
Segmentation
Classification
Segmentation
Does chest x-ray contain pneumothorax?
Does this CT contain vertebral fracture?
melanoma?
Figure 1:
Illustration of three medical image analysis problems that can be framed using either classification or segmentation.
trained networks are coming online 12
Together, these new methods, datasets, and pretrained backbones are enabling users to develop networks for new segmentation tasks with less and less annotation burden-a trend that will likely continue.
[5]
In light of this progress, we reexamine the convention around when to employ segmentation networks. Specifically, investigate using segmentation networks for medical image classification, exploring segmentation's fine-grained annotation and output can lead to benefits compared to classification" coarse annotation and output. Our contributions include:
Exposition. analyze why classification and segmentation networks may perform differ-
ently on the same datasets. We show that segmentation leads to more separable and robust embedding spaces, guiding what benefits we should expect to see (Section 2).
Best practices. We implement multiple methods obtain classification labels from segmen- tation networks, which we call segmentation-for colssssfioclsssiaatioo (Section 33. We empirically
confirm our analysis (Section 4.1 and compare methods across datasets, tasks, and training conditions to build up best practices for training segmentation caraatioooooo--llsiiiilllttte classification networks (Section 4.2) We show regmentation-for-claeee corcclasifiiatiin can improve aggregate performance compared to traditional classification models by up to 16.2% (from 0.74 AUROC to 0.86 AUROC). Finally, we explore semi-supervised cateeentttion-ooo-llasffiiiiiii existing label-efficient training methods for segmentation can directly benefit segmentation- for-classification.
showing
Trade-ofTs. We pull together our analysis and experiments to provide consolidated and expanded evidence of the benefits of using aegmeeettttion-foo-llllfffctiiii improved sample efficiency, enabling improved performance small datasets, on low- prevalence classes, and on certain rare subgroups (up to 161.1% improved recall); improved robustness to spurious correlations (up to 44.8% improved robust AUROC); and improved model interpretability, evaluation, and error analysis (Section
including:
In summary, conventional wisdom driving the choice between classification and segmentation was major bottleneck building neural networks. With recent
formed when labeling burden progress in label-efficient training, we should reconsider conventions. Ultimately, we show that the choice of training classification vs. segmentation network not only impacts how the model can be used but changes the properties of the model itself, and we show that leveraging segmentation models can lead to higher quality classifiers in common settings. These results indicate segmentation may be more natural way to train neural networks to interpret and classify images, even though it's ''s not the conventional way humans classify images.
Related work. Our study is motivated by past works that employ segmentation networks in classifi- cation problems and report various benefits of doing [16 related work included Appendix AT
17
18
19
20
21
A full discussion of
Analysis
In this section we develop an understanding of why segmentation and classification networks may perform differently given the same dataset and overarching task (i.e., to find class-of-interest). We cover task specification and notation in Section 2.T then build up intuition for the benefits of segmentation in Section 2.2

---

Positive class
Negative closs
Positive class
Negative closs
Classification
Segmentation
Figure 2: Visualization of the positive and negative class distributions that classification segmen- tation networks aim to discriminate. In classification, supervision is applied at the image-level; in segmentation, supervision is applied at the pixel-level.
2.1 Preliminaries
Task specification. We first specify the classification tasks we consider replacing with segmentation. We study classification tasks where the following properties are true: the class-of-interess can be localized in the foreground and does occupy the entire image; the class-of-interest appears if and only if the classification label is positive; the background should not inform the classification label. Example tasks that meet these specifications are shown in Figure [ Examples of tasks that don't meet these specifications include classifying the imaging modality or radiographic bone age, where global features of an image inform the class label and there no clear segmentation target.
Notation. We consider dataset of N images, class labels, and segmentation masks = }. For simplicity of
{(X','. (X',v'''') ,S)) Each image X' is made up of L pixels, X2. notation, assume binary classification, the image- classification label is y E {0. where 1 is the positive label. The segmentation mask S contains pixel-level labels S2......... indicating which pixels contain the
The image-level class label and pixel-level segmentation mask are related by y² = { E' S 1 Put simply, this means if the class label positive, there is least one pixel labeled as positive in the segmentation mask and vice versa. We train a segmentation network f(-) to produce predicted segmentation mask f(xt) S by minimizing the cross entropy loss CE(S'. S*). For notation's 's sake, we will drop the sample index i, thus referring to the 7th image X simply as X and the pixel X simply as Xx..
Data generating process. To study segmentation vs. classification. we take an information theoretic approach and use the following simplified data generating process. First, the pixel-level segmentation labels [Si,.... Sr} are drawn from joint distribution Ps. Then, each pixel X2 is drawn from the distribution Px (.) As above, the image-level classification label y can be defined in terms the segmentation mask, y {
S
2.2 Analyzing segmentation classification
To understand why segmentation and classification may perform differently on the same dataset, we consider the distributions that classification segmentation networks are trained to discriminate. As an example, we'll use chest x-ray pneumothorax classification. In classification, supervision is applied at the image-level the network seeks to discriminate images with pneumothorax vs. images without pneumothorax (Figure 2 left). In segmentation, supervision applied at the pixel- level-the network seeks identify pixels that contain pneumothorax (Figure right). Looking these distributions, differences emerge.
Sample complexity. First, see that segmentation has more annotated input-output pairs, each pixel is annotated. Moreover, we see that classification's 's positive and negative classes share more features than segmentation' eegmentations-iimplle s-implying segmentation may have greater class separability. We capture this notion of separability using the Kulbback-Leibler (KL) divergence and show the following:
Proposition 1 It holds that
Pr(u = 1) DKY. (Pr(Xaly 0)] Pr(Xxly = 11).
DRI (Xx O)][Pr(Xx|SS, 1))
Pr(S = 1)
(I)
3

---

define
g(+)
vs.
Classification workflow
Segmenntatonfforllassffccann workflow
Figure 3: Diagram showing how a chest x-ray disease classification problem can be framed using either standard classification or workflow, the input image is processed by classification network, which outputs classification label vector. In the gesmentaiion-for-casiiffee
the traditional classification
workflow, an input image is processed by segmentation network, which outputs binary masks showing where each abnormality is found. These masks are then converted into a classification label vector via a summarizing function, g()).
In other words, the KL divergence between data given pixel-level labels is done in segmentation) is greater than or equal to the scaled KL divergence given image-level labels (as is done in classification). We note that the scaling factor Pr(y=1) is always greater than equal proof for Proposition D in Appendix A3.1 confirming our intuition that segmentation supervision results in more separable data distributions. Because of the greater quantity of input-output pairs and the greater KL divergence, we expect segmentation's 's discrimination function be easier to learn, requiring fewer images simpler function class 22 23 Since we study the setting in which the
We provide the
Pr(S.=])
segmentation mask directly indicates the classification label (y = 1 {{ S 2 1)), we similarly expect iwwentationnforcccassffciii limited data regime. Finally, from Proposition]] | we expect segmentation-fr coreataatioo-oo--clasiiiiatiiiii to impart more benefit for tasks with small targets; for large targets, the terms Proposition approach one another.
to have higher performance than standard classification in the
Robustness to spurious correlations. The second observation make from Figure is that, because background features appear in classification's positive class, it possible for the classification network to rely background features to identify positive samples if those background features spuriously correlate with the class label. Such failure modes have been observed in prior work. For example, classifiers relying metal L/R tokens to identify pneumonia (24] classifiers looking for chest drains to classify pneumothorax 25 and classifiers looking to image edges (which can change with patient positioning and radiographic projection) to classify COVID-19 (26). Segmentation should be more robust to such background features, since background features are less correlated with the target class. This notion has been formalized in past work [21], which we recall here.
Specifically, for spurious features that do not overlap with the segmentation target, the mutual infor- mation between positive features and spurious features decreases with image-level labels to pixel-level labels. Since the spurious feature less informative of the pixel-level labels than the classification label, expect segmentation to be more robust to the presence or absence of spuriously correlated background features.
transition from supervising
Methods
To segmentation for classification tasks, need to determine how to segmentation networks to obtain classification labels. We'll do this by defining a summarizing function, g(-). which takes input the image and segmentation network and outputs class label g(X, f(-)) J. The cewpenatttiin-ooo-lasiiiiii...
workflow is shown in Figure 3 In this section, we describe the summarizing functions that evaluate. We note these methods be used for both binary class problems and multiclass problems, as long as the classes interest follow the task specification set above. We provide multiclass results in Appendix A5,6
3.1 Rule-based summarizing functions
We first consider a simple, rule-based summarizing function. This method is intuitive: if there positive region in the predicted segmentation mask, we infer positive classification label.

---

Synthetic
CANDID
ISIC
SPINE
Figure 4: Visualization of the datasets used in this manuscript: the synthetic dataset, where the task is to classify if navy blue circle is preset; CANDID, pneumothorax classification task; ISIC, lesion classification task; and SPINE, cervical fracture classification task. For each dataset, we show a positive and negative image and their corresponding segmentation masks.
Specifically, inference, we binarize the probabilistic mask output from the segmentation network using threshold f. If the binarized mask contains over threshold т total positive pixels, we return positive classification label. To compute the class probability, we average the pixel-wise probabilities of all pixels in the class, Algorithms for this method are provided in Appendix A2.1
3.2 Trained summarizing functions
Next we consider summarizing functions trained transform the segmentation information into classification labels. The intuition behind these methods is that trained functions may learn more nuanced differences between true positive and negative predicted segmentation masks. We train these summarizing functions using the same splits used to train the segmentation network, and freeze the segmentation networks before training the summarizing functions.
We first consider trained summarizing functions that operate on the predicted segmentation masks. We consider three architectures for the summarizing function, each with increasing complexity: fully connected layer, a global average pooling layer followed by a fully connected layer, and existing image classification architectures such as SqueezeNet (27 and ResNet50 28
We also consider summarizing functions that operate on segmentation network embeddings, similar to traditional transfer learning. We consider both shallow and deep embeddings and two different summarizing function architectures: simple architecture consisting of pooling and fully connected layers, and a more complex classification head proposed in past work (20).
We fully describe these summarizing functions and training processes in Appendix A2.2
Experiments
In this section, first confirm the takeaways from our analysis in Section 2 on synthetic dataset (Section 4.1). Then, we evaluate the summarizing functions we described in Section 3] medical imaging datasets (Section (.2). We briefly describe the datasets and training below; provide full details of the synthetic dataset and training in Appendix A4]and the medical datasets and training in Appendix A5
Datasets.
The synthetic dataset consists of many colored shapes on a gray background; the task is to determine if a navy blue circle is present. We also evaluate three medical imaging datasets: CANDID, which we aim to classify pncumothorax in chest x-rays (n=29))77) [29] ISIC, in which aim to classify melanoma from skin lesion photographs (n=2.750) (30]: and SPINE, in which aim to classify cervical spine fractures CT scans (n=2,018, RSNA 2022 Cervical Spine Fracture Detection Challenge). These datasets visualized in Figure 4 The SPINE dataset is an example of where the growing number of off thels networks enables segmentation segmmenttion--forccaasfffiieee cosslaasiiiiicaiinn we use pretrained CT segmentation network generate segmentation targets for this task, detailed in Appendix[A5
Training.
To establish baseline classification performance, we train standard supervised classifica- tion network. To establish baseline segmentation-clasfffcatioo performance, we train a multitask learning network where the classification and segmentation networks trained concurrently and
5

---

Balanced eccurecy
Balanssas acourney
Balanced accumcyy
0.
88
Balanced
15:00 30.00 45,000
0.01
6a2
0.69
0d4 0.s
690 u.2 0.64 cle bos 1,50
5.00
6.04 0.8 012
(4)
No. training images
(6)
Class provalance (low) (6)
Class prevalence (high) (d)
Small target provalance
during training
Balenced accuracy
1.0
Bulunced eccurasy
1.0
0.0
0.
07
08
Unlanced
0.6
0.5
Seg feg-for-class
Classification
06
05
0.80
0.86 6:90 0.5
1.00
00 92
04 0.6 08 1.0
(e)
Target redius
Spurious correlation strength
(g)
Prevalence center during
training
Figure 5: Results from experiments on the synthetic datasets. Each panel shows classification vs. performance, as measured by the balanced accuracy, on the y-axis as a characteristic of the training dataset changed on the x-axis.
share the same network backbone using the Y-Net structure 31 classification performance, we implement each summarizing function as described in Section 3 We train classification, multitask, and segmentation networks by minimizing the cross entropy loss between the predicted and ground truth labels using the Adam optimizer. We provide additional train- ing details and information on hyperparameter tuning in Appendix [A4.2) and A5.2 and provide ablation experiments controlling for segmentation vs. classification model capacity in Appendix
To obtain segmentation-for
A5.3
4.1 Experiments on synthetic data
We first perform experiments on synthetic data to confirm the takeaways from our analysis. By using synthetic dataset, we have precise control over dataset and task characteristics, allowing us isolate and adjust these variables to assess model performance. We provide highlights of our synthetic experiments below; full description the experimental procedure on synthetic data can found in Appendix A4 In these synthetic experiments, we use the rule-based summarizing function.
The first takeaway from Section is that citmenntttion-oooollassiiatiiiii should require fewer images to achieve high performance, particularly for tasks with small targets. We examine this with our synthetic dataset in the following experiments.
We sweep the number training images from 1,000 to 100,000 and plot classification and www.aaiiioooooooassffiiiiiiiiaae
balanced accuracy in Figure 5h, where for-classification has improved performance in the limited data regime.
segmentation-
Similar learning with small datasets, also expect segmentation perform better with high or low class prevalence, which we see Figure (5p and Figure 5p.
Segmentation's ability to learn with fewer examples should also extend to subtypes-vw expect higher performance on class subtypes that appear infrequently during training. To evaluate this, train small, medium, and large navy blue circles and evaluate network performance on the small circles as they are made increasingly rare during training (Figure 5).). We see segmentation-for coraaionnooo-lllsiicctttooo achieve higher performance on this data slice with few training examples.
We vary the object size the train and test dataset and find segmemationn-forcllssiicccaai provides more boost with smaller targets (Figure 5b))
The second takeaway from analysis is that segmentation-fort coree-oo--aaane classisiifii should be more robust to background features that spuriously correlate with the class of interest. We evaluate this with two synthetic experiments.
We first evaluate an obvious case of spurious correlation: when background object correlates with the class label. We make a large, pink square increasingly correlated with

---

CANDID
ISIC
0.73
0.95
0.70
0.90
0:68
0.83
9.55
0.00
0:62
0.79
0.70
0.65
0:53
Limited training data.
All trining
Limxted triiing data
All tuining
SPINE
0.72
Barnheld
sulput
2..4
sulpet
rutput
dastiiiaiion retwook
0.66
Deep ensheddiggs
head
0.60
Skallow entocilinge simple clas koud Stallow eniocilings comples chas bead
Dorp crchndding exmplex dlast hoad
0.58
0.56
Figure 6: Comparison of different ad SPINE datasets in both the limited and abundant training data settings. As baselines, we include classification and multitask network performance. FC: fully connected.
methods the CANDID, ISIC,
the target (a small navy blue circle) in symthetic dataset- then remove this correlation in the test dataset-and show that segmentation is robust to this correlation (Figure (
Spurious background correlations can also include less obvious features, like where the target object occurs; in other words, we expect segmentation to be more robust to target location. We explore this by changing where the target navy blue circle appears during training, but allowing the navy blue circle to appear anywhere in the test dataset. As the target is increasingly restricted particular location during training, classification is less able to generalize to finding the target in other regions of the image (Figure 5g..
4.2 Experiments on medical imaging data
Given the promising results on synthetic data, next evaluate performance the medical datasets and evaluate saameentatioo-fo--llssfaacac performance of different summarizing functions, classification networks, and multitask networks. In Section 4.2.2 we evaluate semi-supervised segmenationnfor-Cassicccaiiii
training methods. In Section 4.2.1 we compare the
4.2.1 Comparing summarizing functions, classification, and multitask training
We evaluate each summarizing function and model on the three medical datasets two regimes: limited labeled data, which consists of 10% of randomly sampled training data, and all labeled training data. We report mean AUROC on the test set Figure 6 We observe that the simple, rule- based segmentation-fore wowpoooo-ooo---a-ooiiiii classification method achieves higher mean AUROC than both the traditional classification and multitask networks for all tasks and amounts of training data. Specifically, we observe 10.0%, 6.9%, and 2.8% improvement in mean test AUROC in the limited training data regime for CANDID, ISIC, and SPINE respectively over traditional classification, and 2.6%, 0.6%, and 6. % performance improvement in the all training data setting.
Comparing summarizing functions, we find that summarizing functions which operate over segmenta- tion network outputs generally outperform those that operate over embeddings. Among summarizing functions that operate over segmentation outputs, three perform similarly well: thresholding the segmentation output, learning a fully connected layer, and training a full classification network on top of the segmentation output. The threshold-basee based summarizing function has additional advantages: it does not require training, thus saving training effort and avoiding potential overfitting to training data; fast to compute inference; and the results are intuitive- there segmentation mask, there is an associated positive class label. For these reasons, we recommend simple rule-based summarizing functions for converting a segmentation mask into a classification label. We compare the error modes of these methods in Section 5
7

---

0.96
AUROC
Seg-forcless
Classhicatioon, (reference)
Seg fon-llsss, semi supervision
Sog clas, bazated scmi supcrvision
0.88
10% Percent of training data with segmentation labels
15%
100%
Figure 7: cumi-supervised trained with additional segmentation labels; the semi-supervised model also trained with unlabeled data, and the boosted semi-supervised model is also trained with data that only has classification labels. Each line plots the average AUROC of the test set with shaded 95% confidence intervals.
As the x-axis increases, all models are
4.2.2 Semi-supervised segmentation-forccassffice
As discussed in the Introduction, historical obstacle widespread application of segmentation is the difficulty of curating labeled training datasets. However, recent work has led to more public datasets, pretrained networks, and methods reduce the labeling burden of training segmentation models. We posit that this progress in segmentation will also benefit In the previous section, we showed that an off-the-sheil spine segmentation model could be used to generate segmentation labels for training the SPINE segmentation-for pnmreettttoonfo--cllsffiee for-classification models. In this section, we verify that semi con-ssupreeseed segmentation method can improve performance of models. We consider two common settings:
Setting additional unlabeled data. Because unlabeled data often abundant, practition- ers may have a subset of their dataset labeled with segmentation masks and larger set of unlabeled images.
Setting 2, additional classification data. Other practitioners may have a subset of their dataset labeled with segmentation masks and a larger of images with classification labels.
For Setting 1, we apply a gerviously-publisiie semi-supervised segmentation method which takes a small set of labeled data and large set of unlabeled data input and trains segmentation model using a combination of data augmentation, consistency regularization, and pseudo labeling [32
For Setting 2, we update the rrviously-prooossee methods by using the classification labels to improve the pseudo segmentation labels, which we call "boosted" semi-supervision. The original semi-supervised approach uses pseudo labeling, in which a small amount of labeled segmentation data used to train an initial segmentation network; the initial network predicts pseudo segmentation masks for all unlabeled data; and final network is trained on the pseudo segmentation masks. In the "boosted" algorithm, we use the available classification labels to modify the pseudo segmentation masks. Specifically, for images that have negative classification labels, we zero out the pseudo segmentation masks: because we know the classification label is negative, we know that there should not be any positive region in the segmentation mask. For images that have a positive classification label, we check to see if the pseudo segmentation mask has positive segmentation region; if it does not have a positive segmentation region, we know that the pseudo segmentation mask is wrong and we exclude the sample from training of the final segmentation network.
These semi-supervised approaches learn from the unlabeled data to train better segmentation model, which we hypothesize will also improve segmecntatoonfoorclasiiicccciii classsssifiiatoonn performance. In these experiments, we use the threshold- based summarizing function and train models using balanced training dataset. We report model performance for varying amounts of labeled training data in Figure
We report additional performance metrics and the number of labels for each method at the most limited and abundant data settings in Table A8
From these results, we observe that all of the segmentation-for wwr--oosssificattinn models trained with 10% of the training data labeled exceed the performance of the classification model trained with 100% of the training data labeled. These results confirm one of the takeaways from our analysis, which is that segmentation- regmmenationnfoo-ccassisiee for.classiiicatioo can achieve higher performance with fewer images. Additionally, we performance,
see that the semi-supervised training further improves conpeeetattoo-fo-lllssssese

---

confirming that semi-supervised segmentation methods can directly be used for segmentation-for for- classification. Finally, we see that the boosted semi-supervised approach indeed improves perfor- mance beyond standard semi-supervision in the limited data regime.
Benefits and drawbacks of segmeenaaiion-fo-cdaatfiiiton
In this section, we tie together our analysis, synthetics, and experiments on real data to summarize the benefits and drawbacks of using gggnentation-foo-ccaasffiiiiilli
Segmentation can improve aggregate performance, particularly with small datasets. Using models compared to traditional classi-
can lead to bigher-performen fication. Our analysis in Section shows this benefit is due in part to greater divergence between segmentation's positive and negative classes and higher quantity of annotation (we evaluate the contributions of each of these factors with additional experiment in Appendix (A4.3) We observe this takeaway empirically in Figure showing segmentation- wwwpentttiioooolllllssiiiieieeoe closcaassiiiiatoon improves mean perfor- mance all datasets and training settings. Our analysis suggests greater performance differences occur in the limited data regime, which can include:
Small datasets. We expect greater boosts from cotpenatttiioooo--lssfiiiaaaaaaa dataset sizes. We observe this with syntheties Figure 5p and on two of the three medical datasets in Figure 6 Small datasets often occur when developing models custom data or when training models for rare diseases.
with limited
Low class prevalence. Segmentation can also achieve higher performance than classification when the class balance is very low or high (shown with synthetic datasets in Figure pp.c) since segmentation is able to better learn from the few examples of the cow-prevalence class. Rare subtypes. We expect peannentationfoo-classtt
models to have improved ability identify rare subtypes, which appear infrequently in the training dataset. We observe this on the synthetics (Figure (dt) and see this property revealed in the medical datasets well: on small lesions that positive for melanoma, which are less commonly seen during training. traditional classification achieves a recall of 0.18 and segmentation-for classification achieves a recall of 0.47.
Finally, the analysis in Section 2 suggests segmentation provides greater benefit for tasks with small targets. We observe this with synthetics in Figure Se. On the real data experiments, models trained with the full datasets reflect this property as well: the SPINE dataset has the lowest average target-to-biid bobk--goubb-- size ratio (with the target taking up average 0.6% of the image) and achieves the greatest performance boost with segmentation- -or-clasisffication while the ISIC dataset has the highest -aak--oobba size ratio (with the target taking up an average 26.3% of the image) and sees the lowest performance boost.
Segmentation reduces susceptibility to spurious correlations. A worrying failure mode of classifi- cation networks is that instead of learning to identify the pathology of interest, the model instead looks for essier-to-identie features that spuriously correlate with the true target-such as the examples given in Section 2 Segmentation-Fon -oaas--fiocassiiaation should be more robust to background features spuriously correlated with the target task.
Empirically. we observe this in our synthetic data (Figure 5 and evaluate a spurious correlation on the medical datasets here. Specifically, in the CANDID dataser, we evaluate performance under known spurious correlation condition: most pneumothoraces chest x-ray datasets co-occur with a chest tube, which are used to treat pneumothorax 255. Thus, it is important that the pneumothorax classification algorithms are able to find pncumothoraces without chest tubes, as those are the patients who have not yet been identified and treated. To evaluate each model's robustness to this spurious correlation, we evaluate model performance on patients who do not exhibit the spurious correlation (i.e., patients with pneumothorax but chest tube or patients without pneumothorax but do have chest tube) and observe that segmentation- cwwpnaatiionooocllsssiiiiiieoiei forscaassiiiiation achieves an AUROC of 0.84 while the classifier's 's AUROC drops to 0.58.
We also expect segmentation- coapaiioooooo--llsiicctttooooooo to be more robust to the target location, which is another type of spurious correlation. We observe this in the synthetics (Figure and on the medical data: recall of the CANDID classifier drops up to 22.5% conditioned on the lung region in which the pneumothorax occurred (dropping from 0.92 recall in the right lower hemithorax to 0.71 recall on the

---

left upper hemithorax) while segmentation-foo for- cor-claasificattoon recall drops at most 7.5% (from 0.94 in the left lower hemithorax to 0.87 in the right upper hemithorax).
Segmentation facilitates human assessment. location information. While there are methods to probe classification models for which areas of an
inherently delivers
image contribute to the model's prediction (e.g., saliency maps [33], class activation maps [34 35
these methods are often unreliable; shorteomings are further discussed in prior work [36] 37 By
providing location information as well as classification information, legmeenatioon--or-llaffiiciaia models enable the user to more effectively adjudicate model findings. This location information is more pertinent for some applications where the abnormality may be hard to find (e.g., pneumothorax, fracture) versus use cases where the visual features used by the model are obvious (e.g., skin lesion).
Segmentation enables more precise model evaluation. We can perform more detailed model evaluation and error analysis using segmentation-for--ee for-classification's location and size information. For example, we can evaluate models with a stricter definition of recall: instead of identifying a true positive as when the image-level label is positive and correct, we can also require that the correct region of the image is identified. When we compute the stricter recall metric for models trained with all labeled data, we see reductions in recall: CANDID drops from 0.87 0.85 and SPINE drops from 0.74 to 0.55. We can't similarly assess the classification algorithms with this stricter definition of recall because of the lack of reliable methods to acquire accurate location information from classifiers. However, precise model evaluation is important for low-failure- -olllrrr-- applications such as medicine, and segmentation- cettttatiiooo---lsssses for-classification provides an obvious means of doing so. Finally, we can perform additional error analysis using segmentation-for-et caw.aatioooooo--lassfaaiiioooooi 's size and location information, such as assessing the average abnormality size and region.
Segmentation has higher per-image labeling cost. The burden of generating segmentation training labels is drawback of training segmentation networks and can change for different tasks (e.g.. 3D images have a higher labeling burden). In comparison, classification is cheap to generate labels for particularly for cases where classification labels can be pulled from electronic health records (38). In this work, we ve explored two ways of reducing segmentation's 's labeling burden while improving classification performance: using an off-the- fff-the-shelf segmentation model and using cemi-supervised training methods. As new tools emerge and make it casier to generate segmentation training labels we expect the labeling burden gap between classification and segmentation to lessen. Still, the increased annotation cost of segmentation given available tools should be weighed against the expected benefits (detailed above) of using wwwpeettttooo-ooolllsssiiiiiioot
13 14 15),
on a given dataset.
6 Conclusion
Historically, segmentation networks have been employed only when required due to their high annotation cost. Given the increasing ease of training segmentation networks, we aim to better flesh out the decision space between training a segmentation develop intuition and formally analyze why segmentation and classification networks can perform differently. In Section 3 and we explore methods to implement segmentation-for coriioo-ooo-classiiaatiioooo and empirically evaluate segmentation-for coaattioonfo--clasffcatttooo over many datasets and training conditions. Finally, in Section 5 we discuss the benefits and drawbacks of segmentation-for- -o-sssifftoalssiicaattoooo We show that segmentation-for -ol----osaaassiiaation can lead to more performant, robust, and interpretable models. In clinical settings, one may prefer more these upsides despite the added annotation cost.
classification network. In Section 2 we
Broadly, these results underline the potential of deep learning- segmentation as general tool for automating image analysis. Segmentation data provides an information-rich environment for deep neural networks to learn from, which we' 'e shown results in separable, robust embedding spaces. In turn, segmentation network outputs capture detailed information about images, which can support many downstream analysis tasks.
In future work, we are interested in how to leverage segmentation for targets that don't fit the task specification we study here, perhaps utilizing anatomical or coarser forms of segmentation still reap some of the benefits of training segmentation models. Additionally, we are interested in reader studies focused on quantifying the value of the location information contained in segmentation masks for radiologists working with AI algorithms. Finally, we are interested in how to leverage segmentation to benefit other machine learning applications in radiology, extending our study of how task framing impacts a model's 's properties.
10

---

Acknowledgements
We thank Dr. Leslie Smith and Dr. Darvin Yi for their discussion and ideas related this topic.
Sarah Hooper supported by the Fannie and John Hertz Foundation, the National Science Foundation Graduate Research Fellowship under Grant No. DGE-165651 18,6 and Texas Instruments Fellow under the Stanford Graduate Fellowship in Science and Engineering. Additionally, we gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1 563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W91 (FF-11222220 (Interactive Human-Al Teaming); ONR under No. 0000141712266 (Unifying Weak Supervision); ONR N00001-2000 1--00-1----22--0-0-0010000 20-1-2480: Understanding and Applying Non- Von-Euclidean (NEPTUNE): NXP, Xilinx. LETI-CEA, Intel. IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. Figures in this paper were created with BioRender. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
Geometry in Machine Learning: NO001
References
[1] Phillip M Cheng, Emmanuel Montagnon, Rikiya Yamashita, Ian Pan, Alexandre Cadrin- Chênevert, Francisco Perdigón Romero, Gabriel Chartrand, Samuel Kadoury, and An Tang. Deep learning: an update for radiologists. Radiographics, 1155)11427-1445, 2021.
[2] Simeng Zhu, Marissa Gilbert, Indrin Chetty, and Farzan Siddiqui. The 2021 landscape of fda-approved artificial inteliggencemmachiee learning- -eablled medical devices: an analysis of the characteristics and intended use. International journal of medical informatics, 165:10 2022.
[3] S Niyas, SJ Pawan, M Anand Kumar, and Jeny Rajan. Medical image segmentation with 3d convolutional neural networks: A survey. Neurocomputing, 493:397-413, 2022.
[4] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviee vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[6] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414, 2022.
[7] Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[8] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 1oXiv:200800725: 2021.
[9] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29, 2016.
[10] Ting Chen, Simon Komblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi- cui-suueeeeiee learners. Advances neural information processing systems, 13:22233-22555 2020.

---

[11] Jakob Wasserthal, Manfred Meyer, Chhns-Christtann Breit, Joshy Cyriac, Shan Yang, and Martin Segeroth. Totalsegmentator: robust segmentation of 104 anatomical structures ct images. arXiv preprini urXiv:2208.05866 2022.
[12] Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, al. Vindr-cxr: An open dataset of chest x-rays with radiologist's 's annotations. arXiv preprint arXiv:2012.1502) 2020.
[13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.0264. 020444 2023.
[14) Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.0613/ 2023.
[15] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything weerywheee all at once. arXiv preprini arXiv:2304.06711 2023.
[16] Gautam Rajendrakumar Gare, Andrew Schoenling, Vipin Philip, Hai V Tran, P deBoisblanc Bennett, Ricardo Luis Rodriguez, and John Michael Galcotti. Dense pixel-labeling for reverse- transfer and diagnostic learning on lung ultrasound for covid-19 and pneumonia detection. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1406 1410. IEEE, 2021
[17] Gautam Rajendrakumar Gare, Jiayuan Li, Rohan Joshi, Rishikesh Magar, Mrunal Prashant Vaze, Michael Yousefpour, Ricardo Luis Rodriguez, and John Michael Galeotti. W-net: Dense and diagnostic semantic segmentation of subcutaneous and breast tissue in ultrasound images by incorporating ultrasound rf waveform data. Medical Image Analysis, 76: 102326, 2022.
[18] Michal Heker and Hayit Greenspan. Joint liver lesion segmentation and classification via transfer learning. arXiv preprint arXiv:2004.1233: 2020.
[19] Nooshin Mojab, Philip S Yu, Joelle A Hallak, and Darvin Yi. Cvs: Classification via segmenta- tion for small datasets. arXiv preprint arXiv:2111.0004 2021.
[20] Ken CL Wong, Tanveer Sycda-Mahmood, and Mchdi Moradi. Building medical image classifiers with very limited data using segmentation networks. Medical image analysis, 19:105-116, 2018.
[21] Khaled Saab, Sarah Hooper, Mayee Chen, Michael Zhang, Daniel Rubin, and Christopher Ré. Reducing reliance spurious features in medical image classification with spatial specificity. 2022.
[22] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[23] Martin Anthony and Peter I.. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.
[24] John R Zech, Mareus A Badgeley, Manway Liu, Anthony Costa, Joseph Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(1 :):e)eee02225 2018.
Titano, and
[25] Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden strat- ification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151 2020.
[26] Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610 619, 2021.
[27] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb 2016.
model size. arXiv preprint
12

---

[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[29] Sijing Feng, Damian Azzollini, Ji Soo Kim, Cheng- Kai Jin, Simon P Gordon, Jason Yeoh, Eve Kim, Mina Han, Andrew Lee, Aakash Patel, et al. Curation of the candid-ptx dataset with free-text reports. Radiology: Artificial Intelligence, 3(6), 2021.
[30] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittlr, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168-172. IEEE, 2018.
[31] Sachin Mchta, Ezgi Mercan, Jamen Bartlett, Donald Weaver, Joann G Elmore, and Linda Shapiro. Y-net: joint segmentation and classification for diagnosis of breast biopsy images, In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 893-901. Springer, 2018
[32] Sarah M. Hooper, Sen Wu, Rhodri H. Davies, Anish Bhuva, Erik B. Schelbert, James C. Moon, Peter Kellman, Hui Xue, Curtis Langlotz, and Christopher Ré. Evaluating semi-supervision methods for medical image segmentation: applications in cardiac magnetic resonance imaging. Journal of Medical Imaging, 10:024007, 2023.
[33] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[34] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[35] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Gra:-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[36] Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareck, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, and Pranav Rajpurkar. Benchmarking saliency methods for chest x-ray interpretation. Nat Mach Intell, 2022.
[37] Nishanth Arun, Nathan Gaw, Praveer Singh. Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al. Assessing the trustworthi- ness of saliency maps for localizing abnormalities in medical imaging. Radiology: Artificial Intelligence, (()):200226, 2021.
[38] Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L Rubin, et al. Cross-modal data programming enables rapid medical machine learning. Patterns, 1(2):100019, 2020.
[39] Kim-Han Thung and Chong- Yaw Wee. A brief review on multi-task learning. Multimedia Tools and Applications, 77(22):9705-297722 2018
[40] Puyang Wang, Vishal Patel, and lIker Hacibaliloglu. Simultaneous segmentation and classification of bone surfaces from ultrasound using multi-feature guided cnn. In International conference on medical image computing and computer-assisted intervention, pages 134-142. Springer, 2018
13

---

[41] Xiaofei Wang, Lai Jiang, Liu Li, Mai Xu, Xin Deng, Lisong Dai, Xiangyang Xu, Tianyi Li, Yichen Guo, Zulin Wang, et Joint learning of 3d lesion segmentation and classification for explainable covid-19 diagnosis. IEEE transactions on medical imaging, 40(9):2633-276. 2021.
[42] Ozan Ciga and Anne L Martel. Learning to segment images with classification labels. Medical Image Analysis, 68: 101912, 2021.
[43] Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, (2(55):1122-1133 2018.
[44] Jo Schlemper, Ozan Oktay, Michiel Schaap. Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Attention gated networks: Learning to leverage salient regions in medical images. Medical image analysis, 53:197-207, 2019.
[45] Yanming Guo, Yu Liu, Theodoros Georgiou, and Michael S Lew. A review of semantic segmentation using deep neural networks. International journal of maltimedia information retrieval, 7 2018
[46] Saeid Asgari Taghanaki. Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: review. Artificial Intelligence Review, 54(1):137-178, 2021.
[47] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient supi-supe learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, page 896, 2013.
[48] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[49] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arX1v:2010.00747, 2020.
[50] Krishna Chaitanya, Ertune Erdil, Necrav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 13:12546-12554 2020.
[51] Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv preprint 1904v:1904.0000 2019.
[52] Yang Wen, Leiting Chen, Yu Deng, and Chuan Zhou, Rethinking pre-training on medical imaging. Journal of Visual Communication and Image Representation, 78:103145, 2021.
[53] Mohammad Amin Morid, Alireza Borjali, and Guilherme Del Fiol. A scoping review of transfer learning research on medical image analysis using imagenet. Computers in biology and medicine, 128: 104115, 2021.
[54] Zhuo Chen, Ruizhou Ding, Ting- Wu Chin, and Diana Marculescu. Understanding the impact of label granularity on enn-based image classification. In 20/8 IEEE international conference on data mining workshops (ICDMW), pages 895 904. IEEE, 2018
[55] Tanveer Syeda-Mahmood, Ken CL Wong. Joy T Wu. Ashutosh Jadhav, and Orest Boyko. Extracting and learning fine-grained labels from chest radiographs. In AMIA Annual Symposium Proceedings, volume 2020, page 1 199.. American Medical Informaties Association, 2020.
[56] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF international conference computer vision, pages 3478-3488, 2021.
14

---

[57] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-gramee aineggraned robustness in coarse- graeeerraindddeeeee classification problems. Advances in Neural Information Processing Systems, 33:19339- 19352, 2020.
[58] Evan Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
[59] Fabian Isensee, Paul Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Heit nnu-net: a self-coonig method for deep learning-base based biomedical image segmentation. Nature methods, 18(2):203-211, 2021.
[60] William Falcon et al. Pytorch lightning. http:///ttt..oooo/yyoooogiiinnnt 3, 2019.
GitHub.
Note:
[61] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[62] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation, Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[63] Gautam Machiraju, Sylvia Plevritis, and Parag Mallick. A dataset generation framework for evaluating megapixel image classifiers and their explanations. In European Conference on Computer Vision. pages 422 442. Springer, 2022.
15

---

Appendices
A1 Related work
Segmentation to benefit classification. Our study is inspired by past works that employ seg- mentation networks in classification problems and report various benefits of doing so. Improved overall accuracy. For example, [16] uses a segmentation network to generate segmentation masks of different tissue types in lung ultrasound images, then trains sequence of global average pooling and fully connected layers to convert the tissue masks into classification labels. The authors report higher aggregate performance compared to training classification model from scratch for their lung ultrasound classification task. In [7]. the authors expand this approach by including diagnostic tissue segmentation targets (i.e., segmenting benign vs, malignant tissue different segmentation classes), then converting the segmentation masks into class labels via a likelihood score over the segmented tissue types. Similarly, [8] jointly segments and classifies liver lesions in CT scans by segmenting the lesion as one of multiple possible lesion types and finds that the segmentation approach results in higher classification performance than traditional classification or multitask learning. Training with fewer labels. Other reports focus segmentation networks' ability to learn from limited labeled data. For instance, [19] converts segmentation outputs into classification labels via likelihood score and compares the aggregate performance of the segmentation network against classification and multitask learning networks when trained with varying dataset sizes. The authors find that the segmentation network achieves higher classification scores in the limited data regime. Similarly, [20 uses features from a UNet pretrained on a segmentation task to fine tune a classifier and again finds higher classification performance using the pretrained segmentation weights in the limited data regime. Improved robustness. Finally, [2I] shows that training a segmentation network instead of a classification network results in predictors more robust to spurious correlations.
Multitask learning. Other works have combined segmentation and classification via multitask learning, in which the classification and segmentation networks share the same backbone but dif- ferent task heads trained concurrently [39 40
31 42]. The multitask approach differs from the approaches discussed above, which each train a segmentation network, freeze it, then use the segmen- tation network's is embeddings or outputs to produce classification label. In multitask frameworks, the classification task can impact the weights learned for the segmentation task via the shared backbone, and training schedules and task weighting must be managed.
41
Classification to benefit segmentation. Other studies focus on the reverse of what we are inter- ested in by first training a classification network then using that classifier to generate segmentation information. These approaches typically aim either to generate segmentation labels for "free" by extracting location information from the classification models or to use this location information to improve a classification network's 's interpretability. Examples of this class of related work include using saliency maps [33, class activation maps [34]] 35), occlusion maps [43), and classifiers with built-in attention mechanisms [4).
Semantic segmentation. A great deal of work has focused on improving semantic segmenta- tion methods 45 46), which underlie the segmentation- -orlassf-i--lssiicattooo methods we explore here. frameworks contain an additional step converting semantic seg- mentation network or output into an image-level classification label, particularly for tasks which not typically framed as segmentation tasks. Additionally, many recent works have advanced our ability to train semantic segmentation networks with limited labeled data, including using self- supervised networks, semi-supervision, and foundation models 4
5 6 7 47 8) many of which These methods make training
have been adapted specifically to medical imaging 48 49
50
segmentation-for for.iionoooo--llssfiaatttooooooo models less labor-intensive.
Transfer learning. The caapeeentatioo-f----asssiii learning. While many transfer learning studies have focused on settings where the task type stays consistent (e.g., transferring one classification network to a different classification task), some studies have looked at transferring classification weights to segmentation tasks or vice versa [51 Some cegmentation--oorllssiiffee
framing can be viewed as type of transfer
52 (3).
methods can be considered transfer learning, where
16

---

Algorithm Compute binary label
Require: Mask e [0, l| t [0, E 1: BinSegMask t ProbSegMosk 2
2: if' summ (um(BineggMMak)) 2 r then
3: return
4: else
5: return 0
6: end if'
the segmentation network is trained first and much of the segmentation network is frozen before transferring the learned weights to a classification task.
Label granularity. A related line of work explores the right label granularity for training classifica- tion models (e.g., training "herbivore vs. carnivore" compared to training "horse vs. deer vs. cat vs. dog vs..."
54 55. Similar to our findings, these works find that more granular labels can lead to improved network optimization and generalization [54]
Modified classification networks. Finally, many works have modified straightforward supervised classification training schemes. For example, self- and semi-supervised learning aim to improve performance with limited labeled data [56], and past work has proposed various modifications to classification training to improve robustness [7 58
In this manuscript, we use straightforward classification we are interested in exploring fundamental differences between classification and segmentation. We observed that many performance benefits can be achieved simply by using a segmentation model, improving performance along many axes without specialized tooling. Conversely, classification modifications are typically different for different objectives (e.g., methods for reducing reliance on spurious correlations are different from methods that handle class imbalance are different from methods that improve interpretabillity. We believe the fact that segmentation results in many benefits without specialized tooling for each major strength of the proposed approach.
Our contributions. We first analyze why we see performance differences, supporting and helping to explain the empirical results that and others have observed as well as the conditions under which we can expect to see benefits. Next, we implement both previously proposed and new methods for using iegmentation-fr--oee for-classification and compare these approaches across multiple datasets using varying amounts of training data. We perform this head-to-head comparison to build up recommendation of best practices for using segmentation networks in classification problems, including: recommending summarizing function; showing an off-the-shelf segmentor can achieve improved classification performance; showing existing semi-supervised training method can improve performance; and adapting that semi-supervised method specifically to the segmentation-for classification setting. Finally, we aggregate and expand the benefits of switching from classification to segmentation. We expand on benefits by showing improved performance with low class prevalence, rare subtypes, and location spurious correlations, and we discuss these quantitative benefits in conjunction with qualitative benefits like improved interpretability and model evaluation. We use this consolidated evidence make the broader case that many classification problems would benefit from segmentation framing.
A2 Summarizing functions
Below we provide additional details on the summarizing functions we evaluate in the main text.
A2.1 Rule-based summarizing functions
we introduce a rule-based summarizing function using thresholds to transform the segmentation output into a classification label. Here we give further details of that summarizing function. Specifically, in Algorithm [ we give the algorithm we use to compute a binary, image-level label from a probabilistic segmentation mask. Given probabilistic segmentation
In Section 3.1
17

---

Algorithm 2 Compute probabilistic label
Require: arsk E [0, 10,1)xxwxx ,t [0. 1], BinaryLabel E {0,1}.
1:
if BinaryLabel = then
2: PosProbs + Probseeg.MaakkPooSSeeeeee
2
3:
return mean Pos mean(PosProbb))
4: else
5:
NegProbs <- ProbS <
6:
return mean Ceeam(NegProdiee Probs)))
7:
end if
contains the segmentation network output, the pixelwise probabilities indicating where the class of interest was found- we first binarize the segmentation mask using threshold t. At default, t = 0.5. We then sum all pixel values the binary segmentation mask; if there are more than threshold positive pixels in the segmentation mask, we return positive binary image-level label. In this work, we set 7 100.
In Algorithm 2 we provide the method we use to compute a probabilistic image-level label from probabilistic segmentation mask. We use these probabílistic labels compute the AUROC. Given probabilistic segmentation mask, we first compute the binary label using Algorithm I If the binary label positive, we compute the probabilistic image-level label the mean of pixel values that are greater than threshold f. Similarly, if the binary label is negative, the probabilistic image-level label the mean of pixel values less than At default, takes value 0.5.
A2,2 Trained summarizing functions
In Section 3.2 introduce trained summarizing functions to transform the segmentation output into a classification label. We give further details of those summarizing functions here.
Summarizing functions that operate over segmentation outputs. After first training the segmen- tation network, we generate predicted segmentation masks S for all training and validation images. Note that we do not binarize these masks, but retain the pixel-wise probabilities. Then, using the same dataset splits used for training the segmentation network, we train the summarizing function g(-) to take the predicted segmentation mask S as input and produce a class label vector g = ý by minimizing the cross entropy loss CE. y). At inference, the segmentation network first generates predicted segmentation mask for the test image, then the predicted mask fed into the summarizing function which produces class probabilities.
We consider three architectures for trained summarizing functions that operate over segmentation network outputs. The first and simplest architecture is a fully connected layer on the flattened predicted segmentation mask $, which is used to produce class scores that normalized with softmax function.
The next summarizing function we consider is global average pooling layer followed by fully connected layer. We take the output segmentation mask Ś and produce vector with values represent- ing the average of each class's segmentation mask. The fully connected layer then transforms these class averages into scores for each class, which are normalized with softmax function. This method similar in spirit to thresholding as described above but with a threshold Icarned from the values observed in the train set. This global average pooling to fully connected approach previously proposed in [16].
Finally, we consider known image classification architectures summarizing functions. At a high level, the task of transforming segmentation mask into a classification label is exactly an image classification task: does the "image" represented by the predicted segmentation mask correspond to a positive or negative class label? So, we investigate if existing image classification architectures are suited for learning mappings from predicted segmentation masks to class labels. In this work, we consider SqueezeNet [27] and ResNet50 28 as summarizing functions on top of the segmentation output.
18

---

Model
Deep embedding layer name
Shallow emhedding layer name
Modela. wew.laettggentttioo...m.o wtt.mmentttion.nnnnnee..e r
backbone.. 2raxbome.layer22
backbone layerd
Ceneric_ UNet
15
conv_ blockssocs_____ boockks_occlizztioon [2]
conv han_bbocks__in lorccksssoclliitiionn [4]
Table A1: Layer names we pull embeddings from for the trained summarizing functions. These
names are from public networks available in Pytorch.
Summarizing functions on pretrained segmentation embeddings. We next consider summarizing functions that operate over segmentation network embeddings. We first train the segmentation network f()). then store embeddings for all training and validation images. We train the summarizing function g(-) to take the segmentation embeddings as input and produce a class label vector ÿ as output by minimizing the cross entropy loss C' (Ú, y). inference, the image passed through the pretrained segmentation network and embeddings from the segmentation network are stored. These embeddings are then processed with the summarizing function to produce class probabilities.
We consider two types of embeddings, shallow and deep. The intuition for exploring different embedding depths is that different depths contain different information about the input image, which may vary in their usefulness for the classification task. We give the name of the layer in the Pytorch model that we pull embeddings from in Table AT We also consider two different architectures to classify the segmentation network's ' embeddings. First, we define a simple network architecture consisting of global average pooling layer followed by three fully connected layers, which produces class scores that we normalize with softmax function. Second, we consider more complex classification head proposed previously in (20] The classification head takes the segmentation embeddings as input, then processes the embeddings with three blocks of convolution- batch normalliztton-ReeLU-maiaai ReL.U-mxiiuum pooling operations. The convolutional kernels are size 3x3 and each produce 16 channels; the maximum pooling operation operates over 2x2 window. The resulting embedding is flattened. Then, we process the flattened embedding with two blocks of linear transform-bater dropout layer has a probability of dropout set to 0.25. Finally, the resulting feature map is processed with single linear layer produce the final output, vector equal in length to the number of classes, which can be normalized with the softmax function.
Hrmmalzzaioo-RRllU-drorro
The linear layer produces 100 features and the
A3 Supporting analysis
A3.1 Proof of Proposition
Proof. (x(ll 1)
Pr(3 a.1. 83 - 1,
Esp.... 52.5...555..... P((S.....S)))PPCC (1.)..,8))
P a.l. 8,
(A1)
5 8y
Pv(3 a.t. 8y -)
E8 9 2,21
8.) ((X.)))))))
(A2)
Pr 8, =1
Es_
1, Pi
1) ES
Pr(Se = -
PH a.. 1)
Pr(Ss = 1) =1 P) Es 1 Pr(S2
8_k)
(A4)
Pr j. 1)
XP
P) 0).
where P:73 Px(8g=1)
2y= Therefere,
writen probutiiity of pixel value given the image-level label, Pr(xaly 1),
tire of the probuhility of the pisel value given the piset-level libel, Pr(Xg X2 [8
1) and Pr(Xa = 0). Next, we can write Pe(Nglu Nglv = 0)
Pr(Xx1S1
Sg = Pr(X&|8E = by our this generating process.
Putting together Pe( (xxyy = and Pr X byy= the right hand side of the incquality in Propocition J
Dii PRr,Pe(Xyyy 0)] Px(X, 1))
< (Pr(X 0)] Pe(Xy]S 1)) (S PX(A (St, 1) (2) 0))
(A6)
(P Ss, O] 0))
(A7)
ADa (Pr(X, 15. = iii 15 1)) $ Da (P. 0) Pe(Xx18 1))
(A8)
where we have ubtained the desired inoquaiit due convexity the KL. divergmoe. Note that by keeping A the incquality we can make the stronger claim
Pr(y= 1) 15 15 Pr(S, (Pr((el 0)] Pr(xal 1)).
(A9)
19

---

Variable name
Destription
Default valae
Roby
Number of total objects in the image
50
Remacn
of truining images
Roa!
Number of validation images
20, 000
Pleat
Number images
Ctarget
Color of the target object
niny blue
Tearget
targer object
Starget
Shape of the target object
cirote
hearacty
Possible comnlinstes target objject
learaety
Possible courdinates for the target object
unif (0, 224)
Cepurtinns
spurious object
pink
Tepurttou
Radins of the spurious objecct
Pepurtous
Shape spurious object
square
lepuriouxs
Possible courdinates for the spurious object
uni 2242
lepartoucy
coardinases spurious object
unz/ {0, 2243
Bbaokground
Colors of the background objocts
E (yellow, green, pink, red, orange, purple, eight shades blue)
Ponckground
Radi of the background objects
Fouckground
Shapes of the background objects
(circle, restangle, diamend, octagon, square) unif (0, 224}
bachground
Possible courdinates for the background object
buskgroundy
cocrdinates background object
umi 224)
Ppos
Probabuity of positive class tie. class balance). At consains positive sumples; Ppax the datasel contains only positive sumples.
the dasaset
0.5
Peper
Strength of spurious conrelation. At only co-oecurs the target co-occurs with target obyect, appears every target obyect abseat, = 0.5, the spurious object his no depandesee on the tager objecr.
1 the spuñiouss object always and spurious object
0.5
Payue
Table A2: Variables that define synthetic dataset. All experiments from the default values unless specified otherwise.
with datasets constructed
20

---

Experiment
Adjusted varable
Number training images
Perasnn varied
100,000
Class balance
Class balance is changed in the train data by varying Ppox E [0.01, D1.,1 1). Note that class balance in the valičation and tedt data chunged from default.
Size bios
Instead of all larget objects having the same rudius, we draw madios from discribttion incressingly bizsed lowards larger malii Note that the valisanion and lest datinnes only show small objects.
Spurious comelaaion
Spurious
varying Papur changed to 38. Note that spurious correlition strenghh in validation test data is changed from defirilt.
spength changed
spurious
Locasion bias
The location of the object is changed in the train data varying leargety and fearpetyy from unif (112. 112) ~ unif (0. 224). Note that location distributions are not changed from default in validation data.
Turget size
The size of the target and background objects are swept from to change Terras 100000
Number of objectss
The di ficulty of the classifiction task changed by sweeping the number of unique background objects from to 10: objects nooy remains 50. We change Aerain
Table A3: Experiments we run on synthetic datasets. For each experiment, we adjust the variable values indicated above (leaving the other variables default) and assess how classification and segmentation-foo-clssiicce performances are impacted.
A4 Synthetic dataset and experiments
We conduct experiments on synthetic dataset to evaluate how dataset and task characteristics impact the performance classification and cimeentttioo-oor-clafffaaiiii models. Here we provide additional details on our synthetic dataset and experiments. The synthetic dataset is visualized in Figure
A4.1 Synthetic dataset
The synthetic dataset is made of images containing gray background and many objects of varying shapes and colors. The task is to classify if given image has a certain colored shape present. Each synthetic dataset is defined by the set of parameters given in Table A2
To construct synthetic image, initialize gray background of size (224, 224, 3). We then draw the class label y ~ Bernoulli(Ppoa). We sample the spurious label s according to the strength of the spurious correlation Pspuar Specifically, we define and sample the random variable If match = 1, we set the spurious label y, else, we set 8 = = If 1, we create the spurious object specified by (Cspurious: I'spurious: Espurions) the object's color, radius, and shape-and place the spurious object at the coordinates drawn from the distributions - We follow similar processes to generate and place the Roby yy- s background objects and target object. We place the target object last to ensure it is placed in the foreground.
malch ~ Bernoulli(pspur
To run experiments the synthetic dataset, vary dataset variable, train the classification and wwweenttttiooooo--llasifiiiiecie
models, and evaluate performance. The variable we adjust for each experiment is described in Table A3 Additionally, visualize the training dataset for each experiment in Figure AI
A4.2 Training details for synthetic datasets
We train ResNet50 to classify the synthetic data and ResNet50 with fully convolutional segmentation head to segment the synthetic data. We train the classifier and segmentation networks by minimizing the cross entropy loss between the predicted and ground truth labels. We do not use data augmentation on the synthetic data. We train each network with an Adam optimizer and learning rate of le-4, tuned from [le-6, 1e-5, le-4, 1e-3]. We checkpoint the networks the maximum balanced accuracy on the validation set. We use the threshold bassed summarizing function.
21

---

Experiment samples training
Experiment
Experiment
balance
training
Synthetic dateset visualization Positives contain navy blue circle
Experiment Strong location
Experiment Increasing targat
Experimant Increasing background varfance
comelation the
Figure Al: Visualization of the synthetic dataset task and experiments. The task is to identify if the image contains a navy blue circle. Each experiment changes characteristic of the training dataset observe how classification vs. segmentaaionn-for-llliffeeiiiii performance is impacted.
22

---

Classification supervision
Seg. supervision, with subsampling
Seg. supervision, without subsampling
Input image
0
Figure A2: Visualization of the annotation quantity experiment. The input image is grid of colored squares; classification supervision receives class label per image; segmentation supervision with subsampling receives one class label per colored square; segmentation supervision without subsampling receives class label per pixel.
Balanced accuracy
0.9
0.8
Class
Sep class, with subcompiing Seg class without sussampling
1000 2000 3000 4000 5000 No. training images
Figure A3: Results from annotation quantity experiment. This graph shows that providing segmenta- tion supervision for every pixel the most sample efficient, though providing supervision for every colored square still improves sample efficiency compared providing one label for the entire image. This graph supports that is both annotation quantity and the differences in class distribution leading segmennatiion-oorcaasffcccaiaai improved performance in the limited data regime.
A4.3 Additional synthetic experiments
In the main text, focus the divergence between positive and negative classes why classification and segmentation perform differently. We do not explicitly model the benefit from denser annotation during training, instead taking given that the denser annotation should help segmentation learn with fewer images. To show that annotation quantity is not the only component contributing to segmentation'ss improved performance, we run the following experiment.
We study a simplified setting visualized in Figure A2 Specifically, we divide 112 x 112 grid into sixteen isotropic squares. The task is identify if any of the squares navy blue. Class balance is set to 0.5, meaning an image is as likely to have the target class as not. The remaining square colors drawn randomly. In this simplified setting, still expect divergence between segmentation classes to be greater than divergence between classification classes, following our analysis in Section
Classification supervision is applied at the image-level, per usual. To understand the influence of annotation quantity, we evaluate we provide segmentation supervision with only one pixel per square. This level of supervision results in 16 annotated pixels per image, but equal number of positive class annotations as the classification supervision. Second, we provide segmentation supervision over all pixels (this the usual segmentation-for-chee clrssssssiiiaation setting), in effect providing 12,544 annotated pixels per image and 784x as many positive class annotations. The supervision for each of these models is also visualized in Figure A2
performance in two settings. First,
In effect, this experiment separates the impact of just divergence vs. divergence+higher quantity of (positive) annotations. We sweep the number of training images and report results in Figure A3
23

---

We see that segmentation without subsampling outperforms segmentation with subsampling outper- forms classification. This result confirms that both the denser annotation and greater divergence during training benefit aigmentttion--oorclasificiiiiiii
performance.
A greater number of annotated samples should improve most MI. tasks, as has been extensively studied before. We focus on characterising the divergence as it is specific to segmentation-for classification and less studied in prior work. Future work may study the relative benefits of these two aspects of supervision.
A5 Real datasets and experiments
We run experiments on three public medical imaging datasets, described below and fully detailed in their original studies. The medical datasets visualized in Figure 4
A5.1
Medical imaging datasets
CANDID: classifying pneumothorax in chest x-rays 29
Classification task. Our aim is to classify whether a chest X-ray contains a pneumothorax: ground truth classification labels are provided with the original dataset.
Segmentation target. Our aim is to segment pneumothorax when it appears in chest x-ray; ground truth segmentation masks are provided with the original dataset.
Dataset. This dataset contains 19,237 chest x-rays. The images are two-dimensional. one-channel images of size 1024 x 1024, and we normalized each image to the range [0, 1]. We split this dataset randomly into 60% training images, 20% validation images, and 20% test images. In this dataset, 16.6% of images are positive for pneumothorax.
ISIC: classifying melanoma from photographs of skin lesions
30
Classification task. Our aim classify whether a skin lesion melanoma or not; ground truth classification labels are provided with the original dataset.
Segmentation target. Our aim to segment the skin lesion when is melanoma; ground truth segmentation masks are provided with the original dataset.
Dataset. This dataset contains 2750 images of skin lesions. The images are wwo-dimensional, chrnnechannel images that we resized to 224 224. We use the splits provided by the ISIC challenge organizers, resulting in 2000 training images, 150 validation images, and 600 test images. 18.9% of images positive for melanoma.
SPINE: classifying cervical spine fractures in CT scans (RSNA 2022 Cervical Spine Fracture Detection Challenge)
Classification task. Our aim to classify whether there is fracture in the cervical spine; ground truth classification labels are provided with the original dataset.
Segmentation target. Our aim is to segment the vertabrae in which the fracture occurs. Unlike the previous two datasets, the ground truth segmentation masks were not available in this dataset for all training images. Instead, we used an off-the-shelf CT segmentation network [11] generate segmentation masks of the vertebra. For each positive classification label, we create a corresponding segmentation mask which delineates the vertebrea in which the fracture occurs. This use case shows how off-the-shelf models can be used convert some classification problems into segmentation problems without any additional labeling.
Dataset. This dataset contains 2018 CT scans of the spine. The images are diree-dimessional volumes with one channel. We applied CT bone window then normalized each scan to the range [0, 1]. We resized each axial slice to 128 x 128 voxels using bilinear interpolation and resampled the axial dimension to 128 slices around the spine, resulting in 128 x 128 x 128 image volumes. 47.6% of these seans contain cervical spine fracture. We randomly split this dataset into 60% training, 20% validation, and 20% test.
24

---

Datrset
Architoctuee
Learning
Weight docey
Segmentation
ResNe50, SEResNet50, DeepLabyS,
1e-6, 1o-5, 1o-4
1o-6, lo4, 16-2,
Segmestation
SPINE
ReiNet50 3d, UNet Sc, NNoNet 3d
1e-6.1e5 le-5. 1o-4,
le.3
Ic-h, 1e-4, 1o-2,
Segmentation
ISAC
RoNNe50 SFRecNarSi, Dexpl SbyS, UNer
1o4,, los5.
1o-4,
le-3
1e-6, lo-4. 16-2.0
Clossification
CANDID
ResNecto, SEResNer50, DenseNet1E1, ConvNext
1e-5,
1c-4,
164, Io-4.
Chesification
SPINE
5e UNe encuder. NNaNet encoder
1e66, le-5. lo-4.
le-3
le-6. lo-. 16-2.
Clociliiation
ISKC
Ro SFResNet50, DenseNet121. 21, CorvNest
10-6, Ie-5. 1o-4,
le-3
Ie-6, lo-4. To-2.
Multitask
CANDID
ResNet50
1o-6, lo-5,
Tc-4,
le-3
16-6, Io-4, 16-2.
Muliuokk
le-5,
1c-6.
Muhiiakk
ResNe150
16-6, 1o-5, To-4,
1c-6, lo-4. 1o-2.
Table A4: Hyperparameters tested for the backbone networks; chosen hyperparameterrs
bold.
CANDID
SPINE data)
ISIC data))
CANDID (high
SPINE (high
ISIC
Summarizing Fanction
(high
outgat Seg, uulpul - avg pool folly connected layer
connected layer
le-5
1e-h
lo-3
1e-6
1e-4
le-2
le-2
le-3
le-2
le-2
le-4
Seg. outpui - SqueezeNet
le-5
le-5
le-5
1o-5
le-6
le-6
Deep seg emhastdings - simple hend
Te-3
1e-3
1o-3
16-3
16-2
le-3
Shullow veg. embesklings
simple Deep seg embodlings - complex hend
le-4
Ie-4
1e-2
lo-3
164
16-2
Shallow seg, embeekinngs complex head
Ie:3
1e:3
le-1
le3
le:3
le.2
Table A5: Hyperparameters selected for training the summarizing functions.
A5.2 Training details for medical imaging datasets
In Table A4 we list each hyperparameter we evaluated for the backbone networks trained on the medical datasets. We chose hyperparameters that maximized the validation AUROC. Chosen hyper- parameters are in bold.
For the trained summarizing functions, we only tuned the learning rate. We evaluated learning rates [1e-6, 1e-5, le-4, 1e-3, 1e-2] for each summarizing function and chose the learning rate that maximized the validation AUROC. In Table A5 we list the chosen learning rate for each summarizing function and dataset.
To evaluate performance, we report mean AUROC over the test split of each dataset. We binarize the probabilistic labels to report additional performance metrics by setting the operating point on the ROC curve that maximizes the Younden Index. We compute 95% confidence intervals using bootstrapping (1000 iterations). We conduct our experiments using Pytorch Lightning (Pytorch version 1.9.0, Lightning version 1.5.10) [60 61].
A5.33 Ablations on model capacity
To report results in the main text, we tested multiple architectures for each task and dataset and selected the best-performing architecture for both segmentation and classification (Table We did this to give each task and dataset the best performance, as it's not clear the best architecture for segmentation on a given dataset is the same that is best for classification.
However, segmentation networks are often higher capacity than classification networks due to their dense, pixel-wise outputs and corresponding decoders. To show that it is not simply difference in model capacity that is leading to the observed performance differences between segmentation and classification, we perform the following ablations.
First, we use the same backbone for both segmentation and classification (Table A6 We use Resnet50 the 2D backbone as it is common backbone known to be useful for classification and segmentation. The Resnet50 classification head is a linear layer, while the Resnet50 segmentation head is the head described in the Fully Convolutional Network paper [62]. We use the UNet from the nnU-Net paper [59] for the 3D backbone. We see that segmentation shows the same or greater performance enhancement than what appeared in the original paper when using these standardized backbones. We further note that all of our synthetic experiments were run with a standardized backbone between classification and segmentation.
However, even when using the same backbone, segmentation models still typically have more param- eters due to the convolutional decoder head. We next provide results comparing performance using
25

---

Backbone
Method
AUROC
Classification
0.80
CANDID
ResNe150
Seg-for-class
0.91
Classification 0.56
ISIC
ResNet50
Seg-for-class
0.67
Classification 0.60
SPINE
nnU-Net
Seg-for-class
0.62
Table A6: Performance comparison when classification and segmentation models are trained with the same network backbone. For the SPINE task, the nnU-Net encoder is used for the classification task.
Backbone
Method
AUROC
ResNet101
Classification
0.82
CANDID
ResNett0
Seg- Seg-for-class
0.91
ResNet101
Classification
0.59
ISIC
ResNet50
Seg-for-class
0.67
Table A7: Performance comparison when classification has higher-capacity model (i.e., contains
more parameters) compared to segmentation.
a higher capacity classification Mondll-ResnetlOn) which has more parameters than the Resnet50 segmentation model-ipg in Table A7 Again, we improved performance with segmentation.
Finally, we emphasize that we matched training procedure for classification and segmentation networks, including matching the input data, augmentations, and codebase (which standardizes model checkpointing, loss function, hyperparameter tuning, etc.). Together, these results show it is not just model capacity training procedure that lead to segmenatioon-frrclaasfiiccaal improved performance.
A5.4
Additional performance metrics
In Table A8
we report additional performance metrics using standard classification and segmentation- for-classffrcation (including semi-supervised methods) on the CANDID dataset.
A5.5 Results on a natural image dataset
To show the proposed method extends to natural image dataset, we include results classifying dog and cat breeds in the Oxford Pets dataset in the limited data regime (50 training images per class). We trained these models using ResNet50 architectures and the rule-based summarizing function. We observe segmentation-fir -or--fooso--icatiiiiee achieves AUROC of 0.96 while standard classification achieves an AUROC 0.81. These results show segmentation cor-classification improves average AUROC by 18.5% natural image dataset, mirroring our results medical imaging datasets.
A5.6 Results on multiclass datasets
While we focus on single-class tasks in the main text, we show here that segmentation-for classification can directly be used for multiclass tasks. Instead of having binary mask output from the segmentation network, user simply needs to specify multiple output channels. First, we note the Oxford Pets dataset contains classes dog and cat breeds; as shown in the previous subsection, we see segmentation outperforms classification by 18.5% on this multiclass dataset. Further, we extend one of our medical imaging datasets to the multiclass setting and perform class classification on ISIC, classifying lesions as benign nevi, melanoma, or seborrheic keratosis. which achieves balanced
Again, we see improved performance with pnreeetttiion-ooolllssieeeeeeeee AUROC of 0.61 compared to classification's 's 0.50 in the limited data regime.
26

---

images with and clas labels
images with unly class labels
imagos with labels
AUROC
Hal. Accuracy
Recall
Precision
Specificity
Classifcation (limuted)
198
0.74
(1.76)
(0.66
0.68
(0.56-0.4))
0.40
(0.29-0.34)
0.31
(0.44-0.77)
0.76
Seg-for Seg-for-class (limited)
198
0
(0.84 (1.88)
86
(0.76 0.50)
(0.71-0.78)
0.75
(0:39-045)
0.2
(0.00-082)
0.81
Seg class
198
0
3780
01.99
(0.70
0.81
(0.7) -0.71-0.78)
0.75
(0488-055)
0.51
(0.56-088)
0.87
Seg for class, boosted semi stup
198
3780
(0.89 (.92)
(0.81 (.84)
(0.70 0.77)
(0.59
0.62
(091 -193)
Classification (abundant)
3978
(0.89-
0.90
0.82
0.50
(046--052))
0.49
(084--186)
0.85
Seg-far-class (abundear)
3978
(0.96 0.97)
0.96
(0.8) 0.90)
0.90
(0.56 0.93) (0.3
0.89
0.73
0.67
(0.91 -093)
0.2
Table A8: Comparison of classification vs. semi-supervised
perfor-
mance the CANDID dataset in the most limited and abundant training data conditions. These models were trained with a balanced dataset. We report the average of each metric over the test set with 95% confidence intervals in parentheses.
A6 Limitations
A limitation of this work includes the maximum sizes of the datasets. With much larger datasets, we expect overall classification performance to improve. However, the dataset sizes we use this manuscript (2,000-20,000 for medical datasets; 100,000 for synthetic dataset) are not uncommon medical imaging dataset sizes, particularly for new applications or datasets specific to an institution. Further, expect many benefits hold in the very large data regime (e-g., robustness spurious correlations, performance on very rare subsets, human assessment, and model evaluation). As an additional limitation, the seggnennttioonfoo-lllsffieeeee to abnormalities that can be segmented. Images for which even experts do not know which regions directly correspond the class labels (e.g., some histopathology images 63 have less obvious segmentation targets. In these situations, classification in combination with methods like GradCAM can help with feature discovery, for which segmentation-fort colcssssiiiiiiiaoooon may be less well-suited. Similarly. classification labels which are more global in nature (e.g., age prediction from radiographs, atrophy of the brain in a head CT) not have clear segmentation targets and may not benefit as much from the segmentation signal.
methods explored here are most applicable
A7 Broader impacts
This work focuses on automated medical image analysis. Automated image analysis algorithms have the potential to support radiologists by providing decision support, improving analysis efficiency, and improving analysis reproducibility. The work presented here helps to make progress towards these benefits by improving the performance of automated medical image analysis algorithms; improving the quality of model evaluation; and providing location information to the radiologist to facilitate human assessment. However, medical image analysis is low failure tolerance application. The work presented here is meant as a contribution to shift the way we consider segmentation vs. classification algorithms in building medical image analysis tools; the specific neural networks trained here meant for clinical use.
27


Reviews:
[{"review_id": "zVw1LjYtMe", "reviewer": "Reviewer_D8HP", "summary": "In this work, the authors explored the applications of deep learning in radiology, specifically focusing on image classification and segmentation tasks. The authors investigated the performance differences between classification and segmentation models on the same dataset and task using an information theoretic approach. They proposed a method called \"segmentation-for-classification\" that utilizes segmentation models for medical image classification. The authors compared this approach with traditional classification on three retrospective datasets, consisting of 2,018 to 19,237 samples. Through their analysis and experiments, they highlighted the benefits of segmentation-for-classification, including improved sample efficiency, better performance with fewer labeled images (up to 10 times fewer), particularly for low-prevalence classes and rare subgroups (up to 161.1% improved recall). They also demonstrated improved robustness to spurious correlations (up to 44.8% improved robust AUROC), as well as enhanced model interpretability, evaluation, and error analysis.\n", "questions": "As shown in Figure 4, why is there such a significant difference between segmentation and classification on small datasets? Are the model parameters and training processes for segmentation and classification strictly controlled to be consistent?\n\n", "limitations": "Just as mentioned in the weakness section, the method proposed in the paper is limited to binary classification.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Through theoretical analysis, the benefits of using segmentation for learning classification datasets are validated, and extensive analysis and validation are conducted on small datasets.\n", "weaknesses": "1.The proposed method is not specifically designed for medical images; it is a general method but lacks validation in some common settings.\n2.The method proposed in the paper is limited to binary classification requirements, which limits its scalability.\n3.The method does not introduce a new module specifically designed to extend segmentation for classification tasks, lacking novelty.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and questions about our paper. Below, we respond to each of your points to answer your questions, provide new results, and describe how we are updating our submission in response to your comments. We would be happy to answer any additional questions or hear more suggestions during the discussion.\n\n\n*Q1: The proposed method is not specifically designed for medical images; it is a general method but lacks validation in some common settings.*\n\nIn the global response, we clarify why we focus on medical images as a strong motivating application space. That being said, we agree with the reviewer that the methods can extend to natural images and show segmentation-for-classification improves average AUROC by 18.5% on a natural image dataset in the limited data regime. We will add these natural image results to the Appendix.\n\n\n*Q2: The method proposed in the paper is limited to binary classification requirements, which limits its scalability.*\n\nThe method is not limited to binary classification; it can directly be used for multiclass settings. Instead of having a binary mask output from the segmentation network, a user simply needs to specify multiple output channels. We provide new experiments on multiclass data in the global response, showing segmentation-for-classification improves average AUROC compared to traditional classification by 18.5% in a natural image multiclass setting and by 22.0% in a medical image multiclass setting. \n\nTo make sure this misunderstanding does not arise for future readers, we will describe the multiclass case in Section 3 and include the multiclass results in Section A5.\n\n\n*Q3: The method does not introduce a new module specifically designed to extend segmentation for classification tasks, lacking novelty.*\n\nOur contributions are in explaining why and when we see performance boosts using segmentation-for-classification (supported by our theoretical analysis), providing best practices for using segmentation-for-classification, and showing the number of tradeoffs between a traditional classification vs. segmentation-for-classification task framing. \n\nTo develop best practices, we do test eight modules designed to extend segmentation for classification, but find that the simplest deterministic method works best (Figure 5). We believe this simple solution will help boost the usability of segmentation-for-classification, as practitioners can implement it with no extra computational cost or tuning costs as is required by more complex modules. Additionally, we do propose and evaluate a boosted semi-supervision method (Section 4.2.2), which has not been shown before and demonstrates how to adapt semi-supervised segmentation to the segmentation-for-classification setting.\n\n\n*Q4: As shown in Figure 4, why is there such a significant difference between segmentation and classification on small datasets?*\n\nOur analysis in Section 2.2 suggests this is because segmentation has a higher KL divergence between classes. Since the data is more separable using a segmentation task framing, the decision boundary between the two classes can be learned with fewer data points—leading to improved performance in the limited labeled data regime.\n\n\n*Q5: Are the model parameters and training processes for segmentation and classification strictly controlled to be consistent?*\n\nThank you for bringing up this point, we agree it is important to clarify. Please see the global response for details on this question and new results on model capacity and performance. We will add these details to the revised manuscript clarifying this point for future readers in Section A5.\n"}, {"review_id": "CBlDYrKw83", "reviewer": "Reviewer_rmFZ", "summary": "The paper describes a set of insights obtained when using segmentation networks for a classification task. Classification was the task of choice due to issues with obtaining appropriate segmentation labels. However, with wider availability of datasets with appropriate labels, this is no longer the case. To facilitate consideration of using segmentation or classification networks, the paper presents a set of best practices and tradeoffs. ", "questions": "Have the authors evaluated the effect of network size (# of parameters) vs performance? It seems like a smaller classification network and a larger segmentation network should in theory learn the same information. \n\nSeveral weaknesses of classification networks and associated strengths of segmentation networks are discussed (e.g., in Figure 4, segmentation nearly always outperforms classification). When is the reverse true? Would there be cases where segmentation networks more easily overfit? \n", "limitations": "The paper presents several interesting insights into performance of classification via segmentation vs classification networks. However, I am not sure if the presented conclusions will generalize to other datasets/modalities. The paper could do a better job in clarifying the extent of its contributions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The manuscript is very thorough.\n\nAn important problem and best practices to solve it are discussed. In particular, the insights benefit the study of smaller datasets, rare subgroups, and robustness, all of which are important concerns in medical image analysis. \n\nThe supplementary material is very thorough and includes analysis of a synthetic dataset and three real datasets (X-ray, CT, skin). \n\nFinally, some theoretical analysis evaluating separability of positive and negative segmentation and classification classes is provided. \n\nI believe the analysis provided in the paper would be helpful to those trying to design an analysis task on medical image data. \n", "weaknesses": "After reading through the manuscript and supplementary material, I believe that some of the contents of the supplementary material (e.g., visualizations, choice of datasets) and the description of the pipeline can be summarized in a more detailed overview Figure (instead of Figure 3). While a lot of work (e.g., comparisons across networks) has been put into the project, not all of it is clearly explained in the main manuscript.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time reviewing our manuscript and for your suggestions. We were glad to hear you thought the work was thorough, addresses an important problem, and would be helpful to future readers. We were also glad to receive your comments and suggestions for improving the manuscript. Below, we answer your questions and describe how we have improved our submission as a result of your comments. We would be happy to answer any additional questions or hear additional suggestions.\n\n\n*Q1: After reading through the manuscript and supplementary material, I believe that some of the contents of the supplementary material (e.g., visualizations, choice of datasets) and the description of the pipeline can be summarized in a more detailed overview Figure (instead of Figure 3). While a lot of work (e.g., comparisons across networks) has been put into the project, not all of it is clearly explained in the main manuscript.*\n\nThank you for this suggestion. We had trouble with the page limit and as a result placed a lot of content in the Appendix. We understand there is an additional page allowed for the camera ready. With that in mind, we plan on making the following changes and are open to additional suggestions:\n- Bring details from Section A2 into the main text in Section 3, including:\n     - Algorithms 1 and 2 (combined into one algorithm box).\n     - Description of summarizing function architectures, including additional descriptions in Section 3 (moved from their current location in Section A2) and depicting the summarizing functions in a new Figure 3. If the reviewer had specific content in mind they think would be useful to include in a modified Figure 3, we would appreciate their input!\n- Expand the “Training” subsection in Section 4 to include model architectures, loss function, and optimizer (currently in Section A5.2).\n- Move Table A6 to Main Text.\n- Expand the Summary of Related Work in the Introduction to be more thorough, while keeping the full Related Works in the Appendix. \n- To make space for these changes, we can compress Sections 5 and 6 as well as remove Figure 2.\n\n\n*Q2: Have the authors evaluated the effect of network size (# of parameters) vs performance?*\n\nThis is a good question and we appreciate the suggestion to clarify this point. Please see our new results to answer this question in the global response, which we will include in the Appendix of the revised manuscript.\n\n\n*Q3: Several weaknesses of classification networks and associated strengths of segmentation networks are discussed (e.g., in Figure 4, segmentation nearly always outperforms classification). When is the reverse true? Would there be cases where segmentation networks more easily overfit?*\n\nThis is an interesting question and one we investigated, though we did not find (either empirically or theoretically) settings within our defined scope (set in Section 2.1) in which segmentation-for-classification achieved worse performance than traditional classification. It is true that with very small datasets segmentation-for-classification models overfit, but the traditional classification networks were also not able to learn a more performant decision boundary in these settings. \n\nOutside of the scope we set, for example for tasks which require classifying the global image (e.g., classifying imaging modality), classification may achieve higher performance out-of-the-box than the segmentation-for-classification networks we use in the paper. Future work may investigate how to leverage segmentation networks for these global tasks; we will add discussion on this direction in our future work section. The future work is currently in Appendix A7, but we can move the future work paragraph into the main text’s Conclusion.\n\n\n*Q4: The paper presents several interesting insights into performance of classification via segmentation vs classification networks. However, I am not sure if the presented conclusions will generalize to other datasets/modalities. The paper could do a better job in clarifying the extent of its contributions.*\n\nWe have included new results in the global response showing that the proposed method can be directly used in additional settings, including natural images and multiclass settings. We will add the multiclass and natural image results to the Appendix of the revised manuscript. \n\nTo summarize, through our original experiments and this rebuttal, we have shown that the presented conclusions extend to different imaging modalities (RGB images, CT, x-ray), natural and medical images, 2D and 3D images/networks, and single class and multi class tasks. We have also explored trends thoroughly with synthetic datasets (Figure 4), showing settings in which performance differences emerge. Further, our theoretical analysis in Section 2 helps explain and support why one should expect the observed benefits to transfer to other datasets. Please let us know if there are additional generalizations or clarifications you are interested in discussing further.\n"}, {"review_id": "l6oDPU6UYn", "reviewer": "Reviewer_RFD8", "summary": "This paper provides an intriguing and somewhat disruptive approach to medical image classification tasks. It implies that due to advancements in weakly-supervised, self-supervised, and semi-supervised segmentation techniques, the historical inclination towards image classification due to ease of training and label acquisition might be reconsidered. The authors argue for a shift towards segmentation, traditionally a more complex task, and demonstrate this by implementing \"segmentation-for-classification\" models. Using three retrospective datasets, they show these models outperforming traditional classification techniques in sample efficiency, performance on low-prevalence classes and rare subgroups, robustness to spurious correlations, and interpretability. However, the authors limit their consideration to tasks where the class of interest can be localized.", "questions": "1. How do the authors justify the much higher cost and time commitment associated with pixel-level annotation as opposed to class-level annotation? Given the substantial difference, could the authors provide insights on the practicality of their approach in real-world scenarios?\n2. Is it possible that the improved performance of the segmentation models over the classification models is simply due to them being inherently more powerful, possibly because they have more parameters? Could the authors elaborate on this aspect and discuss how they ensured a fair comparison between the two models?\n3. Could the authors elaborate on the decision to relegate critical content to the appendix? Would it not improve the logical flow and comprehensiveness of the paper if key aspects such as related works, algorithm details, and results were included in the main text? Conversely, could parts of the discussion on the differences between segmentation and classification be moved to the appendix to make space for the aforementioned content in the main text?\n4. Have the authors considered re-running their results multiple times to ensure they are reproducible?\n\n", "limitations": "The authors acknowledge their consideration only for classification/segmentation tasks where the class of interest can be localized. They do not address situations where the entire image must be evaluated, which is a significant limitation. Additionally, while the authors advocate for segmentation, they fail to adequately address the issue of pixel-level annotation's increased complexity relative to image-level annotation.\n\nThe paper does not discuss any potential negative societal impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper presents an innovative idea that can potentially change the current paradigm. If the results withstand, this reviewer sees a strong argument in saying that all existing segmentation models can easily be turned into powerful classification models by adding a small g() function at the end.\n* The simplicity of the language used in the paper aids understanding.\n* Figure 2 does an excellent job of visualizing the difference between classification and segmentation problems in the medical domain.", "weaknesses": "1. The most important issue this reviewer has discovered is how readily the authors dismissed the cost of providing pixel-level annotation. The difference between pixel-level and image-level (i.e. classes) annotation can be orders of magnitude. For example, in the study, this reviewer has been part of, an expert spent 279 s on average to produce pixel-level annotation for a single image, while it took only 2 s to generate a class label. Therefore, even a need to produce pixel-level annotations for even an absolute minority of images may be devastating. So far, from the literature and experience, it seems that such data is still needed, albeit in smaller quantities.\n2. It is also important to point out a worry related to the study's design. In this work, the performance of augmented segmentation models is compared to the classification models, but no apparent effort is taken to ensure that the results obtained are not due to segmentation models being more powerful (which is often the case). Even the simplest segmentation architecture with the most naïve thresholding may have more parameters than the corresponding classification model. This is important to comment on in the main paper.\n3. This brings another important observation - a lot of critical content, such as related works and algorithm details for summarising the probability maps, how multi-layer networks were trained and what architectures were used, the results for 4.2.2 etc., are relegated to the appendix, while arguably less crucial content occupies the main body of the text. This hampers the logical flow and comprehensiveness of the paper. For example, the discussion on the inherent differences between segmentation and classification, though interesting, feels redundant as it seems obvious that different methods would yield different results. This space could be better utilized to expand on related work and other methodological bits mentioned above. Section 5 can be greatly compressed, as it seems repetitive and less important.\n4. Most figure captions are insufficiently informative, requiring the reader to refer back to the main text. Figure captions should be independent of the rest of the text.\n5. The presentation of the results can be expended, with key information (e.g., number of true positives and false negatives) at this point missing.\n6. The last two sections (4 and 5) feel more rushed and less systematic, reducing the overall readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your careful read of our paper and many suggestions, which strengthened our submission. Below we respond to your points and describe how we updated our manuscript in response. We would be happy for further discussion or to answer any additional questions.\n\n*Q1: How do the authors justify the much higher cost…associated with pixel-level annotation?*\n\nWe agree with you, per-image annotation cost is a major difference between classification and segmentation. Our goal with this paper is to better flesh out the decision space between segmentation vs. classification—it is not only the annotation cost that is different, but fundamental network behavior is different. As we show, segmentation-for-classification can lead to more performant, robust, and interpretable models. In clinical settings, one may prefer more performant or reliable models despite the added annotation cost. \n\nWe do explore ways to ameliorate the annotation cost. We show in Section 4.2.2 that by using semi-supervision methods, we can use unlabeled data and <10% of the training data labeled with segmentation masks to achieve the performance of a model trained with 100% of the data labeled with classification labels. Further, we believe the landscape of image labeling is changing. In the last six months, the Segment Anything Model [1], SEER [2], and medical-imaging-specific extensions [3, 4] have been published. This changing landscape is part of the motivation for this submission: we want to consider the tradeoffs of segmentation vs. classification as tools that make labeling less burdensome become increasingly available. \n\nThat being said, we do not mean to dismiss the high annotation cost required to produce segmentation labels. To address the reviewer’s concerns, we will adjust paragraphs 2 and 4 in the Introduction to more carefully position the paper with respect to the labeling burden of segmentation. Further, we will expand the discussion of labeling cost in Section 5 with more space given to comparative labeling burdens per image. \n\n[1] Kirillov, A., et al. (2023). Segment anything. arXiv:2304.02643.\n\n[2] Zou, X., et al. (2023). Segment everything everywhere all at once. arXiv:2304.06718.\n\n[3] Butoi, V. I., et al. (2023). Universeg: Universal medical image segmentation. arXiv:2304.06131.\n\n[4] Wang, C., et al. (2023). SAM-MED: A medical image annotation framework based on large vision model. arXiv:2307.05617.\n\n*Q2: Is it possible that the improved performance of the segmentation models over the classification models is simply due to them being inherently more powerful, possibly because they have more parameters?*\n \nThank you for bringing this up, it is important to clarify. Please see the global response to all reviewers where we include two new tables showing that the improved performance is not simply due to increased model parameters. \n\n*Q3: Could the authors elaborate on the decision to relegate critical content to the appendix?*\n\nThank you for the feedback. We had trouble fitting the paper into the 9 page limit, and as a result moved a lot of content to the Appendix. As we understand, there is an additional page allowed for the camera ready. With that in mind, we plan on making the following changes based on your (and other reviewers’) feedback and we are open to additional suggestions:\n- Bring details from Section A2 into the main text in Section 3, including:\n     - Algorithms 1 and 2 (combined into one algorithm box).\n     - Description of summarizing function architectures, including additional descriptions in Section 3 (moved from their current location in Section A2) and depicting the summarizing functions in a new Figure 3.\n- Expand the “Training” subsection in Section 4 to include model architectures, loss function, and optimizer (currently in Section A5.2).\n- Move results Table A6 to Main Text.\n- Expand the Summary of Related Work in the Introduction to be more thorough, while keeping the full Related Works in the Appendix. We agree the Related Works (currently located in the Appendix) is an important section, but it is lengthy (~1.3 pages) in order to be comprehensive, so we do not believe we can put the entire Related Works in the main text.\n- To make space for these changes, we can compress Sections 5 and 6 as well as remove Figure 2 and related references to the Figure.\n\n*Q4: Most figure captions are insufficiently informative, requiring the reader to refer back to the main text. Figure captions should be independent of the rest of the text.*\n\nThank you for this suggestion, we will update all figure captions to be more comprehensive.\n\n*Q5: The presentation of the results can be expended, with key information (e.g., number of true positives and false negatives) at this point missing.*\n\nThank you for the suggestion. In the original submission, we have additional performance metrics (balanced accuracy, recall, precision, specificity) in Table A6 for the CANDID dataset, which are indicative of performance beyond the AUROC. In the revised submission, we will include similar tables for the ISIC and SPINE datasets along with raw TP, TN, FP, FN numbers.  \n\n*Q6: Have the authors considered re-running their results multiple times to ensure they are reproducible?*\n\nWe are including new results for the CANDID segmentation and classification tasks with different random initializations in the limited data regime (Table R5). We observe a standard deviation of 0.0024 for segmentation-for-classification and 0.0054 for classification AUROC. In this rebuttal period we do not have time to rerun every experiment, but we can continue running experiments to have multiple seeds for the camera ready, if accepted. \n\nAdditionally, due to reviewer D8HP’s suggestion, we trained multiclass and natural image models with the proposed method. Both of these applications worked out-of-the-box, suggesting the method and findings extend to additional datasets and settings (details in the global response).\n"}, {"review_id": "Qb22gjYkeL", "reviewer": "Reviewer_GK2b", "summary": "This paper explores the benefits and drawbacks of using segmentation-based methods for classification tasks, particularly in medical imaging. This approach, known as segmentation-for-classification, has been shown to outperform traditional classification models, especially when the available dataset is small or when the classes are imbalanced. It can also handle rare subtypes better than conventional models.", "questions": "Please address the questions mentioned in weakness.\nAlso, I am wondering how the segmentation-for-classification performs in 3D segmentation scenario.", "limitations": "Add more baseline comparison. Specific mechanism for mitigating spurious correlations. 3d data performance.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Improve aggregate performance: by enhancing the divergence between segmentation's positive and negative classes and increasing the quantity of annotation, segmentation-for-classification provides greater performance differences in situations with limited data, such as in small datasets, scenarios with low class prevalence, and rare subtypes.\n- Reduced susceptibility to spurious correlations: Segmentation-for-classification is generally more robust to background features that are spuriously correlated with the target task, thus making it more reliable in classification tasks.\n- Location information: Segmentation-for-classification inherently delivers location information, facilitating human assessment of the model's predictions. This location information is particularly useful in medical imaging, where abnormalities can be challenging to find.\n- Comprehensive experiment: It is impressive to see the breadth of datasets used, covering synthetic data as well as three distinct medical datasets. The thorough exploration of different training regimes, including fully labeled, semi-supervised, and a boosted semi-supervised approach, greatly contributes to the depth and robustness of this investigation.", "weaknesses": "- Lack of Baseline Comparisons: While the study establishes that segmentation-for-classification performs better than traditional classification in certain scenarios, it does not benchmark these findings against other state-of-the-art methods for handling imbalanced datasets or for performing classification with limited data. Without these comparisons, it's hard to evaluate the true value of the proposed approach.\n- Handling of Spurious Correlations: While the authors claim that segmentation-for-classification is less susceptible to spurious correlations, the empirical evidence provided seems limited. The specific mechanisms through which the proposed method avoids or mitigates these correlations could be elaborated on more clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and questions in your review. We were glad to hear the reviewer appreciated the many potential benefits of segmentation-for-classification Below, we answer your questions point-by-point. We would be glad for any additional discussion with or suggestions by the reviewer.\n\n*Q1: Lack of Baseline Comparisons: While the study establishes that segmentation-for-classification performs better than traditional classification in certain scenarios, it does not benchmark these findings against other state-of-the-art methods for handling imbalanced datasets or for performing classification with limited data. Without these comparisons, it's hard to evaluate the true value of the proposed approach.*\n\nWe do use straightforward classification and segmentation networks in the original submission, as we were interested in exploring fundamental differences between the two task framings. We observed that segmentation-for-classification is able to achieve higher performance in the limited data regime (including with imbalanced datasets), improved robustness to spurious correlations, and other benefits without any sophisticated methods—simply by using a segmentation model, one is able to attain these benefits. We agree with the reviewer that additional classification baselines for these settings exist, but we note that they are different for different objectives (e.g., methods for reducing reliance on spurious correlations are different from methods that handle class imbalance are different from methods that improve interpretability). We believe the fact that segmentation handles all of these applications without specialized tooling is a major strength of the proposed approach. \n\nTo address the reviewer’s concern, we are happy to include additional references and discussion of prior work in classification aimed at addressing these applications, for example [1, 2, 3]. Finally, we note that the SPINE dataset is balanced and we still observe improvements using segmentation-for-classification.\n\n\n*Q2: Handling of Spurious Correlations: While the authors claim that segmentation-for-classification is less susceptible to spurious correlations, the empirical evidence provided seems limited. The specific mechanisms through which the proposed method avoids or mitigates these correlations could be elaborated on more clearly.*\n\nWe will add the mathematical statement and support (with proper reference to prior work) of why we expect segmentation-for-classification to be more robust to spurious correlations to the Appendix in Section A3, along with additional intuition and discussion elaborating on this point. Intuitively, spurious features carry less information about segmentation labels than image-level labels. Empirically, in the original submission we observed improved robustness with the CANDID dataset, where we saw performance on a difficult subset of patients improve from 0.58 to 0.84 AUROC. We also swept the strength of the spurious correlation in Figure 4(f) with the synthetic dataset, which showed classification’s performance dropoff with the increasing spurious correlation strength. \n\n\n*Q3: Also, I am wondering how the segmentation-for-classification performs in 3D segmentation scenario.*\n\nThe SPINE dataset is a 3D dataset, for which we use 3D convolutional networks. We see similar trends as we saw in 2D. We note there isn’t anything in our analysis or method that is specific for 2D segmentation, supporting the observed results that hold in both 2D and 3D. \n\nAdditionally, during this rebuttal period we have generated new results for a natural image dataset and for multiclass datasets. Both of these new applications worked out-of-the-box with the method described in the paper, suggesting the method and findings extend to additional datasets and settings the reviewer may be interested in; we provide details in the global response to reviewers.\n\n\n[1] Sohoni, N., Dunnmon, J., Angus, G., Gu, A., & Ré, C. (2020). No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems, 33, 19339-19352.\n\n[2] Liu, E., Haghgoo, B., Chen, A., Raghunathan, A., Wei, P., Sagawa, Shiori., Liang, Percy., Finn, Chelsea. (2021.) \"Just train twice: Improving group robustness without training group information\", Proc. Int. Conf. Mach. Learn., pp. 6781-6792.\n\n[3] Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., ... & Norouzi, M. (2021). Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3478-3488).\n"}]

Author rebuttal:
We thank the reviewers for their time and helpful suggestions, which helped us strengthen our submission. We were glad to hear the reviewers recognized the benefits and potential impact of segmentation-for-classification (reviewers rmFZ, RFD8, GK2b), found the paper thorough (reviewers D8HP, rmFZ, GK2b), and considered the approach innovative (reviewer RFD8). In our global response, we respond to common questions and report additional experiments. In the individual responses, we address each reviewer’s questions point-by-point. 

**Method generality.** Reviewers D8HP, GK2B, and rmFZ asked about the method's generality, both within medical imaging and in other areas. First, we highlight our intentional focus on medical imaging. Then, we present additional experiments requested by the reviewers showing the method’s applicability to natural, multiclass, and 3D images. 

We focus on medical images as segmentation-for-classification helps address problems that are particularly acute in this setting:
- Many medical image datasets collected from within an institution are small or contain rare subgroups [1, 2, 3]; we show segmentation-for-classification improves performance in this limited data regime (e.g., up to 16.2% improved aggregate performance; up to 161.1% improved recall on a rare subset). 
- Sensitivity to spurious correlations is particularly worrisome in low-failure-tolerance applications like medical imaging [1]. We’ve observed segmentation improves robust AUROC up to 44.8%.
- Segmentation inherently delivers location information about abnormalities, helping clinicians interpret the results of the classifier in clinical workflows. 

Thus, we believe medical image analysis is a strong motivating setting and well-suited application space for segmentation-for-classification. The improved performance and reliability in this risk-intolerant setting as well as the close alignment with clinical workflows helps justify the added cost of segmentation, although we note recent work in self-supervision and foundation models helps ameliorate the cost of labeling segmentation data [4, 5, 6]. 

That being said, we agree with reviewer D8HP that the methods extend to natural images and may be of general interest. To show the proposed methods extend to a natural image dataset, we include new results in Table R1 classifying dog and cat breeds in the Oxford Pets dataset in the limited data regime (50 training images per class). We see segmentation-for-classification improves average AUROC by 18.5%. We will include these results in the revised Appendix.

To show the method applies to multiclass settings (reviewer D8HP), the Oxford Pets dataset contains 37 classes of dog and cat breeds; as above, we see segmentation outperforms classification by 18.5%. Further, we extend one of our medical imaging datasets to the multiclass setting and perform 3-class classification on ISIC. Again, we see improved performance with segmentation-for-classification (Table R2). We will describe the multiclass case in Section 3 and include the multiclass results in Section A5.

To show the method applies to 3D data, we emphasize the SPINE dataset in our original submission is a 3D dataset, and we observe the same trends in 3D as we do in 2D.

**Model capacity.** Reviewers D8HP, rmFZ, and RFD8 ask about differences in model capacity between the classification and segmentation. We first clarify how we selected architectures in the original submission. Then, we present new results showing that model capacity does not explain the observed performance differences.

First, we clarify that in the original paper we tested multiple architectures for each task and dataset (Table A4). We did this to give each task and dataset the best performance, as it’s not clear the best architecture for segmentation on a given dataset is the same that is best for classification. 

To address the reviewer’s questions, we present two new results. In Table R3 we use the same backbone for both segmentation and classification. We use Resnet50 as the 2D backbone as it is a common backbone known to be useful for classification and segmentation. We see that segmentation shows the same or greater performance enhancement than what appeared in the original paper. We further note that all of our synthetic experiments were run with a standardized backbone.

However, even when using the same backbone, segmentation models still have more parameters due to the convolutional decoder (as noted by reviewer RFD8). We next provide results comparing performance using a higher capacity classification model—Resnet101, which has more parameters than the Resnet50 segmentation model—in Table R4. Again, we see drastically improved performance with segmentation. 

Finally, we clarify that the training procedure is matched for classification and segmentation, including matching the input data, augmentations, and codebase (which standardizes model checkpointing, loss function, hyperparameter tuning, etc.). 

Together, these results show it is not model capacity or training procedure that leads to segmentation-for-classification’s improved performance. We will add these new details to the revised manuscript for future readers. 


[1] Oakden-Rayner, L., et al. (2020). Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. ACM CHIL.

[2] Ng, D., et al. (2021). Federated learning: a collaborative effort to achieve better medical imaging models for individual sites that have small labelled datasets. Quantitative Imaging in Medicine and Surgery.

[3] Varoquaux, G., et al. (2022). Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ DM.

[4] Kirillov, A., et al. (2023). Segment anything. arXiv:2304.02643.

[5] Zou, X., et al. (2023). Segment everything everywhere all at once. arXiv:2304.06718.

[6] Butoi, V. I.,  et al. (2023). Universeg: Universal medical image segmentation. arXiv:2304.06131.

Meta-review: The submission was reviewed by four reviewers. The reviewers agree that the ideas presented in the paper are interesting, well validated and worth publishing. AC notices that one of the reviewer that recommends to reject did not engage in the discussion with the authors. AC concludes that the criticisms of this reviewer were resolved by the authors' response.  Thus, AC agrees with the majority of the reviewers and recommends to accept the submission.
Decision: Accept (poster)