Conditional Networks
Authors: Anthony Ortiz
Kris Sankaran
Olac Fuentes
Christopher Kiekintveld
Pascal Vincent
Yoshua Bengio
Doina Precup
Venue:  (2021)
Keywords: Conditional Computation
Generalization

Abstract: In this work we tackle the problem of out-of-distribution generalization through conditional computation. Real-world applications often exhibit a larger distributional shift between training and test data than most datasets used in research. On the other hand, training data in such applications often comes with additional annotation. We propose a method for leveraging this extra information by using an auxiliary network that modulates activations of the main network.  We show that this approach improves performance over a strong baseline on the Inria Aerial Image Labeling and the Tumor-Infiltrating Lymphocytes (TIL) Datasets, which by design evaluate out-of-distribution generalization in both semantic segmentation and image classification.

Under review as a conference paper at ICLR 2021
CONDITIONAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this work we tackle the problem out-of-distribution generalization through conditional computation. Real-world applications often exhibit a larger distribu- tional shift between training and test data than most datasets used in research. On the other hand, training data in such applications often comes with additional an- notation. We propose a method for leveraging this extra information by using an auxiliary network that modulates activations of the main network. We show that this approach improves performance over a strong baseline on the Inria Aerial Im- age Labeling and the Tumor Infiltrating Lymphocytes (TIL) Datasets, which by design evaluate out-of-distribution generalization in both semantic segmentation and image classification.
INTRODUCTION
Deep learning has achieved great success in many core artificial intelligence (AI) tasks (Hinton al., 2012 Krizhevsky et al. 2012 Brown et al. 2020) over the past decade. This is often attributed to better computational resources (Brock al. 2018 and large-scale datasets (Deng et al.) 2009).
Collecting and annotating datasets which represent sufficient diversity of real-world test scenarios for every task or domain is extremely expensive and time-consuming. Hence, sufficient training data may not always be available. Due to many factors of variation (e.g.. weather, season, day- time, illumination, view angle, sensor, and image quality). there is often a distributional change or domain shift that can degrade performance in real-world applications (Shimodaira) 2000) Waag & Schneider 2014 Chung et al. 2018). Applications in remote sensing, medical imaging, and Earth observation commonly suffer from distributional shifts resulting from atmospheric changes, sea- sonality, weather, use of different scanning sensors, different calibration and other variations which translate to unexpected behavior at test time (Zhu et al. 2017 Robinson et al. 2019 Ortiz et al.) 2018).
In this work, present novel neural network architecture to increase robustness to distributional changes (See Figure T). Our framework combines conditional computation (Dumoulin et al. 2018 2016 De Vries et 2017, Perez et al., 2018) with task specific neural architecture for better domain shift generalization.
One key feature of this architecture is the ability to exploit extra information, often available but seldom used by current models, through a conditioning network. This results in models with bet- ter generalization, better performance in both independent and identically distributed (i.i.d.) and non- i..dd settings, and in some cases faster convergence. We demonstrate these methodological in- novations on an acrial building segmentation task, where test images are from different geographic areas than the ones seen during training (Maggiori et al. 2017) and on the task of Tumor Infiltrating Lymphocytes (TIL) classification (Saltz al. (0118).
We summarize our main contributions as follows:
We propose a novel architecture to effectively incorporate conditioning information, such metadata.
We show empirically that our conditional network improves performance in the task of semantic segmentation and image classification.
We study how conditional networks improve generalization the presence of distributional shift.

---

Under review as a conference paper at ICLR 2021
Input Imagery (Xp)
Conditioning feature represenlation (Zn(X,))
Conditioning (auxiliary task) Network
MLP predicting for CBN/CGN params (v", p*))
Conditional (main task) network
Loss Lmain_ttatt task Lambda Lconditioning
Conditioning info (t,) (not required inference)
Main task prediction
Figure 1 Conditional Networks
2 BACKGROUND AND RELATED WORK
Self-supervised learning. Self-supervised learning extracts and uses available relevant context and embedded metadata as supervisory signals. It is representation learning approach that exploits variety of labels that come with the data for free. To leverage large amounts of unlabeled data, it is possible to set the learning objectives such that supervision is generated from the data itself. The self- supervised task, also known as pretext task, guides us supervised loss function (Gidaris al.) 2018 Oord et al. 20188 He et al. 2019; Chen et al. 2020). However, in self-supervised learning we usually do not emphasize performance on this auxiliary task. Rather we focus on the learned intermediate representation with the expectation that this representation can carry good semantic or structural meanings and can be beneficial to a variety of practical downstream tasks. Conditional networks can be seen as sulf-supervision approach in which the pretext task is jointly learned with the downstream task.
Our proposed modulation of network architecture based on an auxiliary network's intermediate representation can also be seen as an instance of knowledge transfer (Hinton al. 2015) Urban et al. 2016; Buciluu et al.) 2006). Because the auxiliary network has an additional task signal metadata prediction information about this task can be transferred to the main task network.
Conditional Computation. Ioffe and Szegedy designed Batch Normalization (BN) technique to accelerate the training of deep neural networks (Ioffe & Szegedy 2015). BN normalizes given mini-batch (F..... Jn=1 of N feature maps Fr..... described by the following Equation:
Fn,C.h.w [ (N Bc) = %o VarB F.....] +
Bc.
where c, l and w are indexing the channel, height and width axis, respectively, and Be are train- able scale and shift parameters, introduced to keep the representational power of the original net- work, and is constant factor for numerical stability. For convolutional layers the mean and variance are computed over both the batch and spatial dimensions, implying that each location in the feature map is normalized in the same way.
De Vries et al. 20117) 7) Perez et al. 20188) introduced Conditional Batch Normalization (CBN) as method for language-vision tasks. Instead of setting %e and B. in Equation directly, CBN defines conditioning input 4.- Note that
them as learned functions 3n.c = (e(%)) and "n.c = (c())) this results in different scale and shift for each sample in mini-batch. Scale (%n,e) and shift (3n,e) parameters for each convolutional feature are generated and applied to each feature via an affine transformation. Feature-wis wise transformations frequently have enough capacity to model complex
2

---

Under review as a conference paper at ICLR 2021
(b) AMLL U-Net
(a) Iupur
(c) Fully Cond. U-Net
T E5 D |1B
U D WE
(d) Input
(e) AMLL U-Net
( Fully Cond. U-Net
Figure 2: Qualitative Results. (a) Input from East Tyrol city, (b) AMLL U-Net segmentation results
for (a), (c) Fully Cond. U-Net CGN segmentation results for (a), (d) Input from Bellingham city, (e)
AMLL U-Net segmentation results for (d), (f) Fully Cond. U-Net CGN regmentian results for (d).
phenomena in various settings (Dumoulin et al. 2018). For instance, they have been successfully applicd to neural style transfer (Dumoulin et al. 2016) and visual question answering (Perez et al,, 2018). This kind of conditional computation scheme is not tied to the type of normalization used. Wu & He Hel(H0e11101)) (2018) recently proposed Group Normalization (GN). GN divides feature maps into groups and normalizes the features within each group. GN only uses the layer dimension, hence its compu- tation is independent of batch size. Ortiz et al. (2020) proposed Local Context Normalization (LCN) encourage local contrast enhancement by normalizing features based spatial window around it and the filters in its group. Recently, Michalski et al. (2019) showed that Conditional Group Nor- malization (CGN) offer performance similar CBN. In this work, we show results using CBN and CGN. Conditional normalization methods have been applied to tasks related to generalization, such as few-shot learning (Jiang et al., 2018 Tseng et al. (0200) and domain adaption (Su al. 2020). [Su et al.] propose to use conditional normalization and an adversarial loss for domain adaption in object detection. In contrast to this work, we propose method for implicit conditioning on an auxiliary task to leverage available metadata.
3 FORMULATION AND NETWORK ARCHITECTURE
3.1 PROBLEM ABSTRACTION
We first establish notation. Let x be an image input, associated with a ground-truth target y for the main task (e.g. a segmentation mask). Let available extra annotation for be denoted by t. The main network is trained to predict y given x and contextual information from an auxiliary network. The auxiliary network learns to predict I, also given æ. Features of an intermediate layer of the auxiliary network are used to transform the main task network's 's layers using conditional normalization parameterized by a learned function of 2.
The motivation for this method of implicit conditioning is the following:
1. Since i's are only used as training targets, auxiliary annotation is not required at test time.
2. During training, the auxiliary network learns (via backpropagation) to visually capture in- formation predictive of l. At test time, the auxiliary network reasons about the unavailable
unavallaaa
3

---

Under review as a conference paper at ICLR 2021
Cong Ue Blec
Bleck
Yn
SoyA, BeLckcck
Comy blook
Mia fool bxt
CHES conou
blocx
Carrs
Comry blocs
p
Zn(Xk)
Mond Uifect tteck N-Z
Liserr
Goou
Cord. UNie UNies BiocN
Rell
Linear
MLP
gh-
Cood u Block wft
Coadiion lafa
184
GN BN
ReLLU
Figure 3: Conditional Network Architecture for U-Net
in terms of visual patterns that correlate with auxiliary annotations of training data. Note that this allows the distribution of auxiliary information at test time to differ from the train- ing data (see for example our experiments on out-of -iit----ditiittionn generalization in remote sensing in Section 4.1)
While the first statement is true for any multi-task architecture, the second statement describes the flexibility of the proposed method leveraging auxiliary information of varying degrees of rele- vance. Obviously, the modulation will help most, if the auxiliary information is maximally relevant for the main task. Since the mapping from 2 to the modulation parameters is trained with the main task's training signal, the network can learn discard components of that are not useful for the main task. It is also possible for the network to learn a constant identity transformation of the main network's features case no correlation is found. This reduces the potential of negative transfer learning between unrelated tasks common in multi-task learning (Ruder 2017).
To provide an example how this method can help to exploit inexpensive metadata. Consider the task of segmenting satellite imagery of different regions on the globe. We can use the prediction of geographic coordinates, which are often logged by default when building satellite imagery datasets, as the auxiliary task. In this case, the auxiliary network may learn to capture visual characteristics that are distinctive for each region the training set, such as a predominance for smaller buildings. This would provide useful inductive bias for the segmentation network, even for regions with very different coordinates. By using feature modulation to integrate this contextual information, we hypothesize that the main network can learn more general purpose features, which can be attended to based on the context.
3.2 NETWORK ARCHITECTURE
Our proposed architecture modification transforms any standard neural network with normalization layers into one that incorporates conditioning information t. In each convolutional block of the neural network we substitute the normalization layer by it's conditional counterpart. We refer to this family of networks as Conditional Networks. Figure 3 shows this extension applied to the popular U-Net RRonneberger et al. (2015) architecture. U-Net is an encoder-decoder network architecture with skip connections. Figure shows the auxiliary network on the left modulating the modified U-Net on the right. The conditioning network is convolutional architecture (LeCun et al., 1998) followed by fully-connected layer that predicts metadata t, function of the input image In-- The pre-activation features before the output layer are used as (z,)) The functions (2 (x,)) and 7n (xr)) mapping (xx) to the scale and shift parameters implemented with multi-
4

---

Under review as a conference paper at ICLR 2021
Lelt glas
16:380
Wiiddn
10-366
a.
b.
Figure 4: Heatmap of activations for (a) AMLL U-Net and (b) conditional U-Net. Cell ij in row i and column j gives the activation of features patch Rows and columns sorted so that those that are more similar to one another appear side-by-side. Note that the color legends have different scales.
layer perceptron (MLP). Using the latent representations instead directly using in allows us to leverage combinations of features that were useful in localizing images from previously seen data, potentially improving generalization.
Because all its parts differentiable, conditional networks be trained end-to-end using gradient-based optimization. Our full objective is described in Equation / where a is a hyper- parameter balancing the main and auxiliary losses. Lmmin_task represents standard main task loss. Lmminntakk depends on the task, such as Jaccard, cross-entropy, and dice for semantic segmentation. conditiooing ensures the conditioning networks correctly predicts tn.
Leaand = Lmmai + a Lecoditioning
(2)
EXPERIMENTS
We study the following hypotheses:
H1: Generalization through context.
Explicit incorporation of conditioning information im- proves generalization in semantic segmentation and image classification tasks.
The features learned by the conditioning and the main task network reflect context-specific and context-invariant information, respectively.
H2: Interpretability.
4.1 CONDITIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF AERIAL IMAGES
To study hypotheses H1 and H2 we focus on the Inria Aerial Image Labeling Dataset. This dataset was introduced test out-of-distribution generalization of remote-sensing segmentation models (Maggiori et (017). It includes imagery from 10 dissimilar urban areas in North America and Europe. Instead of splitting adjacent portions the same images into training and test sets, the splicting was done city wise. All tiles of five cities were included in the training set and the remain- ing cities are used as test set. The test set also has variation illumination, landscape, and time, making it well-suited to evaluate out-of- iitto--distribution generalization. The provided imagery comes orthorectified and has a spatial resolution of 0.3m per pixel covering 8 810km² (405km? for training and 405km² for the test set) of building and not building. We use geographical coordinates the images as target data for the auxiliary network (see Section
evenly spaced grid. Images were labeled for the semantic classes
5

---

Under review as a conference paper at ICLR 2021
Table 1: Inria Test Set Performance.
Method
Bellingham
Bloomington Acc.
Innsbruck
San Francisco
East
Tyrol
Overall
IoU
Acc.
IoU
IoU
Acc.
loU
Aec.
IoU
Acc.
IoU
Acc.
AMLL U-Net
65.37
96.53
55.07
95.83
67.62
96.08
72.80
91.00
67.00
96,91
67.98
95.27
U-Net + GN
55.48
93.38
55,47
94.41
58.93
93.77
72.
89,56
62.27
95.73
63.71
93.45
U-Net + LCN
63.61
96.26
60.47
96.22
68.99
96.28
75.01
91.46
68.90
88.90'1
97.19
69.90
95.48
Cond. U-Net CBN
60.38
95.91
48.84
95.20
65.99
95.97
72.77
90.88
69.70
97.24
66,54
95,04
Cond, U-Net CGN
63.10
96.15
56.43
95.84
68.
96.23
74.33
91.28
66.71
97.03
68.66
95.31
Fully Cond. U-Net CBN
65.91
96.49
57.24
95.95
63.14
95.71
73.89
90.83
69.24
97.21
68.50
95.24
Fully Cond. U-Net CGN
66.98
96.52
63.27
96.33
69.80
96.24
73.88
90.75
70.77
97.35
70.55
95.44
nont, Cond. U-Net
62.97
96.10
53.08
95.03
65.61
95.73 77.87
90.32
65.66
97.75
66.33
94.78
Table 2: Distribution Shift Generalization Gap.
Method
Val. set
Test set
Gen. Gap
IoU (%)
IoU (%)
IoU (%)
AMLL U-Net
71.87
67.98
3.89
U-Net GN
71.38
63.71
7.67
Cond. U-Net CBN
72.15
68.50
3.65
Cond. U-Net CGN
72.77
70.55
2.22
For H1, we compare model performances using the standard benchmark training-test split. For H2, we perform an exploratory visualization of the feature-activation maps for the different models.
Generalization via conditioning. We used the Inria standard train- split to see whether condi- tioning information helps out-of-distributio generalization. From the training set we reserved five images of each city for the validation set as suggested by Maggiori al. (2017).
For this set of experiments we trained our conditional U-Net presented in Figure 3 end-to--en from scratch. We used as segmentation network the AMLL U-Net described by |Huang et al. (2018), which is a version of U-Net with fewer filters. The AMLL U-Net was the winning entry and top of the Inria leaderboard and use it as baseline for comparison in this section.
The Conditional U-Net was trained exactly as AMLL U-Net, but without any data augmentation technique. We used the standardized latitude and longitude of the center pixel of each patch as the conditioning information to be predicted by the conditioning network and the mean-squared error (MSE) as the conditioning loss Leonditioing and cross-entropy as segmentation loss Lmmin All training details and figure showing histograms of final Intersection- unienn (IoU) scores of the different models can be found in the Appendix sections A.I and A.2
Table shows the performance of the AMLL U-Net baselines and different variations of our pro- posed architecture on the test set. Conditioning uniformly improved segmentation performance over the corresponding conditioning-free models. This empirically validates our hypothesis H1 about generalization through context. We identify as Cond. U-Net those models which both the encoder and decoder are modulated, which yielded small gain in performance over just modulating the en- coder or decoder alone. We recommend conditioning all blocks since the extra computational cost is very small. Modulating using CGN consistently outperforms CBN.
in Matters. An experiment using the conditioning network without the auxiliary task of predicting in shows a deterioration of performance relative to the baseline U-Net as shown in Table [] We see this as evidence for the importance of guiding the learning process of the latent conditioning representation 2 (z,))
Table 2 shows the generalization gap as the difference between the validation set (i.id.) and the test set performance. Notice how the models' performances consistently degrade when we evaluate them cities not seen during training. Conditioning substantially reduces the generalization gap induced by the distribution- between the training and test sets, yielding evidence for hypothesis H1.
Figure [ shows a qualitative comparison in the performance of the proposed network. The baseline labels beach (2a) and power lines (2d) as buildings while the Conditional U-Net does not.

---

Under review as a conference paper at ICLR 2021
4.2 INTERPRETATION OF CONDITIONING FEATURES.
To evaluate hypothesis H2, we analyze patterns of activations across these experiments. This hy- pothesis is interesting for several reasons,
It serves a sanity check for the proposed architecture, ensuring that supervision from conditioning information leads to features that do indeed distinguish between cities.
It sheds light on the potential of using conditioning information facilitate learning of generalizable features by intentionally learning context-dependent features.
We can begin to characterize how generalization occurs, by identifying which training cities test patch "looks" like.
As our first approach towards characterizing feature context-dependence, we associate activations with underlying conditioning information. We apply the following procedure for few models,
Compute all activations pre-specified layer in the network for all patches within an epoch.
Compute the norm of activations for each feature map in that layer.
Arrange these values into patch x feature matrix, and visualize using heatmap.
For U-Net models, we focus on the "bottom" of the U, which has large number of filters with small spatial extent. In the conditional U-Net, we additionally compute the activations from the last layer before prediction of patch coordinates If we notice separation of patch activations according to con- ditioning information, we deduce that the learned feature maps are not invariant that conditioning context.
The learned activations are displayed in Figure We find, that individual features that activate for patch in one city tend to activate in a large fraction of patches across all cities. Further, across similar cities, patterns of activation are similar, for both conditioned and unconditioned models. Consider relative similarity between Chicago and San Francisco, for example
For the conditioned model, the large majority of features are zero, across all patches. However, when they are nonzero, their values tend to be larger than the typical activation in unconditioned models. From these observations, we conclude that the improved generalization ability of the con- ditional U-Net is not due to any ability to learn features that more invariant the identity of the corresponding city. Instead, it appears that the conditional U-Net learns a smaller collection of features that are ultimately more useful in the downstream segmentation task. We speculate that having fewer active features for any prediction allows for sharper predictions, preventing "blurring" that could result from averaging across feature maps. More details around H2 are presented in the analysis of feature variance and Figure in the Appendix section A.3] In summary, Figure 4 gives evidence against H2, suggesting that conditional network architectures do not neatly segre- gate context-specific and context-dependers features. Figure 10 both suggest that the learned features are qualitatively different between the architectures.
and
4.3 CONDITIONAL NETWORKS FOR TUMOR INFILTRATING LYMPHOCYTES CLASSIFICATION
We also test Conditional Networks for the task of tumor infiltrating lymphocytes (TIL) Classifi- cation. During the cancer diagnosis and treatment process, a patient may have a biopsy, which produces diagnostic tissue sample. Using this sample, slide is prepared and examined under microscope by a pathologist to understand both how to treat the disease and to provide a prognosis for the patient's future. Virtually all cancer patients undergo these biopsies, producing large volumes of these pathology slides.
significant feature these images is tumor infiltrating lymphocytes (TILs), which are types of immune cells that move into a tumor to try to attack the cancer. The quantification of TILs is
Since these features have no spatial extent, we do not take any norms.
"Tyrol/train refers to West Tyrol city, Tyrol/Validation refers to East Tyrol city as established the Inria
dataset
We also present the associated t-SNE projection in section A.3

---

Under review as a conference paper at ICLR 2021
BRCA
COAD
LUAD
PAAD
PRAD
Negative
vositive
Figure 5: Example of negative and positive TIL patches for different cancer types. The desired task is to properly classify patch as TIL negative or TIL positive independently of the cancer type.
well known to have prognostic value in many contexts (Fridman et al. 2012 Angell & Galon
2013] because understanding patient immune response to tumors becoming increasingly important with the growth of cancer immunotherapy. Features such as TILS can be quantified through image analysis and deep learning algorithms (Saltz et al., 2018 Klauschen et al., 2018). In Saltz et al. (2088), convolutional neural network (CNN) architecture systematically optimized to carry out classification of nuclei from pathology images. This led the release of a dataset consisting of TIL maps corresponding to roughly 5,000 whole slide images from The Cancer Genome Atlas (TCGA). Individual slide images were split into 100 x 100 patches and the tasks to classify TIL patches as positive negative in the exact same dataset setup as Saltz et al. (2018). The training set consists of 86,154 patches that were manually annotated with TIL classification (Saltz et al.) (2018). There 64,381 TIL negative patches and 21,773 TIL positive patches. All patches are in 100 100 pixel resolution, 20 times magnification, and are annotated as TIL. positive or TIL negative. Examples of the images and their labels given in Figure [
These training images represent seven different cancer types: invasive carcinoma of the breast (BRCA), colon adenocarcinoma (COAD), lung adenocarcinoma (LUAD), pancreatic adenocarci- noma (PAAD), prostate adenocarcinoma (PRAD). skin cutaneous melanoma (SKCM), and endome- trial careinoma of the uterine corpua (UCEC). The cancer type is the conditioning informations or metadata (t,)) used to train the conditioning network. We use another 652 patches as our validation set, and 900 manually annotated patches from twelve cancer types in total as the testing set. The twelve cancer types are the seven listed above, well as five novel ones never seen during training (urothelial carcinoma of the bladder (BLCA), cervical squamous cell carcinoma and endocervical adenocarcinoma (CESC), lung squamous cell carcinoma (LUSC), rectal adenocarcinoma (READ), and stomach adenocarcinoma (STAD)).
Methods tested. We train the VGG 16-layers network as baseline (Simonyan & Zisserman ). ) VGG networks have been shown to work well for pathology image classification (Xu al. (2015): Hou et al. (2016)). We then build the conditional version of VGG16 which we refer to Conditional VGG16. Conditional VGG16 is created in similar way as conditional U-Net and share a similar conditioning network. In this experiments the conditioning network trained to perform image classification for the type of cancer (t,,) using seven- seven-dimensional softmax layer. As with conditional U-Net, we predict [n (2n)) and 7n.e(2 (Ix)) from features in the conditioning network. 3n,e(2 (xr)) and "n,e(2 (x,)) are then used to modulate VGG16 architecture. We use binary cross-entropy as Lumann and multiclass cross-entropy as Leonditionngg Training details are provided in the supplemental material.
Results. Table 3 shows the results for the Tumor infiltrating Lymphocyte classification task using different approaches. Augmenting VGG16 using conditional networks improves its performance by large margin allowing obtain state-of-the-art performance in the task, even improving previous top performing approaches. is also important to remember that t, (cancer type) is only required during training. Inference only requires the input image, in competing methods.
To show the importance of the conditioning network we trained a version of conditional VGG16 where conditioning parameters predicted directly from the cancer type softmax prediction from
8

---

Under review as a conference paper at ICLR 2021
Table 3: TIL Classification Test (generalization set)
Results
Model
AUC (%)
Spatial Saltz et al. (2018); 8): Hou et al. (2019))
86.16
Inception (Szegedy et al. (201 6): Patton et al. )2019))
89.9
MENNDL (Patton
(2019))
86.16
Conditional VGGI6 (ours)
92.91
Direct Cond. VGGI6
50.16
Cond. VGGT6 alpha Q
77.18
the conditioningaauxilary network and we refer to it as "direct cond. VGG16" on Table 3] As expected, doing so does not work well. Properly predicting for the conditioning task is also very important for conditional networks to work well. When only optimize for Lmain_task performance degrades as it is shown on Table 3 under "Cond. VGG16 alpha
DISCUSSION AND CONCLUSIONS
We have presented conditional networks, family of neural networks that leverages conditional in- formation for improved performance and better generalization. Conditional networks can be applied any network architecture that uses normalization layers. We showed how the performance of two widely adopted network architectures (U-Net and VGG16) can be greatly improved by applying conditional networks for both semantic segmentation and image classification. We have shown that conditional networks consistently reduce the generalization gap observed when there is a shift of the underlying distribution at test time. After carefully studying the network feature activations, we found that the improved generalization ability of the proposed network is not due to the ability of learning more invariant features. appears, instead, that the conditional network learns a smaller collection of features more relevant to the task. Conditional networks exploits available extra anno- tation in while training, however t is not required during inference. is not always obvious which choice of helps the most to better estimate žn Future work involves studying how the choice of in influences performance in a larger set of tasks.
REFERENCES
Helen Angell and Jérôme Galon. From the immune contexture to the immunoscore: the role of prognostic and predictive immune markers in cancer. Current opinion in immunology, 25(2): 261-267, 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 4655, 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, ACM, 2006.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.055 005000000 2020.
Yeounoh Chung, Peter J Haas, Eli Upfal, and Tim Kraska. Unknown examples & machine learning model generalization. arXiv preprint wao.::888000990 2018.
Harm De Vries, Florian Strub, Jérémic Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp.
2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.

---

Under review as a conference paper at ICLR 2021
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv: 2016.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron
Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018. doi: undefined. https///distill.ub/2000 o
Wolf Herman Fridman, Franck Pagès, Catherine Sautes-Fridman, and Jérôme Galon. The immune contexture in human tumours: impact on clinical outcome. Nature Reviews Cancer, 12(4):298- 306, 2012.
Spyros Gidaris. Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations (ICLR), 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 191105572 2019.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1XX3::5533.02331 2015.
Le Hou, Kunal Singh, Dimitris Samaras, Tahsin M Kure, Yi Gao, Roberta Seidman, and Joel H Saltz. Automatic histopathology image analysis with cnns. In 2016 New York Scientific Data Summit (NYSDS), pp. 1-6. IEEE, 2016.
Le Hou, Vu Nguyen, Ariel B Kanevsky. Dimitris Samaras, Tahsin M Kurc, Tianhao Zhao, Rajarsi R Gupta, Yi Gao, Wenjin Chen, David Foran, al. Sparse autoencoder for unsupervised nucleus detection and representation in histopathology images. Pattern recognition, 86:188-200, 2019.
Bohao Huang, Kangkang Lu, Nicolas Audeberr, Andrew Khalel, Yuliya Tarubalka, Jordan Malof, Alexandre Boulch, Bertr Saux, Leslie Collins, Kyle Bradbury, al. Large-scale semantic clas- sification: outcome of the first year of inria aerial image labeling benchmark. In IGARSS 20/8- 2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 6947-6950. IEFE, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. Preceedings of the International Conference in Machine Learning (ICML), 2015.
Xiang Jiang, Mohammad Havaei, Farshid Varno, Gabriel Chartrand, Nicolas Chapados, and Stan Matwin. Learning to learn with conditional class dependencies. In International Conference on Learning Representations, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Frederick Klauschen, K-R Müller, Alexander Binder, Michael Bockmayr, M Hägele, P Seegerer, Stephan Wienert, Giancarlo Pruneri, de Maria, S Badve, al. Scoring of tumor-infilrrating lymphocytes: From visual estimation to machine learning. In Seminars in cancer biology, vol- ume 52, pp. 151-157. Elsevier, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. tradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2224, 1998.
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic la- beling methods generalize to any city? the inria aerial image labeling benchmark. In JEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE, 2017.
10

---

Under review as a conference paper at ICLR 2021
Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, and Doina Precup. An empirical study of batch normalization and group normalization in conditional computation. arXiv preprint arXiv: wrXh:11008.0001,, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint /
2018.
Anthony Ortiz, Alonso Granados, Olac Fuentes, Christopher Kiekintveld, Dalton Rosario, and Zachary Bell. Integrated learning and feature selection for deep neural networks in multispec- tral images. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition Workshops, pp. 1196-1205, 2018.
Anthony Ortiz, Caleb Robinson, Mahmudulla Hassan, Dan Morris, Olac Fuentes, Christopher Kiek- intveld, and Nebojsa Jojic. Local context normalization: Revisiting local normalization. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2020.
Robert M Patton, Travis Johnston, Steven R Young, Catherine D Schuman, Thomas E Potok, Derek C Rose, Seung-Hwar Lim, Junghoon Chae, Le Hou, Shahira Abousamra, et al. Exascale deep learning to accelerate cancer research. arXiv preprint arXiv:1909 12291, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
C. Robinson, Le Hou, Kolya Malkin, Rachel Soobitsky, Jacob Czawlytko, Bistra Dilkina, and Nebo- jsa Jojic. Large scale high-resolution land cover mapping with multi-resolution data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp.
Springer. 2015.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv: 05099808..008, 2017.
Joel Saltz, Rajarsi Gupta, Le Hou, Tahsin Kure, Pankaj Singh, Vu Nguyen, Dimitris Samaras, Ken- neth R Shroyer, Tianhao Zhao, Rebecca Batiste, et al. Spatial organization and molecular corre- lation of tumor-infiltrainng lymphocytes using deep learning on pathology images. Cell reports, 23(1):181-193, 2018.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2014.
Peng Su, Kun Wang, Xingyu Zeng, Shixiang Tang, Dapeng Chen, Di Qiu, and Xiaogang Wang. Adapting object detectors with conditional domain normalization. arXiv preprint arXiv:2003.0707,, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Hung-Yo Yu Tseng, Hsin-Yine Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned fcature-wise transformation. arXiv preprint -=Xiv:2001.08735 2020.
Gregor Urban, Krzysztof Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, AbdeIrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convolu- tional nets really need to be deep and convolutional? arXiv preprint 1XX.vv:60330566 2016.
Xuezhi Wang and Jeff Schneider. Flexible transfer learning under support and model shift. In Advances in Neural Information Processing Systems, pp. 1898-1906. 2014.
11

---

Under review as a conference paper at ICLR 2021
Yuxin Wu and Kaiming He. Group normalization. arXiv preprint arXiv:1803.08494. 2018.
Yan Xu, Zhipeng Jia, Yuqing Ai, Fang Zhang, Maode Lai, Eric, and Chao Chang. Deep volutional activation features for large scale brain tumor histopathology image classification and segmentation. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 947-951. IEEE, 2015.
Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer. Deep learning remote sensing: comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4):8-36, 2017.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Training Details Aerial Image labeling Experiments and Conditional U-Net We trained all networks for the task of aerial image labeling using 572x572 randomly sampled patches from all training image tiles. We used the Adam optimizer (Kingma & Ba 2014) with a batch size of 12. All networks were trained from scratch with learning rate of 0.001. Every network was trained for 100 epochs. We keep the same learning rate for the first 60 epochs and decay the rate to 0.0001 over the next 40 epochs. In every epoch 8,000 patches are seen. Binary Cross- cooss-Entropy was used as the segmentation loss function. For the conditioning part of the conditional network we used the latitude and longitude of every patch center pixel as t. Latitude and longitude were standardised to be from -1 to +1. The dimension is 4374 (9x9x54). z is the input of the MLP predicting for ? and B. We used 1024 hidden units the MLP.
mudel
model lous
wildddoon
43:32
34:22"
7x30
22
22
90
200
60
300
spocn
(a) Train loss AMLL U-Net
(b) Train loss Cond. U-Net
Figure 6: (a) Training loss AMLL U-Net (b)Training loss AMLL U-Net Cond. U-Net
Figure 6 shows the learning curves of the baseline U-Net and the fully conditional U-Net (CGN). The AMLL U-Net seems more prone overfitting than our conditional U-Net.
Training Details Conditional VGG16. We trained all training 100 x 100 TIL patches from the training set. We used the Adam optimizer (Kingma & Ba 2014) with a batch size of 16. All networks were trained from scratch for 20 epochs with learning rate of 0.001. Binary Cross-Entropy loss was used for the Lmain_taek (TIL classification) and multiclass cross-entropy for the Leonditioning..tan (cancer type classification) loss function. For the conditioning part of the conditional network used the cancer type as metadata t. a for the conditional loss was 0.8. No data augmentation was performed.
A.2 DETAILED PERFORMANCE CONDITIONAL U-NET
Figure 7 shows the overall and per-city performance of the models. The conditionti U-Net variant using CGN consistently outperforms the other models on every city in the test set. The CGN variant where only the encoder is conditioned has stronger overall performance and gener- alizes significantly better than the conditional U-Net variants using BN. This is consistent with the observation of Wu & He (2018) that (regular) GN outperforms BN in segmentation tasks.
A.3 NETWORK FEATURE ACTIVATIONS
Figures 8 and 9 are the t-SNE figures obtained from inspecting the activations of the bottom of the
in the U-Net for both the AMIL U-Net and Conditional U-Net CGN. Figure g shows the t-SNE associated to sample images from the same cities in the training set
+ Figure shows the t-SNE
associated to sample images from both training and test set
"Tyrol legend refers to West Tyrol as it refers to the training set TTyrol+Train refers to West Tyrol and Tyrol when train is false refers East Tyrol
13

---

Under review as a conference paper at ICLR 2021
Performance per City in Transfer Set
80
Overall Performance
70
60
30
Bellinghan
Bloomington
Inncbruce
San Frandaco
East Tyrol
Trardder 501
CRN
Wood COM
Colly
(a) Histogram of overall performance Figure 7: (a) Histogram of overall performance for both validation and test set (b) Histogram of per city performance
(b) Histogram of per city performance
2
(a) t-SNE AMLL U-Net train set Figure 8: t-SNE based on activations obtained using image tiles from the cities in the training set. (a) AMLL U-Net train set (b) Cond. U-Net CGN train set
(b) t-SN Cond. U-Net train set
In light of the differences in scale and sparsity of feature activations in the conditional and AMLL U-Nets in Figure 4 we zoom into individual features of interest Figure 10
not surprising that the overall y-axis scale differs between the two networks this is clear from the legend of Figure [ However, for both rows Figure [0] we observe that there are more instances of high variation in activation across patches in the conditional U-Net, compared to the AMLIL U-Net. In the conditional U-Net, there seem to be features that, while typically active, are often exactly or near zero, and similarly, features that typically inactive, but occasionally spike. This type of variation is even observed within individual cities. This suggests type of specificity in the learned features. Rather than activating slightly more or slightly less across all patches, features seem sensitive to particular features within the patches that they activate. While conditional U-Net features are not invariant to conditioning data, they do appear to be more specialized.
"We refer this figure while discussing hypothesis in the section 4.2
14

---

Under review as a conference paper at ICLR 2021
2
(b) I-SNE Cond. U-Net train and test sets Figure 9: t-SNE based on activations obtained using image tiles from cities in both training and test
(a) I-SNE AMLL U-Net train and test sets
sets. (a) AMLL U-Net train and test sets (b) Cond. U-Net CGN train and test sets
2
10
AMLL
500
2.6
Aclivation
0.0
25
0.0
Feature
7
10
Aclivalion
10
CUNer
Feature
bloomington
Innsbruck
city
bellngham
chicago
kiswpp
tynol
Figure 10: The (top) least and (bottom) most variable features, according to interquartile range, in the AMLL and conditional U-Nets, after filtering to those features that are active in at least 10% of patches. Columns 1-10 give the 10 most and least variable features, for least and most variable features, respectively. The y-axis is the activation for each feature. Patches are split into individual cities, and a boxplot of each city's activation values given.
15


Reviews:
[{"review_id": "lR7IWwr8Q8j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper intends to tackle the problem of out-of-distribution generalization through conditional computation.\nThe proposed framework combines conditional computation (exploiting extra information available about the problem data) with a task specific neural architecture.\nPractically, a conditioning network is trained using image embeddings extracted from input images in order to predict extra annotations available about said images. It’s subsequently used to modulate layers of the main task network.\nThis type of Self-supervised learning in the framework will use available relevant context and embedded metadata as supervisory signals.\nThe main stated goal is the evaluation of how conditional networks improve generalization in the presence of distributional shift.\nThe experiments claim that the conditional network improves performance in certain tasks, namely semantic segmentation and image classification . \nExperiments for these tasks are carried using Inria Aerial Image Labeling (aerial building segmentation task) and the Tumor-Infiltrating Lymphocytes (classification task).\n\nQuestions:\n-\tIn 3 FORMULATION AND NETWORK ARCHITECTURE/ 3.1 PROBLEM ABSTRACTION: you state :” To encourage this, we train an auxiliary network to predict the …. Consider the example of building segmentation discussed in Section 4.1, where we use geographical coordinates as extra information. Imagine the model is presented with an image from a new city, which might have features that are visually similar to cities from the training set, but has very different coordinates. If we conditioned on the location alone, we would not be able to generalize to this new city.” This begs three questions:\n-\tIs this kind of example enough to justify using the intermediate high-level features ^z (xn) as a proxy for zn.?\n-\tIn the given example, wouldn’t your framework equate to the main task network (No conditioning part used)? \n-\tIsn’t this testament to the fact that the conditioning addition to the main task network usefulness is highly dependent on the relevance of the extra information used to the main task?\n-\tIn 4.1 CONDITIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF AERIAL IMAGES: You state “The transfer set also has variation in illumination, landscape, and time, making it well-suited to evaluate out-of distribution generalization.” Can you clarify how those variations were obtained? Image selection for the second set of cities etc.\n-\tIn 4.2/Methods tested: you state:  “the conditioning network is trained to perform image classification for the type of cancer (tn).” Can you give more detail on how that was done?", "extra_scores": {}, "extra_text": {}}, {"review_id": "lE_MDmxKeB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper propose a method for leveraging additional annotation by using an auxiliary network that modulates activations of the main network. The method proposed achieves significant improvements over a strong baseline on two datasets.\n\n##########################################################################\n\nReasons for score: \n\nves: \n+ This paper tackle the problem of out-of-distribution generalization which is crucial for machine learning applications.\n+ The idea of leveraging additional information to enhance domain shift generalization is interesting and make sense to me.\n+ The proposed conditional networks is practical.\n+ Overall, the paper is well written and is easy to follow.\n\ncons:\n- My main concern about the paper is the novelty. The paper seems the incremental work based on Conditional Batch Normalization, which limits the contribution. Overall, the novelty and contribution of this work are marginal.\n\n##########################################################################\n\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Smw0RaLHgZz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThis submission proposes an approach to modulate activations of general convolutional neural networks by means of an auxiliary network trained on additional metadata to a dataset. The specific goal is to improve out-of-distribution (OOD) generalisation. This *conditional network* approach is illustrated for two standard convolutional neural network (CNN) architectures, U-Net and VGG, on two benchmark datasets suitable for OOD detection, the Inria Aerial Image Labeling Dataset and the Tumor Infiltrating Lymphocytes classification dataset. The conditional network approach yields favourable results compared to competing segmentation as well as classification networks and exhibits a reduction of the generalisation gap compared to the baseline methods. \n\n### Strengths:\n- Significance / Novelty: The conditional network uses a latent / intermediate representation *z* of an auxiliary network solving an auxiliary task *T* (as usually considered in the context of self-supervised learning) to learn parameters used to perform an affine transformation of feature map activations in the main neural network solving the main task *Y*. The approach combines self-supervised learning with conditional normalisation of activations. As far as I can judge, the proposed idea is original and novel. \n- A particular benefit of the proposed approach is that, at test time, only the latent representation *z* is required (provide by the trained auxiliary network) without the necessity of providing metadata *t*. \n- Relevance / Potential impact: The presented idea seems to improve performance and training of standard CNN architectures and can be used in general CNN architectures, which might render the approach applicable in a great variety of segmentation / classification tasks and relevant for a wider audience. \n- Clarity: In my opinion, this paper is very well-written, self-contained and offers a good description of the approach and adequate experimental evaluation of the made claims with some exceptions outlined below. \n\n\n### Weaknesses:\n- Technical quality:  My major concern with respect to the evaluation is the comparison to the baseline. In the paper by Huang et al. on AMLL U-Net higher IoU scores and accuracies on the transfer (test) set are reported (table 2, page 5 in their paper) compared to table 1, which in fact exceed the performance results of the proposed method. This raises the question whether the baseline methods are properly trained. From the description in the main text and appendix (section A.1), the only difference w.r.t. Huang et al. consists in using data augmentation. Is there a particular reason why the training deviates from the settings outlined in Huang et al.?\n- In connection to the previous point, also no details on hyperparameter tuning are provided and all networks were trained with the same settings. However, the differences in the methods might require different hyperparameter settings to provide the best performance. Were other hyperparameters considered in the outlined experiments?\n- Using the geocoordinates as metadata *t_n* does not strike me as a particularly useful choice, as it appears unlikely that a representation of “location” is learned but rather some random features are extracted which are known to be beneficial, too (as shown e.g. by Rahimi and Recht). In my opinion, relative closeness in geocoordinates should have little information on input aerial regions. The second example with *t_n* being the cancer type is much more convincing.\n\n- Clarity: Figure 2 and 4 are not explicitly addressed, put into context with other results or discussed in the main text (fig. 4 mentioned in the appendix). After carefully studying the paper, one might make a connection between the results in table 1 and figure 2. But although paragraph *”Interpretation of conditioning features”* (p. 7) seems to relate to figure 4, the results are difficult to interpret without any further discussion or comments on it. Could the authors elaborate more on Hypothesis 2 and Figure 4? I believe a more explicit / clearer discussion would improve the quality of the paper quite a bit. \n- In connection to the previous point, the reasoning for the following conclusion (p. 8) should be made more clearer (and maybe connected to section A.3 and figure 10 in the appendix): *“After carefully studying the network feature activations, we found that the improved generalization ability of the proposed network is not due to the ability of learning more invariant features. It appears, instead, that the conditional network learns a smaller collection of features more relevant to the task.”* \n- I might have missed it, but I am not sure what the difference between *“Cond. U-Net”* and *“Fully Cond. U-Net”* is. As the former already modulates all blocks in the encoder and decoder, what does *“fully cond.”* implicate?\n\n\n### Additional Feedback:\n- Figure 2 caption: *“(e) Fully Cond. U-Net CGN”* should read *“(e) AMLL U-Net”*.\n- Figure 4: I would suggest using the same scale range for both plots, to make the difference more prominent. This change might be beneficial for the points mentioned above, too.\n- Page 5, last sentences: *“[…] (see Section 4.1).”* This is a self-reference to the same section. Quite likely the reference was supposed to be *“Section 3.1”*.\n- Tables 1 and 2, (overall) IoU score on transfer set for “U-Net + GN”: There is probably a small typo what concerns the values *“63.71”* (table 1) and *“63.79”* (table 2). If I understand correctly, these values should be exactly the same. \n- Page 8, results paragraph, first sentence: *“[…] Tumor-infiltrating Lymphocyte […]”* while before *“[…] Tumor Infiltrating Lymphocyte […]”* was used (without the hyphen).\n\n### Recommendation:\nI enjoyed reading this submission and think that the idea of “conditional networks” constitutes a novel and relevant contribution to improve OOD generalisation. In particular, I believe the proposed approach might be impactful in a variety of related methods and applications as it can be used in general CNN architectures. However, there are some concerns and questions outlined above which I believe need to be addressed / adapted in order to accept this paper. My initial rating is weak accept, but I am willing to raise my rating if the authors can address these aspects. \n\n### Post-Rebuttal:\nI would like to thank the authors for addressing the questions and concerns. I still believe that the general idea of “conditional networks” might pose a relevant contribution to improving OOD generalisation. However, after rereading the submission, reading the other reviews, and taking the rebuttal into consideration, I think there are some aspects which need some revision and clarification. I comment on this in more detail below. Therefore, I stand with my initial rating of borderline, but I would like to encourage the authors to revise their paper taking the points raised by the reviewers into consideration and submit again.\n\n- **Section 4.2, discussion of figure 4 and hypothesis 2:** I thank the authors for the added discussion. However, the results are a bit at odds with the premise of the proposed approach, in my opinion. The bullet points which detail why hypothesis 2 could be interesting, are refuted by the results presented below in that section. I believe the result that activations look different (what concerns scale of activation and which features are active) by itself is less surprising as the approach tackles the normalisation of activations explicitly. But more importantly, other than that, I would say the activation patterns for different cities look qualitatively similar in both models and does not align well with the story of the paper. So, in my opinion, the activation patterns on the left (AMLL U-Net) in figure 4 look very similar for all cities, and also the patterns on the right (Cond. U-Net) look quite alike for different cities. Therefore, the interpretation of these results in the context of conditioning on auxiliary information remains speculative. I believe this part of the submission requires a careful reconsideration. \n\n- **Geocoordinates as metadata *t_n*:** A potential reason for the difficulties in the previous point could be the choice of metadata *t_n* in the segmentation example. I thank the authors for elaborating again on the choice. Still, I am not convinced that this is the most suitable choice to present the advantages of the proposed approach. If the conditioning network really just performs city ID classification, I would not be sure that any kind of useful (generalisable) features are extracted. This could be a potential explanation why no “conditioning influence” on the results in figure 4b is observed.\n\n- **Difference “Cond. U-Net”:** I should’ve been more explicit in my question. On page 6, last paragraph of “Generalization via conditioning” it reads: *”We identify as Cond. U-Net those models in which both the encoder and decoder are modulated, which yielded a small gain in performance over just modulating the encoder or decoder alone.”* If I see correctly, “Cond. U-Net” should be replaced by “Fully Cond. U-Net”, as the provided answer suggests, too.\n\n\n### References: \n- Rahimi and Recht, “Random Features for Large-Scale Kernel Machines”, NeurIPS 2007.\n- Huang et al., \"Large-scale semantic classification: outcome of the first year of inria aerial image labeling benchmark\", 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MYwlU1C4dI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary:\n\nThis paper aims to tackle the out-of-distribution generalization problem where a model needs to generalize to new distributions at test time. The authors propose to utilize some extra information like the additional annotations as the conditional input and output the affine transformation parameters for the batch normalization stage. This extra information helps the backbone network get a more general representation from the training set thus the model is robust to the distribution shift when testing. Experiments are conducted on the Aerial Image Labeling and the Tumor-Infiltrating Lymphocytes datasets which correspond to the image segmentation and classification task respectively.\n\nComments:\n1. The Conditional Batch Normalization (CBN) mechanism has been used in some generalization relevant tasks like domain adaptation detection [1] and few-shot learning [2]. The paper misses the insightful ideas, and only simply combines the CBN into different backbones and lacks theoretical analysis or empirical evidence about how the additional information and the conditional network can help to improve the model’s generalization ability. I think more ablation studies to demonstrate the effectiveness of the proposed conditional network and choices of the “metadata” or additional information in a dataset are needed.\n[1] Adapting Object Detectors with Conditional Domain Normalization\n[2] Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation\n\n2. The paper demonstrates the experiments on two tasks (i.e. semantic segmentation, image classification), but the proposed conditional network is different and does not have a unified architecture. So the concept of the conditional networks is over-packaged.\n\n3. The motivation proposed framework is not clear. The conditional learning assumption is a straight-forward idea for machine learning or deep learning, so what is the main novel mechanism proposed in the paper is not clear.\n\nQuestions:\n\n1. Is there now an existing standard benchmark of the out-of-distribution (OOD) generalization task? I noticed that this paper does not compare with other OOD generalization methods.\n\n2. What is the meaning of Figure 4? It seems that the explanation is not clear.\n\n3. From this paper, the additional information of the Inria Aerial Image Labeling dataset is the geographical coordinates. But what is the additional information or “metadata” of the Tumor-Infiltrating Lymphocytes dataset? I think it is an important issue but I did not find an answer from this submission.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to address the out-of-distribution generalization problem by means of conditional computation in form of a feature modulating module.
While the approach is interesting and brings a new take on how to perform feature modulation (although initially felt too similar to Conditional Batch Normalization) some major concerns about the experiments and validation of the approach are raised by all reviewers. Some of the hypothesis made are also challenged due to lack of proper validation.
Although the discussion clarified some points I am afraid many open questions are left unanswered and would require a more work to be fully addressed before acceptance.
Decision: Reject