Dynamic Scale Inference by Entropy Minimization
Authors: Dequan Wang
Evan Shelhamer
Bruno Olshausen
Trevor Darrell
Venue:  (2020)
Keywords: unsupervised learning
dynamic inference
equivariance
entropy

Abstract: Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.

Under review as a conference paper at ICLR 2020
DYNAMIC SCALE INFERENCE BY ENTROPY MINIMIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which not itself totally immune to the scale variation it is meant to counter, or select scales by fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.
INTRODUCTION
The world is infinite in its variations, but models are finite. While inputs differ in many dimensions and degrees, a deep network is only so deep and wide. To nevertheless cope with variation, there are two main strategies: static enumeration and dynamic adaptation. Static enumeration defines set of variations, processes them all, and combines the results. For example, pyramids enumerate scales (Burt & Adelson 1983, Kanazawa al. 2014) and group-structured filters enumerate orientations
(Cohen & Welling 2017). Dynamic adaptation selects a single variation, conditioned on the imput, and
transforms processing accordingly. For example, scale-space search (Lindeberg) 1994] Lowe (004) selects scale transformation from input statistics and end-to-end dynamic networks select geometric transformations (Jaderberg et al. 20155] Dai et al. (017), parameter transformations (De Brabandere et 2016), and feature transformations (Perez et 2017) directly from the input. Enumeration and adaptation both help, but are limited by computation and supervision, because the sets enumcrated and ranges selected are bounded by model size and training data.
Deep networks for vision exploit enumeration and adaptation, but generalization still limited. Networks enumerative, by convolving with set of filters to cover different variations then summing across them to pool the variants (LeCun et al.] 1998 Krizhevsky et al.) 2012 Zeiler & Fergus 2014). For scale variation, image pyramids (Burt & Adelson 1983) and feature pyramids (Shelhamer et al.) 2017] [in et al., (2077) enumerate scales, process each, and combine the outputs. However, static models have only so many filters and scales, and may lack the capacity or supervision for the full data distribution. Dynamic models instead adapt to each input (Olshausen et al. 1993). The landmark scale invariant feature transform (Lowe 2004) extracts a representation adapted scales and orientations predicted from input statistics. Dynamic networks, including spatial transformers (Jaderberg et al.) 2015) and deformable convolution (Dai al. 2017). make these predictions and transformations end-to-end. Predictive dynamic inference is however insufficient: the predictor may be imperfect its architecture or parameters, or may not generalize to data was not designed or optimized for. Bottom-up prediction, with only one step of adaptation, struggle to counter variations in scale and other factors that are too large or unfamiliar.
To further address the kinds and degrees of variations, including extreme out-of-distribution shifts, we devise a complementary third strategy: unsupervised optimization during inference. We define an unsupervised objective and constrained set of variables for effective gradient optimization. Our novel inference objective minimizes the entropy of the model output to optimize for confidence. The variables optimized over task parameters for pixel-wise classification and structure parameters

---

Under review as a conference paper at ICLR 2020
prediction
entropy
prediction
image prediction entropy
(A) test
(b) Test 3x scale
(c) dynamic prediction
(d) dynamic optimization (ours)
Figure| Generalization across scale shifts between training and testing conditions is difficult. Accuracy is high and prediction entropy is low for training and testing at the same scale (left). Accuracy drops and entropy rises when tested 3x the training scale, even when the network is equipped with dynamic receptive fields to adapt to scale variation (middle). Previous approaches are limited one-step, feedforward scale prediction, and are unable to handle a 3x shift. In contrast our iterative gradient optimization approach is able to adapt further (right), and achieve higher accuracy by minimizing entropy with respect to task and scale parameters.
for receptive field udaptation, which are updated together to compensate for scale shifts. This optimization functions as top-down feedback iteratively adjust feedforward inference. In effect, we update the trained model parameters to tune a custom model for each test input.
Optimization during inference extends dynamic adaptation past the present limits of supervision and computation. Unsupervised optimization boosts generalization beyond training by top-down tuning during testing. Iterative updates decouple the amount of computation, and thus degree of adaptation, from the network architecture. Our main result is to demonstrate that adaptation by entropy optimization improves accuracy and generalization beyond adaptation by prediction (see Figure D), which show for semantic segmentation by inference time optimization of dynamic Gaussian receptive field model (Shelhamer et al. 2019 on the PASCAL VOC (Everingham et al. 2010) dataset.
ITERATIVE DYNAMIC INFERENCE BY UNSUPERVISED OPTIMIZATION
Our approach extends dynamic scale inference from one-step prediction to multi-step iteration through optimization. For optimization during inference, require objective to optimize and variables to optimize over. Lacking task or scale supervision during inference, the objective must be unsupervised. For variables, there are many choices among parameters and features. Our main contribution is an unsupervised approach for adapting task and structure parameters via gradient optimization minimize prediction entropy.
Note that our inference optimization is distinct from the training optimization. We do not alter training in any way: the task loss, optimizer, and model are entirely unchanged. In the following, optimization refers to our inference optimization scheme, and not the usual training optimization.
To optimize inference, base dynamic inference method is needed. For scale, we choose local receptive field adaptation (Dai et al. 2017 Zhang et al. 2017 [Shelhamer et al. 2019), because scale varies locally even within a single image. In particular, we adopt dynamic Gaussian receptive fields (Shelhamer et al. 2019) that combine Gaussian scale-space structure with standard "free-form" filters for parameter-efficiet spatial adaptation. These methods rely on feedforward regression to infer receptive fields that we further optimize.
Figure illustrates the approach. Optimization is initialized by feedforward dynamic inference of Gaussian receptive fields (Shelhamer et al., 2019). At each following step, the model prediction and its entropy are computed, and the objective is taken as the sum of pixel-wise entropies. Model parameters iteratively updated by the gradient of the objective. resulting in updated predictions and entropy. Optimization of the parameters for the Gaussian receptive fields is instrumental for adapting scale.
2

---

Under review as a conference paper at ICLR 2020
minimize
minimize
minimize
entropy
entropy
entropy
Figure Overview. Dynamic receptive field scale (top) optimized according to the output (bottom) at test time. We optimize receptive field scales and filter parameters to minimize the output entropy (middle). Optimizing during inference makes iterative updates shown from left to right: receptive field scale adapts, entropy is reduced, and accuracy is improved. This gives modest refinement for training and testing at the same scale, and generalization improves for testing at different scales.
2.1 OBJECTIVE: ENTROPY MINIMIZATION
Unsupervised inference objectives can be bottom-up, based the input, top-down, based on the output. To augment already bottom-up up prediction, we choose the top-down objective of entropy minimization. In essence, the objective to reduce model uncertainty.
More precisely, for the pixel-wise output Y [0, XxC for C classes and image of height H and width W, we measure uncertainty by the Shannon entropy (Shannon 1948):
(8)= P = c) log P(v.i = =c)
for each pixel at index i to yield pixel-wise entropy of the same spatial dimensions as the output.
Entropy is theoretically motivated and empirically supported. By inspection, we see that networks tend be confident on in-distribution data from the training regime. (Studying the probabilistic calibration of networks (Guo et al. (2017) confirms this.) In our case, this holds for testing scales similar to the training scales, with high entropy on segment contours. On out-of-distribution data, such scale shifts, the output entropy is higher and less structured. For qualitative examples, see Figures and 2
This objective is severe, in that its optimum demands perfect certainty (that is, zero entropy). As more stable alternative, we consider adaptively thresholding the objective by the average entropy across output pixels. We calculate the mean entropy at each iteration, and only take the gradient of pixels with above-average entropy. This mildly improves accuracy.
Our final objective is then:
L()- H))( for S = (i,j Haj > Hp)
$.jES
where S is the set of pixels with entropy above the average H.,. At each step, we re-calculate the average entropy and re-select the set of violating pixels. In this way, optimization is focused updating predictions where the model is the most uncertain.
2.2 VARIABLES: TASK AND STRUCTURE PARAMETERS
We need to pick the variables to optimize over so that there are enough degrees of freedom to adapt, but not many that overfitting sets in. Furthermore, computation time and memory demand minimal set of variables for efficiency. Choosing parameters in the deepest layers the network satisfy these needs: capacity is constrained by keeping most of the model fixed, and computation is
3

---

Under review as a conference paper at ICLR 2020
(C) prediction (b) entroyy (a) input/truti
16
Figure 3: Iterative dynamic inference by our entropy minimization. We optimize output entropy
with respect to task and scale parameters. (a) Input and ground truth. (b) Output entropy. (c) Output prediction. Our optimization reduces entropy and improves prediction accuracy.
reduced by only updating a few layers. The alterantive of choosing all the parameters, and optimizing end-to-end during inference, is ineffective and inefficient: inference is slower and less accurate than feedforward prediction.
We select the task parameters &oore of the output classification filter, for mapping from features to classes, and the structure parameters Oscalo of the scale regression filter, for mapping from features to receptive field scales. Optimizing over these parameters indirectly optimizes over the local predictions for classification scores Y and scales E.
Why indirectly optimize the outputs and scales via these parameters, instead of direct optimization? First, dimensionality is reduced for regularization and efficiency: the parameters are shared across the local predictions for the input image and have fixed dimension. Additionally, this preserves dependence the data: optimizing directly over the classification predictions admits degenerate solutions that are independent of the input.
2.3 ALGORITHM: INITIALIZATION ITERATION, AND TERMINATION
Initialization The unaltered forward pass of the base network gives scores (0 and scales (0)
Iteration For each step f, the loss the sum of thresholded entropies of the pixel-wise predictions (() The gradient of the loss is taken for the parameters @ and g(t)) The optimizer then updates both to yield ((++1) and ((++1) Given the new parameters, partial forward pass re-infers the local scale scales and predictions for (++1) and ((++1), This efficient computation is a small fraction of the initialization forward pass.
Termination The number of iterations is set and fixed to control the amount of inference computation. We do so for simplicity, but note that in principle convergence rules such as relative tolerance could be used with the loss, output, or parameter changes each iteration for further adaptivity.
Figure shows the progress of our inference optimization across iterations.
EXPERIMENTS
We experiment with extending from predictive to iterative dynamic inference for semantic segmenta- tion, because this task has a high degree of appearance and scale variation. In particular, we show results for iterative optimization of classifier and scale parameters in dynamic Gaussian receptive field model (Shelhamer et al. 2019) on the PASCAL VOC (Everingham et al.) 2010) dataset. By adapting both task and structure parameters, approach improves accuracy in in-distribution inputs and generalizes better on out-of out-oo--isttibution scale shifts. We ablate which variables to optimize and for how many steps, and analyze choices by oracle and adversary results. These experiments
4

---

Under review as a conference paper at ICLR 2020
establish the efficacy of entropy minimization during inference for scale adaptation, while oracle results show opportunity for further progress.
Data and Metric PASCAL VOC (Everingham et al. 2010 well-established semantic segmenta- tion benchmark with 20 semantic classes and a background class. The original dataset only has 1,464, 1,449 and 1,456 images with segmentation annotations for training, validation, and testing, respec- tively. As is standard practice, we include the additional 9,118 images and annotations from Harlharan al. 2011), giving 10,582 training samples in total. We measure accuracy by the usual metric of mean intersection-over-unioo (IoU). We report our results on the validation set.
Architecture We choose deep layer aggregation (DLA) Yu et al. 2018) strong, representative fully convolutional network (Shelhamer et al. 2017) architecture. DLA exploits the feature pyramid inside the network via iterative and hierarchical aggregation across layers. We will release code and the reference models implemented in Py Torchh (Paszke et al. 2017).
Training We train our model on the original scale of the dataset. We optimize via stochastic gradient descent (SGD) with batch size 64, initial learning rate 0.01, momentum 0.9, and weight decay 0.0001 for 500 epochs. We use the "poly" learning rate schedule (Chen al., 2018) with power 0.9. For the model with no data augmentation ("w/o aug"). the input images are padded to 512 512. As for the "w/ aug" model, data augmentation includes (1) cropping to 512 512, (2) scaling [0.5, 2). (3) rotation in [-10-, 10"]. (4) color distortion (Howard 2013), and (5) horizontal flipping.
Testing We test our model on different scales the dataset in the [1.5, range. We optimize the model parameters for adaptation via Adam (Kingma & Ba 2015), batching all image pixels together, and setting the learning rate to 0.001. The model is optimized episodically to each input, and the parameters reset between inputs. No data augmentation is used during inference to isolate the role of dynamic inference by the model.
3.1 RESULTS
We compare the semantic segmentation accuracy of our optimization with a prediction baseline and optimization by oracle and adversary. The baseline is one-step dynamic model using feedforward scale regression to adapt receptive fields following (Shelhamer et al. 2019). We train on a narrow range of scales and test a broader range of scales measure refinement, the improvement for the training scales, and generalization, the improvement for the new scales. This baseline is the initialization for iterative optimization approach: the output and scale predictions for the initial iteration are inferred by the one-step model. For analysis results, the oracle and adversary optimize during inference to respectively minimize/maximan the cross cross-entropy loss of the output and the truth.
As reported in Table [ our method consistently improves on the baseline by ~22 points for all scales, which indicates that our unsupervised optimization for iterative inference helps the model generalize better across scales. When the scale shift is larger, there is likewise a larger gap.
To evaluate the effect of data augmentation, experiment with ('w/ aug") and without ("w/o aug"). Data augmentation significantly improves generalization across scales. Note that our optimization during inference still improves the model with data augmentation by the same amount.
1.5x
2.0x
2.5x
3.0x
3.5x
4x
scale regression
68.2
59.3
50.2
41.8
34.0
27.5
w/o aug
entropy optimization (ours)
69.0
60.1
51.9
43.5
35.8
29.2
oracle
72.0
64.4
55.8
47.5
39.2
32.1
scale regression
74,2
70.8
65.8
59.8
53.5
46.8
w/ aug
entropy optimization (ours)
74.6
71.7
67.7
61.8
56.0
49.0
oracle
78.0
75.7
72.3
67.8
62.4
55.6
Table 1: Comparison of our method with the feedforward scale regression baseline and the oracle. Results are scored by intersection-over-uiion (higher is better). "w/o aug" excludes data augmentation, where "w/ aug" includes scaling, rotation, and other augmentation. Even though data augmentation reduces the effect of scale variation, our method further improves accuracy for all scales.
5

---

Under review as a conference paper at ICLR 2020
1.5x
2.0x
2.5x
3.0x
3.5x
4x
step 0
scale regression
68.2
59.3
50.2
41.8
34,0
27.5
entropy optimization (ours)
69.0
60.1
51.9
43.5
35.8
29.2
step 32
oracle
72.0
64.4
55.8
47.5
39.2
32.1
entropy optimization (ours)
69.0
60.3
52.1
43.5
35,2
28.5
step 128
oracle
73.3
68.6
61.8
54.0
45.7
38.5
Table 2: Ablation of the number of iterations: entropy minimization saturates after 32 steps.
Sanituion
pestictin
optinizttiotttoooooooooo predetion
Figure 4: Analysis of dynamic receptive field sizes across scale shift. (a) plots the distribution of dynamic receptive fields, confirming that optimization shifts the distribution further. (b) is the prediction 1x scale while (c) and (d) are the prediction baseline and iterative optimization 3x scale. (c) and (d) are visually similar, in spite of the 3x shift, showing that the predictor has failed adapt. Optimization adapts further by updating the output and scale parameters, and the receptive fields are accordingly larger. For visualization darker indicates smaller, and brighter indicates larger.
3.2 A ABLATIONS
We ablate the choice of parameters to optimize and the number of updates to make.
We optimize during inference adapt the task parameters (score) of the classifier and structure parameters (scale) the scale regressor. The task parameters map between the visual features and the classification outputs. Updates the task parameters the most direct way alter the pixelwise output distributions. Updates to the structure parameters address scale differences by adjusting receptive fields past the limits of the feedforward scale regressor. From the experiments Table both are helpful for refining accuracy and reducing the generalization gap between different scales. Optimizing end-to-end, over all parameters, fails achieve better than baseline results.
Iterative optimization gives simple control over the amount of computation: the number of updates. This is trade-off, because enough updates are needed for adaptation, but too many requires excessive computation. Table sshows that 32 steps are enough for improvement without too much computation. Therefore, we set the number of steps as for all experiments this paper. For our network, one step of inference optimization takes the time of full forward pass.
3.3 ANALYSIS
We analyze the distribution of scales in Figure show qualitative segmentation results in Figure
While better compensating for scale shift is our main goal, our method also refines inference on in-distribution data. The results in Table for 1x training and testing show improvement of ~1 point.
We analyze our approach from an adversarial perspective by maximizing the entropy instead of minimizing. To measure the importance of parameter, consider how much accuracy degrades

---

Under review as a conference paper at ICLR 2020
when adversarially optimizing it. The more performance degrades, the more it matters. Table shoows that adversarial optimization of the structure parameters for scale degrades accuracy sigmificantly, indicating the importance of dynamic scale inference. Jointly optimizing over the task parameters for classification further degrades accuracy.
test on 1x
test on 3x
score
scale
both
score
scale
both
scale regression
69.8
69.8
69.8
59.8
59.8
59.8
entropy optimization (ours)
70.2
70.7
70.6
61.1
61.8
62.3
oracle
73.7
75.6
77.7
63.9
67.8
71.3
adversary
67.4
55.9
52.4
57.4
47.4
44.4
Table 3: Analysis of entropy minimization (compared to oracle and adversary optimization) and ablation of the choice of parameters for optimization (score, scale, or both). The orarae/adversary optimizations minimize/maximize the cross-entropy of the output and truth to establish accuracy bounds. The adversary results show that our method helps in spite of the risk of harm. The oracle results show there are still better scales to be reached by further progress on dynamic inference.
4 RELATED WORK
Dynamic Inference Dynamic inference adapts the model to each input (Osshausen et al. 1993).
Many approaches, designed (Lindeberg) 1994 Lowel (004) and learned (Jaderberg et al. 2015
De Brabandere et al. [2016] Dai et al., 2017, Perez et al. 2017 [helhamerr et al.] (019). rely on bottom-up prediction from the input. Our method extends bottom-up prediction with top-down optimization to iteratively update the model from the output. Recurrent approaches to iterative inference (Pinheiro & Collobert 2014,, Carreira al., (0166) require changing the architecture and training more parameters. Our optimization updates parameters without architectural alteration.
Entropy Objective We minimize entropy during testing, not training, in effect tuning different model to each input. The entropy objectives in existing work are optimized during training, especially for regularization. Entropy is maximizzd//iiiiizze for domain adaptation (Tzeng et a 2015| [ Long et al. 2016 Vu et al. 2018 Saito al. 2019) and semi-supervised learning (Grandvalet & Bengio 2005 Springenberg 2016). In reinforcement learning, maximum entropy regularization improves policy optimization Williams & Peng 1991 AAhmed et al. 2019). We optimize entropy locally for each input during testing, while existing use cases optimize globally for a dataset during training.
Optimization for Inference We optimize unsupervised objective output statistics update model parameters for each test input. Energy minimization models (LeCun et al.) 2006) and prob- abilistic graphical models (Koller & Friedman, 2009; Wainwright & Jordan] 2008) learn model parameters during training then optimize over outputs during inference. The parameters of deep energy models (Belanger et al.] 2017 Gygli al.) 2017) and graphical models are fixed during testing, while our model is further optimized on the test distribution. Alternative schemes for learning during testing, like transduction and meta meta- leamnng differ their requirements. Transductive learning [Vapnik] 1998: Joachims 1999) optimizes jointly over the training and testing sets, which can be impractical at deep learning scale. We optimize over each test input independently. hence scalably, without sustained need for the (potentially massive) training set. Meta-learning by gradients (Finn ctt al. 2017) updates model parameters during inference, but requires supervision during testing and more costly optimization during meta-training
5 CONCLUSION
Dynamic inference by optimization iteratively adapts the model to each input. Our results show that optimization to minimize entropy with respect to score and scale parameters extends adaptivity for semantic segmentation beyond feedforward dynamic inference. Generalization improves when the training and testing scales differ substantially, and modest refinement is achieved even when the training and testing scales the same. While focus entropy minimization and scale inference, more optimization for dynamic inference schemes are potentially possible through the choice of objective and variables.
7

---

Under review as a conference paper at ICLR 2020
(a) image
(b) 1 prediction (c) 3x prediction (d) 3x opt. (ours)
(e) truth
Figure 5: Qualitative results from the PASCAL VOC validation set. Our model is trained on 1 1x scale and tested on 3x scale. (a) and (e) are the input image and ground truth. (b) indicates the reference in-distribution prediction on scale. (c) the duitt--dit prediction for the dynamic prediction baseline. (d) is the out-of-distribution prediction for our iterative optimization method. Our method corrects noisy, over-segmented fragments and false negatives in true segments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. ICML. pp. 151-160, 2019.
David Belanger, Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In 2017.
2017.
Burt and E. Adelson. The laplacian pyramid as a compact image code. Communications, IEEE Transactions on, 31(4):532-540, 1983.
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In CVPR, June 2016.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, 2018.
Taco S Cohen and Max Welling. Steerable enns. ICLR, 2017.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In NIPS, 2016.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. arXiv preprint urXiv:/706.0455 2017.
Michael Gygli, Mohammad Norouzi, and Anelia Angelova. Deep value networks learn to evaluate and iteratively refine structured outputs. In ICMI., pp. 1341-1351, 2017.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.
Andrew G Howard. Some improvements deep convolutional neural network based image classifi- cation. arXiv preprint arXiv: 1312.5403 2013.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. NIPS, 2015.
Thorsten Joachims. Transductive inference for text classification using support vector machines, In ICML, volume 99, pp. 200-209, 1999.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv; 1412.5104, 2014.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
A. Krizhevsky, Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012.

---

Under review as a conference paper at ICLR 2020
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-base learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23 22224, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. A tutorial on energy-based learning. Predicting structured data, 2006.
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer Science & Business Media, 1994.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. In NIPS, 2016.
D.G. Lowe. Distinctive image features from scale-invarian keypoints. IJCV, 2004.
Bruno A Olshausen, Charles H Anderson, and David C Van Essen. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. Journal of Neuroscience, 13(1 13(11):4700-4719 1993.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPSW, 2017.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. arXiv preprint 17Xiv:170..0787, 2017.
Pedro HO Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. In ICML, 2014.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. arXiv preprint arXiv: 0e04199444498 2019.
Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 1948.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. PAMI, 2017.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize and adapt receptive fields. arXiv preprint arXiv: 1487, 2019.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In ICLR, 2016.
Eric Tzeng, Judy Hoffmaan, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In ICCV, December 2015.
V. Vapnik. Statistical Learning Theory. Wiley, 1998.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Mathieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. arXiv preprint arXiv: 1811.18811.128... 2018
M.J. Wainwright and M.I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR, 2018.
MD Zeiler and R Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. Scale-adaptive convolutions for scene parsing. In ICCV, 2017.
10


Reviews:
[{"review_id": "ryxrlQ_Lcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe following work proposes a test-time optimization over scales to improve semantic segmentation. Specifically, at test time, they iteratively optimize over the score and scale parameters of Shellhamer et al 2019, where a Gaussian receptive field is used to allow for dynamic scale adaptation of each convolutional layer. They optimize the parameters with respect to an entropy minimization objective. Experiments on PASCAL VOC, reported at multiple scales, demonstrate improvements in IOU over the vanilla architecture.\n\nStrengths:\n-The work was well-motivated\n-Formulation is pretty elegant and outperforms the baseline\n\nWeaknesses:\nWhile I liked the somewhat elegant formulation of dynamic test-time scaling proposed by the following work, I don't think this work introduced many novel results nor insights\n-Multiscale test-time inference is already standard in state-of-the-art architectures such as DeepLab. Specifically, DeepLab runs inference at multiple scales, then max pools the logit responses across all scales. (see Table 3 of https://arxiv.org/pdf/1511.03339.pdf)\n-Using entropy as a measure of network uncertainty is a good idea, but also not a novel finding.\n-COCO and Cityscapes would probably have been better choices for datasets with larger variations in scale\n\nImprovements:\n-Try comparing against the multiscale logit-max-pooling inference procedure as a baseline -- or demonstrate that the proposed technique can further improve upon the results of the logit-pooling technique.\n-Some details weren't very well explained. Specifically, what is the $\\theta_{score}$ task parameter for?\n\n** Post Rebuttal Response\nI'd like to thank the authors for clarifying some points in their response. Overall, I maintain that I think the optimization-based scale inference solution they present is interesting from an implementation standpoint, but the findings in this work did not yield sufficient new insight for the task. While I agree that their approach is fairly different from common approaches such as discrete image pyramids, a thorough quantitative comparison of these differences would make this work a lot stronger. As such, I maintain my original rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJeSrHy0FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focused on the problem of semantic segmentation. The author proposed to minimize the output entropy to dynamically predict the scales when doing inference. The entropy minimization strategy is achieved by iterative optimization. Experimental results are reported on the PASCAL VOC dataset.\n\nClarity:\nI think this paper is moderate. The idea of dynamically predicting the scale or receptive field is interesting. However, this issue can be addressed through multi-scale training/testing or the deformable kernels. The experimental results are not that convincing. The method is only evaluated on one dataset and one backbone. The paper could be improved with more convincing experiments.\n\nLimitations:\nThe optimization process may take a certain number of forward and backward steps. In Sec 3.2 the author shows this will introduce 3x more time, this will much decrease its popularity when compared to other scale processing methods like deformable kernels.\n\nExperiments:\n1. The proposed method is evaluated on the PASCAL VOC dataset with the DLA segmentation backbone. The chosen backbone is not as strong as the most popular frameworks like DeepLab and PSPNet. Thus the baseline results as shown in Table 1 are not that high. I would like to see the relative improvements introduced by the proposed method over a stronger baseline.\n\n2. The experimental dataset is PASCAL VOC only. I would be more convinced with more datasets like Cityscapes or ADE20K.\n\n3. The reported experimental results are with models trained on a narrow range of scales. What the results and relative improvements would be if trained with regular multi-scales like [0.5, 2.0]? Will the scale issue be easily addressed by a multi-scales training strategy?\n\n4. The number of optimization steps may be hard to control, 32 is used for DLA on PASCAL VOC. Will this number be changed for different models on different datasets? If yes, can the author find a more elegant way to decide when to end the optimization process?\n\nMisc:\nIt is better to give a brief introduction of structure parameters scale and dynamic Gaussian receptive fields as in Sec 2.3.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1x7EQ7AFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to dynamically adapt some structural features of a semantic image segmentation model at inference time based on the entropy of the predictions.\n\nUsing a model that explicitly controls the size of the filters at each layer, they show that running a small number of SGD steps on the scale and final prediction parameters in the last layer to minimize the entropy of least confident predictions for a specific example leads to better performance overall, and especially better generalization when there is a size discrepancy between training and test set.\n\nStrengths: The method is inspired, and leads to significant improvements. The dynamic inference setup is clearly explained, and well motivated for the case of the scale parameters. Extensive ablation experiments and the inclusion of an oracle system help understand the contributions of each component of the setup, and the potential of inference-time optimization of the considered parameters.\n\nWeaknesses: Some information is missing from the description of the experimental setting. A quick review of the DLA model would be welcome, to get a better sense of the roles of \\theta_{scale} and \\theta_{score}. The authors should include published numbers for a relevant baseline and the current or recent state-of-the-art for the considered dataset (Table 1should also report 1x numbers in both settings). Finally, while the authors make a strong case for dynamic adaptation of the scale parameters, the prior reason for adapting \\theta_{score} is less obvious and would require further explanation.\n\nQuestions and miscellaneous remarks:\n“As reported in Table 1, our method consistently improves on the baseline by ∼2 points for all scales” > this statement is a little misleading, since the improvement is ~2 points on average, not ~2 points for each scale.\n\nWouldn’t simply multiplying \\theta_{score} by a large number decrease the entropy of the predictions? Do you do anything to prevent that from happening?\n\nSimilarly, couldn’t the adversary simply rotate \\theta_{score} to reduce IoU? Is the adversary optimized for long enough?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper constitutes interesting progress on an important problem.  I urge the authors to continue to refine their investigations, with the help of the reviewer comments; e.g., the quantitative analysis recommended by AnonReviewer4.
Decision: Reject