Neural ODEs for Image Segmentation with Level Sets
Authors: Rafael Valle
Fitsum Reda
Mohammad Shoeybi
Patrick Legresley
Andrew Tao
Bryan Catanzaro
Venue:  (2020)
Keywords: neural odes
level sets
image segmentation

Abstract: We propose a novel approach for image segmentation that combines Neural Ordinary Differential Equations (NODEs) and the Level Set method.  Our approach parametrizes the evolution of an initial contour with a NODE that implicitly learns from data a speed function describing the evolution.  In addition, for cases where an initial contour is not available and to alleviate the need for careful choice or design of contour embedding functions, we propose a NODE-based method that evolves an image embedding into a dense per-pixel semantic label space. We evaluate our methods on kidney segmentation (KiTS19) and on salient object detection (PASCAL-S, ECSSD and HKU-IS). In addition to improving initial contours provided by deep learning models while using a fraction of their number of parameters, our approach achieves F scores that are higher than several state-of-the-art deep learning algorithms

Under review as a conference paper at ICLR 2020
NEURAL ODES FOR IMAGE SEGMENTATION WITH LEVEL SETS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel approach for image segmentation that combines Neural Or- dinary Differential Equations (NODEs) and the Level Set method. Our approach parametrizes the evolution of initial contour with NODE that implicitly learns from data forcing function describing the evolution. In cases where an initial contour is available contour embedding functions, we propose using NODEs to directly evolve the embedding of input image into pixel-wise dense semantic label. We evalu- ate our methods on kidney segmentation (KiTS19) 19) and on salient object detection (PASCAL-S, ECSSD and HKU-IS), addition to improving initial contours pro- vided by deep learning models while using fraction of their number of parame- ters, our approach achieves Fg scores that are higher than several state-of-the-ee deep learning algorithms.
alleviate the need for careful choice or design of
INTRODUCTION
Figure 1: Transversal slices of CT scans. Leftmost image: initial contour provided by UNet model. Other images: intermediate steps of the evolution of the initial contour with our Neural ODE model.
Image segmentation is the task of delineating pixels belonging to semantic labels. The ability to automatically segment objects is important because accurate labeling is expensive and hard (Vit- tayakorn & Hays 2011 ZZang et al. 20188). Automatic image segmentation can have large impact many domains, e.g. obstacle avoidance in autonomous driving and treatment planning in medical
imaging.
Accurate classification of pixels in close proximity inter-class boundaries remains challenging task in image segmentation. Object boundaries can have high curvature contours or weak pixel intensity that complicate separating the object from surrounding ones. In deep CNNs (Simonyan & Zisserman 2014 Zeiler & Fergus) 2014 Szegedy et al. 2013 He et al. 2016 Chen et al. 2017). the object-of interest and surrounding competing objects can provide equal context receptive field of boundary pixel, which can make accurate classification difficult. Humans also find it difficult to accurately label pixels near object boundaries.
Level Set methods (Zhao et al. 1996 Brox et al., 2006) and Active Shapes (Paragios & Deriche 2000 Chan & Vese 2001) have been proposed to incorporate shape and image priors to mitigate boundary ambiguities (Tsai et al. 2003 Rousson & Paragios 2002). The Level Set method for image segmentation evolves an initial contour an ofject-of- on-teres along the normal direction with forcing function. A contour is represented by an embedding function, typically signed distance function, and its evolution amounts to solving a differential equation (Osher Sethian 1988).

---

Under review as a conference paper at ICLR 2020
In this work, extend the formulation of the level set method. Inspired by the recent progress in Neural Ordinary Diferential Equations (NODEs) (Chen et al. 2018 Dupont et al. 2019 Gholami et al. 2019), propose to use NODEs to solve the level set formulation of the contour evolution, thus learning the forcing function an end-to-end data driven manner. Unlike earlier attempts in combining the level set method with CNNS, we benefit from NODEs parametrization of the deriva- tive of the contour because it allows us to incorporate external constraints that guide the contour evolution, e.g. by adding regularization penalty the curvature of the front or exploiting images at the evolving front by extracting appearance constraints in a non-supervised way.
Finally, similar to experiments in (Chen et al. 2018). alleviate the need for careful choice or design of contour embedding functions, we propose NODE-based method that evolves an image embedding into a dense per-pixel semantic label space.
To the best of our knowledge, this work is the first to apply Neural ODEs to real world problems. We validate our methods on two 2D segmentation tasks: kidney segmentation transversal slices of CT scans and salient object segmentation. Given an initial estimate of kidney via existing algorithms, our method effectively evolves the initial estimates and achieves improved kidney segmentation, as we show in Figure On real life salient objects, in addition to contour evolution, we use our method to directly evolve the embedding of an imput image into a pixel-wise dense semantic label.
Following (Hu et al., 2017), we compare against the results in (Wang et al. 2017 Li et al. 2016 Li
& Yu 2015 Zhao et 20155 Lee et al. 2016, Wang et al. 2015 Hu et al. 2017) and achieve w- F-F scores, AASCAL-SO..665 and ECSSD 0.868, that higher than several state-of-the art algorithms. Our results suggest the potential of utilizing NODEs for solving the contour evolution level set methods the direct evolution of image embeddings into segmentation maps. We hope our findings will inspire future research in using NODES for semantic segmentation tasks. We foresee that our method would allow for intervention on intermediate states the solution of the ODE, allowing for injection of shape priors or other regularizing constraints.
In summary, our contributions are:
We propose to use NODEs to solve the level set formulation of the contour evolution.
We propose using NODEs to learn the forcing function in an end-to-end data driven way.
We show NODEs can also evolve image embeddings directly into dense per-pixel semantic label spaces, which may alleviate the need for careful choice or design of contour embed- ding functions.
2 METHODS
Suppose I is a 2D image, S is the contour of object we want to segment, and Φ is contour embedding function, defined as a distance map, such that S = {( p)) y) = 0}. We assume an initial but rough contour of the object is given by human operator or by existing algorithm. A level set segmentation (Osher & Sethian 1988) solves a differential equation to evolve a contour along its normal direction with speed function F as:
dø
dt = [ fort E [0, 1],
(1)
where the initial value d0 y) is defined as a signed Euclidean distance from (xr v) the closest point on the initial contour S". The speed function F is often modelled to be a function of the target image I, the shape statisties of the object contour (derived from training shapes), or a regularizing curvature term (V Vé
[
In Neural ODEs, we parametrize the derivative of the hidden state using a neural network fo parametrized by A:
dh = (h, dt
(2)
The relationship between Eq. [ and Eq. 2 is immediate. the next section, we propose two approaches that adapt NODES the level set method for image segmentation.
2

---

Under review as a conference paper at ICLR 2020
2.1 CONTOUR EVOLUTION WITH NODES
We propose to solve a more general form of Eq. to evolve an initial contour estimate for image segmentation. We define the state of the NODE to be à augmented with the input image's embedding, h. We then advance the augmented state, ? = (ó. h), using NODEs, which can be interpreted as estimating the speed function described in Eq. Mathematically.
? h),
dy
dt fo(T, 1) for [0.
(3)
7 (0) - ())), Φ ( (7)
where is the time step in the evolution, y is the augmented state the NODE, f is a neural network parametrized by 0, 3(0) is the initial value of the distance map, (0 the initial value of the image embedding, vb is a learned function and ø is the dense per-pixel distance map prediction. Figure 2a schematically illustrates our initial contour evolution approach. Throughout this paper, we will refer to this method as Contour Evolution.
2.2 IMAGE EVOLUTION WITH NODES
In our first approach, we obtain final optimal contour by evolving an initial estimate. In our second approach, inspired by Chen et al. (018)), we evolve an image embedding h and project it into dense per-pixel distance map o, whose zero level set defines the final segmentation map, ((t) = ((x, y) ()) (x, y) = 0} Mathematically,
dh = fo(h, t) for E [0,
dt ( X(T),
お (h(
where t is the time step in the evolution, is a neural network parametrized by 0, / is an image. À is a learned image embedding function and a learned function that maps an image embedding to distance map. Figure 2b 2bjchmmma illustrates our direct image evolution approach. Throughout this paper, will refer to this method as Image Evolution.
IMPLEMENTATION
In the following subsections, we describe our design choices in loss function and their regularization terms, architectures, strategies for emphasizing the evolution of the contour on a region of interest. We also detail our model initialization strategies prevent drifting from the sub-optimal initial value, and choices of error tolerances and activation normalization.
3.1 LOSS FUNCTION AND REGULARIZZTION TERMS
We optimize the parameters of our NODE models, described in Figures 2a and 2b to minimize the empirical risk computed as the Mean Squared Error (MSE) between the target (Φ) and predicted (ő) distance maps. We remind the reader that although our techniques can access intermediate NODE states, which could allow injection of priors or other constraints, do not explore this in current experiments, and relay it to future work.
3.2 NARROW BAND AND RE-INITIALIZATION
In the level set formulation, all levels that describe the propagating contour tracked. Adalsteins- son & Sethian (995) proposed limiting the evolution to the subset levels within narrow band of the zero level contour.

---

Under review as a conference paper at ICLR 2020
(
Image
Image
Prenet (A)
Prenet (A)
h(0)
Neural ODE (fo)
Neural ODE (fo)
h1))
4()
(())
Postnet (2))
Postnet (2))
(a) Contour Evolution
(b) Image evolution
Figure 2: Diagrams for the contour and image evolution methods described in sections 2.1 and 2.2 Superscripts 0 and represent initial value and numerical solution of the NODE respectively.
In our approach, we obtain the equivalent of a narrow band by applying hyperbolic tangent non- linearity on the evolved distance map. It effectively attenuates the contribution levels in the optimization process. This transformation is especially valuable in refinement setups because it weights the gradients of the loss according to the proximity to the contour]
Re-initialization of & is another common practice in classical level set methods. It ensures the states in the trajectory of the numerical solution remain valid distance maps. (Sussman et al. [994] Hartmann et al. 2010)) propose to first extract zero level set of an evolving state, and re-calculate a distance map of that contour. In our experiments, we found that our optimization is not sensitive to non valid distance maps, and we did not find it necessary to reinitialize é.
3.3 PARAMETER INITIALIZATION AND LEARNING RATE RAMPUP
In tasks where the initial value is already close to the desired solution, not initializing the model parameters represent the identity function and not using learning rate ramp up can slow down the optimization process as the model predictions can immediately drift away from the initial value.
In addition using learning rate rampup, we prevent this issue by setting the weights and biases on the last layer of the NODE and Postnet layers to zero. This approach has been successfully used in normalizing flow models (Kingma & Dhariwal 2018 |renger al. 2019).
3.4 ADAPTIVE SOLVERS AND ERROR TOLERANCES
In ordinary differential equations, adaptive step solvers vary the step size according the error estimate of the current step and the error tolerance. If the error estimate is larger than the threshold, the step will be redone with smaller size until the error is smaller than the error tolerance.
The error tolerance etu given the current state is the sum of the absolute error tolerance azot and the infinity norm of the current state h weighted by the relative error tolerance Trat:
é atol + Tool ||
(5)
For the hyperbolic tangent, the gradients decrease as it moves away from zero, which represents the contour the zero level
4

---

Under review as a conference paper at ICLR 2020
Given that do not know in advance the infinity norm of h², which in our contains the image we set the contribution of the relative error tolerance
embedding as described Equations 3 and 4 term to zero and adjust the absolute enror tolerance.
3.5 ACTIVATION NORMALIZATION
When the batch size too small for using batch size dependent normalization schemes like Batch- Norm [loffe & Szegedy 2015), researchers rely on dataparallel multi-ppoocssse training setups with BatchNorm statistics reduced over all processes, for example SyncBatchNorm the APEX library
Saroffee et al., 2019).
When training in multi processor environments with data parallelism and NODEs with adaptive step solvers, the number of NODE function evaluations each processor differ. Consequently, the number BatchNorm calls inside each NODE layer will be dependent on the number of function evaluations, thus making reduction over processes complex.
In our setup, we circumvent this issue by using GroupNorm (Wu & He 2018) in layers where no convolution groups are used and LayerNorm (Ba al. 2016) when convolution groups are used.
4 EXPERIMENTS
4.1 DATASETS AND TASKS
The KiTS19 Challenge Data (Heller et al. 2019) consists of CT scans from 210 patients with tumour and kidney annotations. The MSRAIOK dataset (Cheng al., 2014) consists 10000 images with pixel-level saliency labeling from the MSRA dataset. The PASCAL-S dataset (Li et al. 2014) consists of free-viewing fixations on a subset of 850 images from PASCAL VOC, the ECSSD dataset Yan et al. 2013) consists of 1000 semantically meaningful but structurally complex images and the HKU-IS (Li & Yu 2015) consists of 4447 including multiple disconnected salient objects.
For the kidney segmentation task, we prune images that do not contain kidney and resize the data to 200 by 200, without any loss of generality given that the original images are interpolated with the same affine transformation for each patient. We divide the dataset between 7108 images from 168 patients for training and 1786 images from another 42 patients for validation.
For the salient object detection task, we train the MSRAIOK dataset and use 512 by 512 image crops, padding where necessary and masking the loss accordingly. We use data augmentation pro- cedures such scale, horizontal flip and change in brightness. We use all images for training and compute validations scores on the other salient object detection datasets.
4.2 EVALUATION METRICS
In experiments we use three metrics: Intersection Over Union (IOU), adaptive Fg (a-Fg) and weighted F'g (w-F3) described in (Margolin et al.) 2014).
For computing IOUs, we rely on the definition from the PASCAL-VOC challenge (Everingham ++ + Fp + FN), where Tp, Fp, and Fy represent true positive, false positive and false negative pixels determined over the whole validation set.
et al. 2015) and compute as Tp (Tp(TP-
The a-F3 metric is computed as the weighted F1 score, Fi (1 + /3) where we set 82 = 0.3 (Hu al.) 2017) and compute Fi over the entire validation set For computing the weighted Fg. we use the MatLab code provided by (Margolin et al. (0044), compute scores per image and average over all images. We understand these are the mechanism used to compute the scores reported in Hu et al. (017)).
4.3 TRAINING SETUP
All models are trained in PyTorch (Pazzke et al., 2017). We use the Adam optimizer (Kingmal & Ba (014)) with default params and learning rates between le-3 and le-4. We anneal the learning rate once the loss curves start plateau.
5

---

Under review as a conference paper at ICLR 2020
We use the Runge-Kunn Kutta (RK-45) adaptive solver and the adjoint sensitivity method provided in (Chen 2019). We set the relative error tolerance to zero and explore absolute error tolerances be- tween le-3 and le-5.
The model architectures we evaluate include the UNet (Ronneberger et al. 2015), NODEs parametrized by UNets (NodeUnet), and architecture inspired by DeepLabV3 ( Chen al. 2017 in which we stack NODEs (NodeStack) with Squeeze and Excitation modules (Hu et al. 2018) 88) fol-
lowed by Atrous Spatial Pyramid Pooling Layer (Chen et al. 2017).
The kidney segmentation experiments were conducted on a single NVIDIA DGX-1 with 8 GPUs. single
The salient object detection experiments NVIDIA DGX-I with 8 GPUs, and the experiments on NodeStack were conducted on 4 NVIDIA DGX-1 nodes with 32 GPUs total. We used the largest possible batch size given memory constraints.
UNet and NodeUNet were conducted
The code for replicating our experiments and pre-trained weights will be made available on github.
4.4 RESULTS
In this section, provide comparative results between methods (contour evolution and image evolution) and other methods over multiple datasets. In our experiments, our contour evolution method focuses on using NODE to refine suboptimal contours obtained from regression model trained to predict distance maps from an image; our image evolution method focuses on using a NODE to learn to evolve an image embedding kidney segmentation and then provide results on salient object detection.
a distance map. We first provide results on
4.4.1 KIDNEY SEGMENTATION
In this task we compare three setups: the first trains a UNet regression model that maps an image to a distance map; the second uses our contour evolution method to refine the prediction of the aforementioned ÜNet model with NODE parametrized by a UNet (NodeUNet): the third uses our image evolution method to train a NodeUnet that evolves an image into distance map.
We chose the UNet model checkpoint used in the contour evolution experiment by selecting the checkpoint with the lowest validation loss on the first 8 samples of the validation set right before the UNet starts overfitting the training data and the validation loss starts going up.
We use the same training and validation setup for all models and provide results over the validation set below on Table The UNet models provide the worse IOU scores. Our image evolution method produces goods results, showing evidence that it is possible to use NODEs to evolve image into a distance map.
Lastly, our contour evolution method is able to improve the suboptimal initial contour provided by the UNet model and represents our best results this experiment. These promising results show evidence that our method could generally be used to improve suboptimal models that underfit or overfit the training data. This is specially valuable for domains with scarcity of data. We provide results of the NodeUNet model trained with the contour evolution method in Figure 3
Model
Method
Parameters
IOU
UNet
Regression
5M
0.8762
UNet
Regression
15M
0.8494
NodeUNet
Image Evolution
5M
0.8832
NodeUNet
Contour Evolution
5M
0.8866
Table I: Validation IOU scores from three methods using similar model architectures with simi- lar and different number of parameters. Image Evolution represents evolution from an image distance map and Contour Evolution represents refinement of an initial contour.

---

Under review as a conference paper at ICLR 2020
Figure 3: Kidney segmentation results. Black and orange contours ground truth and model prediction, respectively. Blue, white and orange images are narrow band distance map prediction.
4.4.2 SALIENT OBJECT DETECTION
In this task we replicate the training setup described in Hu et al. (017): we train our models on the MSRAIOK dataset and compute their validation Fg scores on PASCAL-S, ECSSD and HKU-IS.
We first evaluate the effect of different methods and model architectures on F3 scores. We compare results from regression model trained using the UNet architecture, a contour evolution model using the NodeUNet architecture and an image evolution model using the NodeStack architecture. We choose the UNet model for contour evolution by repeating the procedure described in 44.41
We provide results over the validation set below on Table wherein the UNet model provides the baseline F3 scores. all experiments, our refinement method provides 5% relative improvement over our UNet baseline, which has 3x more paremeters than our NodeUNet (5M vs 15M). We foresee that this trend will continue and models with more parameters will yield better refinement results.
UNet 15M
NodeUNet 5M
NodeStack 41M
Dataset
Metrics
Regression
Contour Evolution
Image Evolution
PASCAL-S
a F8
0.745
0.762
0.780
PASCAL-S
W F3
0.696
0.704
0.698
ECSSD
a F8 0.736
0.771
0.848
ECSSD
w F3 0.719
0.745
0.768
HKU-IS
w F3
0.672
0.701
0.734
Table 2: Scores for different methods and models with 15, 5 and 41 million parameters. All scores are computed on binary maps produced by thresholding the ground truth saliency maps 0.
Finally, we compare our best model against a contrast based model DRFI (Wang et 2017 and recent deep learning based models such as MTDS (Li et al. 2016). MDF (Li & Yu 2015), MCDL
(Zhao al. 2013). ELD (Lee al., 2016), LEGS WWang al. 2015). We also compare against DLS (Hu et al. 2017). deep learning model based on level sets.
Whenever possible, we use the original code provided by the authors for computing scores or collect the scores from their publications. We reproduce the setup in Hu et al. (2017) by computing our PASCAL-S scores on binary maps produced by thresholding the ground truth saliency maps 0.5.
Figure below illustrate our model performance PASCAL-S, ECCSD and HKU-IS.
Table 3 below shows that the NodeStack model (Ours) achieves the best results on all but one metric.
Dataset
Metrics
DRFI
MCDL
LEGS
MTDS
MDF
ELD
DLS
Ours
PASCAL-S F3
~ 0.6
~ 0.7
0.7
0.7
0.7
~ 0.7
0.7
0.740
PASCAL-S
Fg
0.514
0.573
0.596
0.537
0.582
0.658
0.651
0.668
ECSSD
F3
0.7
~ 0.8
0.8
0.8
0.8
~ 0.8
0.8
0.848
ECSSD
F38
0.0
0.679
0.682
0.663
0.692
0.756
0.766
0.768
HKU-IS
Fg
0.514
0.634
0.607
0.711
0.567
0.718
0.748
0.734
Table 3: F3 Scores. PASCAL-S scores are computed on binary maps produced by thresholding the ground truth saliency maps at 0.5.

---

Under review as a conference paper at ICLR 2020
Figure 4: Salient object results. Black and orange contours ground truth and model prediction, respectively. Blue, white and orange images are narrow band distance map prediction.
5 DISCUSSION
In this paper, extend the level set segmentation method to use NODEs to solve the contour evolu- tion problem. We learn forcing function an end-to-end data driven manner. We demonstrate that techniques can effectively evolve rough estimates of contours into final segmentation of objects. Our techniques can also evolve input image's 's embedding into a pixel-wise dense semantic label.
Experimental results several benchmark datasets suggest using NODEs for image segmenta- tion task is viable. Compared to state-of-the-ait methods, our proposed techniques also produce favourable segmentation results.
Although we benefit from NODEs' parametrization of the derivative of the contour, in this paper we do not explore the incorporation of external constraints guide the contour evolution and that is area for future exploration. We also foresee that our method can generalize to 3D images.
Finally, in architecture with different learning rates and error tolerances yielded similar losses but largely dif- ferent number of NODE function evaluations, prohibitively increasing wall clock time. This hyper- parameter search can be replaced with automated approaches such as the Gated Info CNF described Nguyen et al. (2019). where the errr tolerances are estimated by the model.
cases during hyperparameter search found that training the same model
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Adalsteinsson and James A Sethian. A fast level set method for propagating interfaces. Journal of computational physics, 118(2):269-2) 277, 1995.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: 1e..16607704400 2016.
Thomas Brox. Andrés Bruhn, and Joachim Weickert. Variational motion segmentation with level sets. In European Conference on Computer Vision, pp. 471-483. Springer, 2006.
Tony F Chan and Luminita A Vese. Active contours without edges. IEEE Transactions on image processing, 10(2):26--277, 2001.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arX1v:1706.05587, 2017. 2017.
Tian Qi Chen. torchdiffeq. https: httpppr//stttuo.ooooo/////ataaat
2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, 2018.
Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu, Global contrast based salient region detection. IEEE Transactions on Puttern Analysis and Muchine Intelligence, 3783):569-58,, 2014.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. arXiv:1904.01681, 2019.
Augmented neural odes. arXiv preprint
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, (11(11)98-116) 136, 2015.
Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory-efficient gradients for neural odes. arXiv preprint arXiv: /902. 10298, 2019.
Daniel Hartmann, Matthias Meinke, and Wolfgang Schröder. The constrained reinitialization equa- tion for level set methods. Journal of Computational Physics. 229(5):1514-1533 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Nicholas Heller, Niranjan Sathianathen, Arveen Kalapara, Edward Walezak, Keenan Moore, Heather Kaluzniak, Joel Rosenberg, Paul Blake, Zachary Rengel, Makinna Oestreich, et al. The kits19 challenge data: 300 kidney tumor cases with clinical context, semantic segmentations, and surgical outcomes. arXiv preprint
2019.
Jie Hu, Li Shen, and Gang Sun. Squeezo-and-excitatn networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Ping Hu, Bing Shuai, Jun Liu, and Gang Wang. Deep level sets for salient object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2300-2309, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141. 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.

---

Under review as a conference paper at ICLR 2020
Gayoung Lee, Yu-Wing Tai, and Junmo Kim. Deep saliency with encoded low level distance map and high level features. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 660-668, 2016.
Guanbin Li and Yizhou Yu. Visual saliency based on multiscale deep features. Proceedings of the IEEE conference computer vision and patlern recognition, pp. 5455-5463, 2015.
Xi Li, Liming Zhao, Lina Wei, Ming-Hsuan Yang, Fei Wu, Yueting Zhuang, Haibin Ling, and Jingdong Wang. Deepsaliency: Multi-task deep neural network model for salient object detection. IEEE Transactions on Image Processing, 25(8):3919-335) 3930, 2016.
Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan Yuille. The secrets salient object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 280-287, 2014.
Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 248-244222 258, 2014.
Tan M Nguyen, Animesh Garg, Richard G Baraniuk, and Anima Anandkumar. Infoenf: An efficient conditional continuous normalizing flow with adaptive solvers. Workshop on Invertible Neural Nets and Normalizing Flows, 2019.
Stanley Osher and James A Sethian. Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi formulations. Journal of computational physics, 79(1):12-49, 1988.
Nikos Paragios and Rachid Deriche. Geodesic active contours and level sets for the detection and tracking of moving objects. JEEE Transactions on pattern analysis and machine intelligence, 22 (3):266-280) 2000.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. ICASSP 2019-2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 3617-3621, IEEE, 2019.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Mikael Rousson and Nikos Paragios. Shape priors for level set representations. In European Con- ference Computer Vision, pp. 92. Springer, 2002.
Christian Sarofeen, Michael Carilli, and Marek Kolodziej.
Apex.
htttp:://itthu. com/
[ttgichhe/torchalfffe 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. 2014.
Mark Sussman, Peter Smereka, and Stanley Osher. A level set approach for computing solutions to incompressible two-phase flow. Journal of Computational physics, 114(1):146-159) 1994.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Andy Tsai, Anthony Yezzi, William Wells III, Clare Tempany, Dewey Tucker, Ayres Fan, W Eric Grimson, and Alan S Willsky. A shape-based approach to the segmentation of medical imagery using level sets. 2003.
Sirion Vittayakom and James Hays. Quality assessment for crowdsourced object anmotations. In BMVC, pp. 1-11, 2011.
10

---

Under review as a conference paper at ICLR 2020
J Wang, H. Jiang, Z Yuan, MM Cheng, X Hu, N Zheng, and SO Detection. A discriminative regional feature integration approach. IEEE Int. J. Comput. Vis, pp. 1-18, 2017.
Lijun Wang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Deep networks for saliency detection via local estimation and global search. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3183-3192, 2015.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. Hierarchical saliency detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 1155-1162, 2013.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
Lishi Zhang, Chenghan Fu, and Jia Li. Collaborative annotation of semantic objects in images with multi-granuland supervisions. 2018 ACM Multimedia Conference on Multimedia Conference, pp. 474-482. ACM, 2018.
Hong- Kai Zhao, Tony Chan, Barry Merriman, and Stanley Osher. A variational level set approach to multiphase motion. Journal of computational physics, 127(1):179-19))) 1996.
Hong-Kai
Rui Zhao, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. Saliency detection by multi-context deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 1265-1274, 2015.


Reviews:
[{"review_id": "rylKU1IVKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to utilize Neural ODEs (NODEs) and the Level Set Method (LSM) for the task of image segmentation.  The argument is that the NODE can be used to learn the force function in an LSM and solve the contour evolution process. The authors propose two architectures and demonstrate promising performance on a few image segmentation benchmarks. \n\nWhile I like the attempt to combine deep learning with traditional CV algorithms, I do not think this submission is suitable for acceptance. My major critique is that both the two proposed models (Figure 2) are essentially two fully differentially NODE based architecture trained in a purely supervised way (by minimizing MSE). This makes the flavor of the proposed methods drastically different from what an LSM does, whereas the latter heavily rely on rich priors embedded in the design of the force function. To be more specific, the image evolution method (Figure 2(b)) essentially has nothing to do LSM. The contour evolution method bares more similarity to LSM but only in the sense that it learns to iteratively refine a contour estimation with an NODE. In the experimental evaluations, it seems that the image evolution model works favorably compared to the contour evolution method. This suggests that the main benefit of the proposed method comes from applying an NODE based architecture to a supervised learning task, rather than the inductive prior brought by LSM.\n\nGiven this, I think a much more proper way of presenting this work should be from the view of applying an NODE to the supervised image segmentation task. This reduces the novelty but increases clarity, and may still make the work a useful empirical reference for these benchmarks.\n\nSome more detailed comments:\n1. Equation 3&4 are not super easy to follow. For example, \\gamma^{(1)} is not defined or explained in main text.\n2. A related work section will be useful, especially for readers who are not familiar with LSM and NODE related literature.\n3. Experiments with more careful control needed. For example, from Table 1, it seems that the baseline Unet 15M performs inferior to the 5M model (explanation needed too), while Table 2 only compares against the 15M model. This makes it difficult to interpret the results.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyljiI4e5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to integrate Neural ODEs into image segmentation using level sets. I think the paper makes a good methodological contribution, however I don't think that ICLR is the best conference to publish this as the topic (image rather semantic segmentation) is too narrow and in my humble opinion, it won't attract the interest of ICLR audience. Moreover, it seems to me that the saliency object detection experiment is not a very convincing one as the methods compared are a bit old (mainly from 2015-2016).  I strongly recommend the authors try to publish this at MICCAI focusing on kidney segmentation or any other related medical imaging application.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJx6B8qpYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to apply the Neural ODE framework (Chen et al 2018) for image segmentation. The method relies on contour delineation through Level Sets. Since contour estimation requires to solve an ODE, this naturally allows to apply the work presented in (Chen et al 2018). The method is here applied in two segmentation tasks: kidney segmentation and salient object detection.\n\nThe concept underlying the paper is interesting, and leverages on very recent advances in the field. The idea of learning the dynamics required to evolve segmentation contours is original and certainly appealing. Unfortunately the content of this work seems quite preliminary in terms of presentation and experiments.\n\nFirst, the methodology is only sketched, while the motivation underlying the modelling rationale is often missing. For example, it is not clear what is the difference between \"image evolution\" and \"contour evolution\" models, besides the implementation details, and what motivates the definition of these two different modelling approaches in parallel. \nSecond, the experimental paradigm is controversial. The application on medical imaging is overly simplistic, as the authors do not consider the original 3D image stack, but rather the set of corresponding 2D slices modelled independently. The paper seems to ignore the large variety of body organ segmentation methods already available to the community, most of them working in 3D (e.g. [1-4]). The paper should necessarily compare with respect to these approaches and, even more importantly, with respect to standard level sets methods. \n\nFrom the practical perspective, the proposed method builds upon the results obtained with the UNet, and therefore is characterised by an additional computational burden. Given that the the training of neural ODE is not straightforward and computational expensive, the use of this model for achieving a tiny accuracy improvement seems overkill for this kind of application. Moreover, the segmentation accuracy is still computed slice-by-slice in the 2D images, and no information is available for the consistency of the reconstruction in 3D (regularity over the vertical axis).  \n\nFinally, the results reported in Table 3 are not clear, why the metrics of the competing methods are approximated (e.g. ~ 0.7), while for the proposed methods are given up to the 3rd decimal term?\n\n\n[1] 3D Kidney Segmentation from Abdominal Images Using Spatial-Appearance Models. Fahmi Khalifa, Ahmed Soliman, Adel Elmaghraby, Georgy Gimel'farb, and Ayman El-Baz 1. \n\n[2] Automatic Detection and Segmentation of Kidneys in 3D CT Images Using Random Forests. Rémi Cuingnet, Raphael Prevost, David Lesage, Laurent D. Cohen, Benoit Mory, Roberto Ardon. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part III\n\n[3] Multi-organ localization with cascaded global-to-local regression and shape prior. Medical image analysis. Gauriau, R., Cuingnet, R., Lesage, D., & Bloch, I. (2015), 23(1), 70-83.\n\n[4] Joint Classification-Regression Forests for Spatially Structured Multi-object Segmentation. Ben Glocker, Olivier Pauly, Ender Konukoglu, Antonio Criminisi. ECCV 2012 pp 870-881", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the classic medial image segmentation by combining Neural Ordinary Differential Equations (NODEs) and the level set method. The proposed method is evaluated on kidney segmentation and salient object detection problems. Reviewer #1 provided a brief review concerning ICLR is not the appropriate venue for this work. Reviewer #2 praises the underlying concept being interesting, while pointing out that the presentation and experiments of this work is not ready for publication yet. Reviewer #3 raises concerns on whether the methods are presented properly. The authors did not provide responses to any concerns. Given these concerns and overall negative rating (two weak reject and one reject), the AC recommends reject.
Decision: Reject