End to End Trainable Active Contours via Differentiable Rendering
Authors: Shir Gur
Tal Shaharabany
Lior Wolf
Venue:  (2020)

Abstract: We present an image segmentation method that iteratively evolves a polygon. At each iteration, the vertices of the polygon are displaced based on the local value of a 2D shift map that is inferred from the input image via an encoder-decoder architecture. The main training loss that is used is the difference between the polygon shape and the ground truth segmentation mask. The network employs a neural renderer to create the polygon from its vertices, making the process fully differentiable. We demonstrate that our method outperforms the state of the art segmentation networks and deep active contour solutions in a variety of benchmarks, including medical imaging and aerial images.

END TO END TRAINABLE ACTIVE CONTOURS VIA DIFFERENTIABLE RENDERING
Shir Gur & Tal Shaharabany
School Computer Science, Tel Aviv University
Lior Wolf Facebook Al Research and Tel Aviv University
ABSTRACT
We present an image segmentation method that iteratively evolves polygon. At each iteration, the vertices of the polygon are displaced based on the local value 2D shift map that is inferred from the input image via an encoder-decoder architecture. The main training loss that is used is the difference between the polygon shape and the ground truth segmentation mask. The network employs a neural renderer to create the polygon from its vertices, making the process fully differentiable. We demonstrate that our method outperforms the state of the art segmentation networks and deep active contour solutions in variety of bench- marks, including medical imaging and aerial images. Our code available at com con//shirgur/AADRNet
INTRODUCTION
The importance of automatic segmentation methods is growing rapidly in a variety fields, such as medicine, autonomous driving and satellite image analysis, to name but few. In addition, with the advent of deep semantic segmentation networks, there is a growing interest in the segmentation of common objects with applications in augmented reality and seamless video editing.
Since the current semantic segmentation methods often capture the objects well, except for occa- the image boundaries seems to
sional inaccuracies along some of the boundaries, fitting be an intuitive solution. Active contours, a set of techniques that given an initial contour (which can be provided by an existing semantic segmentation solution) grow iteratively to fit an image boundary. Active contour may also be appropriate in cases, such as medical imaging, where the training dataset is too limited to support the usage of high-capacity segmentation network.
Despite their potential, the classical active contours fall behind the latest semantic segmentation solutions with respect accuracy. The recent learning- based active contour approaches were not yet demonstrated to outperform semantic segmentation methods across both medical datasets and real world images, despite having success in specific settings.
In this work, we propose to evolve an active contour based on 2-channel displacement field (cor- responding to 2D image coordinates) that inferred directly and only once from the input image. This is, perhaps, the simplest approach, since unlike the active contour solutions in literature, it does not involve and balance multiple forces, and the displacement is given explicitly. Moreover, the architecture of the method is that of straightforward encoder-decoder with two decoding networks. The loss is also direct, and involves the comparison of two mostly binary images.
The tool that facilitates this explicit and direct approach is neural mesh renderer. It allows for the propagation of the intuitive loss, back to the displacement of the polygon vertices. While such renderers have been discovered multiple times in the past, and were demonstrated be powerful solutions in multiple reconstruction problems, this is the first time, as far as we can ascertain that this tool is used for image segmentation.

---

Our empirical results demonstrate state of the art performance in wide variety of benchmarks, showing a clear advantage over classical active contour methods, deep active contour methods, and modern semantic segmentation methods.
2 RELATED WORK
Neural Renderers A neural mesh renderer a fully differential mapping from a mesh to an image. While rendering the 3D 2D shape given vertices, faces, and face colors is straightforward. the process involves sampling on a grid, which is non-differentiable. One obtains differentiable rendering, by sampling in a smooth (blurred) manner (Kato et al. 2018) or by approximating the gradient based on image derivatives, as in (Loper & Black] (014)) Such renderers allow one to Backpropagate the error from the obtained image back to the vertices of the mesh.
In this work we employ the mesh renderer Kato et al. (2018). Perhaps the earliest mesh renderer was presented by Smelyansky et al. (2002). Recent non-mest differential renders include the point cloud renderer of Insafutdinov & Dosovitskiy (2018) and the view-based renderer of Eslami et al. (2018). Gkioxari et al. (2019) use a differentiable sampler to turn a 3D mesh to point cloud and solve the task of simultaneously segmenting a 2D image object, while performing 3D reconstruc- tion of that object. This is a different image segmentation task, which unlike our setting requires a training set of 3D models and matching 2D views.
Active contours Snakes were first introduced by Kass et al. 1988), and were applied in a variety of fields, such as lane tracking (Wang et al.) 2004). medicine Yushkevich et al. (006) and image segmentation (Michailovich et (007). Active contours evolve by minimizing an energy function and moving the contour across the energy surface until it halts. Properties, such curvature and area of the contour, guide the shape of the contour, and terms based on the image gradients or various alternatives attract contours to edges. Most active contour methods rely initial contour, which often requires a user intervention.
Variants of this method have been proposed, such using balloon force to encourage the con- tour to expand and help with bad initialization, e.g.contours located far from objects (Kichenassamyy al. 1995 Cohen 1991). Kichenassamy et al." (1955) employ gradient flows, modifying the shrink- ing term by function tailored to the specific attracting features. [Caselles et al. (1997) presented the Geodesic Active Contour (GAC), where contours deform according to an intrinsic geometric measure of the image, induced by image features, such as borders. Other methods have replaced the use of edge attraction by the minimization of the energy functional, which can be seen as a minimal partition problem (Chan & Vese 2001 Marquez-Neii Neila et al., 2014)).
The use of learning base models coupled with active contours was presented by Rupprecht et al. 2016), who learn to predict the displacement vector of each point the evolving contour, which would bring it towards the closest point on the boundary of the object of interest. This learned network relies on local patch that is extracted per each contour vertex. This patch moves with the vertex as the contour evolves gradually. Our method, in contrast, predicts the displacement field for all image locations at once. This displacement field is static and it is the contour which evolves. In our method, learning is not based on supervision in the form of the displacement to the nearest ground truth contour but on the difference of the obtained polygon from the ground truth shape.
A level-set approach for active contours has also gained popularity in the deep learning field. Work such (Wang al. 2019 Hu al. 2017: Kim et al. 2019) use the energy function as part of the loss in supervised models. Though fully differentiable, the level-sets do not enjoy the simplicity of polygons and their ability to fit straight structures and corners that frequently appear in man-made objects (as well as in many natural structures). The use of polygon base snakes in neural networks, as a fully differentiable module, was presented by Marcos et al. (2018): Cheng et al. (2019) in the context of building segmentation.
Building segmentation and recent active contour solutions Semi-automatic methods for urban structure segmentation, using polygon fitting, have been proposed by Wang et al. (2006); (Sun et al. 2014). Waag et al. (2016); Kaiser et al. (2017) closed the gap for full automation, overcoming the unique challenges in building segmentation. |Mrcos et al. (2018) argued that the geometric prop- erties, which define buildings and urban structures, are not preserved conventional deep learning semantic segmentation methods. Specifically, sharp corners and straight walls, are only loosely
2

---

learned. In addition, a pixel-wise model does not capture the typical closed-polyee structure of the buildings. Considering these limitations, Marcos et al. (2018) presented the Deep Structured Active Contours (DSAC) approach, in order to learn a polygon representation instead of the segmentation mask. The polygon representation of Active Contour Models (ACMs) is well-suited for building boundaries, which are typically relatively simple polygons. ACMs are usually modeled to attract to edges of the reference map, mainly the raw image, and penalize over curvature regions. The DSAC method learns the energy surface, which the active contour attracts. During training, DSAC in- tegrates gradient propagation from the ACM, through dedicated structured-rrediction loss that minimizes the Intersection Anversection-Ove--re
Cheng et al. (0119). extended the DSAC approach, presenting the Deep Active Ray Network (DAR- Net), based on polar representation of active contours, also known as active rays models of Denzler & Niemann (1999). To handle the complexity of rays geometry, Cheng et al. (2019) reparametrize the rays and compute L.1 distance between the ground truth polygon, represented as rays and the pre- dicted rays. Furthermore, in order to handle the non-convex shapes inherit in the structures, Cheng ct al. 2019) presented the Multiple Sets Active Rays framework, which is based on the deep watershed transform by Bai & Urtasun 2017).
Concurrently to our work, Haakkkizaaiz et al. (2019) presented DCAC, a Deep Convolutional Active Contours method which extends DSAC, and performs instance segmentation in acrial images using a differential renderer.
Castrejón et al. (017) and AAcua et al. (2018) presented polygonal approach for objects segmen- tation. Given an image patch, the methods use a recurrent neural network to predict the ground truth polygon. AAcuna et al. (2018) additionally made use of graph convolution networks and re- inforcement learning approach. Ling et al. (2019). introduced the Curve-GCN, an interactive or fully-automated method for polygon segmentation. learning an embedding space for points, and us- ing Graph Convolution neural networks to estimate the amount of displacement for each point. Ling al.) (219) also presented the use of differentiable rendering process, using the non-differentiable OpenGL renderer, and the method of Loper & Black (2014) for computing the first order Tylor approximation of the derivatives. Curve-GCN supports both polygon and spline learning, but also added supervision by learning explicit location for each point, and edge maps.
Both DSAC and DARNet enjoy the benefit of a polygon-based representation. However, this is done using elaborate and sophisticated schemes. Curve-GCN on the other hand, benefits from the use of differentiable rendering. but suffers from a time consuming mechanism that restricts it to apply it only in the fine-tuning stage. Our method is considerably simpler due to the use of a fast differentiable renderer, and a 2-D displacement field in the scale of the input. It uses only ground truth masks supervision, and two additional loss terms that are based time-tested pulling forces from the classical active contour literature: the Balloon and Curvature terms.
3 METHOD
Our trained network includes an encoder E and decoder D. In addition, a differential renderer is used, as well as a triangulation procedure L. In this work use the Delaunay triangulation.
Let be the set of all training images. Given an image I e ex an initial polygon contour pd is produced by oracle A. and the faces of this shape retrieved from a triangulation procedure L, which returns a list of mesh faces, each a triplet of polygon vertex indices. In many benchmarks in which the task is to segment given Region of Interest (ROI) extracted by a detector marked by a user, the oracle simply returns a fixed circle in the middle of the ROL. Fig. 3 illustrates the initial contour generation process.
The contour evolves for T' iterations, from p8 to pl and so on until the final shape given by pf, It where each vertex is two dimensional
is represented by a list of k vertices pt [pf P coordinate. This evolution follows a dual-channel displacement field:
..., ..pg)..,,,,,,,,
= =D((E())) e xexxx
For every vertex j = 1..k, the update follows the following rule:
p = 2タ J(P
(2)
3

---

T iterations
Points
Faces
Forward
Skip Connections
Gradients
Decoder
Grid
Neural Renderer
Update
Sampler
Figure 1: Illustration of our method. The input image is encoded using network E' and decoded back by the decoder D to provide 2D displacement field J. The vertices of the polygon time t-1 are updated by the displacement values specified by J, creating the polygon of the next time step- During training, neural renderer reconstructs the polygon shape, based on the polygon vertices and the output of triangulation process. A loss is provided by comparing the reconstructed shape with ground truth segmentation mask.
Convolation
Faces
Dropout
Concolation
Skip --
Stip
Concalirationn
Initial
BN, Masiiinia
IN. Relli Relll, UpSamping
Points
(
(b)
Figure 2: Illustration of the (a) encoder and (b) decoder blocks.
Illustration of the initial contour generation process.
Figure 3:
where -1 ) is a 2D vector, and the operation J(-) denotes the sampling operation displacement field J, using bi-linear interpolation at the coordinates of p/-1.
Coordinates which fall outside the boundaries of the image are then truncated (the following uses square brackets to refer to indexed vector elements):
pJ [1] min ma. pi (1))) 1/12] min(w, max (2))
The neural renderer, given the vertices and the faces returns the polygon shape mask, where all pixels inside the faces F' are ones, and zero otherwise:
M RRP', R kix
(4)
where M is the output segmentation at iteration t. This mask is mostly binary, except for the boundaries where interpolation occurs. Because the used renderer works 3D space, we project all points to 3D by setting their axis to 1, and use orthogonal projection.
This segmentation mask A compared to the ground truth mask M at each iteration, and accu- mulated over T' iterations to obtain the segmentation loss:
T
L
Σ
I - M
(5)
1=1
where || ||2 is the MSE loss applied to all mask values.
The curve evolution in classic active contour models is influenced by two additional forces: bal- looning force and a curvature minimizing force. In our feed forward network, these are manifested as training losses. The Balloon term Lg. maximizes the polygon area, causing it to expand:
Ls (1- M
(6)
4

---

Algorithm Active contour training of networks E, D. Shown for batch size of one.
Require: {4)]=1>
Input images, {M,} Matching ground truth segmentation masks, Initial guess oracle, R differential renderer, L triangulation procedure, & number of vertices, T'; number of iterations, A1. A2 a weighting parameter.
1: Initialize networks E, D
2: for multiple epochs do
3: for i 1....n do
4:
p0 = p.. P..-p...P. - A(T)
D Initialize the polygon using the oracle Triangulation obtain the mesh faces
5:
F L(PO)
6:
J + D(E(
7:
for I....T do
8:
Let P = [p] P2 J J 1
P2..., Pkl
9:
forj= = l...k
10:
PJ Pj
Set the vertices of polygon p
11:
M = R( F)
> The polygon shape an image
- C + | Ml1e+ λ1 hxw
+ X2 k Σ p 2p5 1
13:
Backpropagate the loss and update E,
where h and are the segmentation height and width, and æ denotes a single pixel Mr Second, the Curvature term Lx minimizes the curvature of the polygon, resulting in a more smooth form:
LK E
2p} + +f1+1ll2
where the L2 norm is computed on 2D coordinate vectors.
The complete training loss therefore:
L + AL + AeCis
(8)
for some weighting parameter A₁ and A2. It is applied after each evolution of the contour (and not just the final contour). See Alg. [ for listing of the process.
3.1 ARCHITECTURE AND TRAINING
We employ Secoder--ecodee architecture with U-Net skip connections (Çiçek al. 2016), which link layers of matching sizes between the encoder sub-network and the decoder sub-network, can be seen Fig. The encoder part, can be seen in Fig. a), is built from blocks which mirror-like versions of the relative decoder blocks, as can be seen in Fig. bb), connected by skip connection.
An encoder block consists of (i) three sub-blocks of convolution layer followed by dropout with probability of 0.2, (ii) ReLU, (iii) batch normalization and (iv) max-pooling, to down-sample the input feature map.
decoder block consists of (i) batch normalization, (ii) ReLU, (iii) bi-linear interpolation, which up- samples the input feature map to the size of the skip connection, (iv) concatenation of the input skip connection and the output the previous step, (v) three sub-blocks of convolution layer, followed by dropout with probability of 0.2. For the last decoder block, we omit the dropout layer, and up- sample to the input image size using bi-linear interpolation. To get the pixel-wise probabilities, we employ the Sigmoid (logistic) activation.
For the initial contour, unlike DARNet (Cheng et al. 2019) which multiple initializations, we simply fixed circle centered at the middle of the input image, with diameter of 16 pixels, across all datasets.
the ADAM optimizer Kingma & Ba (2014) with a learning rate 0.001, batch size varies depending of input image size, for 64 x 64 we use 100, 128 x 128 50. We set X 10-2 and X2 5 10-1
For training the segmentation networks,
5

---

(b)
(c)
(d)
Figure 4: Qualitative results of DARNet (Cheng et al. 2019) and our method. Columns (a)-(c) show results from the Vaihingen dataset (Rottensteiner et al.), and (d)-(() show results from the Bing huts (Marcos et al. (2188) dataset. (b) and (e) DARNet (Cheng et al., 2219), (c) and (f) Ours. Blue Initial contour. Yellow Final contour. Green GT Mask.
Quantitative results the buldings datasets of Vaihingen (Rottensteiner al.) and Bing (Marcos et al. 2018). t denotes the use of DSAC as backbone, and 3 denotes the use of DARNet as backbone.
Table 1:
Vaihingen
Bing
Method
FI FI-Score
mloU
WCov
BoundF
FI-Score
mloU
WCov
BoundF
FCN
81.09
81.48
64.6
69.88
73.36
30.39
FCN#
87.27
86.89
76.84
74.54
77.55
37.77
DSAC!
71.10
70.76
36.44
38.74
44.61
37.16
DSAC
60.37
61.12
24.34
57.23
63.09
15.98
DARNet
93.65 95.62
88.24
88.16
75.91
85.21
75.29
77.07
38.08
Ours
91.74
89.03
79.19
91.04
84.73
82.23
58.29
4 EXPERIMENTS
For evaluation, the common segmentation metrics of Fl-score and Intersection-over-Uniioo (IoU). Additionally, for the buildings segmentation datasets, we use the Weighted Coverage (WCov) and Boundary F-score (BoundF), which is the averaged Fl-score over thresholds from to 5 pixels around the ground truth, as described by [Chnng et al. (2019).
4.1 BUILDING SEGMENTATION
We consider two publicly available datasets in order to evaluate our method, the Vaihingen Rotten steiner et al.) dataset, which contains buildings from a German city, and the Bing Huts dataset (Mar- cos et al.) 2018), which contains huts from a village in Tanzania. A third dataset named TorontoCity, proposed by Marcos et al. (2018); Cheng et al. (2019) is not yet publicly available. The Vaihingen dataset consists of 168 buildings extracted from ISPRS Rottensteiner et al.) All images contain centered buildings with a very dense environment, including other structures, streets, trees and cars, which makes the task more challenging. The dataset is divided into 100 buildings for training, and the remaining 68 for testing. The image's size is 512 x 512 x 3, which is relatively high. We experi- ment with different resizing factors during training. The Bing Huts dataset consists of 606 images,

---

Figure 5: Qualitative results on the mammographic and cardiac datasets. Top INBreast (Moreira
al. 2012), Middle DDSM-BCRP (Heath al. 1998). Bottom SCD (Radau 2009). Blue Initial contour. Yellow Final contour. Green GT Mask.
Quantitative results on the two mammo- graphic datasets of INBreast (Moreira at. 2012) and DDSM- -CRP (Heath et al. 1998). Reported results the FI-Score.
Table 2:
Table 3: Results on the cardiac MR
Left Ventricle segmentation dataset of SCD (Radau et al., 2209)). F1-Score on the entire test set.
Method
INBreast
DDSM-BCRP
Method
FI-Score
Ball & Bruce 2007
90.90
90.00
Queirós et al. (2014)
0.90
Zhu et al. (2018)
90.97
91.30
et al. 201
0.92
Li et al. 2018
93.66
91.14
Avendi et al. (2016)
0.94
Singh et (2020)
92.11
Ngo et al. 2017
0.88
Ours
94.28
92.32
Ours
0.95
335 images for train and 271 images for test. The images suffer from low spatial resolution and have the size of 64 64, contrast to the Vaihingen dataset.
We compare our method to the relevant previous works, following the evaluation process, as de- scribed in [Cengg al. (2019), using the published test/val/train splits. The evaluated polygons are scaled, according to the original code of Cheng et al. (2019). For both datasets, we augment the training data (of the networks) by re-scaling in factors of [0.75, 1, 1.25, 1.5], and rotating by [0, 15, 45, 60, 90, 135, 180, 210, 240, 270] degrees.
can be seen in Tab. Oour method significantly outperforms the baseline methods on both building datasets. Fig. Acompares the results of our method with the leading method by Cheng et al. (019
4.2 MEDICAL IMAGING
We evaluate our method using two common mammographic mass segmentation datasets, the IN- Breast (Moreira et al. 2012), DDSM-BCRP (Heath al., 1998), and cardiac MR left ventricle segmentation datasets, the SCD (Radau et al. 2009). For the mammographic dataset, we follow previous work and use the expert ROIs, which were manually extracted, and the same train/test split as Zhu et al. (2018) Li et al. (0118) INBreast dataset consists of 16 accurately annotated masses, with mass size ranging from 15mm² to 3689mm². The dataset is divided into into 58 images for train and 58 images for test, done in previous work. DDSM-BCRP dataset consists of 174 an- notated masses, provided by radiologists. The dataset is divided into into 78 images for train and 5788 images for test, done in previous work. SCD dataset The Sunnybrook Cardiac Data (SCD),

---

Barlien Yarle
(bike)
(bus)
(motoreycle)
(person)
(truck)
Figure 6:
Sample results from the Cityscapes dataset.
Model
Bicycle
Bus Person Train Truck Motorcycle
Car
Rider Mean
Polygon-RNNe +NNN++ (with BS)
63.06
81.38 72.41 64.28 78.90
62.01
79.08 69 69.95 71.38
PSP-DeepLab
67.18
83,81 72.62 68.76 78.76 80.48
65.94
80.45 70.00 73.66
Polygon-GCh (with PS)
66.55
85.01 72.94 60.99 79.78
63.87
81.09 71 71.00 72.66
Spline-GCN (with PS)
67.36
85,43
73.72 64.40 80.22
64.86
81.88 71.73 73.70
Ours
68.08
83.02 75.04 74.53 79.55
66.53
81.92 72.03 75.09
Table 4: Cityscapes dataset. Quantitative results reported in mean IoU. BS indicates that the model uses beam search, PS indicates that the model train with explicit points supervision.
the MICCAI 2009 Cardiac MR Left Ventricle Segmentation Challenge data, consist of 45 cine-MRI images from a mix of patients and pathologies. The dataset is split into three groups of 15, resulting in about 260 2D-images each, and report results for the endocardial segmentation.
Tab. and show that our method outperforms all baseline methods on the three medical imaging benchmarks. Sample results for our method are shown in Fig.
4.3 STREET VIEW IMAGES
Following Ling et al. 2019), employ the Cityscapes dataset (Cordts et al. 2016) to evaluate our model in the task of segmenting street images. The dataset consists of 5000 images, and the experiments employ the train/val/test split of oastrreoon et al. (201)). Results are evaluated using the mean loU metric.
The Cityscapes dataset, unlike the other datasets, contains single objects with multiple contours. Similarly to Curve-GCN (Ling et al., 2019), we first train our model on to segment single contours, given the patch that contains that contour. Then, after the validation error stops decreasing, we train the final network to segment all contours from the patch that contains the entire object. Note that the network outputs a single output contour for the entire instance. We follow the same augmentation process as presented for the buildings datasets, using an input resolution of 64 x 64.
We compare our method with previous work on semantic segmentation, including PSP- DeepLab (Chen et al. 2017). and the polygon segmentation methods Polygon- RNN+++ (Acuna al.) 2018 and Curve- GCN (Ling et al. 2019).
As can be seen in from Tab. 4 our model outperforms and all other methods in six out of eight categories, and achieves the highest average performance across classes by sizable margin that is bigger than the differences between the previous contributions (the previous methods achieve average IoU of 73.70, we achieve 75.09) Unlike Curve-Net (Ling et al., 2019), we do not use additional supervision in the form of explicit point location and edge maps. Sample results can be seen in Fig. 6
4.4 MODEL SENSITIVITY
To evaluate the sensitivity our method to the key parameters, we varied the number of nodes the polygon and the number of iterations. Both parameters are known to effect active contour models.
Number of Vertices We experimented with different number of vertices, from simple to complex polygons- [4, 8, 16, 32, 64, 128). In Fig. top row, report the Dice and mloU on all datasets,
8

---

Input image
4-vertices
8-vertices
16-vertices
32-vertices
64-vertices
Figure 7: Varying number of vertices. Yellow Our method. Green - DARNet (Cheng et al. 2019
Va triggen
100
Bing
[Xtireant
230
DOSNY-BAP
200
5C3
50
20
30
50
10
Our F3 Score
Duas Score
Cun miU
Cura miU
EARINe
TL
Ozn FL-Geare
Can F3-5pre
18
126
25 Verticx
otvartces
Vartices
# VerVoas
hingggn
Bing
300
100
170
330
200
Our
mMU
Cus
Ourumbed
55-
75
35
55
65
bratioes
Figure 8: Top different number of vertices, and Bottom different number of iterations. Results for DARNet (Cheng et al. 2019) are available for the buildings datasets.
including results on DARNet (Cheng et al., 2019) on their evaluation datasets. As can be seen, seg- menting with simple polygons yields lower performance, while the number of vertices increases the performance quickly saturated at about 32 vertices. A clear gap in performance visible be- tween our method and DARNet (Cheng al. 2019), especially with low number of vertices. Fig. illustrates that gap the two buildings datasets.
Number of Iterations As can be seen from Fig. g Bottom), the effect of the iterations number is show to be moderate, although a mean increase is seen over all datasets, saturating at about iterations. is also visible that our model, trained only by a single iteration, has learned to produce displacement map that manipulate the initial circle close to optimal.
Training Image Resolution All of our models are trained at the resolution of 64 64 pixels. We have experimented with different input resolutions on the Vaihingen dataset. As can be seen Tab. 55] there steep degradation in performance below 64, which we attribute to the lack of details. When doubling the resolution 128 x 128, the performance slightly degrades, however, that model could improve with further training epochs.
Ablation Study In Tab. 6 we show the effect of different loss combinations on model perfor- mance on the Vaihingen benchmark. The compound loss C is better than its derivatives. Each the ballooning loss improves performance by itself does not. We note that even without the auxiliary losses, with a single straightforward loss term, method outperforms the state of the art.
not using auxiliary losses at all, while the eurvature loss

---

Table 5: Evaluation of different image resolutions during training the Vaihingen dataset.
Resolution
16
32
64
128
FI-Score
13.11
54.86
95.62
95.13
mloU
7.13
39.50
91.74
90.85
Table 6: Evaluation of different loss combinations the Vaihingen dataset.
Loss combination
Ses
SEE + Lx
LseG + LB
F1 F1-Score
94.94
94.80
95.13
95.62
mloU
90.31
90.20
90.82
91.74
5 CONCLUSIONS
Active contour methods that are based on global neural network inference hold the promise of improving semantic segmentation by means of an accurate edge placement. We present novel method, which could be the most straightforward active contour model imaginable. The method employs a recent differential renderer, without making any modifications to it, and simple MSE loss terms. The elegance of the model does not state of the art results on a wide variety of benchmarks, where each benchmark, it outperforms the relevant deep learning baselines, well as all classical methods.
at the expense of performance, and it achieves
ACKNOWLEDGMENT
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974). The contribu- tion the first author is part of a Ph.D. thesis research conducted at Tel Aviv University.
REFERENCES
David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmen- tation datasets with polygon-mn++.. 2018.
MR Avendi, Arash Kheradvar, and Hamid Jafarkhani. A combined deep-learning and deformable- model approach to fully automatic segmentation of the left ventricle in cardiac mri. Medical image analysis, 30: 30:108-119, 2016.
Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5221-5229, 2017.
John E Ball and Lori Mann Bruce. Digital mammographic computer aided diagnosis (cad) using adaptive level set segmentation. 2007 29th Annual International Conference of the IEEE En- gineering in Medicine and Biology Society, pp. 4973-4978 IEEE, 2007.
Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. International journal of computer vision, 22(1):61-79, 1997.
Lluis Castrejón, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Annotating object instances with a polygon-rnn. CVPR, 2017.
Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Annotating object instances with a polygon-rnn. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5230-5238. 2017. 2017.
Tony F Chan and Luminita A Vese. Active contours without edges. IEEE Transactions on image processing, 10(2):266-277, 2001.
10

---

Liang-Chin ShingCChihh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Dominic Cheng, Renjie Liao, Sanja Fidler, and Raquel Urtasun. DARNet: for building segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7431-7439, 2019.
Deep active ray network
Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u- net: learning dense volumetric segmentation from sparse annotation. International conference on medical image compuling and compuler-asisttdd intervention, pp. 424-432. Springer, 2016.
Laurent D Cohen. On active contour models and balloons. CVGIP: Image understanding, 53(2): 211 211-218, 1991.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213- 3223, 2016.
Joachim Denzler and Heinrich Niemann. Active rays: Polar-transformed active contours for real- time contour tracking. Real-Time Imaging, 5(3):203-213, 1999.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka. Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, (MM6639))1204-1220 2018.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. International Conference on Computer Vision (ICCV), 2019.
Ali Hatamizadeh, Debleena Sengupta, and Demetri Terzopoulos. End-to-end deep convolutional active contours for image segmentation. arXiv preprint arXiv: 13359, September 2019.
Michael Heath, Kevin Bowyer, Daniel Kopans, Kegelmeyer, Richard Moore, Kyong Chang, and Munishkumaran. Current status of the digital database for screening mammography. In Digital mammography, pp. 457-460 Springer, 1998.
Ping Hu et al. Deep level sets for salient object detection. In CVPR, 2017.
Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervised learning of shape and pose with differ- entiable point clouds. In Advances in Neural Information Processing Systems, pp. 2802-2812, 2018.
Pascal Kaiser, Jan Dirk Wegner, Aurélien Lucchi, Martin Jaggi, Thomas Hofmann, and Konrad Schindler. Learning acrial image segmentation from online maps. IEEE Transactions on Geo- science and Remote Sensing, 55(11):6054-606) 2017.
Michael Kass, Andrew Witkin, and Demetri Terzopoulos. Snakes: Active contour models. Interna-
tional journal of computer vision, (4):321-331, 1988.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 3907-3916, 2018.
Satyanad Kiehenassamy, Arun Kumar, Peter Olver, Allen Tannenbaum, and Anthony Yezzi. Gradi- ent flows and geometric active contour models. In Proceedings of IEEE International Conference on Computer Vision, pp. 810-815. IEEE, 1995.
Youngeun Kim et al. CNN-based semantic segmentation using level set loss. In WACV, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2 69800 2014.
11

---

Heyi Li, Dongdong Chen, William H Nailon, Mike E Davies, and David Laurenson. Improved breast mass segmentation in mammograms with conditional residual u-net. In Image Analysis for Moving Organ, Breast, and Thoracic Images, pp. 81-89 Springer, 2018.
Huan Ling. Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object anno- tation with curve-gen gen. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5257-5266, 2019.
Yu Liu, Gabriella Captur, James C Moon, Shuxu Guo, Xiaoping Yang, Shaoxiang Zhang, and Chun- ming Li. Distance regularized two level sets for segmentation of left and right ventricles from cine-mri. Magnetic resonance imaging, 34(5):699-706, 2016.
Matthew M Loper and Michael Black. Opendr: An approximate differentiable renderer. In Euro- pean Conference on Computer Vision. pp. 159-169. Springer, 2014.
Diego Marcos, Devis Tuia, Benjamin Kellenberger, Lisa Zhang, Min Bai, Renjie Liao, and Raquel Urtasun. Learning deep structured active contours end-to-end. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 8877-8885, 2018.
Pablo Marquez-Neila, Luis Baumela, and Luis Alvarez. A morphological approach to curvature- based evolution of curves and surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1):2-17, 2014.
Oleg Michailovich, Yogesh Rathi, and Allen Tannenbaum. Image segmentation using active tours driven by the bhattacharyya gradient flow. IEEE Transactions on Image Processing, 16(11): 2787-2801, 2007.
Inés C Moreira, Igor Amaral, Inês Domingues, António Cardoso, Maria Joao Cardoso, and Jaime Cardoso. Inbreast: toward full-field digital mammographic database. Academic radiology, 19 (2):236-248, 2012.
Tuan Anh Ngo, Zhi Lu, and Gustavo Carneiro. Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance. Medical image analysis, 35:159-171, 2017.
Sandro Queirós, Daniel Barbosa, Brecht Heyde, Pedro Morais, João I. Vilaça, Denis Friboulet, Olivier Bernard, and Jan D D'hooge. Fast automatic myocardial segmentation 4d cine cmr datasets. Medical image analysis, 18(7):115-1111, 2014.
P Radau, Y Lu, K Connelly, G Paul, A Dick, and G Wright. Evaluation framework for algorithms segmenting short axis cardiac mri. The MIDAS Journal-Cardiac MR Left Ventricle Segmentation Challenge, 49, 2009.
Franz Rottensteiner, Gunho Sohn, Jaewook Jung, Markus Gerke, Caroline Baillard, Sébastien Bénitez, and U Breitkopf. International society for photogrammetry and remote sensing, 2d semantic labeling contest. labeling.html.
URL http: hatt:://www/ http::///www...mm isprs org commi sa sansns///sss.. /ttom//cc/mmmmmm/mmm/////// comm3/ 4
semantic- label ing. html
Christian Rupprecht, Elizabeth Huaroc, Maximilian Baust, and Nassir Navab. Deep active contours.
arXiv preprint arXiv: 1607.00007000 2016.
Vivek Kumar Singh, Hatem A Rashwan, Santiago Romani, Farhan Akram, Nidhi Pandey, Md Mostafa Kamal Sarker, Adel Salch, Meritxell Arenas, Miguel Arquez, Domenec Puig, et al. Breast tumor segmentation and shape classification in mammograms using generative adversarial and convolutional neural network. Expert Systems with Applications, 139:1 12855. 2020.
Vadim N Smelyansky, Robin Morris, Frank O Kuchnel, David A Maluf, and Peter Cheeseman. Dramatic improvements to feature based stereo. In European Conference on Computer Vision, pp. 247-261 Springer, 2002.
Xiaolu Sun, C Mario Christoudias, and Pascal Fua. Free-shape polygonal object localization. In European Conference Computer Vision, pp. 317-332. Springer, 2014.
12

---

Oliver Wang, Suresh K Lodha, and David P Helmbold. A bayesian approach to building footprint extraction from aerial lidar data. In Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), pp. 192-199. IEEE, 2006.
Shenlong Wang, Min Bai, Gellert Mattyus, Hang Chu, Wenjie Luo, Bin Yang, Justin Liang, Joel Cheverie, Sanja Fidler, and Raquel Urtasun. Torontocity: Seeing the world with a million eyes. arXiv preprint arXiv: 1rX.v::612200422: 2016.
Yue Wang. Eam Khwang Teoh, and Dinggang Shen. Lane detection and tracking using b-snake. Image and Vision computing, 22(4):269-286) 2004.
Zian Wang. David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Object instance annotation with deep extreme level set evolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7500-7508, 2019.
Paul A Yushkevich, Joseph Piven, Heather Cody Hazlett, Rachel Gimpel Smith, Sean Ho, James C Gee, and Guido Gerig. User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage. 31(3):1116-11
2006.
Wentao Zhu, Xiang Xiang, Trac Tran, Gregory D Hager, and Xiaohui Xie. Adversarial deep structured nets for mass segmentation from mammograms. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 847-850. IEEE, 2018.
13


Reviews:
[{"review_id": "r1xFi7sFFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: The rating changed from '1: Reject' to '6: Weak accept' after the rebuttal. See below for my reasoning.\n\nThe submission considers two-class image segmentation problems, where a closed-contour image region is to be specified as the 'object'/region of interest, vs. 'no-object'/background. The approach taken here is end-to-end learning with an active-contour type approach. The main loss, in contrast to other active contour approaches, contains a direct difference of the estimated polygon area vs. ground truth polygon area.\n\nThe applied method seems conceptually quite simple (as admitted by the authors in Section 5), and the neural rendering approach seems quite neat, but both method presentation (Section 3) and evaluation (Section 4) seem incomplete and leave significant open questions.\n\nOne of my main concerns is related to the fact that the displacement field is static and, according to Figure 1 and Algorithm 1, is evaluated only once per image.\nIf the displacement field J is not conditioned on the current polygon shape (and this does not seem to be the case), then I am wondering why T iterations in the sampling/rendering part are necessary at all. When only considering L_seg, the optimal solution should be found within one iteration, since the displacement field will be able to provide the optimal answer. So maybe these iterations are only necessary when L_B and L_K are incorporated?\nIn any case, it is unclear why even L_seg is accumulated (using unweighted mean) over all T iterations before being backpropagated. Does this mean that these iterations are not meant to yield shape improvements? Why is ||M^t-M|| not evaluated per iteration, for the purpose of minimization?\nIt is also not sufficiently clear whether M^t in Equation 4 is a filled polygon mask, or if the mask is just related to the boundary (with a certain width). In absence of explanatory image material, I am assuming the former.\nOverall the method description remains weak, since obvious questions/concerns such as the above are not addressed.\n\nThe experimental results look good from a quantitative point of view, and indeed, the strongest baselines, e.g. DARNet, are outperformed significantly in many cases.\nSection 4 mostly focuses on quantitative evaluation and lots of picture examples, but fails to give insight into particular behaviors, failure cases, etc.\nThe evaluation procedure is cast a bit into doubt by two things: 1) In Figure 4, the initializations (blue circles) between the DARNet method and the proposed method are very different in size. I am wondering if this then still constitutes a fair comparison, and I have some doubts there. 2) In Figure 6, the proposed method consistently looks much worse than the DARNet baseline (and, in contrast to the baseline, completely fails for 4 vertices), unless the colors were swapped in the description.\n\nOverall, I do not think the submission is in a good enough shape for acceptance.\n\nMinor remarks:\n- The values for lambda_1 and lambda_2 seem to come out of thin air, and they also seem quite small. It needs to be mentioned how they were determined.\n- Data augmentation by rotation seems to be missing several values (between 270 and 260 degrees) and also not evenly spaced. Is this a typo or on purpose? In the latter case, an explanation is needed, since this seems weird.\n- Section 4.3: There is no \"Figure 4.2\", I assume you mean Figure 6, which otherwise remains unreferenced.\n- Section 4.3, Ablation Study: Don't use the word \"derivatives\" when you're talking about variations.\n- Section 4.3, Ablation Study: \"even without no auxiliary loss\" -> remove \"no\" or change \"without\" -> \"with\"\n\n-------------\nPost-rebuttal comments:\n\nI have read the revised version, as well as the other reviews and all authors' comments. The inclusion of an evaluation on a larger-size data set is highly appreciated, and seems to indeed validate the robustness of the method. Typos were fixed, including the switched color descriptions in Figure 7 (which should not have passed initial submission in the first place, if the text had been proofread properly).\n\nSeveral of the open questions (e.g. \"Why is L_seg accumulated before backpropagation?\", \"Why is the algorithm iterative if the displacement map is computed only once, if not for the other loss terms?\", \"Choice of values for lambda_1, lambda_2\", Initial diameter of initialization\") have been somewhat addressed by the authors in the rebuttal comment, though not in great detail.\n\nBased on the quality of the results across data sets, and because I believe that the timely publication of this rather simple method can benefit further research in this area, I have adjusted my score to a 'Weak accept'. That said, I still do not think it is a good manuscript, and my score should be seen as a massive benefit of the doubt toward the authors.\n\nMost importantly, above questions have NOT been adequately addressed in the actual revised text. The authors claim they have \"improved the manuscript considerably\", but yet I see more reasoning for certain choices described in the comment here than in the actual manuscript. Most of the changes are in Section 2 and the new Section 4.3, but not much relevant to my comments changed in Section 3.\n\nFor example, balloon and curvature losses aside, it is still not clear why an iterative approach would be helpful past the first iteration. An ideal displacement map that is not conditioned on the polygon should point, for each pixel, straight to the closest contour pixel. It is clear to me that this may not be what is being learned when multiple iterations are forced, yet it is not addressed why multiple iterations should be beneficial. (I could see why they could be beneficial if the approach was conditioned on the polygon vertices, to avoid vertex collapsing, but it's not.)\n\nA good submission preempts these kinds of questions by addressing them carefully. What seems crystal clear to the authors will not be crystal clear to every reader. The authors should be more careful to include their reasoning in the actual text, which I believe this is essential for proper, easy understanding of the paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syl8NXg6tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates an image segmentation technique that learns to evolve an active contour, constraining the segmentation prediction to be a polygon (with a predetermined number of vertices).  The advantage of active contour methods is that some shapes (such as buildings) can naturally be represented as closed polygons, and learning to predict this representation can improve over pixelwise segmentation.\n\nThe authors propose to learn an image-level displacement field to evolve the contour, and a neural mesh renderer to render the resulting mask for comparison with the ground truth mask.  The performance compared to prior learning-based active contour methods is impressive.\n\nIn section 4.3, there’s a reference to a “gap in performance” between the proposed method and DARNet and a reference to a \"low number of vertices,\" but a comparison between the two methods as the numbers of vertices is varied seems to only be present in Fig. 6 -- it would be interesting to see an explanation of the discrepancy for the lower number of vertices seen in this figure.\n\nOverall, due to the relative simplicity of the approach and impressive performance compared to prior learning-based approaches I recommend to accept.\n\nPost-rebuttal:  I maintain my recommendation.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xhlGTQqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a straightforward method for end-to-end learning of active contours, based on predicting a dense field of 2D offsets, and then iteratively evolving the contour based on these offsets. A differentiable rendering formulation by Kato et al is employed to make the process of aligning a contour to a GT mask differentiable. \n\nThe model shows rather compelling results on small datasets, and is very simple, with very strong parallels to active contours, which is a strength. The results improve those of DARNet, which to the best of my knowledge is the main published work in the space other than Curve-GCN. One thing that would be helpful, is  to have an experiment on a large dataset, such as Cityscapes -- right now all the datasets are testing the model in only the small-data regime. Perhaps in a supplement, it would also help to do ablation of how input image / dense deformation resolution affects the result quality -- the input can be subsampled by powers of 2 for the experiment. \n\nAs Amlan Kar helpfully points out, the work heavily overlaps with his approach \"Fast Interactive Object Annotation with Curve-GCN\", CVPR 2019, which is not cited or compared to. Curve-GCN similarly utilizes differential rendering (only a different variant) to match the GT masks. To me, the main difference wrt Curve-GCN is that explicit dense displacement fields are generated by the net and used directly for the iterative refinement steps, while Curve-GCN leverages implicit feature embeddings and uses GCN layers for their iterative updates. A second main difference is that Curve-GCN supports splines and interactive editing, while the proposed approach does not. Beyond these, there are multiple other differences that the authors point out, but those are more of a technical nature. Unfortunately, without a more direct comparison, it is very difficult to evaluate the design choices in the two approaches, which I feel is necessary for proper understanding of the paper. \n\nAFTER REBUTTAL: The authors made additions that covered my concerns, so I have switched my recommendation. \n\nA few more minor clarity / presentation issues. \n-- “The recent learning-based approaches are either non-competitive or proven to be effective in the specific settings of building segmentation\". It's not exactly clear what the point is in the context. Which \"learning-based approaches\"? \n-- Typo 'backpropogation'. \n-- A little better explanation of how a differentiable renderer of Kato works would have been helpful. \n-- Figure 3 is not referenced in the text, takes a little bit of thought why it is relevant (helps explain Fig 1, but maybe better to show it prior to Fig 1). \n-- In Eq 4 it’s not clear what F is.  (I see it is explained in Algorithm box, but that's much later)\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents a differentiable take on classic active contour methods, which used to be popular in computer vision. The method is sensible and the results are strong. After the revision, all reviewers recommend accepting the paper.
Decision: Accept (Poster)