Continuous U-Net: Faster, Greater and Noiseless
Authors: Chun-Wun Cheng
Christina Runkel
Lihao Liu
Raymond H. Chan
Carola-Bibiane Schönlieb
Angelica I Aviles-Rivero
Venue: Accepted by TMLR ()

Abstract: Image segmentation is a fundamental task in image analysis and clinical practice. The current state-of-the-art techniques are based on U-shape type encoder-decoder networks with skip connections called U-Net. Despite the powerful performance reported by existing U-Net type networks, they suffer from several major limitations. These issues include the hard coding of the receptive field size, compromising the performance and computational cost, as well as the fact that they do not account for inherent noise in the data. They have problems associated with discrete layers, and do not offer any theoretical underpinning. In this work we introduce  continuous U-Net, a novel family of networks for image segmentation. Firstly, continuous U-Net is a continuous deep neural network that introduces new dynamic blocks modelled by second order ordinary differential equations. Secondly, we provide theoretical guarantees for our network demonstrating faster convergence, higher robustness and less sensitivity to noise. Thirdly, we derive qualitative measures to tailor-made segmentation tasks. We demonstrate, through extensive numerical and visual results, that our model outperforms existing U-Net blocks for several medical image segmentation benchmarking datasets.

Published in Transactions on Machine Learning Research (04/2024)
Continuous U-Net: Faster, Greater and Noiseless
Chun-Wun Cheng
cwc560ccmm. uk
Deportment of Applied Mathematics and Theoretioal Physics, University of Combridge
Christina Runkel
cr661 @cam..cmmm.cc ac. uk
Department of Applied Mathematics and Theoretical Physics, University of Combridge
Lihao Liu
11610@cam. ac. uk
Department of Applied Mathematics and Theoretical Physics, University of Cambridge
Raymond H Chan
chymond.cham@cittyu.co edw..
Department Mathematics, City University of Hong Kong
Hong Kong Centre for Cerebro-Cardiovaser Health Engineering
Carola-Bibiane Schönlieb
chs31 @com.ac.mm....mmmm... uk
Department of Applied Mathematics and Theoretionl Physics, University of Cambridge
Angelica I Aviles-Rivero
ac. uk
Department of Applied Mathematics and Theoretical Physics, University of Combridge
Reviewed on OpenReview: https: // openreview. net/ bet/forunt forum? id= ongi2oe3Fr
Abstract
Image segmentation is a fundamental task in image analysis and clinical practice. The current stale-offthe-arl Lechniques are based on U-shape type encoder-decoder networks with skip connections called U-Net. Despite the powerful performance reported by existing U- Net type networks, they suffer from several major limitations. These issues include the hard coding of the receptive field size, compromising the performance and computational cost, as well as the fact that they do not account for inherent noise in the data. They have problems associated with discrete layers, and do not offer any theoretical underpinning. In this work we introduce continuous U-Net, novel family of networks for image segmentation. Firstly, continuous U-Net. is a continuous deep neural network that introduces new dynamic blocks modelled by second order ordinary differential equations. Secondly, we provide theoretical guarantees for our network demonstrating faster convergence, higher robustness and less sensitivity noise. Thirdly, we derive qualitative measures to tailor-made segmentation tasks. We demonstrate, through extensive numerical and visual results, that our model outperforms existing U-Net blocks for several medical image segmentation benchmarking datasets.
1 Introduction
Image segmentation is fundamental task in medical image analysis and clinical practice. It is critical component in several applications including diagnosis, surgery-guided planning and therapy. Manual seg- mentation of such medical datasets is time-consuming, and increases financial cost. The advent of deep learning, and in particular Fully Convolutional Neural Networks (FCNNs) (Long et al., 2015), opened the door to automatic segmentation techniques. The current state-of-the-art techniques are based on U-shape type encoder-decoder networks with skip connections (Qin al., 2020; Liu et al., 2020; Zhou al., 2019;
"Equal Contribution.

---

Published in Transactions on Machine Learning Research (04/2024)
Su et al., 2021; Valanarasu & Patel, 2022). This family of networks has demonstrated astonishing results due to their representation learning capabilities, and the ability to recover fine-grained details.
The seminal paper of Ronneberger et al. (Ronneberger et al., 2015) introduced the U-Net model for Biomed- ical Image Segmentation. More precisely, U-net seeks to capture both the context and the localisation features. Firstly, it uses skip connections to provide additional information that helps the decoder to gen- erate better semantic features. Secondly, consists of a symmetric encoder-decoder scheme, which reduces the computational cost. The impressive performance of U-Net motivated the fast development of several U-Net variants e.g., (Qin et al., 2020; Su et al., 2021; Valanarasu & Patel, 2022; Etmann et al., 2020), and its usage in wide variety of clinical data.
Despile the powerful performance reported by existing U-Net. lype networks, they suffer from several major limitations. Firstly, they hard code the receptive field size, which requires architecture optimisation for each segmentation task. Optimized receptive views increase the accuracy (Chen et al., 2018a). However, the limited computational memory forces trade-off between network depth, width, batch size, and input image size. Since deep learning makes the prediction by discretising the solution layer by layer, this has a very high computation cost. Secondly, the current existing U-Net models do not account for inherent noise that affects the predictions. e Thirdly, U-Net is a discrete neural network with discrete layers, but medical image data is continuous. Fourthly, existing U-net variants do not provide any theoretical underpinning.
Can one design a U-type network that overcomes the aforementioned major is- sues of eristing models? This is the ques- tion that we address in this work. Our work is motivated by deep implicit learn- ing, and continuous approaches based on Neural Ordinary Differential Equa- tions (Neural ODEs) (Chen et al., 2018b; Dupont et al., 2019) in particular. The body of literature has explored Neu- ral ODEs mainly for image classification tasks. So far, Neural ODEs have only been used for a small number of applica- tions as designing Neural ODEs for more complex tasks is far from being trivial.
(a) Continuous U-Net
DDG soiver
Dynamic Block
-0h
+00
+0g
*1(1 (200)
atle) -
='0c) (()(())).))))
Discrete Block
(b) U-Net
Figure 1: Visual comparison of our continuous U-Net vs. U- Net (and variants). The zoom-in views display the difference be- tween discrete blocks in U-Net and our proposed dynamic blocks.
We underline that U-Nets (and majority of its variants) are designed discrete neural networks. This is in contrast to the continuous nature of medical data. In this regard, Recurrent Neural Networks (RNNs) are more appropriate. However, non-uniform intervals for handling medical data can compromise the performance. Neural ODEs are suitable for continuous data since they output a vector field. Another major problem of the discrete neural network is the issue of overfiting/undeeritttin underfitiing (Bilbao Bilbao, 2017). However, we show that looking at U-Net from the lens of continuous dynamical systems, this issue can be mitigated.
Another major keypoint is that the performance of image segmentation depends partly the receptive field of the networks. Duc to the limited GPU memory, U-Net and other variants force a trade-off between archi- tecture design and input image size. Different techniques have been developed for this problem. Examples are dilated convolutions (Folle et al., 2019) and reversible blocks (Brügger et al., 2019). These architectures reduce the computational cost because of the reduced number of stored active functions. However, the mem-- ory cost of these architectures is still associated with the depth of the model. In our work, we sook to address this problem using trace operation via the adjoint sensitivity method with O(1) memory cost. That is, for neural networks based on ODEs, reversibility is built into the architecture by construction. Therefore, matter the model's complexity, one can always provide the benefit of constant memory cost. In this work, we propose a novel family of networks that we call continuous U-Nets for medical image segmentation.
2

---

Published in Transactions on Machine Learning Research (04/2024)
An overview of our approach in contrast to standard U-Net can be found in Figure 1. In particular, contributions are:
We propose a new network, called Continuous U-Net for medical image segmentation. Continuous U-Net is continuous deep network whose dynamics are modelled by second order ordinary differential equations. We view the dynamics in our network the boxes consisting of CNNs and transform them into dynamic blocks to get solution. We introduce the first U-Net variant working explicitly in higher-order neural ODEs. We highlight:
Faster Convergence. By modelling the dynamics in a higher dimension, we provide more flexibil- ity to learn the trajectories. Therefore, continuous U-Net requires fewer iterations for the solution, which is more computationally efficient and particular provides constant memory cost. (Proposi- tion 1 & Corollary 1)
More Robustness. We show that continuous U-Net is more robust than other vari- ants (CNNs) and provide an intuition for this (Theorem 3). Moreover, we demonstrate that our dynamic blocks are reliably useful (Theorem 1).
Greater:
Noiseless. We show that continuous U-Net is always bounded by some range whilst CNNs are not. Also, our network smoother than existing ones leading to better handling the inherent noise in the data.
Underpinning Theory, Continuous U-Net is the first U-type network that comes with theoret- ical understanding.
& Open the "box" of ODE-solvers. Existing works lack guidelines for choosing the best ODE solver. We derive qualitative measures for the choice of different ODE-solvers. At the practical level, this means that our framework can be tailor-made for various segmentation tasks (Global Error & Theorem 2).
& We demonstrate, through extensive numerical and visual results, that our proposed continuous U-Net outperforms existing U-type blocks. Moreover, we show that our proposed network stands alone performance-wise without any additional mechanism, and its performance readily competes with other mech- anisms including transformers, nested U-Nets and tokenised MLP.
2 Related Work
The task of image segmentation, for medical data, has been widely investigated in the community, whith state-of-thh-art techniques relying in U-type architectures, alone or in combination with additional mecha- nisms. In this section, we review the existing techniques turn.
2.1 U-type Nets: Block Based Perspective
The gold-standard U-Net model was introduced by Ronneberger et al. (Ronneberger al., 2015) for biomedi- cal image segmentation. U-Net consists of four main components blocks of neural network layers, downsam- pling, upsampling and concatenation operations. The astonishing performance reported by U-Net motivated the fast development of a wide range of U-type variants, where the key difference is largely based on different types of blocks.
The standard U-Net architecture (Ronneberger et al., 2015) uses convolutional blocks, followed by an acti- vation function. Further work in this area was introduced in (Zhang et al., 2018b), where the authors use residual blocks (He et al., 2016) yiclding to ResUNet. To facilitate learning, the input x of each convolu- tional block Fo is added to the output via skip-connections DenseUNet (Li et al., 2018) where densely connected blocks used in the U-Net structure. Their work follows the principles of DenseNet (Huang et al., 2017). A different U-Net variant is based on inception blocks (Zhang et al., 2018a). The notion is based on computing convolutions with varying kernel sizes in parallel to then concatenate their outputs. Common kernel sizes are x 1, 3 x 3 and 5 x 5. This approach aims at computing different levels of features by choosing different kernel sizes.
that y Fo(x) + x. Li et al. introduced
Another U-type network uses pyramid pooling blocks (Zhao et al., 2017), where the idea is to pool the input to get different input sizes (e.g., 1 1x C, 2x2x C and 4 4 x C for the number of channels C)2, and apply
3

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Overview of properties of our continuous U-Net vs, existing U-type networks.
PROPERTIES
continuous U-Net
OTHER U-NETS
Parameters Efficiency
x
Constant Memory Cost
X
Continuous Network
X
Noise Resistant
X
Beyond homomorphic
X
Transformations
Reversible
X
Theoretical Underpinning
/
X
convolutions on each of them before upsampling to the original input size. The upsampled output is then concatenated channel-wir wise. Most recently, the work of Pinckaers et al. (Pinckaers & Litjens, 2019) uses first order neural ODEs in a U-Net setting.
2.2 U-type Nets with Additional Mechanisms
The aforementioned techniques seek to create variants of U-Net solely based on the block types. Another set of techniques has instead focused on creating additional mechanism in U-Net structures. Zhang et al. introduced ResUNet (Zhang et al., 2018b), where residual blocks along with several additional mechanisms such as atrous convolutions (Chen et al., 2017), pyramid scene parse pooling (Zhao et al., 2017) and multi task inference (Ruder, 2017) were used. An attention mechanism was introduced U-Net where the combination of both is known as Attention U-Net (Oktay et al., 2018). Attention gates are used to filter features before concalenating the upsampled input and the skip connection in the decoder part of the U-Net.
In more recent works, DynUNet (Ranzini et al., 2021) was introduced as combination of two works. It takes the heuristic rules and sctting from nnU-Net (Isensee et al., 2019), and the optimisation scheme of Futrega et al. (Futrega et. al., 2022) for searching an (sub-)optimal network structure. Self-atlention mechanisms have also been explored for U-Net. Chen et al. proposed TransUNet (Chen et al., 2021) that combines a transformer encoder, with a self-attention mechanism, and a classical convolutional neural network docoder. The most recent. U-type network, called UNeXt, was introduced in (Valanarasu & Patel, 2022). UNeXt is convolutional multi-layer perceptron (MLP) based network, which uses tokenised MLP blocks with axial shifts. In comparison to transformer based approaches like TransUNet, UNeXt needs only a small number of parameters.
2.3 Existing Techniques & Comparison to Ours
We provide summary of properties for continuous U-Net and existing U-Type nets in Table 1. More precisely, existing U-type networks (Ronneberger al., 2015; Zhang et al., 2018b; Li et al., 2018; Zhang et al., 2018b; Oktay et al., 2018; Ranzini et al., 2021; Valanarasu & Patel, 2022) discretise the solution layer by layer which yields to high computational cost. Contrary, our work is a continuous architecture that can be solved via the adjoint method Chen et al., 2018b), which translates in constant memory cost. Unlike existing U-type blocks (Ronneberger al., 2015; Zhang et al., 2018b; He et al., 2016; Li et al., 2018; Zhang et al., 2018a; Zhao et al., 2017), we propose new dynamic blocks modelled by second-order noural ODEs, which are not restricted to homomorphic transformations. Moreover, opposite to (Pinckaers & Litjens, 2019) our dynamic blocks at least twice contimuously differentiable resulting in being robust to noise.
Most recently, new mechanisms have been used along with U-Net including attention, self-attention, trans- formers, heuristic rules and nested U-Nets (Zhang et al., 2018b; Oktay el al., 2018; Ranzini el al., 2021; Chen et al., 2021; Valanarasu Patel, 2022). Unlike these works, our continuous U-Net does not use any additional mechanism. We instcad introduce new dynamic blocks. Therefore, the philosophy of our work is closer the block based U-Net techniques (Ronneberger et al., 2015; Zhang et al., 2018b; He et al., 2016; Li et al., 2018: 8; Zhang et al., 2018a; Zhao et. al., 2017) and not directly comparable to those with additional
4

---

Published in Transactions on Machine Learning Research (04/2024)
mechanisms. We highlight that our continuous U-Net does not any additional mechanism, opening the door to new research line conlinuous U-type networks. Finally and to the best of our knowledge, this is the first U-type architecture that provides underpinning theory.
3 Proposed Technique
This section contains the key parts of our proposed continuous U-Net (i) we present our proposed dynamic blocks and how we model the dynamics of our network using second order Neural ODEs, (ii) our derived quality measure for maieor-made segmentation tasks and (iii) robustness and noise properties of our network.
3.1 Unboxing Continuous U-net
Continuous U-Net greatly differs from existing U-type networks since it is modelled as a continuous ap- proach. That is, we avoid computing predictions by discretising the solution layer-by-layer, which involves a high computational cost. In particular, our network models the dynamics by taking the CNN boxes and transforming them into ODE blocks. Unlike existing works on Neural ODEs, we go beyond the standard learning setting by designing new U-type architecture using higher order neural ODEs. Why designing higher-order blocks? The standard Neural ODE setting fails to learn complex flows. As the training pro- gresses and the flow becomes more complex, the number of steps required to solve the ODE increases (Chen et al., 2018b; Grathwohl et al., 2018). This is one of the limitation of neural ODEs. Although augmented neural ODEs (ANODE) (Dupont et al., 2019) were proposed to mitigate this issue to some degree, ANODE is still a first order ODE. This property of first order ODE limits the performance in terms of computational speed and learning of the flow. Moreover and unlike our work, those approaches are stand-alone techniques whilst our work uses a different principle building on ODE blocks construct a new U-type network that handle segmentation for complex data as in the medical domain.
Dynamic Blocks. Continuous U-Net is formulated from dynamical systems perspective. We transform the CNN boxes into second order ODEs blocks. Our blocks work under the definition of second order Neural ODEs, which read:
z"(t)= = f(w) z'(t),,((t),tt))) a(to) Xo. x'(to) g(x((t))
(1)
whose velocity is described by a neural network f()) with parameters f, and initial position given by the points of a datasct Xg. We now discuss how our dynamic block can be reliably useful and computational efficient.
Proposition
Any given high-order Neural ODEs can be transformed into a system of first-order Neural ODEs.
Proof.
Consider a first order Neural ODEs:
x' x'(1) (x)()) æ(tu) Xo
(2)
where velocity is described by neural network f(v) with parameters 0y. Define an mhth order neural ODEs
as:
=t) = (()...))(
x(to) = =xn-
(3)
x m- - = g(x((to) 2'
5

---

Published in Transactions on Machine Learning Research (04/2024)
where Z², and f(a) refers to neural network with parameters 8--
=(t)= = x1
x(to) x'(6))
x'(1) x2(1)
Let z(1)= =
.2(10)=
(4)
(1) 2mm (t)
x (to)
then the mhth order Neural ODFs becomes:
x2(()
(t)(1) : ((a) (x1) (t),,,
z((1) =
S (2(1,,,,))
(5)
We therefore can always transform an m th order Neural ODEs into a first order Neural ODEs. Note that f(a) and f(r) denote networks with distinct weight sets.
The way to demonstrate that the th order ODE is reliably useful is through well-posedness:
Theorem
An m th Order Neural ODEs with neural network f(v) is well-posed (in the sense of Hadamard) if f(v)
is Lipschitz.
Proof. Using Proposition 1, we show that any given high order ODE can be transformed into a system of first order ODEs. Since neural network usually Lipschitz (e.g.(Virmaux Scaman, 2018; Fazlyab et al., 2019)) so we can sssumed f(e) is lipschitz. Let f: [0 T] xRd Hg be a continuous function depends on and uniformly Lipschitz in Let Xn Rd by Picard's Existence Theorem, Then there exists unique differentiable x [O,T] Fd satisfying
x'(1)= = (0)())))))) xx xxtn)= = Xo
(6)
It is then to say that if
Continuous U-Net breaks the barrier the receptive field limitation that existing U-type networks have. In existing networks, the key factor affecting the memory cost is the storing of intermediate hidden unit activation functions. Our dynamic blocks only need single point to reconstruct the entire trajectory by forward and backward iterations. That is, continuous U-Net offers constant memory cost for segmentation. We guarantee this property our blocks trace operation via the adjoint sensitivity method with O(1) memory cost.
Corollary 1
Our second order Neural ODE's dynamic blocks can be solved by using the first order adjoint method.
Proof.
Our dynamic blocks are based on second order neural ODEs. We therefore set the order to m in Proposition 1 which implies z' z'(1) = (w) (z))
Our second order dynamic blocks can then solved by first-order adjoint method. At the practical level, this mcans that we do not need to store the layers one-by-one the architocture. We need any single point reconstruct the entire trajectory by forward and backward iterations. Therefore, offer a constant memory cost..

---

Published in Transactions on Machine Learning Research (04/2024)
3.2 Opening the ODE-solver Box
When working with Neural ODEs open question is how to choose the best ODE solver? In this section, we derive qualitative measures for the choice of different ODF-solvers. At the implementation level, this means that our framework can be tailor-made for various segmentation tasks. In this section, we open the ODE-solver 'box' and derive an error analysis for better understanding on how to choose different solver for continuous U-Net
Global Error
Global Error (GE): En = a((t)) In where x(t,) denotes the exact solution and In denotes the numerical solution. GE is the error final time.
Property 1 (en oc lt). If h is small, higher occuracy resulls can be achieved. Moreover, if O(h)), we say z is p-th order where |a| & Ch² C > 0 for all < < he- That is, when is larger, the numerical method converges faster, being better method in terms of convergence rate. In order to be convergent, we need to satisfy the conditions of stability and consistency.
How GE can be used! Euler's method the most simple yet widely used onc. We take as good starting point for more advanced methods and use our defined GE to provide further intution.
Lemma
For any real number a 2 0 then the exponential function e is always 2 1 1 r.
Proof. Let f(x) ex - --- then f(0) = 1 = 0. Moreover, f'(x) ex but e" always 1 implies
(z(x) is positive and f(x) is increasing function. Thus, f(x) f(0). We get f(x) e" - 1 - 2 0 implics ex 2 1l 20
Theorem
Euler's method in our dynamic blocks (based neural ODEs) converges
x1(() =x(t)=)(t)+++++ + x<0)= =1 AEC
f oststy
and the GE al [0,4g] is O(h).
Proof. We follow the idea from (Griffiths & Higham, 2010) and extend to Neural ODEs. Euler's method for our dynamic blocks reads
En In Ahe, h h((()) (tn, (tn, (1+ Xh)a, (tn, 0f)
(8)
applying Taylor expansion to the exact solution, get
2(tn.1) x h (tn, 0,),,,)))))) + (tx)
(9)
We can get the formula of GE from the difference of equation and equation 9:
1a = x((n+1) - Tnal
(10)
=(1+ (h))x))z(tt)) xx)) +Th-1 = (1 Ah)e, + Taai
We know that eo The relation of GE for different step is given in equation 10. We can combine this with the local truncation error. Substituting 1,2 into equation 9 and using en 0, e T and

---

Published in Transactions on Machine Learning Research (04/2024)
22 (1+hA)ez + === (1 1AAT + T2 gives
(n-1) (1)))))0-1) cn (1+ hA) (n-
(1+ n-2 +...
....t +Tn
(11)
11)
By Lemma 1, we know MMN 2 + h/1l 2 11 + hAI. Similarly, eliity 2 c(n-j)MA = cliinoss 2 11 hx We know that [Tyl & Ch² which implies,
1 On Σ (1+ Ah)" ne ch²
(12)
Since nh is finite which implies en is also finite. Thus, lepl s Ch en = O(h) Therefore, Euler's mothod applied to our dynamic blocks converges at a first-order rate, i.e,, p 1.
Our Theorem 2 focuses on the Euler method, which is a simple RK method with order 1. to clarify that since higher-order RK methods are extensions of Fuler's method, proving the theorem for Euler's method establishes a baseline from which the behavior of higher-order methods can be inferred. We remark that this foundational proof is sufficient to demonstrate the theorctical properties we are showcasing, as higher-order RK methods would follow the same conceptual line, with additional complexity that does not change the underlying theory presented.
We would like
Whilst Euler's method is widely used as it is simple and always convergent, it is not accurate, and limited at the practical level. The GE is only (h). We therefore provide further intuition on the family of linear multistep method (LMMs). LMMs provide a higher order p which converges faster than Euler's method.
For example, explicit Adams- Bashforth and implicit Adams- Bashforth Moullon LMMs with a higher convergent rate. The GE of these methods is O(h)). From mathematical point of view, it is known that a convergent LMM is consistent. However, the reverse does not hold, and Zero-stability" is also re- quired (Dahlquist, 1956). That means that Consistent + zero-stability imply convergence for these models. If we want a method to be convergent, we need both conditions. This is the intuition behind why high order in linear multistep methods is not possible. Finally, we discuss the Kunge-Kutta (RK) method, which is another widely used and effective method. Euler's method and midterm method are also type of RK methods with lower order. RK methods are one-step methods composed of a number of stages. The midpoint method is a RK2, i... two stage method while Euler's method is RKI, i.e., a 1 stage method. Is it possible to always find RK method of s stages (order s)? This is not true for s >>44 4.
What method is the best then? RK methods are a one-step methods, and the linear multistep hed is a two step method. Compared to linear multistep, RK does not have to treat the first few steps taken by a single- step integration mothod as special cases. Morcover, RK methods are very stable. Based on our derived error analysis along with the aforementioned factors, we found that our continuous blocks, continuous U-Net for segmentation, greatly benefit performanco-wist when using RK4. Our analysis in this section provides an answer to the open question on the solver when working with Neural ODEs. We remark that existing works on neural ODEs lack such analysis to provide a clear guidence on the solver.
3.3 Continuous U-net: Greater and Noiseless
This section provides the intuition on why our continuous U-Net is more robust and noiseless than existing C-type architectures.
Our dynamic blocks are defined as second order Neural ODEs. Existing works have shown empirically that the properties inherent to ODFs make them more robust. than CNNs in terms of robustness, e.g., (Yan al., 2019). The key idea show such robustness comes from the ODEs integral curves as follows, which is detailed next.
Stability specifically zero stability, relates to the method's capability manage error propagation throughout the compu- tational iterations. This is assessed by the root condition: all roots of the characteristic polyuomial should satisty Ir| with roots where Ir| = being simple. Consistency involves the method's local truncation error reducing to zero as the step size h approaches sero.
8

---

Published in Transactions on Machine Learning Research (04/2024)
Theorem 3
ODE integral curves do not intersect (Coddington & Levinson, 1955, Ch. 1). If 21 (f) and 22()) are two ODE solution with different initial value of the same function, then 21 + (2 for all te [0. oo). The proof follows (Coddinglon & Levinson, 1955).
Why is Theorem 3 relevant? Since ODE integral curves do not intersect, we can prove that some range always bounds the output of our dynamic blocks whilst. the output of CNNs not bounded by any range. This proof leads to Neural ODEs being more robust than CNNs. In particular, for our dynamic blocks based on second order ODEs, we can further support the robustness as follows: If we take a derivative on a function, we require the function to be smooth. A significant difference between the first-order ODE and second-order ODE is that the latter is at least twice contimuously differentiable. However, a first-order ODE requires once continuously differentiability yielding to our dynamic blocks being smoother than first-order ODEs and therefore less sensitive to noise than existing techniques. That is, the inherent smoothness of socond-order ODEs, characterised by at least twice continuously differentiable solutions, contributes to noise suppression by ensuring smoother trajectory of the dynamic block's output. This smoother trajectory is less alfected by small variations in initial conditions, which translates to lower sensitivity to input noise.
Secondly, existing U-type networks can only learn smooth homcomorphisms, which is one of the modelling disadvantages. Our blocks solve this problem by providing extra dimensions through the second order design. Moreover, our dynamic blocks are a physics-based model that better captures the nature of segmentation. We also remark that our second-order dynamic blocks are parameter efficient as no parameters are required 2' (to) = g(x((t)) 0g) That's why second order Neural ODEs is most robust compared first order Neural ODEs.
Why our continuous U-Net improves for segmentation? Our model integrates second-order ODE dynam- ics, which inherently allows for variable receptive field sizes without the need for manual optimisation. This adaptability leads to enhanced accuracy while circumventing the computational trade-offs imposed by limited memory, which traditional U-Nets cannot avoid. The continuous nature of our model aligns more closely with the intrinsic continuity of medical image data, leading to more natural and accurate segmentation outputs. Moreover, the built-in reversibility of our architecture, enabled by the adjoint sensitivity method, maintains constant memory cost regardless of model complexity. This contrasts sharply with discrete U-Nets that lack theoretical underpinning and struggle with memory efficiency. Moreover, we remark that our second-order neural ODE model inherently resists noise due to its smoother numerical solutions, which arise from the model's twice continuously differentiable nature. This smoothness is a structural property of the model itsclf, rather than a result of training on varied data. While data augmentation can indeed increase robustness by introducing more variance during training, it does not in- herently smooth the function that the network learns. Instead, it makes the model more tolerant to the types of variations seen during training.
4 Experiments
In this section, we detail the experiments that we conducted to validate our continuous U-Net model.
4.1 Data Description & Evaluation Protocol
We expensively evaluate our continuous U-Net using six medical imaging datasets. They are highly het- erogenous covering a wide range of medical data and significantly varying in terms of image sizes, fidelity of segmentation masks and datasct sizes. An overview of the datasets used and their propertics can be found in Table 2.
Evaluation Protocol. Following a standard protocol medical image segmentation, we evaluate the performance of the proposed continuous U-Net and existing techniques using three metrics: the Dice score, accuracy and averagod Hausdorff distance. For a fair comparison, we use shared code-base for all experiments. More precisely, set learning rate of 1 x 10-3 a step-based learning rate scheduler with
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Characteristics of the datasets used in our experiments.
DATASET
Samples
#
Train
# Test
Image size
GlaS Challenge (Sirmukknwattana al., 2017)
165
85
80
352x352
STARE (Hoover et al., 2000)
20
16
512x512
Kvasir-SEG (Jha et al., 2020)
1000
800
256x256
Data Science Bowl (Caicedo et al., 2019)
841
707
134
256x256
ISIC Challenge (Gutman et al., 2016)
1279
900
379
512x512
Breast Ultrasound Images (Al-Dhabyani al., 2020)
647
518
129
256x256
GT
Continuous U-Net
GT
Continuous U-Net
GlaS Challenge
Kvasir- SEG
ISIC Challenge
STARE
Figure 2: Visual segmentation results from our proposed continuous U-Net, with the output masks overlaid in yellow for improved contrast and comparison. The left column shows the Ground Truth (GT) images, while the right column presents the corrosponding segmentation by the continuous U-Net. Extended results and comparisons against all techniques are available the supplementary material.
Table 3: Comparison in terms of Dice score, accuracy and average Hausdorff distance for existing U-Net blocks and our dynamic blocks. We denote our continuous U-Net with * The best results are highlighted in green colour.
Block
ClaS Challenge
STARE Datasel
Kvasir-SEG Dataset
DS Bowl Dataset
Type
Dicet
Acct
AHD
Dicef
Acct
AHD
Dicef
Acct
AHD]
Dicef
Acct
AHD]
PLN
0.7616
0.7824
13.27
0.7828
0.9462
8.79
0.4597
0.8595
3.96
0.9156
0.9681
3.68
RSE
0.7893
0.8100
12.15
0.7731
0.9460
8.74
0.6418
0.8754
4.57
0.9131
0.9687
3.67
DSE
0.7264
0.7516
13.93
0.7358
0.9413
8.67
0.6071
0.8649
4.58
0.9181
0.9690
3.63
INC
0.7962
0.8192
11.58
0.7750
0.9505
8.33
0.6856
0.8825
4.30
0.9210
0.9724
3.41
FO
0.8037
0.8270
10.47
0.5005
0.9241
6.28
0.7465
0.9080
3.58
0.8953
0.9625
4.00
PSP
0.5523
0.5699
14.37
0.4805
0.9251
5.77
0.3883
0.8672
0.9060
0.9655
3.74
DB
0.8469
0.8675
9.31
0.8378
0.9568
8.21
0.7922
0.9243
3.32
0.9335
0.9745
3.29
Block
BUST
Dataset.
ISTC
Challenge
Type
Dicef
Acct
AHD
Dice
Acct
AHD
PL.N
0.4724
0.9012
2.56
0.8232
0.8764
6.76
RSE
0.7179
0.9218
2.63
U.8501
0.9003
6.00
U-Nett
DSE
0.7095
0.9196
2.61
0.8376
0.8907
6.32
INC
0.7434
0.9243
2.71
0.8370
0.9050
5.83
FO 0.7893
0.9338
2.44
0.9087
0.9452
4.45
PSP
0.6098
2.46
0.8798
DB
0.8090
0.9447
2.15
0.9094
0.9433
4.46
step size of 1 and gamma value of 0.999. We use a fourth-order Runge-Kutta (RK4) solver, a batch size of 16 and train all networks for 500 epochs.
10

---

Published in Transactions on Machine Learning Research (04/2024)
GlaS Challenge
Kvasi--EG Datasel
DS Bowl Dataset
BUST Dutaset
0.8
0.8
0.6
Loss value
0.6
Loss value
0.6
Loss value
0.6
0.4
Loss
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
50
100
15c
10
30
10
20
40
Epochs
Epochs
Epochs
Epochs
Figure 3: Convergence plots for different types of U-Net blocks. The behaviour of our dynamic blocks is denoted in blue colour, where only few iterations are needed to converge whilst existing blocks need far longer iterations.
Table 4: Performance comparison, in terms of Dice score, on range of noise level We denote our continuous U-Net with *, The best results are highlighted in green colour. Refer to the supplementary material for extended results.
NOISE LEVEL EXPERIMENTS
Block
BUSI
Dutaset
Kvasir-Sog
Dataset
ISIC Challenge
Type
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.5
PLN
0.4724
0.4723
0.4722
0.4721
0.1597
0.4597
0.4597
0.4597
0.8232
0.4064
0.4064
0.4064
U-Not.
RSE
0.7179
0.2373
0,2130
0.2094
0.6418
0.4597
0.4597
0.4597
0,8501
0.4064
0.4064
0.4061
DSE
0.7095
0.1944
0,2321
0.2423
0.6071
0.4597
0.4597
0.4597
0.8376
0.4774
0.4067
0.4064
INC
0.7434
0.1930
0.1769
0.1883
0,6856
0.4597
0.4597
0.4597
0,8570
0.4064
0.4064
0.4064
PSP
0.6098
0.5809
0.5368
0.5255
0.5883
0.4668
0.4597
0.4597
0,8179
0.3979
0.3851
0.3847
FO
0.7754
0.5345
0.5154
0.5131
0.7224
0.6401
0.5964
0,5915
0.9031
0.6941
0.6734
0.6612
DB
0.8090
0.6881
0.5743
0.5569
0.7922
0.6417
0.6164
0.5972
0.9094
0.7055
0.6910
0.6746
Table 5: Ablation study on different types of ODE solvers on four medical image segmentation datasets. Fourth-order Runge-Kutta (RK4) consistently outperforms Euler, Explicit Baans-BBshforth (AB) and Im- plicit
(ABM) Dice score, accuracy and average Hausdorff distance. The best results are highligthed in green colour, Refer the supplementary material for extended results.
Solver
STARE
Dataset
DS Bowl Dataset
Kvasir-SEC SEG Dataset
ISIC Challenge
Type
Dicet
Acct
A1ID
Dice f
Acct
AIID
Dice 1
Acct
AIID
Dice f
Accf
AIID
Euler
0.7697
0.9468
8.57
0.8911
0.9539
4.37
0.5450
0.8596
4.82
0.8487
0.9003
6.91
AB
0.8066
0.9517
8.49
0.9252
0.9726
3.39
0.7850
0.9185
3.46
0.9051
0.9398
4.68
ABM
0.8015
0.9506
8.50
0.9246
0.9730
3.37
0.7714
0.9158
3.50
0.9032
0.9374
4.78
RK4
0.8378
0.9568
8.2149
0.9335
0.9745
3.29
0.7922
0.9243
3.32
0.9004
0.9433
4.46
4.2 Results & Discussion
In this section, we present all experimental results and visualisation conducted to validate our approach.
Comparison to Other Existing U-Type Blocks. We underline that our work is a stand-alone contin- nous network, where the highlight is the new dynamic block. Therefore, our main comparison is based on different types of U-Net blocks. More precisely, our comparison includes plain convolutional blocks (PLN), residual blocks (RSE), dense blocks (DSE), inception blocks (INC), pyramid pooling blocks (PSP), first order ODE blocks (FO) and our dynamic blocks (DB). We start by reporting the global results in Table 3. The displayed numbers show all metrics. Looking at the results more closely, we observe that we achieve significant improvement over most SOTA techniques, most notable on the GlaS and the Kvasir-SEG chal- lenge datasets. There are only three cases where the FO block performs similarly well than our dynamic blocks. However, a closer look at the FO block shows general difficulties to get reliable performance for the remaining datasots as for example in the STARE dataset. In contrast, our dynamic blocks report a stable and high performance across all datasets and metrics. Additionally, Figure 2 highlights visual results for our approach on different datasets.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 6: Performance comparison of continuous U-Net against state-of-the-art U-Net architectures with additional mechanisms. We denote our continuous U-Net with *, Extended results with additional three datasets can be found in the supplementary material.
GlaS Challenge
STARE Dataset
Kvasir-Sh SEG Dataset
TECHNIQUE
MECHANISM
Dicet
Acct
AHD]
Dicet
Acct
AHD
Diect
Acc1
AHDD
Attn. U-Net
Attn. Gates
0.7991
0.8173
12.10
0.8675
0.9637
7.63
0.7290
0.8997
4.35
DyaU-Net
Heuristic-R + Opt
0.8626
0.8828
9.15
0.8508
0.9575
8.25
0.7634
0.9101
4.00
U2Net
Nested U-Nots
0.8465
0.8658
9.18
0.7845
0.9539
7.84
0.7950
0.9171
3.23
UNeXt
Tokenised MLP
0.8911
0.9040
7.72
0.7866
0.9482
8.59
0.7879
0.9260
3.08
TransUnet
Transformers
0,8984
0.9145
7.11
0.8565
0.9607
7.79
0.8749
0,9422
2.70
*
X
0.8469
0.8675
9.31
0.8378
0.9568
8.21
0.7922
0.9243
3.32
Continuous U-Net : Faster & Greater To further support our theoretical findings, we evaluate the convergence of our continuous U-Net against the blocks of other existing techniques. In Figure 3 we display convergence plots for challenging datasets. We observe that our theory agrees with the experiments; our dynamic blocks offer an extra dimension, by the second-order Neural ODEs modelling, which yields to require fewer iterations for the solution than other existing U-type blocks. In closer look, we observe that our dynamic blocks only need a small number of epochs to converge whilst the other ones do not converge even with a higher number of epochs.
Continuous U-Net: Noiseless In Subsection 3.3, we proved that our continuous U-Net is less sensitive to noise than other CNN-basod U-Net architectures. We use pretrained models, adding zero mcan Gaussian noise with varying standard deviations (0.2, 0.4, 0.5) and compute the Dice score for each of them. Table 4 reports the results and highlights the robustness of our approach. Continuous U-Net consistently achieves the highest Dice scores for all noise levels. Additionally, comparing our continuous U-Net approach to the second best performing version - the U-Net with inception blocks (INC) the difference becomes even more obvious. For all datasets, adding even a small amount of noise (slandard deviation of 0.2) to the INC model, the Dice score drops massively, e.g., from 0.7434 to 0.1930 on the GlaS challenge dataset. Our approach, however, is able to deal much better with additive noise, leading to a difference in Dice score of only 0.1209. A similar pattern can be observed for the other two datasets.
Continuous U-Net Opening the ODE-Solver Box Subsoction 3.2 sheds light onto the thcoretical error analysis for different types of ODE solvers. We compare Euler's method, Explicit (AB), Implicit Adams-Bashforth-Me Bashforth-Moulton (ABM) and a fourth-order Runge- Kutta (RK4) Table 5 reports the performance for all methods and datasets. Our derived qualitative measures agree with the empirical results, the Runge-Kutta method outperforms Euler, Adam-Bashforth and Adam-Bashfore Moulton for each of the datasets in all metrics and is thus suggested for segmentation lasks.
Continuous U-Net vs. other U-type with Additional Mechanisms. We underline that main contribution in this paper is to create continuous network with our dynamic blocks. Whilst direct comparison with architectures using additional mechanisms is unfair, we ran set of experiments on current SOTA techniques with additional mechanisms including attention gates, transformers or tokenised MLPs. Firstly, we seck to demonstrate that our network stands alone without any additional mechanism. Secondly, to open the door lo a new research line to design continuous U-lype networks with additional mechanisms- which is far from being trivial. Table 6 provides the results for these experiments. Continuous U-Net is able to outperform at least two methods per dataset (GlaS challenge, STARE dataset, DS Bowl dataset, ISIC challenge), three for the Kvasir-SEG dataset and even four on the BUSI dataset.
We also compare our network's efficiency with SoTA models, emphasising our unique achievement in compu- tational efficiency without relying on complex enhancements. Specifically, our model's performance on the ISIC dataset stands out, requiring only 0.45 GFLOPs per inference, which is notably lower than UNeXt's 0.57 GFLOPs and TransUnet's 's 38.52 GFL.Oss. This efficiency leads to a significant reduction in parameters and model size, alongside a boost in inference speed. Our model outperforms the most efficient discrete U-Nets, including UNeXi, across several benchmarks, without the need for tokenised MLPs other such mechanisms, highlighting our model's ability to guarantee constant memory cost, an advantageous trait in deep learning design.
12

---

Published in Transactions on Machine Learning Research (04/2024)
5 Conclusion
Continuous U-Net is a continuous network modelled by our dynamic blocks using second order neural ODEs. We show that our approach outperforms existing U-Net blocks on six benchmarking datasets, and readily competes or even outperforms famous U-Net architectures with additional mechanisms like attention. By parameterising the model as a continuous function over time, we offer a novel perspective that allows for repeated computations within the same 'layer' conceptual layer defined by the continuous time variable t. This is in contrast to traditional U-Nets that perform single computation per discrete layer. Hence, approach can be seen as improving the capabilities of U-Nets. Our findings open the door to new research line on continuous U-type networks by introducing the to the best of our knowledge first U-type architecture that provides underpinning theory. Future work includes a theoretical extension on higher-order ODEs and integrating additional mechanisms like attention.
6 Acknowledgements
CWC acknowledges support from Department of Mathematics, College of Science, CityU, HKASR reaching out award and funding from CCMI, University of Cambridge. RHC acknowledges support from HKRGC GRF grants CityU1101120 and CityU11309922 and CRF grant C1013-21GF. AAR gratefully acknowledges funding from the Cambridge Centre for Data-Driven Discovery and Accelerate Programme for Scientific Discovery, made possible by a donation from Schmidt Futures, ESPRC Digital Core Capability Award, and CMIH, CCMI, University of Cambridge. CBS acknowledges support from the Philip Leverhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC advanced career fellowship EP/ PPP/V2942//1, EPSRC grants EP /S/026005/1 and EP EP/T003553/1, EP/NO14588/ EP EP/T011961/11 the Wellcome Innovator Awards 215733/2/19// Z and 221633///20//2 Z, the European Union Horizon 2020 research and innovation programme under the Marie Skodowska- grant agreement No. 777826 NoMADS, the Cantab Capital Institute for the Mathematics of Information and the Alan Turing Institute.
References
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020. 10
Tmanol Bilbao and Javier Bilbao. Overfitting problem and the over-training in the era of data: Particularly for artificial neural networks. In 2017 eighth international conference on intelligent computing and in information systems (ICICIS), pp. 173 177. IEEE, 2017. 2
Robin Brügger, Christian F Baungartner, and Ender Konukoglu. A partially reversible u-net for memory- efficient volumetric image segmentation. In International conference on medicall image compuling and computer-assisted intervention, pp. 429-49--90 437. Springer, 2019. 2
Juan C Caicedo, Allen Goodman, Kyle W Karhohs Beth A Cimini, Jeanelle Ackerman, Marzieh Haghighi, CherKeng Heng, Tim Becker, Minh Doan, Claire McQuin, et al. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature methods, 16(12):1247-1253, 2019. 10
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunct: Transformers make strong encoders for medical image segmentation. arXiv preprint ar Xrx::22102. 04306, 2021.
Seang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: xxXx:1706. 1706.05577, 2017. 4
Liang-Chics Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decodor with atrous separable convolution for semantic image segmentation. In Proceedings of the European con- ference on computer vision (FCCV), pp. 801-81 801-818, 2018a. 2
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018b. 2, 4, 5
13

---

Published in Transactions on Machine Learning Research (04/2024)
Earl A Coddington and Norman Levinson. Theory of ordinary differential equations. Tata McGraw-Hill Education, 1955. 9
Germund Dahlquist. Convergence and stability in the numerical integration of ordinary differential equations. Mathematica Scandinavica, pp- 33 53-53, 1956.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Infor- mation Processing Systems, 32, 2019. 2, 5
Christian Etmann, Rihuan Ke, and Carola-Bibiane Schönlieb. iunets: learnable invertible up-and downsam- pling for large-scale inverse problems. 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 6. IEEE, 2020.
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurale estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Lukas Folle, Sulaiman Vesal, Nishant Ravikumar, and Andreas Maier. Dilated deeply supervised networks for hippocampus segmentation in mri. In Bildverarbeitung für die Medizin 2019, pp. 68 73. Springer, 2019.
2
Michal Futrega, Alexandre Milesi, Michal Marcinkiewicz, and Pablo Ribalta. Optimized u-net for brain tumor segmentation. In International MICCAI Brainlesion Workshop, pp. 15-29. Springer, 2022. 4
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, llya Sutskever, and David Duvenaud. Ffjord: Free-form continnous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018. 5
David Francis Griffiths and Desmond Higham. Numerical methods for ordinary differential equalions: initial value problems, volume 5. Springer, 2010.
David Gutman, Noel CF Codella, Emre Celebi, Brian Helba, Michael Marchelti, Nabin Mishra, and Allan Halpern. Skin lesion analysis toward melanoma detection: challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic). arXiv preprint ar xrXiv:1600 01397, 2016. 10
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 770 778, 2016. 3,
AD Hoover, Valentina Kouznetsova, and Michael Goldbaum. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medicul imaging, 19(3): 203 210, 2000. 10
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
4700 4708, 2017. 3
Fabian Isensee, Paul F Jäger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Automated design of deep learning methods for biomedical image segmentation. arXiv preprint 1rXiv:1904.08122 2019. 4
Debesh Jha, Pia H Smedsrud, Michael A Riegler, P&l Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. Kvasir-seg: A segmented polyp dataset. In International Conference Multimedia Modeling, pp. 451 462. Springer, 2020. 10
Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi- Hiin Fu, and Pheng-Ann Heng. H-denseunet: hybrid densely connected unet for liver and tumor segmentation from et volumes. IEEE transactions on medical imaging,
2018. 3,
Lihao Liu, Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Am Heng. v-nntt Stacking densely convolutional Istms for sub-cortical brain structure segmentation. IEEE transactions on medical imaging. 39(9):2806-295) 2817, 2020.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedingg of the IEFE conference on compuler vision and pollern recognilion, pp- 3431-3440, 2015. 1
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprinl or Xrr::188440033 2018. 4
Hans Pinckaers and Geert Litjens. Neural ordinary differential equations for semantic segmentation of individual colon glands. arXiv preprint Xiv: 111111101000 10470, 2019.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern recognition, 106: 107410,, 2020. 1,
Marta Ranzini, Lucas Fidon, Sébastien Ourselin, Marc Modat, and Tom Vercauteren. Monaifbs: Monai- based fetal brain mri deep learning segmentation. Xi preprint arXiv:2103. 133144 2021. 4
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image compuling und compuler-assisted interven- tion, pp. 234-241. Springer, 2015. 2, 3,
Sebastian Ruder. An overview of multi-tas learning in deep neural networks. arXiv preprint arXiv:1706.080 050998 2017. 4
Korsuk Sirinukuwaatanaa, Josien PW Pluim, Hao Chen, Xiaojuan Qi, Pheng- Heng, Yun Bo Guo, Li Yang Wang, Bogdan J Matuszewski, Elia Bruni, Urko Sanchez, et al. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis, 35:489-502, 2017. 10
Run Su, Deyun Zhang, Jinhuai Liu, and Chuandong Cheng, Msu-net: multi-scale u-net for 2d medical image
segmentation. Frontiers in Genetics, 12:639930, 2021. 2
Jeya Maria Jose Valanarasu and Vishal M Patel. Unext: Mlp-based rapid medical image segmentation
network. arXiv preprint arXiv:2203.0400 2022. 2, 4
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Newral Information Processing Systems, 31, 2018. 6
Hanshu Yan, Jiawei Du, Vincent YF Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. Xixe preprint 153.:1910005111 2019. 8
Lu Zhang, Ju Dai, Huchuan Lu, You He, and Gang Wang. A bi-directional message passing model for salient object detection. In Proceedings of the IEEE conference computer vision and pattern recognation, pp. 1751- 2018a. 3, 4
Zhengxin Zhang, Qingjic Liu, and Yunhong Wang. Road extraction by deep residual u-net. IEEE Geoscience and Remole Sensing Lellers, 15(5):749-755) 2018b. 3, 4
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing nelwork. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881 -2990, 2017. 3,
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet+++: Re- designing skip connections to exploit multiscale features in image segmentation. IEEE bransactions on medical imaging, 39(6): (856:1185-88861 1867, 2019. 1
15

---

Published in Transactions on Machine Learning Research (04/2024)
A Appendix
This document extends the practicalities and results presented in the main paper. This is structured follows.
Supplementary Convergence & Noise Results: We provide further results comparisons of our continuous U-Net against other existing U-type networks. We also give further experimental results on noisy data.
Supplementary Solver Results: We provide further results on the ablation study of our model under different, type of ODE solvers and its effect.
Supplementary Performance Comparison & Training Scheme. We give further numerical comparisons for different. type of blocks along with other type of U-Nets with different mechanisms. In the interest clarity and completness, we give an explicit definition the training setting of our model.
A.1 Supplementary Convergence Noise Results
Figure provides convergence plots for different types of U-Net blocks for the STARE dataset and the ISIC challenge. Our continuous U-Net is denoted in blue colour. In closer look, we observe that our model only need few iterations to converge whilst existing blocks need far more iterations.
We also extend our noise results from the main paper using the GlaS challenge, STARE dataset and DS Bowl dataset. Our experients are conducted by adding zero mean Gaussian noise with increasing standard deviation. The results are in Table 8. They shows that except for the DS Bowl dataset, our approach overall performs the best in terms of Dice metric. This is the only dataset where our approach achieves lower scores when adding noise, while it outperforms the other block types for the other datasets, we show that it is less sensitive to noise than the other block types in most of the cases.
STARE Dataset
ISIC Challenge
0.8
0.8
Continuous U-Net
Continuous U-Net
UNET-PLN
UNET-PLN
0.6
0.6
Loss value
UNET-DSE
Loss value
UNET-DSE
T-NT- UNET-PSP
UNET-PSP
0.4
0.4
0.2
0.2
0.0
0.0
0
25
50
75
100
10
20
30
Epochs
Epochs
Figure 4: Convergence plots for different types of U-Net blocks on remaining two datasets. The behaviour of our dynamic blocks is denoted in blue colour, where only few iterations needed to converge whilst existing blocks need far more iterations.
A.2 Supplementary Solver Results
In addition to the experimental results displayed the main paper, we presented some further experiments for the different type of solvers and its effect in our model, displayed in Table 7. The results show that the
16

---

Published in Transactions on Machine Learning Research (04/2024)
fourth order Rungo-Kutta solver performs the best for the BUCSI dataset in terms of accuracy and Average Hausdorff distance. For the particular case of the GlaS dataset, however, the implicit Adam-Pashforth- Moulton solver outperforms the RK4 one. Our theroretical insights reported in the main paper follows the empirical findings showing that RK4 is thcoretically and empirically is the best choice.
Table 7: Ablation study on different types of ODE solvers on the two remaining medical image segmentation datasets. The best results are highligthed in green colour.
Solver
ClaS Challenge
BUSI Dataset
Type
Dicef
Acct
AHD
Dice 1
Aoct
AHD]
Euler
0.7141
0.7374
14.01
0.6612
0.8957
3.36
AB
0.8455
0.8668
9.32
0.8127
0.9419
2.28
ABM
0.8521
0.8696
9.27
0.8203
0.9409
2.34
RK4
0.8469
0.8675
9.31
0.8090
0.9447
2.15
A.3 Supplementary Performance Comparison & Training Scheme
We provide further numerical comparison of our model against other existing U-type Networks using addi- tional mechanisms. The results are displayed in Table using the DS Bowl dataset, BUSI dataset and ISIC challenge. As for the other datasets, our apporach is able to perform as good most of the state of the art approaches while even outperforming some of them, even though despite we do not use any additional mechanisms. Finally and for sake of clarity, we provide the training scheme we for our experiments, which is summarised in Table 10. We remark that we use the same setting for all experiments for fair comparison.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Table 8: Performance comparison, in terms of Dice score, on range of noise levels. continuous U-Net with
We denote
The best results are highlighted in green colour.
NOISE LEVEL EXPERIMENTS
Block
GlaS Challenge
STARE
Dataset
DS Bowl
Dataset
Type
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.5
0.0
0.2
0.4
0.5
PLN
0.7616
0.5898
0.4209
0.3977
0.7828
0.5354
0.4929
0.4840
0.9156
0.7635
0.6563
0.6253
U-Net
RSE
0.7893
0,4813
0.5133
0.5117
0.7731
0.4805
0.4805
0.4805
0.9131
0.5015
0,4892
0.1963
0.3713
0.3613
0.7358
0.4805
0.4805
0.4805
0.9181
0.5307
0.4842
0.4698
INC
0.7962
0.4517
0.4485
0.4697
0.7750
0.5290
0.4895
0.4829
0.9210
0.5172
0.4849
0.4751
PSP
0.5523
0.5523
0.5515
0.5469
0,4805
0,4805
0,4806
0,4805
0,9060
0,4895
0,4690
0.4869
FO
0.8045
0.5345
0.4845
0.4801
0.5112
0.4854
0.4788
0.4761
0.8855
0.4844
0.4745
0.4700
DB
0.8469
0.8063
0.5882
0.4922
0.8378
0.5145
0.5034
0.5017
0,9335
0.4939
0,4678
0.4644
Table 9: Performance results for state of the art U-Net architectures with additional mechanisms, We mark our continuous U-Net with
DS Bowl Datasot
BUSI Dataset
ISIC Challenge
TECHNIQUE
MECHANISM
Diect
Accf
AHD,
Dicef
Acct
AHD1
Dicct
Acet
AHD,
Attn. U-Net
Attn. Gates
0.9388
0.9760
0.7484
0.9208
3.01
0.8987
0.9331
DynU-Net
Heuristic-R + Opt
0.9446
0.9771
3.07
0.7716
0.9270
2.83
0.9105
0.9408
4.48
U2Net
Nested U-Nets
0.4776
0.8723
5.97
0.5096
0.8696
4.12
0.8310
0.8863
7.83
UNeXt
Tokenised MLP
0.6671
0.8943
5.54
0.8064
0.9398
2.26
0.9202
0.9482
4.01
TransUnet
Transformers
0.9351
0.9760
3.17
0.8507
0.9497
2.00
0.9251
0.9529
3.74
0.9335
0.9745
3.29
0.8090
0.9447
2.15
0.9094
0.9433
4.46
Table 10: Overview of training settings for all experiments.
Parameter
Value
Loss function
Binary Cross Entropy Loss
Optimiser
Adam
Learning rate Learning rate schedule
10-4
Multiplication of learning rate with 0.999 every epoch
Epochs
500
Batch size Levels of U-Net architecture
16
4
Number of filters per block
3, 6, 12, 24
Tolerance (for contin. blocks only)
10 3
18


Reviews:
[{"review_id": "Dhv86XCqyC", "reviewer": "Reviewer_5bd4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problems aimed to be addressed in this paper would be of interest to the community.\n- In comparison to the baseline U-net architecture along with some variants, the continuous U-net performs well and is more robust to noise.\n\nWeaknesses:\n- Many claims in the paper feel very overstated. For example, the authors claim that they prove various aspects regarding their model, which is missing from other methods, but the theoretical results are essentially due to other works, e.g., robustness of neural ODEs has already been shown (by the same argument) in Yan et al (2019).\n- The presentation of the paper can be improved, e.g., see comments below regarding typos and lack of rigor in proofs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new variant of the U-net architecture by introducing neural ODE blocks in place of traditional convolutional discrete blocks as layers. Such blocks are second-order neural ODEs. Experimental comparisons with standard U-net architectures show that the proposed network performs well in terms of DICE and accuracy scores, and can be robust to Gaussian noise for segmentation tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I noticed that the noise level experiments did not include comparisons to FO blocks. What was the difference in performance when using first order vs second order blocks?\n- Were there experiments done to showcase how long training took in terms of wall clock time rather than simply epochs? I imagine second order neural ODEs would result in more of a computational burden.\n- Missing the Proposition number at the bottom of page 2\n- strange phrasing of certain terms, e.g., Faster Convergent vs Convergence\n- sometimes mathematical fonts do not match, e.g., in the proof of Theorem 1\n- please be more precise in your statements, such as in the proof of Theorem 1 when you say “neural networks are usually Lipschitz”. Conditions on when the network is Lipschitz should be stated (e.g., by using L-Lipschitz activation functions)\n- the proof of proposition 1 is unclear. Do f^(a) and f^(v) share weights? Is the function g(…) applied recursively to each x^i for 1 < i < m?\n- you require f’(x) to be positive, not an increasing function in the proof of the Lemma (which is true in this case)\n- how are you defining T_j in the proof of Theorem 2 in equation (10)?\n- Can a theorem number be provided in Theorem 3 for the Coddington & Levinson reference for ease of verification?\n- I think it would be good if the authors added more model details in the appendix.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V9yJUQkNKn", "reviewer": "Reviewer_TBGq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- innovative introduction of second-order dynamic blocks to U-Net, replacing the conventional discrete CNN layers.\n- engaged in valuable theoretical exploration of the properties inherent to the proposed continuous U-Net model.\n- extensive experiments demonstrate the advantages of continuous U-Net over other variants of U-Net without additional mechanism in performance and robustness.\n\n\nWeaknesses:\n- regarding the theoretical analysis:\n    - the central assumption that \"neural network is usually Lipschitz\" in the proof of Theorem 1 is not justified.\n    - in the discussion following Theorem 3, the authors postulate that the current dynamic block is less sensitive to input noise due to the additional smoothness constraints imposed by the 2nd-order ODE. Nonetheless, the casual link between smoothness and the sensitivity is not established. Concerning sensitivity, one estimates how the trajectory is bounded given an initial value variation.  The smoothness of the network function alone should not dictate this bound.\n    - the current theoretical investigations are all general arguments concerning a ODE system with specific assumptions. While these analyses indeed support the advantages claimed for the proposed continuous U-Net model, it would be better if there are discussions taking into account the concrete network function in use.\n- regarding the GE metric:\n    - The $f^{(v)}$ in Theorem 2 does not rely on $x(t)$, which is crucial for the proof. In contrast, $f^{(v)}$ in the proposed dynamic block depends on $x(t)$. Then Theorem 2 may not reflect the order of GE of Euler's method in the proposed dynamic block.\n    - what are the results of GE for RK methods?\n- regarding the presentation:\n    - typos, e.g.,\n        1. $f^v$ in Theorem 1 should be $f^{(v)}$ (what does the superscript of $f$ refer to?)\n        2. in Equation 11, should be $(1+h\\lambda)^{n-j}$\n        3. in Equation 12, should be $\\sum_{j=1}^n$\n    - some notations are in consistent, e.g., in Theorem 2, $\\mathcal{O}$ is used while in later text $O$ is used.\n    - in Figure 2, it is not clear what is the predicted mask and what is original figure\n    - it would be better to briefly expand the meaning of \"conditions of stability and consistency\" in Property 1.\n    - it is stated in subsection 3.3 that “our dynamic blocks are a physics-based model that better captures the nature of segmentation.”  What does physics-based model indicate? What is the nature of segmentation? \n    - the word \"Greater\" is too vague and being used in different ways.\n    - the word \"Noiseless\" is overstated.\n- does moving to higher-order block present any challenges to the model compared to the 1st-order case? for example, does it affect the stability of training process? \n- In Figure 2, the fine details in the STARE dataset, such as the branches, seem inadequately represented by the Continuous U-Net. Could this be due to the enhanced smoothness constraint introduced by the 2nd block?\n    - A 2D projection of 3D object can be quite irregular. Would the smoothness constraint potentially causes any problem?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study presents Continuous U-Net, a novel U-Net based models where traditional discrete CNN layers are substituted by second-order NODE blocks. A combination of theoretical analysis and numerical experiments are employed to showcase the advantages of the Continuous U-Net, highlighting its superiority in terms of both performance and robustness for medical image segmentation tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- modify presentation (see weaknesses in the above box) and correct typos.\n- justify the Lipschitz assumption on neural network models and make the connection between smoothness and robustness (insensitive to input noise) more clear.\n- what are the results of GE for RK methods?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UGlKeEkdAN", "reviewer": "Reviewer_xLKL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1) The importance of the problem and the value proposition of the work is clear. Indeed, robustness to the change of resolution and dynamically adjusted receptive field is a highly desirable property of UNet architecture. \n\n2) The approach is validated on a diverse set of medical set of tasks, where it consistently outperforms discrete UNet baselines with different choices of Residual Blocks. The architecture with ODE blocks converges noticeably faster compared to alternatives. \n\n3) Continious formulation allows reducing memory costs to \\mathcal{O}(1) (with respect to depth).\n\n**Weaknesses**\n\n1) I would not fully agree with part of the claims stated in the introduction and the main part of the paper:\n* Authors claim that the continuous formulation is theoretically justified unlike the discrete formulation, but both methods are equally heuristic approaches to solving the task of medical segmentation, and the continuous formulation just suggests certain benefits in terms of performance / efficiency. In practice, one works with discretized inputs of a certain resolution, and so does this work as well.\n* One of the claims of the work is that continuous nets are more noise resistant, but it is not fully clear from Theorem 3. I think that some error bound, depending on the change of the input would be more appropriate here. Moreover, robustness on the noise can be achieved via training on larger amount of data with augmentation. Doesn't more data + augmentation address the problem sufficiently well?\n* One should make it clear, that the Theorems in the main text are not novel, but rather known results from the theory of differential equations. \n\n2) The continuous formulation of UNet was proposed earlier in the cited work [1]. This approach extends this work via using a higher-order equation and more complicated solver, but the choice of both components is not ablated. Which order is the optimal? How does the performance depend on the solver and the number of ODE steps? \n---\n[1] Neural ordinary differential equations for semantic segmentation of individual colon glands. arXiv preprint arXiv:1910.10470, 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a UNet architecture with continuous NeuralODE blocks for semantic segmentation. Unlike prior art in this field, the introduced method adopt 2nd order differential equations and adopts higher-order ODE solver. Approach is validated on several medical segmentation benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**General**\n\n* The paper would greatly benefit from experimental ablation of the design choices for the solver and choice of specific ODE equation form. \n* One should make clear, that the theoretical results are not novel, but borrowed from the literature. One can move the proofs to the Appendix part of the manuscript. \n* More recent designs of UNets with attention mechanisms seem to achieve noticeably better results than the proposed method. Can one combine the components of their architectural design together with the one introduced in the work to achieve stronger results? \n\n**Minor**\n\n* What does one mean by the **boxes** in the sentence: *view the dynamics in our network as the boxes consisting of CNNs*.\n\n* Proposition 1 & ?? (missing reference)\n(end of page 2) \n\n* I would suggest to use brighter colors, more different from the image, to make the segmentation result more salient.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new variant of the U-net architecture for tackling the task of image segmentation (especially on medical images) by introducing neural ODE blocks in place of traditional convolutional discrete blocks as layers. The proposed blocks are modeled as second-order neural ODEs. Experimental comparisons with standard U-net architectures show that the proposed network performs well in terms of DICE and accuracy scores, and can be robust to Gaussian noise for segmentation tasks.

Before the rebuttal, the reviewers mainly raised concerns regarding the over-stated claims, the relation between Yan et al (2019), the connection between smoothness and robustness to noise.

After the rebuttal, the paper received two "Leaning to Accept" and one “Leaning to Reject”. The AE found that the authors were able to solve most issues raised by the reviewers. But most reviewers still think that the theoretical analysis between the second-order ODE and noise-resistent is on the short side. They are mostly leaning to accept the paper due to the empirical results of the proposed model, which the AE also agreed with.

However, the authors shall strive to solve the remaining issues raised by reviewers, especially reviewer TBGq: "the newly added FO results in Table 4 appears inconvincible. Specifically, in Table 4 the Dice scores of FO on the three datasets in the 0 noise scenario differ from those in Table 3, indicating that these outcomes may not be from properly trained models."
Decision: Accept (poster)