On the effectiveness of Rotation-Equivariance in U-Net: A Benchmark for Image Segmentation
Authors: Robin Ghyselinck
Valentin Delchevalerie
Bruno Dumas
Benoit Frenay
Venue: Accepted by TMLR ()

Abstract: Numerous studies have recently focused on incorporating different variations of equivariance in Convolutional Neural Networks (CNNs). In particular, rotation-equivariance has gathered significant attention due to its relevance in many applications related to medical imaging, microscopic imaging, satellite imaging, industrial tasks, etc. While prior research has primarily focused on enhancing classification tasks with rotation equivariant CNNs, their impact on more complex architectures, such as U-Net for image segmentation, remains scarcely explored. Indeed, previous work interested in integrating rotation-equivariance into U-Net architecture have focused on solving specific applications with a limited scope. In contrast, this paper aims to provide a more exhaustive evaluation of rotation equivariant U-Net for image segmentation across a broader range of tasks. We benchmark their effectiveness against standard U-Net architectures, assessing improvements in terms of performance and sustainability (i.e., computational cost). Our evaluation focuses on datasets whose orientation of objects of interest is arbitrary in the image (e.g., Kvasir-SEG), but also on more standard segmentation datasets (such as COCO-Stuff) as to explore the wider applicability of rotation equivariance beyond tasks undoubtedly concerned by rotation equivariance. The main contribution of this work is to provide insights into the trade-offs and advantages of integrating rotation equivariance for segmentation tasks.

Published in Transactions on Machine Learning Research (April/2025)
On the effectiveness of Rotation-Equivarian in U-Net: A Benchmark for Image Segmentation
Robin Ghyselinck
mbin.ghhgellncck@nnammn..
Faculty of Computer Science
NaDI institute
University of Namur
Rue Grandgognage 21, 5000 Namar, Belgium
Valentin Delchevalerie
walettin.eeicheeleeieeemaam
Faculty of Computer Science
NaDI & naXys institutes
University of Namur
Rue de Bruaelles 61, 5000 Nomur, Belgium
Bruno Dumas
bruno. www.odmmms@@unamu....m
Paculty of Computer Science NaDI institute
Universily of Namar
Rue Grundgugnage 21, 5000 Namur, Belgium
Benoît Frénay
Faculty of Computer Science
NaDI institute
Universily of Namar
Rue Grandgugnage 21, 5000 Namaur Belgium
Reviewed on OpenReview: https: /// openreview. net/ pormm? id= UerVAXBaZI
Abstract
Numerous studies have recently focused incorporating different variations of equivari- ance in Convolutional Neural Networks (CNNs). In particular, -uaation-quiivarianee has gathered significant attention due to its relevance in many applications related to medical imaging, microscopic imaging, satellite imaging, industrial tasks, etc. While prior research has primarily focused on enhancing classification tasks with rotation equivariant CNNs, their impact on more complex architectures, such U-Net for image segmentation, remains scarcely explored. Indeed, previous work interested in integrating rotation-equivariance into U-Net architecture has focused on solving specific applications with limited scope. In con- trast, this paper aims to provide a more exhaustive evaluation of rotation equivariant U-Net for image segmentation across a broader range of tasks. We benchmark their effectiveness against standard U-Net architectures, assossing improvements in terms performance and sustainability (i.e., computational cost). Our evaluation focuses on datasets whose orien- tation of objects of interest arbitrary the image (e.g., Kvasir-SEG), but also on more standard segmentation datasets (such COCO-Stuff) bility of rotation equivariance beyond tasks undoubtedly affected by rotation equivariance. The main contribution of this work is to provide insights into the trade-offs and ad advantages of integrating rotation equivariance for segmentation tasks.
to explore the wider applica-

---

Published in Transactions on Machine Learning Research (April/2025)
1 Introduction
Recently, many studies have focused on integrating different forms of equivariance in Convolutional Neural Networks (CNNs) (Weiler & Cesa, 2019; Delchevalorie et al., 2023; Worrall ct al., 2017; Cohen & Welling, 2016; Kumar et al., 2018). This idea is largely inspired by the superior ability of CNNs to solve image-related tasks (e.g. classification, detection, segmentation) compared to fully-connected architectures. Indoed, CNNs inherently leverage translation equivariance (i.e., shifts in position) by extracting features through the use of convolutions that implement weight sharing across translations. This property is particularly desirable for solving image processing tasks. This sharing of weights leads to models that are more efficient in image processing, both in terms of accuracy and computational resources as compared to their fully-connected counterparts.
In previous work on introducing equivariance in CNNs, rotation equivariance has attracted particular atten- tion. As a matter of fact, rotation oquivariance is highly relevant for many applications, espocially when the overall orientation of the image is arbitrary. Such cases arise in many fields, including medical imaging (Or- eiller et al., 2022; Elaldi et al., 2024), microscopic imaging (Chidester et al., 2019b; Graham et al., 2020), satellite imaging (Marcos et al., 2018; Li et al., 2020) and industry 4.0 (Marcos ct al., 2016). In these cases, the overall orientation of the images often contains no meaningful information. Instead, this orientation adds noise, potentially hindering the training process of a model. As a result, constraining CNNs with rotation equivariance can incorporate prior knowledge into the model, thereby improving its performance thanks to a reduction in unnecessary variability.
When applied to relevant tasks, rotation equivariant CNNs can outperform their vanilla counterparts (Weiler & Cesa, 2019; Worrall et al., 2017; Delchevalerie et al., 2023). Specifically, equivariant CNNs often achieve higher accuracy or at least state-of-the-art performance while using fewer parameters. This leads to an improvement from a sustainability point of view, since training models with fewer parameters reduces the overall computational cost. However, only a few studies have evaluated the benefits of these techniques in more complex CNN architectures or on tasks beyond classification (Chidester et al., 2019a; Milton & Murray- Smith, 2021; Oreiller et al., 2022). Furthermore, while it is well-established that rotation equivariance is beneficial for specific applications that involve arbitrary orientations, it may also prove to be useful for more general tasks that do not inherently require global rotation invariance priori. In fact, learning rotation equivariant features can still help in solving such tasks more effectively.
In this work, we explore the impact of rotation equivariance on segmentation tasks, in particular using the popular U-Net (Ronneberger et al., 2015) architecture. While most prior rescarch has focused on evaluating the benefits of equivariant CNNs for classification tasks, we aim to fill the gap by benchmarking their performance on the more complex task of image segmentation. We assess whether introducing rotation equivariance can improve model accuracy, reduce the computational resources required by the model, even both. Moreover, we also evaluate equivariant models for tasks where the orientation of the image is not necessarily arbitrary or not explicitly related to the task, such as objects segmentation (e.g., building or ship) in aerial imagery, where such objects maintain consistent orientations relative to the ground. For example, recognizing buildings or ships in aerial images. This is mainly motivated by the fact that learning rotation invariant features can still be useful for such applications, where the performance of an equivariant model higher. This justifics the use of standard segmentation datasots that focus on general tasks (such as the well-known COCO-Stuff by Lin et al. (2014)) along with other datasets that are more linked to the concept of rotation invariance and focus specific, narrow tasks (e.g., Kvasir-EGG by Jha et al. (2020), a dataset for polyp detection in colonoscopy images). In this work, we provide a comprchensive analysis of the trade-offs and advantages of incorporating rolation equivariance into U-Net, thereby contributing to a broader understanding of its applicability across a range of use cases. The PyTorc implementation of our work is publicly available as an open-source code on Githubt.
Our key findings are that the effectiveness of equivariant U-Nets compared to vanilla U-Nets is highly con- tingent on both object characteristies and segmentation task complexity. In datasets where objects can have arbitrary shapes and do not have a semantically meaningful orientation, equivariant models often outperform vanilla architectures. Conversely, for rotationally symmetric objects, vanilla architectures prove superior.
2

---

Published in Transactions on Machine Learning Research (April/2025)
One explanation is that the rotation equivariance constrains the filters and reduces their expressiveness, hindering the learning process of the model. For general-purpose segmentation, the performance metrics are similar between equivariant and vamilla U-Nets. However, for those general tasks, the performance gap arises in small data regimes, where equivariant models are consistently worse than the vanilla oncs. This observation highlights how low data availability affects the trade-off bet ween architectural constraints and model expressiveness.
Section 2 provides background knowledge on equivariance in CNNs, well the U-Net architecture. Subsequently, Section 3 presents the related work on using equivariant models for segmentation tasks. Then, the datasets and models considered in assessing the usefulness of equivariant U-Nets are presented in Sec- tion 4, while the experiments and results are presented in Section 5. Finally, Section 6 presents a discussion on the results before concluding in Section 7.
2 Background
This section provides background knowledge about the task of image segmentation and the U-Net architec- ture. Furthermore, it presents the current state-of-the-art techniques that allow for the implementation of equivariance CNNs.
2.1 Image Segmentation
Image segmentation is branch of computer vision that aims at separating an image into meaningful objects regions. Typically, four tasks can be considered. First, semantic segmentation (see for example Long et al. (2015)) where each pixel in the image is assigned to specific category based what stuff it represents (e.g., level of blue for a bird, a level of yellow for giraffe, white for the clouds, etc.). Although it is a subset of semanntc segmentation, we specifically mention the lask of binary segmentation (Kittler & Illnnworthh 1986), when each pixel is either background (the black pixel) or foreground, i.c., the object of interest (the white pixel). Second, instance segmentation (see for example, He et al. (2017)), extending semantic segmentation by also making the distinction between individual instances within an object class, answering the questions of how many objects there in image, and what are their extents (e-g., each of the three giraffes is uniquely identified in the image). Third, panoptic segmentation (Kirillov al., 2019), where all the objects in the scene, as well as the remaining stuff are labeled. This goes beyond semantic sesmentation and instance segmentation. Fourth, dense pose estimation, where pixels that belong to people are labeled, together with their body parts (body, head, locations of limb and attitude, see for example Felzenszwalb & Hnttenlocher (2005) and Andriluka et al. (2014)). This work only deals with semantic segmentation tasks.
Many segmentation methods have been developed in the last decades, prior to the introduction of deep learning. To name few, thresholding-basee approaches, such as method by Otsu (1979) that segments images based on pixel intensity values. Edge detection methods, such as the Canny edge detector (Canny, 1986), identify boundaries between different image regions. Region-based segmentation techniques, including region growing (Adams & Bischof, 1994) and watershed transformation (Beucher & Mathmatique, 2000), segment images by grouping pixels with similar properties. Clustering- based techniques, such as Mean Shift (Comaniciu & Meer, 2002), deal with segmentation by iteratively shifting data points toward high- density regions in fcature space. Next, graph-based methods, such as Normalized cuts (Shi & Malik, 2000) and graph cuts (Boykov & Jolly, 2001), represent the image as a graph where pixels are nodes and edges connect neighboring pixels. Finally, energy minimization Bayesian inference techniques, like the Texton- Boost system (Shotton et al., 2007) uses shared boosting to use information from the color distributions of images, information about the location and texture of the object.
With the advent of deep learning, those methods fell out of favor because they relied on the use of hand- crafted fcatures, often requiring a specific domain knowledge or expertise, as well as extensive finc-tuning for each particular task Sfarcia-Garcia et al., 2017). Conversely, deep learning methods have the ability learn feature representations for the pixel labeling problem in end-to-end fashion.
3

---

Published in Transactions on Machine Learning Research (April/2025)
2.2 Image Segmentation with Deep Learning
Deep learning has revolutionized image segmentation, offering models that learn hierarchical features directly from data. Popular architectures include Fully Convolutional Networks (FCNs) (Long et al., 2015), Encoder- Decoder architectures, such as U-Net (Ronneberger et al., 2015) or DeepLab (Chen et al., 2018), pyramids networks, and transformer-bese models.
FCNs are the first architecture that made it possible to semantically label each pixel with one neural network. This method replaces fully connected layers with convolutional layers such that the spatial information is preserved (Long et al., 2015). Although this work showed interesting results, the resolution permitted by the technique is low, ie., the boundaries of the objects are not well defined. Other works added conditional random fields (Zheng et al., 2015) at the end of the network to deal with the problem of poor resolution. Building on this work, DeepLab (Chen et. al., 2018), improves the model's performance with the introduction of atrous convolutions in addition to the conditional random fields.
Another technique, U-Net, introduced for biomedical segmentation tasks, features an encoder-decoder ar- chitecture with fine-level connections that help recover spatial information lost during a downsampling phase (Ronneberger et al., 2015).
Some architectural variations rely on the feature pyramid network (Lin et al., 2017) and use lop-down connections to help semantic information flow to higher- resouutinoo maps (see for example the Pyramid Scene Parsing Network by Zhao et al. (2017)).
In recent years, transforrmer-aased architectures, such as SETR (Zheng et al., 2021), Swin-Transfore (Liu et al., 2021) and SegFormer (Xie et al., 2021), have gained attention because of their ability to model long- range dependencies thanks to the attention mechanism. However, transformer-basee models generally require large datasets for training because of their lack of inductive biases (Dosovitskly al., 2021). Conversely, CNN-based methods, such as U-Net, with its strong inductive biases and efficient use of labeled data, are better suited for those data settings. Furthermore, the incorporation of rotation-equivariant modifications to CNNs provides additional benefits in scenarios where rotational invariance is desirable.
2.3 U-Net
Assigning label to each pixel in an image is essential in various domains such as medical imaging, au- tonomous driving, and satellite imagery, where understanding the spatial structure of objects within image is necessary for predicting segmentation masks. Among these models, U-Net (Ronneberger et al., 2015) has emerged as one of the most popular and effective architectures for image segmentation. This model was originally developed for biomedical image segmentation and has since been widely adopted across different fields. The model has a 'U' shape and is based on an encoder-decoder structure with skip connce- tions (see Figure 1), where the encoder responsible for extracting hierarchical features and the decoder focuses on reconstructing segmentation masks. The ability of U-Net to combine low-level spatial details with high-level semantic information makes it particularly effective for boundary delincation at the pixcl-level, which allows for producing segmentation masks for objects of arbitrary shapes.
The encoder aims at progressively extracting useful information at different scales in the input image. It is responsible for capturing the context of the input image through a series of downsampling convolutional layers. Each layer in the encoder path applies a set of convolutional operations followed by activation functions and pooling operations. This process progressively reduces the spatial dimensions of the input, while increasing the number of feature channels, effectively condensing the image information into much more compact representation. At the bottom of the U-shape (after the encoder), one finds the bottleneck layer. It acts the bridge between the encoder and decoder, and consists of convolutional operations that further distill the compact representation of the image. This allows the network to capture complex and abst patterns within the input data. The decoder then aims lo reconstruct an output image with the initial spatial dimensions. It employs transposed convolutions (also known as up-convolutions) to upsample the feature maps, gradually restoring the spatial dimensions that were reduced during the encoding process. A core feature of the U-Net architecture is its skip connections between the encoder and decoder layers. When the input data moves downward the encoder (e.g., a 224 224 pixels image), it undergoes successive
4

---

Published in Transactions on Machine Learning Research (April/2025)
downsampling operations that increase the receptive field and capture abstract features. Unfortumately, the corollary is that fine spatial details are lost. Nonetheless, the skip connections address this limitation by connecting each encoder layer to its corresponding decoder layer (i.e., there is a one-to-one mapping between each encoder and cach decoder layer) in order to concatenate their feature maps. This allows high-rcsolution spatial information to flow from the early encoder layers (at. the top the U-Net) to the decoder. Moreover, the abstract features learned at the bottom of the U-Net (i.c., in the bottleneck) are preserved. In medical image segmentation for instance, these connections help preserve boundary details and anatomical structures that might otherwise be lost during the encoding process.
12 22
22
44
22
22
224 x
H
112 x 112
172
44
56
172
172
344
86
86
28 x 28
344
1
172
172
X 14
344
Figure 1: The U-Net architecture for the large vanilla U-Net (adapted from Ronneberger et al. (2015)). It illustrates the encoder, bottleneck, and decoder paths with skip connections. The input image is preprocessed by two convolutional layers (not represented in this illustration). Red arrows correspond to downsampling, blue arrows convolutions and green arrows to upsampling. The final images obtained after the last black arrow are the segmentation masks for the corresponding classes.
The U-Net architecture is particularly effective in scenarios with limited training data, which is common in medical imaging and other specialized fields. The extensive use data augmentation, along with the skip connections that help preserve spatial information, allows U-Net to produce precise segmentation maps even when trained on small datasets. Furthermore, its ability work end-to-end makes it a preferred choice for segmentation lasks that require detailed, pixel-level accuracy.
2.4 Equivariance in CNNs
Recently, numerous studies have focused on incorporating equivariance in CNNs (see for example Flaldi al. (2024); Delchevalerie et al. (2023); Mitton & Murray-Smith (2021); Oreiller et al. (2022) or Graham et al. (2020)). Rotation equivariance, in particular, has attracted significant attention due to its relevance in image recognition. The property of rotation equivariance is very important in several domains, like medical, satellite, and microscopy imaging where the orientation of objects can be arbitrary, mcaning that any variation in the orientation of the object should not affect its classification or detection (Weiler & Cesa, 2019). This means that some images do not have a preferred global orientation. For example, this applies a building in satellite imagery, where regardless of the viewing angle from above (i.c., acrial view), it is still the same object. Medical scans are other concrete examples, where tumor should be recognizable regardless of the patient's position in the machine during the capture of the images.
5

---

Published in Transactions on Machine Learning Research (April/2025)
There are three categories of methods that deal with rotation equivariance: (i) those that enhance rotation robustness without providing formal guarantees of equivariance, (īi) those that offer some mathematical guarantees, but only for a discrete set of rotations, such as those defined by the cyclic C₂ and dihedral D, groups, and (ii) those that give mathematical guarantecs of equivariance for contimoous transformations, primarily involving the special orthogonal S0(2) or the orthogonal ((2) groups.
The most widely adopted technique from the first category is data augmentation (Quiroga et al., 2018). The main idea of data augmentation is to artificially increase the size of the training set by incorporating variability in the data that is unrelated to the considered task. In the context of rotation equivariance, one may introduce rotated versions of the images in the training datasct, for example by rotating the entire image by any multiple of 90". However, while robustness can be considerably increased thanks to the use of data augmentation, it is obvious that the model will try to learn the rotation equivariance from training examples instead of being constrained to be rotation equivariant. Unfortunately, this does not lead to any formal guarantee of oquivariance and has severe limitations from an efficiency point of view.
Regarding the second category, the most famous method is probably Proup-CNN by Cohen & Welling (2016). In group-CNN, filters are artificially duplicated in several versions with the application of all the transfor- mations from the considered discrete symmetry group. For example, if the group C2 (all the rotations for a E {0, 90", 180", 270")) is considered, the final number of filters will be multiplied by four after performing 90", 180 and 270" rotations on the initial filters. In other words, filters are learned in the same fashion as for vanilla CNN. Yet, when forwarding an input image, each filter is artificially duplicated according to the transformations contained in the considered symmetry group and convolved with the image. At the end of the method, a symmetry pooling is performed to summarize the activations of the different variants of the same filters. Only one copy of each filter needs to be updated during the back-propagation. They are thus defined as real trainable parameters.
Finally, for the third category, genoral-E((2-oquivvrr CNNs (E(2)-CNNs) by Weiler & Cesa (2019) have become the primary method, achieving equivariance to continuous groups through a finite set of irreducible representations. ions. The idea is closely related to that of steerable filters (Colen & Welling, 2017; Weiler et al., 2018; Graham et al., 2020), which is widespread in the equivariant CNNs literature. Regarding general-E(2)-a (u)u-vqvivarr CNNs, they build on the fact that any reprosentation of a finite compact group can be decomposed into direct sum of irreducible representations. This decomposition in irreducible representations can be used afterwards to constrain the filters. This idea has the advantage of being very general. However, the authors report that irreducible representation for SO(2) or O(2) generally does not lead to better performance than the discrete versions Cn or Dn-.
In this work, we focus on the group-CNN (Cohen & Welling, 2016) method from the second category. This choice is motivated by the fact that methods from the first one do not provide any guarantees on the property of equivariance. With these methods, one cannot be confident that any trained model fully relies on rotation equivariant features. Methods from the third category, like general-E((2-oouivvviia CNNs, are not selected either. This exclusion is based on previous results in the literature showing that approximating the continuous equivariance with a discrete one is generally a good solution, and the use of groups (such as C.) is less computationally expensive (Weiler & Cesa, 2019; Delchevalerie et al., 2023). Consequently, we choose to use the publicly available and general implementation of E(2)-CNNs² that facilitates building Croup-CCNs by leveraging discrete symmetry groups
3 Related Work
Exploiting rotation equivariance has been extensively studied for many classification tasks. A non-exhaustive subset of those application domains includes medical imaging (Oreiller et al., 2022; Elaldi et al., 2024), microscopic imaging (Chidester et al., 2019b; Graham et al., 2020), satellite imaging (Marcos et al., 2018; Li et al., 2020) and industry 4.0 (Marcos et al., 2016). However, a limited number of works try to address segmentation lasks while applying rotation equivariance constraints.
6

---

Published in Transactions on Machine Learning Research (April/2025)
To the best of our knowledge, the first study to consider exploiting rotation equivariance in U-Net archi- tecture is Marcos et al. (2016), which introduced Rotation Equivariant Vector Field Networks Neet) an approach for incorporating rotation equivariance into CNNs through vector field representations. Their method encodes orientation information explicitly within the network architecture by implementing spe- cialized filters that respond to features at multiple orientations and aggregating these responses into veclor fields. The authors show that their architecture results in compact models in terms of parameters size, and is able achieve comparable performance to those of much larger networks. However, this work is limited to the ISBI 2012 2D EM segmentation challenge Eirganda-Carrery et al., 2015), specialized microscopy binary segmentation dataset. This method was fine-tuned for this specific application, and it is not clear how RotE would perform on different or more complex (e.g., higher resolution, importance of global orientation in the image, elc.) data or lasks (e.g., semantic segmentation task).
The work by Weiler et al. (2018) proposed general framework for constructing rotation equivariant CNNs using stecrable filters. They developed a method where convolutional kernels transform predictably under rotations. Thanks to their approach, the network processes rotated inputs with consisteney. By encoding this geometric prior knowledge into the network architecture, they demonstrated substantial parameter efficiency while maintaining or improving performance compared to conventional CNNs. In addition, the authors provide theoretical guarantees of equivariance and practical implementation strategies for their steerable filter basis through circular harmonic decomposition. Similarly to the work by Marcos et al. (2017), this study is limited to the ISBI 2012 2D EM segmentation challenge dataset (Arganda- Carreras et al., 2015).
A semantic segmentation model developed by Winkens et al. (2018) and Limmans et al. (2018) uses rotation and reflection symmetries to improve sample efficiency and robustness to image transformations. This work is an extension of the group equivariant CNN that is able to deal with segmentation tasks thanks to an equivariant (G + convoluton that transforms feature maps from group to planar representations. The authors evaluate their work on tumor localization dataset.
Another work by Chidester et al. (2019a) investigates the use of group convolutions (restricted to the use of the C4 group) in a U-Net architecture for nucleus segmentation in histopathological images. The authors conclude that using equivariance to rotations of 90 "enables the network to learn better parameters that generalize well to simple transformations, namely, translations and rotations".
Other works have considered the use of rotation equivariance, such as the work by Mitton & Murray- Smith (2021), group-CNNs are used for the very specific task of deforestation segmentation. Oreiller et al. (2022) introduce a specific locally rotation invariant bispectral U-Net for the task of multi-organ nuclous segmentation.
Having considered those previous works, we believe that integrating rotation equivariance into CNNs remains umder-cxplored and has to be further researched, more particularly for segmentation tasks using U-Net ar- chitectures. The few previous works only investigate one particular configuration for rotation equivariance, and only tackle one specific task. The current literature lacks insights and thorough evaluations of the per- formance trade-offss and sustainability assessment in terms of computational costs involved in incorporating rotation equivariance into U-Net for segmentation across diverse datasets. The aim of this work is to close this gap by delivering a more comprehensive assessment of the effectiveness of rotation equivariant U-Net models across various segmentation tasks, while shedding more light on the use of rotation equivariant con- straints. By evaluating both rofalion-sensitive and standard datasets, we not only investigate the usefulness of rotation equivariance for tasks inherently linked to orientation arbitrariness, but also consider its potential advantages for segmentation tasks that are more general.
4 Datasets and Models
This work aims to assess the effectiveness of equivariant U-Net models under several practical conditions: nature of the selected task, dataset size, number of classes for the segmentation task, and model size (number of trainable parameters). First, one can train U-Net models on datasets dealing with tasks of different nature. Either those tasks involve a notion of rotation invariance at global or local scales (c.g., polyp segmentation mask) they do not (e.g., segmentation mask of pedestrian). Second, the selection of datasets is performed
7

---

Published in Transactions on Machine Learning Research (April/202)) 2025)
Images
Masks
(a)
(b)
Figure 2: Example of colonoscopy images with their segmentation mask from the Kvasir-SEG dataset. One notice that the polyps vary in shape and size, and located at different positions within the images.
such that smaller and larger data regimes exist. Indeed, the usefulness of equivariant constraints may depend on the quantity of available data for training (Delchevalerie al., 2023). For this reason, in addition to the use of datasets of different sizes, we also consider using different amounts of data for each dataset. Third, both binary (one foreground class and one background class) and semantic segmentation (involving small to large number of distinct classes) are considered. Last, one can consider smaller and larger U-Net models in terms of number of trainable parameters. This is motivated by the idea that equivariant models are constrained by prior knowledge, which may reduce their required number parameters to obtain a model with similar performance (Cohen & Welling, 2016). In agreement with the aforementioned prerequisites, this section presents the datasots well as the different architectures considcred in this work.
4.1 Datasets
Five datasets are considered in this work. The first three datasets are designed for binary segmentation tasks, i.e., masks only contain foreground and background. These datasets are Kvasir-SEG (Jha et al., 2020), NucleiSeg (Janowezyk & Madabhushi, 2016), and URDE (De Silva et al., 2023). Two additional datasets are selected for semantic segmentation, namely COCO-Stuff (Caesar et al., 2018) and iSAID (Waqas Zamir et al., 2019). While the COCO-Stuffs much larger in size than the datasets from the first. category, iSAID has the same order of magnitude in terms of image counts. In addition, COCO-Stuff and iSAID are much more general than the first three and are often used as baseline datasets for benchmarking segmentation models. Further, the number of classes that are featured in these two last datasets is greater than in the first three datasets. Also, some datasets feature objects for which the orientation does not matter (like polyp in Kvasir-SEG), as opposed to other datasets (e.g., a train in COCO-Stuff). Consequently, these five datasets cover a large spectrum of segmentation tasks: smaller /larger datasets, binary/semanie /semantic segmentation, and rotation invariant/ /standard applications. Table 1 presents a summary of key information about each dataset, ic., the number of images that contains, images resolution, and the intended use case for the dataset. The next part this section provides more practical information about them.
saariiant/stadaarr
Kvasir-SEG This first dataset (Jha et al., 2020) contains endoscopic images. Its focus is the segmen- tation of polyps in human colons. It contains 1, 0.0 images with varying sizes, from 625 x 513 pixels for the smallest image to 1920 x 1072 pixels for the largest one. Each image is assigned to a binary mask that shows the items of interest, ie., the polyps. Figure 2 presents three examples of paired images and masks the datasct. This datasct is expected to be very informative on the ability of a model to learn from only a few images. Furthermore, the task is inherently rotation invariant, as polyps may appear with any arbitrary orientation.
8

---

Published in Transactions on Machine Learning Research (April/2025)
Images
Masks
(a)
(b)
(c)
Figure 3: Example of images with their segmentation mask from the NucleiSeg dataset. Notice the multi- plicity of nuclei in single patched images.
NucleiSeg This second dataset (Janowezyk & Madabhushi, 2016) is designed for segmenting cell nuclei in histopathological images. It contains 135 images with multiple nuclei per image. Each image has resolution 2f 000 000 pixels, much higher than the resolution of images from the other binary segmentation dalasets considered in this work. Because of this, each image is pre-processed to generate set of lower resolution images by extracting random patches of 448 x 448 pixels within the original images and masks. Because there are multiple nuclei in each image, the pre-processing will generate sub-images containing multiple nuclei too, such that the images remain useful for the segmentation task. For each amage-patch pair, this operation is repeated 30 times ensure diversity and comprehensiveness. Figure 3 illustrates few paired image-mase examples resulting from the pre-processing. The NucleiSeg dataset is considered to be informative the performance of segmentation models in biomedical images having many foreground items per image. The orientation of the object is not expected to contain any information and objects of interest always have a circular shape.
The name of this third dataset, URDE (De Silva al., 2023), stands for Urban Roadside Environ- ment and is the last involving binary segmentation. The dataset focuses detecting dust clouds induced by unsealed roads. It includes 7, 7,000 images of 1, 024 x 1, 024 pixels with their annotation
URDE
vehicles traveling masks. For illustration, Figure 4 presents a sample of paired images and masks in the dataset. In this case, the problem is more general and orientation may be seen as a particular form of information.
This fourth dataset (Caesar et al., 2018) is large-scale dataset commonly used for general segmentation tasks. It includes over 164, 000 images with 91 stuff categorics (plus class unlabcled) notation masks, providing a diverse and exlensive set of images for evalualing general-pure purpose segmentation algorithms, The resolutions vary between images, with a maximum of 640 x 640 pixels. Figure 5 presents a few examples of paired images and masks in the dataset. The complexity and size of the dataset make it a must-have for benchmarking semantic segmentation tasks. As with the URDE dataset, orientation in COCO-Stuff contains meaningful information rather than being arbitrary.
COCO-Stuff
iSAID This fifth and last dataset (Waqas Zamir et al., 2019) is based the DOTA (Xia et al., 2018) dataset. While the latter focuses on object detection in satellite imagery, the former focuses on segmentation of the same images. iSAID consists of 2, 806 images with annotations for 15 different object classes such as buildings, vehicles, and vegetation. The resolutions the images vary from 353 x 278 pixels for the smallest image 4, 294 x 4, 0000 pixels for the largest onc. Consoquently, the same pre-processing as the one described for the NucleiSeg dataset applied to initial images and masks generate input images and masks of 448 x 448 pixels, 30 times per initial pair of image and mask. In similar fashion to other
9

---

Published in Transactions on Machine Learning Research (April/202)
Images
Masks
(a)
(b)
(c)
Figure 4: Example of images with their segmentation mask from the URDE dataset. One can notice that
the masks have a cloudy, irregular structure.
Images
Masks
(a)
(b)
(c)
Figure Example of images with their segmentation mask from the COCO-Stuff dataset. Those images illustrate the fact that orientation may matter. For instance, tennis players expected to be standing up.
datasots, a few examples of paired images and masks contained in the dataset are presented in Figure For this last. dataset, the global orientation of the images is inherently arbitrary and does not. carry meaningful information. To illustrate this, one can inspect Figure 6 (b), where the cars (dark blue segmentation masks) appear in different orientations across the image. Indeed, cars at the top of the image have given orientation, while cars on the bottom-left have a different orientation. The variability in orientation for the same kind of objects is natural for aerial imagery, where objects photographed from above can appear rotated at any angle, but they are fact the same object. This exactly the idea captured by the property of rotation equivariance.
Table 1: Dataset Characteristics
Datasct
N. images
Resolution (pixels)
Use Case
Orientation Sensitive
Kvasir-SBG
1,000
332 487 1, .200 1,072
Polyp segmentation.
NucleiSeg
135
2,000 x 2, 000
Cell muclei segmentation.
X
URDE
7,000
1, 1,024 x 1, 024
Roadside cloud dust segmentation.
COCO-Stuff
164, 000
to 640 x
General object segmentation.
iSAID
2, 806
353 x 278 to 4, 294 x 4,000
Segmentation in satellite imagery.
X
10

---

Published in Transactions on Machine Learning Research (April/202))
Images
Masks
(a)
(b)
(c)
Figure 6: Example of images with their segmentation mask from the iSAID dataset. The orientation in those images does not matter as they are taken from an aerial perspective.
4.2 Model Architectures
After introducing the considered datasets, this section continues with a presentation of U-Net models that are used in this work and the motivation for their selection. These models can be divided in two main categories: (i) vanilla U-Nets and (ii) equivariant U-Nets. Within each category, we evaluate two model variants a larger version and a smaller version which differ in the number of trainable parameters. Furthermore, several ways of introducing the equivariance into U-Net are considered. To ensure a fair comparison, all models (and all variations of those models) share the same overarching architecture, with differences only in the convolutional approach, and in the number of filters in each layer (in order to compare models with the same number of trainable parameters). The following gives more details about the two categories of U-Net models.
Vanilla U-Net The vanilla U-Net serves as the baseline model in this study. U-Net by Ronneberger el al. (2015) is well-established Convolutional Neural Network (CNN) architecture, commonly used for image sogmentation tasks (more details are presented in Section 2). This network relies on standard convolutional layers that are not inhereutly equivariant to rotations. As a result, this model serves as a baseline to measure the potential benefits of including rotation-equivariant mechanisms in alternative models. As indicated above, small and a large version of this vanilla U-Net are developed.
The architecture of the vanilla U-Net is close to the one from the original paper by Ronneberger et al. (2015). It is made of two convolutional layers, followed by four downsampling (downsampling is achieved through max pooling layers that divide the resolution by a factor of 2) and four upscaling blocks (with bilinear upsampling that increases back the resolution by a factor 2, followed by convolution layers). Note that the use of a bilinear upsampling followed by a convolution is not the most common choice. Most. of the time, transposed convolutions (or up-convolutions) are considered instcad. However, those layers are available in the case of equivariant models. Consequently, to ensure fair comparison, transposed convolutions are replaced by this two steps process of upsampling followed by a simple convolution. In practice, replacing the transposed convolutions with this process does not significantly affect the performance and involves the same number of trainable parameters.
All the convolutions the model use a kernel size 3, and are followed by batch normalization layers and ReLU (Nair & Hinton, 2010) activation functions. The first convolutional layer takes a fixed 224 x 224 input image. Finally, the last convolutional layer reduces the number of channels to the number of classes that are to be predicted.
11

---

Published in Transactions on Machine Learning Research (April/2025)
All the elements regarding the U-Net configuration that are described in the previous paragraph are common for the two vanilla U-Net models (small and large), but they count a different number of trainable parame- ters. The small model contains roughly 500, 000 trainable parameters, while the large one contains roughly 11, 000, 000 of them. In order to create diffcrences in model size, a factor À E R introduced to divide the number of filters each layer of the model, thereby controlling the number of trainable parameters in the model. À has a constant, specific value for each model and size combination. For example, 1.65 for the large vanilla U-Net, and 6.00 for its small counterpart. The final number of filters for each convolutional layer is found in Table 2.
Table 2: Number of filters for the different models, for the two first convolutional layers (convl, conv2), the four downsampling blocks (D1, D2, D3, D4, located the left-hand side of Figure 1), and upsampling blocks (U1, U2 U3, U4, presented on the right-hand hand side of Figure 1).
Small Models
Large Models
convl
conv2
D1
D3
D4
U3
U4
conv
conv2
D1
D3
D4
U1
U2
U3
U4
vanilla
12
12
86
172
86
12
52
52
104
206
410
820
410
206
104
52
C4
4
4
12
24
46
24
14
14
28
56
112
222
112
56
28
14
C8/D4
4
4
10
18
34
18
10
10
10
20
40
78
154
78
40
20
10
The E(2)-U-Net modifies the vanilla U-Net architecture by incorporating equivariant convo- lutional blocks. These equivariant convolutional blocks follow the exact same architecture for the vanilla U-Net, except that the group convolution implemented in E(2)-CNNs (Weiler & Cesa, 2019) is used. In addition, modified versions of ReLU activation functions and batch-normalization layers are also used. This is necessary given the fact that E(2)-CNNs involve a particular representation of the data that requires using specific activation functions and normalization. Furthermore, a final layer is also added at the end of the decoder part to perform the symmetry pooling over the different transformations in the considered symmetry group (see Section 2 for more details). Finally, an important modification also the use of bigger filter sizes compared Cesa, 2019; Worrall et al., 2017; Delchevalerie et al., 2023) shows that using larger filter sizes is required to be able to achieve rotation equivariance (due to the use of a cartesian grid to represent images). This results in E(2)-U-Nets using kernel size of x 9 for convolutions, instead of 3 x 3 for the vanilla counterpart.
E(2)-U-Net
vanilla convolutions. Indeed, many previous work equivariant CNNs (Weiler &
As with the vanilla U-Net, small and large versions the models are defined by modifying the number of filters in each layer. Additionally, several symmetry groups are considered per E(2)-U-Net model:
C4: the cyclic group order 4, which ensures equivariance to rotations of 90%.
Os: the cyclic group of order 8, which extends equivariance to rotations of 45%.
D4: the dihedral group of order 4, which combines 90º rotations and reflections.
This setup results in the creation of six U(2)-U-Net architectures (three small, and three large models). All the configurations are presented in Table 2, together with the number of filters in each layer.
5 Experiments
This section deals with the experiments performed in this study. First, the training setup is outlined, with details on the selected optimizer, loss function, and their byperparameters. Second, details about the evaluation metrics for each task are presented. Third, the results obtained by the vanilla U-Net as well as the different E(2)-U-Net models across the different tasks are presented.
5.1 Training Setup
All models are trained under identical conditions to ensure a fair comparison. The selection of hyperparam- eters based on the initial paper of each dataset, except for NucleiSeg, where the author's implementation deviated too much from approaches used in the four other datasets. Specificially, the authors use batch size
12

---

Published in Transactions on Machine Learning Research (April/2025)
of 128 of patched images having size of 32 x 32 pixels and train their model over 600, 000 steps (Janowezyk & Madabhushi, 2016). Because our model accepts images with input size of 224, the resolution chosen by the authors is too small for our study. As consequence, standard hyperparameters are retained for this dataset. The AdamW (Loshchilov & Hutter, 2019) optimizer is selected for all datasets, with a learning rate and a number of epochs that is specifie each dataset, as presented in Table 3. The Dice Loss (Sudre et al., 2017) is retained as the only loss criterion, because is well-suited for segmentation tasks, particular with imbalanced data. Similarly, the batch size is specific each dataset and is also indicated in Table 3. Finally, the following data angmentation techniques are applied to images and their corresponding mask during training: random rotations of 09, ,990, 1800 or 270 with probability 0.25 each, horizontal and/or vertical flip(s) with probability 0.5 for each, and also a color normalization.
Table 3: Hyperparameters for Each Dataset
Dataset
Batch Size
Learning Rate
N. epochs
Kvasir-SEG
1 10-1
150
NucleiSeg
16
X 10
200
COCO-Stuff
16
1 x 10-3
200
URDE
5 x 10-1
500
ISAID
16
x 10
250
For each model configuration, five-fold cross-validation is used. To do so, the dataset is partitioned into five non-overlapping subsets. For each of the five folds, four subsets are used for training while the remaining one is usod for testing. This approach guarantecs that cach sample appears in the test set exactly once, and provides statistical robustness of the results. As already presented in Section 4, two different quantities of data considered for training for all the different datasets. These two settings are defined by (i) "Large data setting" and (ii) "Small data setting" and correspond to the use of the full 4 folds for training for (i) and the use of only a subset of 10% of each fold for (ii). Thus, (i) corresponds to the use of 80% of the available data for training, while (ii) corresponds to the use of only 8% of the initial dataset (i.e., 10% of the
80% of total data). For both settings, the testing set is held constant, and is the entirety of the remaining fold (made of 20% of the data). All models are trained using one Nvidia Tesla A100 with 40GB.
5.2 Evaluation metrics
The performance of the models is evaluated using metrics that are specific to the task at hand: binary segmentation or semantic segmentation.
For the binary segmentation tasks, one must classify each individual pixel as belonging either the fore- ground class to the background class. Five performance metrics are selected to evaluate the ability of model to correctly identify the foreground class in images.
Dice Score (DS) is a mcasure of the similarity (or overlap) between two sets, i.c., predicted segmentation masks (P) and ground truth masks (G). DS is computed as the ratio of twice the intersection the predicted and ground truth masks to the sum of their total pixel counts. A perfect overlap has a DS of 1 since all pixels are forecasted in the correct class, while a DS of 0 means that there absolutely no overlap between forecasted masks and their ground truth.
2 | |
DS
Intersection over Union (IoU) measures the overlap between the predicted segmentation masks (P) and their ground truth (G). IoU is defined the intersection of the predicted and ground truth segments divided by their union.
PnG
LoU = PUG
13

---

Published in Transactions on Machine Learning Research (April/2025)
Precision is the ratio of True Positives (TP) to the sum of TP and False Positives (FP). This metric indicates the ability of a model to avoid false positives: the greater the ratio, the lower the relative FP.
TP
Precision
TP + FP
Recall is the ratio of TP to the sum of TP and False Negatives (FN), Recall measures the ability of a model to detect the foreground class.
TP
Recall
T
Accuracy the proportion of correctly classified pixels, ie., TP and True Negatives (TN) out of the total number of pixels.
TN Accuracy = TP + TN FP+ FN
order to assess the performance of the models on the semantic segmentation tasks, four widely used evaluation criteria are selected (Caesar et al., 2018; Long et al., 2015; Eigen & Fergus, 2015; Caesar et al., 2016): (i) Mean Intersection predictions and ground-truth class, and average the number over classes, (ii) Frequency-weightee IoU (FW IoU) is the per-class loU, weighted by the frequency (at the pixel-level) of each class, (iii) Mean Accuracy (mAcc), is the average pixel accuracy calculated across all classes, and (iv) Pixel Accuracy (pAce) is the percentage of pixels that are correctly labeled. Note that these criteria are derived from binary segmentation metrics described above.
Union (mIoU) divides how many pixels are in the intersection between
5.3 Results
This section presents the results obtained for the five datasets. For each of them, the results for the small and large models, both for the vanilla and E(2)-U-Net are presented. Each configuration has two settings: a small data setting (i.c., only 10% of each training fold used), and a large data setting (i.c., the full 4 folds are used for training). This means that for each dataset, each of the four models (one vanilla, and three F(()-U-Nett) has four variations: small/large number of trainable parameters (two variations), and small/ llarge data setting (two additional variations). This resulted in the training of 16 models, for each of the five datasets.
Kvasir-SEG Results for the Kvasir-SEG dataset are summarized in Table 4. In the large data setting, and regarding the small modcls, the E(2)-U-Net model based on the use the Cs symmetry group outperforms the others in terms of Dice Score, ToU, precision, and accuracy, whereas the vanilla U-Net achieves the highest recall. Regarding the larger models for the same data setting, the E(2)-U-Net based on C's again shows the best performance in Dice Score, IoU accuracy, and recall. With regards to precision, both Cs and D4 show comparable performance. In the small data setting, and regarding small models, Cs has the highest Dice Score, ToU, precision, and accuracy while C4 has the highest recall. As far as the large models and small data setting is concerned, the overall picture is comparable to that of the small models for the same data setup in terms of DS and IoU, where Cs has the highest metrics. Interestingly, the vanilla model has the best recall for the large model. Overall, the equivariant models show better performance than the vanilla counterparts, both regarding small and large models.
While the above analysis helps in understanding models' performance when there is no constraint training resources, second analysis is performed to assess the performance from sustainability standpoint that considers the resources used. To do so, one can visualize the evolution of performance against the cumulative time during training. Therefore, we also present in Figure 7 the loU with respect to the cunulative training time (in seconds) for the large as well as small models, both for the small and large data setting case. Please note that the total number of epochs the same for all models and all configurations. Under the large data setup in this Figure, one sees the vanilla U-Net is better than the equivariant ones until stops training at 3 x 10 and 7 102 seconds, for small and large models respectively. Thus, if one had to stop training early,
14

---

Published in Transactions on Machine Learning Research (April/2025)
Table 4: Performance metrics for the Kvasir-SEa SEG dataset
Large data setting
Small data setting
Model
Dice
IoU
Pre.
Rec.
Acc.
Dice
IoU
Pre.
Rec.
Acc.
Small models
vanilla
80.5 =2.5
+3.1
79.2 16.1
83.5 =44
93.9 =12
544 16.7
16.0
65.4 #13.3
52.6 #111
87.2 1:2.8
80.6 =3.2
443
84.2 43,8
79.2 14.4
91.2 =11
50.5 19.5
35.3 47.9
55.0 #147
63.1 #140
72.9 =20.2
Cs
+4.0
86.7 +1.5
81.2
95.0 40.9
426
53.0 45.0
89.6 4:05
D2
=3.6
69.0 +4.8
85.0
+4.55
79.0
94.4 =13
#9.2
34.7 48.0
57.6
#149
1154
74.0 =20.0
Large models
vanilla
86.5 =1.6
76.7 +2.3
89.5 #1.1
84.7 22.7
96.0 00.5
66.5 #3.8
50.8 #4.1
76.2 45.8
61.3 +0.9
90.7 10.4
C+
87.2 -1.2
77.8 +1.7
88.8 +2.0
86.6 +1.7
96.2 -044
66.7 +3.
51.1 +3.5
83.9 +5.0
57.4 +3.77
91.4 +0.5
C
88.0 +1.9
79.0 12.
90.2 +12
86.8 =34
96.4 40.7
68.3 +3.9
53.0 14.7
82.7 42.7
60.0 14.8
91.7 10.5
D.
86.4 =1.9
76.7 43.1
90.2
+16
$4.0 #2.7
96.0 =0.7
63.8 #2.7
47.7 22.8
$3.5 47.4
53.4 14.5
90.9 #0.6
the vanilla model would be better, leading to a gain in computational resources. be made about the small data set
The same observation can
Vanilla
C4
C8
D4
Small Models, Large Data
Small Models, Small Data
Large Models, Large Data
Large Models, Small Data
0.8
0.6
Score
0.4
0.2
10a
102
10a
100
10a
10a
10a
102
10a
10a
10a
10a
Training time (seconds)
Figure Evasir-SEG. IoU performance over cumulative time (in seconds) for small and large models small and large data setting, averaged across the 5 folds. The total number of epochs is the same for all models and all configurations.
NucleiSeg Results for the NucleiSeg dataset are summarized in Table 5. Regarding the large data setting for the small models, one can observe that the vanilla U-Net model has the highest Dice Score, ToU and recall. Considering the precision metric, the U-Neet based on the Cg symmetry group achieves the best score. As far the accuracy is concerned, Cs and D4 both show the highest. score. For the large data setting and the large models, the vanilla U-Net model has the best Dice Score, IoU, precision, and recall. C4 has the highest accuracy level. The small data setting reveals that vanilla models for both small and large models have the best Dice Score and loU. The other metrics are dominated by Cs or D4. Overall, the vanilla models seem to perform better than the equivariant counterparts for this dataset.
Figure presents the IoU with respoct to the cumulative time seconds (with a constant number of epochs throughout models and setups). It. shows that vanilla models have the greatest performance throughout. Also, one can notice that the performance starts dropping around time 2 x 102 and x 102 seconds for small and large models respectively, under the large data setup. This is also true for the small data setup, with drop of performance at 1 x 102 and 1 102 seconds.
URDE Results for the URDE datasct are summarized in Table 6. Focusing on the large data setting and small models, the E(2)-U-Net based on C4 and D4 models achieves comparable Dice and loU. The D1 variant also has the highest precision, while the Cg model outperforms the other models in recall. The equivariant configurations all achieve highest accuracy metrics. For the larger models under the large data setting, the E(2)-U-Net based on C2 shows the best Dice Score, IoU, precision and recall. It also achieves the highest accuracy, score matched by Cs. The small data setting shows that the best large model is Ca
15

---

Published in Transactions on Machine Learning Research (April/2025)
Table Performance metrics for the NucleiSeg dataset
Large data setting
Small data setting
Model
Dice
IoU
Pre.
Rec.
Acc.
Dice
loU
Pre.
Rec.
Anc.
Small models
vanilla
28.4 =7.6
17.3 +5.1
37.4
30.1 13.8
97.0 10.7
18.7 =7.3
10.8 #46
25.2 13.2
20.5 110.5
96.6 10.4
33.3
+2.7
+0.3
23.4 +4.3
13.8 +2.8
37.8 134
=5.6
97.3
46.2
7.7 #3.7
30.5 24.2
10.4 16.0
97.4 =0.4
D4
21.5 #5.2
14.5
+3.3
37.6 14.3
21.2 =6.9
97.3 +0.2
12.9 +5.9
7.1 #3.5
22.8 1145
29.2 19.5
78.3 #2.7
Large models
vanilla
39.8 =2.3
25.9 +1.
36.9 117
53.3 +4.7
96.5 1:066
24.9 +8.3
15.1 +5.5
36.1 #7.1
26.0 16.9
97.0 40.8
Ca
314 +1.7
21.6
318
40.1 =5.6
96.8
43.6
9.0 #2.1
38.4 116
114 13.1
97.5 =041
42.3
34.4
+3.6
37.4
D2
+3.1
21.0 2244
33.3
+2.6
66.8
96.4 10.6
+5.2
43.1
37.2
#3.3
9.2 14.4
97.5 =0.5
Vanilla
C4
C8 Large Models, Large Data
Smail Models, Large Data
Small Models, Small Data
Large Models, Small Data
0.20
loU Score
0.10
0.05
10a
10a
10a
10
10a
102
10a
10a
10a
10
10a
10a
10
Training time (seconds)
Figure 8: NucleiSeg: loU performance over cumulative time (in seconds) for small and large models in small and large data sctting, averaged across the folds. The total number epochs is the same for all models and all configurations.
for all performance metrics. Regarding the small models of this same sctting, C4 has the higest Dice Score, IoU, recall and accuracy, whereas C' has the best precision.
Table Performance metrics for the URDE dataset
Large data setting
Small data setting
Model
Dice
ToU
Rec.
Acc.
Dice
loU
Pre.
Roc.
Aoc.
Small models
wandla
78.1 +2,4
70.9 +22
41.4 +3.1
39.3 +3.9
98.5 +0.1
65.9 +4.3
58.6 +4.7
31.7
335
33.2 +4.5
97.9 +0.2
Ca
82.0 #1.6
75.0 +1.7
43.5 42.7
42.1 424
98.7 =0.1
67.9 +2.6
60.4
22.6
37.5
22.7
33.4
42.3
98.2 +0.1
C4
81.6 42.1
74.7
+22
43.5
42.1
42.3 +28
98.7 +0.1
66.4
+4.9
58.7
+4.8
37.8
42.5
32.9
#2.0
98.1 =0.1
Da
82.1 +:1.3
75.0 +1.4
44.0 +2.
41.9 +2.7
98.7 +:011
+2.9 65.9
58.1
+:2.9
37.5 =3.0
32.8
+2.6
98.1 =0.1
Large models
vanilla
81.3 #1.0
74.1 10.8
42.2 11.
40.4 42.3
98.6 10.1
62.1
18.
54.8 18.8
32.4 11.
32.0 13.5
97.7 10.5
C4
84.4 +1.5
77.6 114
44.6 +1.
43.0 43.3
98.8 20.1
71.3 22.2
63.5 11.9
39.1 43.2
33.4 42.9
98.2 10.1
C4
83.6 116
76.6 116
43.2 118
43.1
133
98.7 20.1
70.6
12.9
63.0
12.9
38.0
-1.
33.1 11.5
98.2 10.1
D4
81.2 #1.5
#1.4
44.5 42.2
42.8
42.7
98.7 =0.1
71.0
11.
63.2
119
38.6 225
32.9
98.1 =0.0
Regarding the time-performance plot (with the same total number of epochs for all configurations and models) presented Figure 9, it is clear that for the small data setting as well as for the large data setting, the vanilla U-Net performs significantly worse than the three equivariant models in terms of IoU, both for the small and large models.
COCO stuff Results for the COCO-Stuff dataset are presented in Table For small models in the large data setting, the vanilla U-Net shows better performance than the F(()-U-Nets for all metrics, except for
16

---

Published in Transactions on Machine Learning Research (April/2025)
Vanilla
C4
C8
Small Models, Large Data
Small Models, Small Data
Large Models, Large Data
Large Models, Small Data
0.8
Score
0.6
M1 1170
0.2
102
1000
10a
10a
10a
10a
102
10a
10a
10a
10a
10a
10a
10"
Training time (seconds)
Figure 9: URDE: loU performance over cumtlative time (in seconds) for small and large models in small and large data sctting, averaged across the 5 folds. The total number of epochs is the same for all models and all configurations.
mean accuracy, where its performance is equal to that of C4. However, when considering larger models, C and Cs models close the gap in performance and Cs even achieves the best mean IoU and mean accuracy. The Da model consistently underperforms. In the small data setting, however, the vanilla. model is best across all metrics, both for small and large models.
Table 7: Performance metrics for the COCO-Stufi dataset
Large data setting
Small data setting
Model
Mean loU
Pixel Acc.
Mean Ace.
FW loU
Mean IoU
Pixel Acc.
Mean Acc.
FW IoU
Small models
vanilla
13.6 +0.2
64.2
+0.3
18.9
#0.2
49.1 +0.5
10.8 40.2
60.1 10.3
15.7
10.3
44.6 10.6
C4
13.5 +0.1
64.0 +0.3
18.9 +0.2
48.9 +0.5
9.5 +0.2
56.9 +0.4
14.1 +0.2
41.4 004
Cs
13.0 #0.6
63.0 =1.0
18.4 +0.9
47.7
#1.4
9.6 +0.3
57.6 #0.4
14.3 +0.3
42.6 00.3
D4
12.9 10.5
62.9 0.7
18.2
10.6
10.9 47.4
8.6 10.3
55.1 10.7
12.7 10.3
40.9
0066
Large models
vanilla
17.9 +0.1
68.9
+0.2
23.8 +0.3
56.3 +0.6
10.9 +0.3
60.9 +0.4
15.2 +0.4
47.7
+0.5
C1
18.2 +0.2
67.8 002
#0.2
54.8 10.3
10.6 10.3
60.4 10.4
10.3
47.2 00.4
Cs
18.3 +0.4
67.8 =02
24.2
10.6
54.8 -0.7
10.4 10.1
60.3 +0.3
14.5 10.1
47.2 =0.1
D4
16.7 +-0.3
65.9 0.5
22.5
+0.2
53.4 +0.1
9.1 +-0.2
58.7 +0.4
12.9
+0.3
45.6 -05
The analysis of Figure (similarly to the previous analyses, total number of epochs is constant) shows that under the large data setting, the small vanilla model always better than its equivariant counterparts in terms of mean loU. However, when looking large models, it not always the case. Indeed, between times 1.5 3 and 8 10t, notices that very similar performance between vanilla and C₁ models. Ultimately, Ca and Cs finish at higher mean ToU levels than the vanilla U-Net. For the small data setting, the vanilla model is always the best in terms of mean loU.
iSAID Results for the iSAID dataset are presented in Table 8. As far as both the small and large data settings are concerned, and for both the small and large models, the vanilla U-Net perforrsms the three other models across all metrics, with a clear advantage mean loU and mean accuracy.
The analysis of mean IoU against cumulative time (in seconds) in Figure 11 shows that the vanilla models always the best, for the small and the large data setup, both for small and large models.
17

---

Published in Transactions on Machine Learning Research (April/2025)
Vanilla
C4
C8
Small Models, Large Data
Small Models, Small Data
Large Models, Large Data
Large Models, Small Data
lou Score
0.10
0.05
10a
10a
10
102
10a
10a
100
10a
10a
10a
102
10a
10
Training time (seconds)
Figure 10: COCO-Stuff: mean IoU performance cumulative time (in seconds) for small and large models in small and large data sctting, averaged across the folds. The total number of epochs is the same for all models and all configurations.
Table 8: Performance metrics for the iSAID dataset
Large data setting
Small data setting
Model
Mean loU
Pixel Acc.
Mean Acc.
IW loU
Mean loU
Pixel Acc.
Mean Acc.
IW loU
Small models
vonilla
39.9 1:16
97.2
+..3
43.5
11.7
95.1 10.4
30.3 11.7
95.9
10.5
33.3
11.
93.1 10.8
C4
33.4 12.6
96.8 0.3
36.7
12.8
94.5 10.4
23.5
11.2
95.1
10.4
25.8
112
91.9 0.7
C.
34.9 +2.0
97.1 0.2
38.6
+1.9
94.9
00.4
24.6
+2.0
95.6 +0.3
27.1 +1
92.6 005
D1
30.8 +2.9
96.8 -0.3
34.1
13.2
94.4
10.5
20.3
11.2
95.0
10.4
22.4
113
91.7 =0.7
Large models
vnnilla
48.2 1..
97.6
+0.33
52.7 +1.7
95.8 +0.4
36.2 +2.0
96.9 +0.3
39.7 4-2.0
94.6 +0.4
C4
43.1 +1.
97.5 =0.2
47.1
+1.7
90.4
32.1 +1.0
96.3 +0.3
35.0 #1.0
93.6 00.5
Cs
44.6 11.4
97.6 0.3
48.8 11.6
95.7 10.4
32.3 11.0
96.5 10.3
35.2 11.0
93.9 0.5
D1
40.4
10.2
97.3 0.3
44.2
10.2
95.3 10.4
26.3 10.6
96.1 10.3
10.8
93.2 00.5
6
Discussion
The analysis of the results reveals interesting patterns regarding the usefulness of equivariant models across model sizes, the type of lasks, the dataset specificity (i.e., the size, nature of images, and number of classes) and the data regime. This section sums up the observations from the previous section, and tries to extract
Vanilla
C4
D4 Large Models, Large Data
C8
Small Models, Large Data
Small Models, Small Data
Large Models,
Small Data
0.5
0.4
0.3
0.2
0.1
10a
10c
105
10a
10a
104
10
104
10
10a
10a
10'
Training time (seconds)
Figure 11: iSAID mean IoU performance over cumulative time (in seconds) for small and large models small and large data setting, averaged across the 5 folds. The total number of epochs is the same for all models and all configurations.
18

---

Published in Transactions on Machine Learning Research (April/2025)
some insights about when it could be interesting to consider equivariant U-Net for segmentation tasks. Based on the results and previous studies in the literature (see Section 2), the discussion is subdivided into two parts, depending on the interest one has in using equivariant models: (i) getting an improvement in terms of ovaluation metrics ("are oquivariant models better at solving the considered task?") and (ii) getting an improvement in terms of sustainability ("are equivariant models able to achieve comparable performance while reducing the amount of required computational resources?").
6.1 Discussion based on evaluation metrics
First, we present our key findings on evaluation metrics. We then discuss the overall performance indepen- dently for the different setups in terms of model size and data regime (small or large). In particular, we discuss performance for (i) small models in large data settings, (ii) large models in large data settings, (iii) small models in small data seltings, and (iv) large models in small data settings.
Key findings Our analysis reveals that the effectiveness of equivariant U-Nets compared to their vanilla counterparts largely depends on the nature of the objects as well as the complexity of the segmentation task. For datasets where objects can have arbitrary shapes, and their orientation is not semantically meaningful, such as Kvasir-SEG and URDE, we find that equivariant models often outperform vanilla architectures (see Table 4 and Table 6)- However, when objects are symmetric from rotational standpoint (e.g., nuclei have circular shapes in NucleiSeg), vanilla U-Nets show better performance than their rotation equivariant counter parts. One explanation is that the rolation equivariance constrains the filters and reduces their expressiveness, which hinders the learning process of the model. For general-purpose segmentation, like COCO-Stuff, cquivariant U-Nets match the vanilla ones on performance metrics (sce Table 7). For those general lasks, the performance gap arises in small data regimes, where equivariant models are consistently worse than the vanilla, ones. This shows that low data availability affects the trade-off between architectural constraints and model expressiveness. In particular, D4 consistently shows lower performance across datasets, suggesting that stronger symmetry constraints may be too restrictive for practical segmentation tasks.
Small models in large data settings In terms of evaluation metries, equivariant models can be preferable to their vanilla counterparts depending on the task. For example, it is the case for the Kvasir-SEG dataset and the URDE datasot, where results show that E(2)-U-Net can achieve better performance (only requiring slightly more computation time for Kvasir-SEG). However, far as the NucleiSeg datasel is concerned, vanilla U-Nets seem to be a better choice than equivariant U-Nets, as they achieve better or at least top performance regarding all the different metrics. This can be explained by the fact that object of interest in NucleiSeg are all of circular shape. In such situation where the object of interest already satisfy the rotational symmetry by itself (rotation of the object of interest has no impact on the observed shape), applying constraints on the filters like in group-CNNs is meaningless, and even leads to an unnecessary reduction of the expressiveness of the filters, leading to lower levels of performance.
Regarding the COCO-Stuff and iSAID datasets, the vanilla U-Nets achieve better overall performance. However, for COCO-Stuff, the C4 equivariant U-Net is able to achieve similar performance, only requiring larger training time. Regarding the iSAID dataset, equivariant models seems to struggle achieving the vanilla performance, which is even more true for the D2 equivariant model. This may be explained by the fact that the prior knowledge of rotation equivariance is not worth the reduction in expressiveness in the filter. Indeed, only rotation equivariant features can be extracted, which is a constraint that reduces the search space when optimizing the filters, and which is sometimes not beneficial. This insight can also be confirmed by looking at the D4 models that are always significantly below the other ones in terms of performance, and which further constrain the shape of the filters by including a reflection symmetry.
Large models in large data settings Regarding the first three datasets, i.e., Kvasir-SEG, NucleiSeg and URDE, conclusions are very similar to the previous ones using small models. However, one can point out that large models are always significantly better than their lightweight versions. Now, the results for the COCO-Stuff and iSAID datasets are more interesting. Indeed, for the first one, performance is significantly better, and the C4 and Cs models are now even better than the vanilla models according mean ToU and
19

---

Published in Transactions on Machine Learning Research (April/2025)
mean accuracy. For the iSAID dataset, vanilla U-Net are still the best performing ones and large differences still exist with their equivariant counterparts.
Small models in small data settings The results for this experimental setup really similar to those already discussed. However, one can point out that the gap in performance is sometimes bigger. It means that significantly reducing the amount of data seems to exacerbate the advantage and disadvantage of the different methods.
Large models in small data settings Again, insights are similar to those already provided for the other settings. Using larger models seems beneficial as it significantly increases the overall performance. The URDE dataset is the only case where, for the vanilla U-Net, using a larger model leads to a small loss in performance. The observation of Figure shows that the larger model under small data quickly stagnates al around 55% of ToU, with large up and down peaks in the metric between 5 102 and 3 100 seconds. The behaviour of the smaller model under the small data setting does not share this characteristic, and IoU remains at around 59% from 5 x 102 seconds until the end of training.
Final thoughts Based on the results from Kvasir-SEG and NucleiSeg, one may conclude with first insight that equivariant U-Nets are an interesting choice to achieve better performance in tasks for which local rotations in the image have an impact on the final image, but does not contain any information. It is the case for Kvasir-SEGG as polyps can be of arbitrary shape. However, the particular orientation of polyps in the image should not affect their segmentation. This last point is also true for NucleiSeg, but in this case as nuclei are not of arbitrary shape but are instead always of circular shape, using rotation equivariance is detrimental to the performance. One additional factor of explanation lies with the vast quantity of objects that are present in each image, making the task more difficult for equivariant U-Nets.
Regarding datasets that focus on more general tasks, such as COCO-Stuff, equivariant models can achieve similar performance to their vanilla counterparts. However, if equivariant features are used alone (without introducing non-equivariant features in the model), results show that it is preferable consider sufficiently large models. Indeed, the loss in expressiveness in the filters makes it difficult for the small models capture enough features in the images. Nonetheless, the fact that equivariant U-Nets are able to achieve similar performance when increasing the number of filters indicate that they are able to extract meaningful features. Thus, for tasks that are more general and involve complex object segmentations, one could benefit from new models that are specifically designed to extract both cquivariant and non-equivariant features in parallel.
6.2 Discussion in terms of computational resources
Achieving the best performance in terms of evaluation metrics is of course of primary importance for many tasks. However, it is also important to consider the amount of computational resources (and time) that are involved in the training process of the models. For example, for some applications, it could be interesting to consider a particular model even if it leads to slightly lower metric performance (e-g. lower ToU) if it could save a significant amount of time. For this purpose, E(2)-U-Net are an interesting candidate. This second part of the discussion will focus more on the question of sustainability. In other words: "is possible to obtain a good trade-off between performance and training time?"
For many of the experimental setups that were considered, E(2)-U-Nets are able to achieve better or similar performance. However, only observe for few particular cases (URDE, small and large models large data settings, and iSAID, small models and small data settings) potential gains in terms of computational re- sources. In other cases, the IoU with respect to the training time are similar or convergence is slightly slower for equivariant models. This can be explained by the fact that even rotation equivariance is meaningful ful, using cooppcconvouuiions more expensive than vanilla convolutions. The analysis of the results reveals that adding more computational resources does not always justify the marginal improvements epoch-wise convergence. This finding challenges the assumption that equivariant models are universally beneficial for reducing training time, the optimal choice depends heavily the specific task requirements. The experi-
20

---

Published in Transactions on Machine Learning Research (April/2025)
ments consistently show that both larger models and larger data regimes correlate with higher performance. Nonetheless, the relationship between these factors reveals an important asymmetry: reducing the training dataset size has a substantially more detrimental effect on model performance compared to reducing model size, despite both approaches decreasing training time. If one wants obtain a good trade-off between performance and training time, using smaller models appears to be a better approach over using smaller training datasct size.
7 Conclusion and limitations
In this work, the performance of vanilla and equivariant U-Nets is assessed across five diverse datasets, namely Kvasir-SEG, NucleiSeg, URDE, COCO-Stuff, and iSAID. They vary in size, may or may not have rotation-invariant characteristics, and focus on specific, narrow tasks or on tasks that are more general. This enables the examination of the effects of model size, dataset characteristics, and segmentation task type (i.e., binary semantic segmentation). The intention behind such analysis is to get better understanding of how architectural constraints, such rotation equivariance, impact the performance and computational efficicncy of sogmentation models.
The findings of this study highlight that equivariant U-Nets (and more specifically, rotation equivariant models), particularly E(2)-U-Nets based on the use of discrete symmetry groups like Ca, Cs or D4, provide advantages in datasets with high orientation variability, such as Kvasir-SEG and URDE, where oquivariant models performeed their vanilla counterparts. However, in datasets like NucleiSeg, where the objects exhibit inherent rotational symmetry, vanilla U-Nets achieved higher accuracy and expressiveness. For larger, more complex datasets such as COCO-Stuff and iSAID, vanilla U-Nets consistently demonstrated superior performance. This shows that equivariance constraints may not always translate to improved performance, especially in settings requiring broader feature expressiveness. Moreover, we find that reducing the number of training data (i.e., small data setting with only 10% of each training fold data) does not have a significant impact on the relative performance.
An important future work will be to design and test models that able to build set of both equivariant and non-equivariant features. Our experiments highlight the fact that for more complex datasets like COCO- Stuff or ISAID, equivariance is less useful. Still, they are able to extract useful features and can achieve almost state-of-the-ar performance, Thus, we believe that those equivariant features can extract complementary information that could benefit to the model.
Acknowledgments
Valentin Delchevalerie is supported by SPWR under grant 2010235 ARIAC by DIGITALNALIONIA..A. The present research benefited from computational resources made available on Lucia, the Tier-1 1 supercom- puter of the Walloon Region, infrastructure funded by the Walloon Region under the grant agroement 111910247.
The authors thank Pragati Mitra and Sacha Corbugy for their comments and the fruitful discussions on this paper.
References
Adams and L. Bischof. Soeded region growing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(6):641-61) 647, 1994.
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Ignacio Arganda-Carreras, Carreras, Srinivas C. Turaga, Daniel R. Berger, Dan Cireçan, Alessandro Giusti, Luca M. Gambardella, Jürgen Schmidhuber, Dmitry Taptev, Sarvesh Dwivedi, Joachim M. Buhmann, Ting Liu,
21

---

Published in Transactions on Machine Learning Research (April/202)
Mojtaba Seyedhosscini, Tolga Tasdizen, Lee Kamentsky, Radim Burget, Vaclav Uher, Xiao Tan, Chang- ming Sun, Tuan D. Pham, Erhan Bas, Mustala G. Uzunbas, Albert Cardona, Johannes Schindelin, and Н. Sebastian Seung. Crowdsourcing the creation of image segmentation algorithms for connectomics. Frontiers in Newroanatomy, 9, November 2015. ISSN 1662-5129.
Serge Beucher and Centre Mathmatique. The watershed transformation applied to image segmentation. Scanning. Microsc., 6, 07 2000.
Y.Y. Boykov and M.-. Jolly. Interactive graph cuts for optimal boundary region segmentation of objects in n-d images. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1, pp. 105 112 vol.1, 2001.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Region-based semantic segmentation with end-to-end training. In Computer Vision ECCV 2016, pp. 381 397, 2016.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In 2018 TEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp- 1209- 1218. TEEE Computer Society, 2018.
John Canny. A computational approach to edge detection. IEEE Transactions on Pattera Analysis and Machine Intelligence, PAM1-8(6):679 698, 1986.
Liang-Chiel Chieh Chen, George Papandreou, Tasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pallern Analysis and Machine Intelligence, 40(4):834 848, 2018.
Benjamin Chidester, Thal-Vinh Ton, Minh-Triet Tran, Jian Ma, and Minh N. Do. Enhanced rotation- equivariant u-net for nuclear segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1097 1104, 2019a.
Benjamin Chidester, Tianming Zhou, Minh Do, and Jian Ma. Rotation equivariant and invariant neural networks for microscopy image analysis. Bioinformatics, 35(14):1530-53) 1537, 2019b.
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), volume 48, pp. 2990 2999. PMLR, 2016.
Taco Cohen and Max Welling, Steerable CNNs. In International Conference on Learning Representations (ICLR), 2017.
D. Comaniciu and P. Meer. Mean shift: robust approach toward feature space analysis. JEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603-61) -199 2002.
Asanka De Silva, Rajitha Ranasinghe, Arooran Sounthararajah, et al. A benchmark dataset for binary segmentation and quantification of dust emissions from unsealed roads. Sci. Data, 10(1):14, 2023.
Valentin Delchevalerie, Alexandre Mayer, Adrien Bibal, and Benoît Frénay. SO(2) and O(2) Equivariance in Image Recognition with Bessel-Convolationnl Neural Networks. arXim:2304.09211 092114 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nets 2021.
David Eigen and Rob Fergus. Prodicting depth, surface normals and semantic labels with a common multi- scale convolutional architecture. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2650-2658, 2658, 2015.
22

---

Published in Transactions on Machine Learning Research (April/2025)
Axel Elaldi, Guido Gerig, and Neel Dey. E(3) souui-equi networks for spherical deconvolution in diffusion mri. In Medical Imaging with Deep Learning, volume 227 of Proceedings of Machine Learning Research, pp. 301 319. PMLR, 2024.
Pedro F. Felzenszwalb and Danicl P. Huttenlocher. Pictorial structures for object recognition. International Journal of Compuler Vision, 61(1):55 79, January 2005. ISSN 0920-5691.
Alberto Garria, Sergio Orts Erts-Escolano, Sergiu Oprea, Victor Villena-Martine and Jose Garcia- Rodriguez. A review on deep learning techniques applied to semantic segmentation, 2017.
Simon Graham, David Epstein, and Nasir Rajpoot. Dense steerable filter CNNs for exploiting rotational symmetry in histology images. IEEE Transactions on Medical Imaging, 39(12):4124-4136 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980- 2980-2988, 2017.
Andrew Janowezyk and Anant Madabhushi. Deep learning for digital pathology image analysis: compre- hensive tutorial with selected use cases. Journal of Pathology Informatics, 7(1):29, 2016.
Debesh Jha, Pia H. Smedsrud, Michael A. Ricgler, et al. Kvasir-sog: A segmented polyp datasct. In MMM, pp. 451-462, 2020.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1396-9400 2019.
J. Kittler and J. Illinwwotth. Minimum error thresholding. Pattern Recognition, 19(1):41 47, January 1986. ISSN 0031-3203.
Abhay Kumar, Nishant Jain, Chirag Singh, and Suraj Tripathi. Exploiting sift descriptor for rotation invariant convolutional neural network. In IEEE India Council International Conference (INDICON), pp. 1-5, 2018.
Linhao Li, Zhiqiang Zhou, Bo Wang, Lingjuan Miao, and Hua Zong. A novel CNN-based method for accurate ship detection in HR optical remote sensing images via rotated bounding box. IEEE Transactions on Geoscience and Remote Sensing, (9)))668 699, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/ 10ss1400.001,, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 936-944, 2017.
Jasper Limmans, Jim Winkens, Bastiaan S. Veeling, Taco S. Cohen, and Max Welling, Sample efficient semantic segmentation using rotation equivariant convolutional networks. 2018
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVE International Conference Compuler Vision (ICCV), pp. 9992-10002. IEEE, October 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion In 2015 IEEE Conference on Computer Vision and Pattern Recognâtion (CVPR), pp- 3431-3440, 2015.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
23

---

Published in Transactions on Machine Learning Research (April/202))
Diego Marcos, Michele Volpi, and Devis Tuia. Learning rotation invariant convolutional filters for texture classification. In International Conference on Pallern Recognalion (TCPR), pp. 2012-2013 2016.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, October 2017.
Diego Marcos, Michele Volpi, Benjamin Kellenberger, and Devis Tuia. Land cover mapping at very high resolution with rotation equivariant cnns: Towards small yet accurate models. ISPRS Journal of Pho- togrammetry and Remote Sensing, 145:96-107, 2018.
Joshua Mitton and Roderick Murray-Smith. Rotation oquivariant deforestation segmentation and driver classification. NewrIPS 2021 Workshop on Tuckling Climale Change with Machine Learning, 2021.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Pro- ceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp. 807 814, 2010.
Valentin Oreiller, Julien Fageot, Vincent Andrearczyk, John O. Prior, and Adrien Depeursinge. Robust multi-organ mucleus segmentation using locally rotation invariant bispectral u-net. In Proceedings of The 5th International Conference Medical Imaging with Deep Learning, volume 172 of Proceedings of PMLR, 2022.
Machine Learning Research, pp.
Nobuyuki Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62 66, 1979.
Facundo Quiroga, Franco Ronchetti, Laura Lanzarini, and Aurelio F Bariviera. Revisiting data augmentation for rotational invariance in convolutional neural networks. In International Conference on Modelling and Simulation in Management Sciences, pp. 127-141, 2018
O. Ronneberger, P.Fischor, and T. Brox. U-net: Convolutional networks for biomedical image sogmentation. In Medical Image Compuling and Compuler-Assisled Intervention (MICCAT), volume 9351 of LNCS, pp. 234 241, 2015.
Jianbo Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888 905, 2000.
Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for image understand- ing: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. International Journal of Computer Vision, 81(1):2 23, December 2007. ISSN 1573-1405
Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M. Jorge Cardoso. Generalised dice overlap deep learning loss function for highly unbalanced segmentations. Tn Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp.
Cham, 2017.
Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolci Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In IEEE Conference on Computer Vision and Pattern Recogrétion (CVPR), pp. 849 858, 2018.
Jim Winkens, Jasper Linmans, Bastiaan S. Veeling, Taco Cohen, and Max Welling, Improved semantic segmentation for histopathology using rotation equivariant convolutional networks. 2018.
24

---

Published in Transactions on Machine Learning Research (April/2025)
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetor, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedingg of the IEEE Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 5028 5037, 2017.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale datasct for object detection in acrial images. In 2018 IEEE/CVP Conference Compuler Vision and Pallern Recognilion (CVPR), pp. 3974 3983, 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Segformer: Simple and elficient design for semantic segmentation with transformers. Tn Advances in Neural Information Processing Systems, volume 34, pp. 12077-12090, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In 2017 IEEE Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 6230-6239, 2017.
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredess Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip H. S. Torr. Conditional random fields as recurrent neural networks. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, December 2015.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2021.
25


Reviews:
[{"review_id": "bJor9eLejO", "reviewer": "Reviewer_zi1b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n\nS1: The authors approach a relevant topic \n \nS2: The authors use five-fold cross-validation and report results and errors. This leads to statistical robus results and should be standard opractice in machine learning but is not done by an most modern papers. \n\n\nMajor Weaknesses\n\nW1: Limited Architectural Scope: The study exclusively investigates the U-Net architecture, restricting the generalizability of the conclusions. Exploration of additional segmentation models would provide stronger evidence of rotation-equivariance effectiveness.\n\nW2: Inappropriate Comparison: Sec. 2.1. “This end-to-end architecture is capable of producing segmentation masks for objects of arbitrary shapes, distinguishing it from other contemporary architectures such as YOLO (Redmon et al., 2016), which are typically limited to generating bounding boxes.” The comparison between U-Net (segmentation architecture) and YOLO (object detection architecture) is conceptually flawed and irrelevant to the segmentation-focused context of this study.\n\nW3: Lack of Depth and Analysis: The paper provides limited related work coverage and lacks comprehensive analysis on critical aspects, such as explaining metric performance differences among symmetry groups, showing failure cases, and suggesting possible improvements.\n\nW4: Limited Novelty:\nThe paper analyzes a single, somewhat outdated architecture (U-Net), applies only one line of existing approaches for rotation-equivariance, and provides limited in-depth analysis or novel insights. This raises concerns about the novelty and overall contribution of the work.\n\n\nMinor Weaknesses:\n\n- Statements in Sec. 2.2 such as “Recently, numerous studies have focused on incorporating equivariance in CNNs” require references.\n\n- Informal and unscientific wording, e.g., Sec. 2.2: “The most famous technique.”\n\n- Figure formatting could be improved, especially aligning text with images (Figures 2–6).\n\n- Ambiguous phrasing, e.g., “different magnitude in terms of trainable parameters,” is uncommon and unclear.\n\n- Unclear reference to \"performance\" in Sec. 6.2 (“leads to slightly smaller performances”), without specifying if referring to metric performance or computational efficiency.\n\n- Limited number and inconsistent formatting of references (only 32 citations, inconsistent abbreviation and naming conventions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the impact of integrating rotation-equivariance into the U-Net architecture for image segmentation. The study benchmarks rotation-equivariant U-Net models against the vanilla U-Net architecture across various segmentation datasets, considering both rotation-sensitive and general-purpose datasets. Results indicate that rotation-equivariance enhances performance in tasks involving arbitrary object orientations. The paper provides insights into when rotation-equivariance is beneficial.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper would significantly benefit from restructuring and rewriting. Given the limited novelty and small number of experiments, the current length of over 18 pages is unjustified. A more precise structure and concise presentation, ideally condensing the content to approximately 6-8 pages plus supplementary material, would be appropriate. Furthermore, providing deeper analysis—such as detailed insights into why certain symmetry groups perform better on specific datasets, and illustrating this with additional experiments—would substantially strengthen the work. This feedback is intended to help improve the paper and should not discourage the authors. The study has potential and could become a valuable contribution with these adjustments. However, in its current form, the length is disproportionate to the content provided.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iTnDWSTeHZ", "reviewer": "Reviewer_ts19", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Benchmarking equivariant CNNs on segmentation is indeed, as the authors note, underexplored, and is an appreciated contribution.\n\nThere are a few claims that, I feel, the authors should revise:\n\n1) There are some earlier works on equivariant CNNs for segmentation that have not been acknowledged by the authors, such as Weiler et al. 2018, as well as:\n\nMarcos, D., Volpi, M., Komodakis, N. and Tuia, D., 2017. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision.\n\nLinmans, J., Winkens, J., Veeling, B.S., Cohen, T.S. and Welling, M., 2018. Sample efficient semantic segmentation using rotation equivariant convolutional networks. arXiv preprint arXiv:1807.00583.\n\nWinkens, J., Linmans, J., Veeling, B.S., Cohen, T.S. and Welling, M., 2018. Improved semantic segmentation for histopathology using rotation equivariant convolutional networks. Medical Imaging with Deep Learning.\n\n2) The authors limit themselves to three discrete symmetry groups in their experiments, leaving out continuous groups. They claim this is because continuous groups result in more expensive models. Could the authors expand on this? I understand methods line Harmonic Networks and Steerable Filters tend to require larger convolutional kernels, but this could be compensated by needing less filters.\nIn addition, the results do show that accounting for eight rotations, rather than four, results in better accuracy. Wouldn’t it be possible then that bringing this to the extreme (continuous rotations) may lead to even better results? Weiler and Cesa 2019, for instance, reported better results using 16 rotations and Weiler et al. 2018 report improved performance with steerable filters. Did the authors attempt at benchmarking some of these methods? If not, what was the rationale for their choice?\n\n3) The authors claim that they explore the impact of equivariance in “more complex architectures”, although I would say that U-Net is not really more complex than a regular feed forward model (the only major difference are the feature map upscaling ops) and using equivariant convolutions within a U-Net is mostly plug-and-play. I believe there is a lack of reflection on  the relevance of this study in a time when, even in semantic segmentation tasks, non-CNN models (like transformers- and Mamba-based) are taking over CNN-based models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compare, on five segmentation datasets with different characteristics, the performance of adding three variants (four rotations, four rotations + reflection, eight rotations) of equivariant behaviour to a U-Net segmentation model. Each of these experiments in performed in four different settings, combining small/large dataset size with small/large model size. Their results suggest that, in most cases, it is better to use a vanilla U-Net, with only one dataset benefiting from the equivariant behaviour (particularly to eight rotations).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following the mentioned weaknesses, I’d see it positively if authors:\n1) Expanded the related works on equivariant CNNs for segmentation.\n2) Ideally, added at least some experiments moving towards equivariance to continuous rotations (like steerable filters of 16 discrete rotations). A clear rationale of why not doing this would otherwise be important.\n3) Added a reflection on the on the relevance of this study in a time when, even in semantic segmentation tasks, non-CNN models (like transformers- and Mamba-based) are taking over CNN-based models.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z7ewYiFqQQ", "reviewer": "Reviewer_uWjN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a comprehensive benchmark evaluating rotation-equivariant U-Net variants across multiple segmentation datasets, offering broad insights into their effectiveness across medical, remote sensing, and general segmentation tasks with varying rotation invariance requirements.\n\n2. By comparing performance in both large and small data settings, the paper highlights how rotation-equivariant models behave in low-data scenarios, providing practical guidance for segmentation tasks with limited annotated data.\n\nWeaknesses:\n\n1. The authors aim to build a benchmark for image segmentation tasks. However, in the background section, the focus is primarily on U-Net and equivariance in CNNs, with no introduction or context provided for image segmentation itself. Adding a brief overview of image segmentation as a task, its importance, and common challenges would help better position the benchmark within the broader research landscape.\n\n2. In the first paragraph of related work, the author firstly shows several related works about classification tasks. However, there is no logic that can show there is a limited number of works in segmentation tasks. The author should show more related work in segmentation tasks.\n\n3. In section 4, the author states “This is motivated by the idea that equivariant models are constrained by a prior knowledge, which may reduce their required number of parameters to obtain a model with similar performance” in the first paragraph, however, this statement lacks a citation to support it. To improve credibility, the authors should provide references to prior theoretical or empirical works that support this claim about equivariance reducing parameter requirements.\n\n4. For section 4.1, the author states “Consequently, these five datasets cover a large spectrum of segmentation tasks: smaller/larger datasets, binary/semantic segmentation, and rotation invariant/standard applications”. However, before the sentence, the author only discusses the number of classes of different datasets and has no information about  invariant/standard applications. \n\n5. In paragraph of Kvasir-SEG in section 5.3, the author states “In small data setting, the small models show better Dice Score, IoU, precision, and accuracy for C8 while C4 has the highest recall”. However, in table 4, it shows that in small data settings, large models have better performance than small models, which is in contrast to what the author writes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive benchmark study evaluating the effectiveness of rotation-equivariant U-Net architectures for image segmentation across a diverse set of datasets, including medical imaging, remote sensing, and natural scene segmentation. The authors systematically compare vanilla U-Net with several rotation-equivariant variants (e.g., group-equivariant U-Nets), assessing their performance under both large-data and small-data settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes\n\n1. The author should add background and related works for image segmentation in section 2 and 3.\n\n2. In section 4, the author should provide references to support the statement as I show in weakness 3.\n\n3. The authors should ensure that their statements are consistent with the results presented in the tables to avoid confusion and improve the clarity of their analysis. See details in weakness 5.\n\n1 and 3 are critical to securing my recommendation for acceptance. 2 would simply strengthen the work in my view.", "broader_impact_concerns": "1. The selected datasets, while diverse, may underrepresent segmentation challenges in low-resource regions, introducing bias that could steer future research away from culturally or geographically underrepresented segmentation tasks requiring different invariance properties.\n\n2. The benchmark’s focus on controlled datasets may overlook deployment challenges, such as sensor noise, varying image quality, or domain shifts, limiting its applicability to real-world segmentation tasks in unpredictable environments.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the impact of rotation equivariance in training U-Net–like architectures for image segmentation. It is an experimental study that investigates the potential of group convolutional networks (Group-Convnets), which replicate filters across a set of discrete orientations and pool their outputs accordingly. Experiments are conducted on five datasets (Kvasir-SEG, NucleiSeg, URDE, COCO-Stuff, and iSAID), covering both orientation-sensitive and orientation-insensitive data, as well as binary and semantic segmentation tasks. Each experiment is performed under four different settings, varying in data regime (full vs. low) and model size (small vs. large).

The paper initially received mainly positive reviews, with reviewers acknowledging the value of the experiments. However, they also pointed out several limitations, including the restriction to U-Net architectures, presentation issues, limited novelty, and a lack of clear positioning with respect to the related literature. The authors responded to the reviewers’ comments and revised the paper accordingly. After the discussion period, Reviewer ts19 recommended acceptance; Reviewer zi1b recommended rejection — primarily because their suggestion to significantly reduce the length of the main paper had not been implemented; and Reviewer uWjN also recommended rejection, citing a lack of novelty — a limitation they had not mentioned in their initial review.

The AE has thoroughly reviewed the submission and the discussion. The AE acknowledges that the paper addresses an important problem and that the experiments provide valuable insights for the community. The AE also considers that the choice of Group-Convnets is well justified in the updated version of the paper, and that the positioning of the approach is convincing. Moreover, focusing on U-Net architectures is coherent, since most equivariant models have been introduced for convolutional networks. Finally, although the paper is long, the AE finds the reading flow pleasant. Therefore, the AE recommends accepting the paper. As a final suggestion, the AE encourages the authors to include the main findings earlier in the introduction — allowing readers to grasp the key messages and engage more smoothly with the experimental details.
Decision: Accept (poster)