$\epsilon$-Seg: Sparsely Supervised Semantic Segmentation of  Microscopy Data
Authors: Sheida RahnamaiKordasiabi
Damian Dalle Nogare
Florian Jug
Venue: NeurIPS 2025 poster ()
Keywords: semantic segmentation
microscopy
representation learning
variational networks
Primary area: machine_learning_for_sciences

Abstract: Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences.
EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming.
We introduce $\epsilon$-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering-free label prediction.
Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse ($0.05$\% of the total image data or less).
For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster w.r.t. the semantic classes we wish to distinguish. 
Finally, instead of clustering latent embeddings for semantic segmentation, we propose a MLP semantic segmentation head to directly predict class labels from latent embeddings.
We show empirical results of $\epsilon$-Seg and baseline methods on $2$ dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. 
Our results show that $\epsilon$-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.
Code available at https://github.com/juglab/eps-Seg.

E-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data
Sheida Rahnamai Kordasiabi Damian Dalle Nogare', Florian Jug'
'Human Technopole, Milan, Italy Technical University of Dresden, Germany
Abstract
Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge the life sciences. EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming. We introduce c-Seg, a method based on hierarchical variational autoencoders (H (HV) employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering- label prediction. Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels sparse (0.05% of the total image data or less). For optimal performance, we employ CL and GMM prior to shape the latent space the HVAE such that encoded input patches tend to cluster w.r.t. the semantic classes we wish to distinguish. Finally, instead of clustering latent embeddings for semantic segmentation, propose MLP semantic segmentation head to directly predict class labels from latent embeddings. We show empirical results of c-Seg and baseline methods on 2 dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. Our results show that e-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available. Code available
Introduction
Electron Microscopy (EM) comes in multiple flavors and is without doubt the tool of choice for high-resolution investigations of biological samples [12]. Today, microscopists can capture fine cellular structures nanometer resolution [22 3 Although this opens unprecedented possibilities
for studying the very fabric life, it also means that such microscopes are producing an unfathomable amount of raw image data that then available to be analyzed [36].
key module nearly every analysis pipeline is the segmentation step, where specific structures of interest must be found in the entire body of captured image data. Performing this step manually, is
typically not feasible as it takes an impossibly long time [16] 36 22]. Unfortunately, even semantic
segmentation of EM data of biological samples remains a challenge 3 31..
Ideally, methods for segmenting EM data should (i) lead sufficiently good segmentation results for the downstream analysis tasks at hand with as few training labels as possible, (ti) generalize well to different imaging conditions and image tissue types and/or be able to fine-tune on moderate amounts of new training data (9) (iáx) be able to benefit from sparse labeled data via supervised contrastive learning approaches, and possible (iv) operate on a hierarchy spatial scales to distinguish objects not only by either detailed textures larger scale shapes, but both.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

With this in mind, we introduce e-Seg, a novel and sparsely supervised semantic segmentation framework for EM images that reduces the 'hunger' for labeled data by using a powerful hierarchical VAE (HVAE) 28 21 with GMM prior instead of regular Gaussian one. Furthermore, our method uses center-region inpainting and contrastive learning to enhance feature consistency and segmentation robustness, even when training data is scarce. Hence, e-Seg learns structured latent space representations with effective feature separation for the semantic classes of interest. Once such features are learned, they can be clustered to obtain meaningful semantic segmentations. However, since this process is computationally intensive, we integrate a dedicated semantic segmentation head that directly produces segmentation labels, thereby improving both accuracy and runtime.
2 Related Work
Sparse Supervision. Deep learning has transformed microscopy image segmentation. The U-Net 26 has long been a standard architecture, achieving strong results when trained in a fully supervised setting. However, such approaches rely dense annotations, which are costly and time-consuming to obtain. At the other extreme, self-supervised methods such as MAESTER (34] learn directly from raw data without labels, offering excellent scalability but typically the cost of reduced segmentation accuracy compared to fully supervised approaches. Between these extremes lies growing body of work on sparse or weak supervision, which seeks to achieve label efficiency while maintaining good performance. We aim to surpass self-supervised methods in accuracy while requiring only fraction of the annotations needed by fully supervised methods. Comprehensive reviews on segmentation methods in large-scale EM with deep learning are available [3, with representative examples including slicewwise pseudo-label propagation for neuronal membranes (4S) 30 or
domain adaptation variants of U-Net designed for -anitationno settings [4.
Hierarchical Variational Autoencoders. Hierarchical architectures, like HVAES 28
appear to be an interesting choice for segmenting biological microscopy data. Based on variational autoencoders [20), these powerful models learn full approximate posterior. but are limited by the typically used Gaussian prior, making us wonder if a Gaussian mixture would not be more suitable choice for the semantic segmentation task hand. While the above-mentiones methods pursue label efficiency through different strategies, they do not explicitly enforce semantically disentangled latent representations. In contrast, we explicitly enforce semantically disentangled latent representations by combining GMM prior with contrastive learning. ensuring that each latent component aligns with a distinct object class. This motivates our focus on HVAEs, which progressively encode features from fine to coarse across network layers. As higher-level semantic structure emerges in deeper layers, the latent space can be disentangled and aligned with semantic classes, enabling efficient segmentation and downstream biological analysis.
Gaussian Mixture Models (GMMs). GMMs have been extensively used to model multimodal distributions and are a key component for many clustering methods 88 27 5 J0] Many approaches
integrate GMMs within autoencoder- architectures, either explicitly as a clustering module (5) or by enforcing multimodal latent structure through GMM prior [8] [0]. In VAEs, GMM priors enable structured latent spaces where each mixture component represents a distinct cluster or class [10 8). Some methods employ direct optimization of GMM objectives alongside autoencoders [5). while others leverage categorical latent variables within GMVAE frameworks, using discrete reparameter- ization techniques such the Gunbel-Softmax [99 relaxation to improve scalability 88 These techniques effectively combine deep generative models with Gaussian mixture priors, enhancing unsupervised representation learning and clustering performance in high-dimensional data spaces.
Contrastive Learning (CL). CL has gained attention for its ability to refine feature representations by maximizing similarities between related samples and minimizing them between unrelated ones. Methods like SimCLR 66 and MoCo [5] demonstrated their effectiveness in many applications. In the context of EM segmentation, CL enables better alignment of latent representations with subcellular structures. We will use CL to ensure that each GMM component corresponds to a distinct semantic class, not just in the highest level of the hierarchy we learn.
Next, we present our proposed method, which integrates hierarchical variational autoencoders with GMM- priors and contrastive learning to achieve accurate and label-efficient EM segmentation.

---

batch
GT
CCL
CE
F.SMM
#L(2)
9
h
HLPP
2 B
(Ah)
LL
P
θ
Nocleca
Gcenoles
Mitochondrria
peice
X
Figure The overall pipeline e-Seg which trained on an inpainting task (of center-region masked inputs). & and 0 are encoder and decoder of the network, respectively. Dotted arrows show sampling from distribution (Gumhel-Softuaa (Categorical--ide distribution) for segmentation head and Normal distribution for conditional posterior). h is an intermediate feature embedding of input r coming from the encoder é. f(h) a logit vector and [f(b]] = with being the number of different classes/GMM prior components (equal to 4 for "BetaSeg" (22)). 3 and 7 are feature-wise linear modulation (FiLM's (23) parameters (shifting and scaling factors) of features h are the posterior distribution's 's parameters and are divided into chunks shown as μz (z) and a (x) by с being the corresponding label of the masked center region of each input patch a in the batch. #1 is sample from A (pz (x), of (x)). y is differentiable sample from Gumbel-Softma distribution. Green arrow shows positive pair of patches having similar labels, and red arrows show negative pairs of patches having dissimilar labels. LCL is then computed on μas (further explanation can be found section
[19]
3 For L inpainting loss, LOl contrastive loss, LCe cross-entropy loss
and LK refer Equations
16 14 and 15 respectively.
3 Methods
The method we propose is based on Hierarchical VAE (HVAE) backbone similar to the ones We modify the standard HVAE setup by (i) using Gaussian mixture model
described in 28 24 (GMM) instead of the default Gaussian, every semantic class we want to distinguish has its own predetermined Gaussian region, and by (ii) adding a contrastive loss (CL), we further ensure that latent encodings are grouped by their semantic similarity through all hierarchy levels.
As the basis for our work, we used the openly available HVAE backbone of Hierarchical DivNoising (HDN) 24] HVAEs, as introduced elsewhere (28 32 24 consist of a bottom-up path (encoder) and top-down path (decoder) with trainable parameters and 0, respectively. The encoder extracts features from a given input æ at progressively coarser scales, creating a hierarchical latent encoding that splits into sub-spaces 2;, 1. L. with L being the number of hierarchy levels, or latent layers, in the HVAE. The decoder network in regular HVAES reconstructs starting from the topmost latent variables 2T., Here, we first switch from reconstructing æ to inpainting a masked central region in as described next.
Autoencoding vs. Inpainting. In contrast to regular VAEs and HVAES that use a reconstruction loss on full input patches æ, we are using masked autoencoding instead [18]. Since our aim is to learn semantic features that can be used for pixel-level semantic segmentation, the zero-masking we employed asks the network to only reconstruct the masked region, effectively learning features that best represent the masked semantic class. We conducted experiments with masked regions of various sizes and have always ensured that all masked pixels were from the same semantic class, see Table g

---

The model is trained to reconstruct the masked center pixel(s) using an MSE-based inpainting loss on X, a training batch of inputs, of size B, as
4 B
Σ ø
(x mask
& maak) 2
where a mask is the inpainted masked region the decoder predicted, and a mask is the mask region of the respective input patch prior zero-masking.
HVA with Gaussian Priors.
The Gaussian prior of regular VAEs only applies to the topmost
hierarchy level in HVAEs, where it remains (0,0 7) depicted in Figure S1
The latent variables HVAE are split into L layers 2;, E [I,.... LJ so that
L-1
p=( = p=( )
II
po (z z(41),
(2)
1=1
po(z - (zr, I),
(3)
po 2 = (2: o (2++1)) and
(4)
po = / (2 (2.0
(z)))
(5)
where no(z)) and (z)) represent the and the variance of the latent encoding, parameterized by For each layer the approximate posterior 46 (2//2, = computed by the encoder, is defined as (=,|2, zcx) Ni (zr, =<x));
where (2, zz3) and (2, =eح) are functions parameterized by d, and are the mean and variance conditioned on the input and the latent variables from lower layers j denoted by Zei-
The KL divergence term for each layer in the Evidence Lower Bound (ELBO) is
E
[LL (40 z<x) po(2
where zyi are all 2j for
HVAES with a GMM Prior. When replacing the topmost prior Po (2t.) ) in an HVAE with a Gaussian mixture model (GMM), the prior becomes a weighted sum of Gaussians
(=(2 = Σ Te Nizz Mes o.),
(8)
where C' is the total number of Gaussian components and also the number of semantic classes we want to distinguish, Te are the mixing coefficients the GMM with Ee= Te = 1, and N(z Pe. o²) is Gaussian component with mean He and standard deviation de.
Note that there is a one-to-one correspondence between Gaussian components of the GMM and the semantic classes e-Seg is supposed to distinguish. This would ensure that the latent variable follows categorical distribution over the semantic classes; we ideally want mixture assignment (I.,...,, TC) to act as a one-hot vector, i.e. one should be 1, and the rest should be 0.
However, practice, learning a fully discrete π is challenging because the standard VAE framework with a GMM prior typically results in soft assignments [10]. To encourage hard assignments, one could (i) use Gumbel-Softma
[19] trick approximate categorical sampling while maintaining [8) (ii) introduce an entropy loss to encourage т. values to be closer to either 0 or 1. In our experiments, we used the Gumbell-Sfftaa during training, while reverting to the standard softmax at inference time. We also introduced an entropy loss term as form of self- self-supervision, which yielded moderate improvements the noffel-Softtax-- -basd results (see Supplementary Material), but did not lead to significant gains w.r.t. the best-performing softmax configuration. We therefore report the softmax-ban based results as our main findings, without the additional training phase using the entropy loss. In future work, we plan to investigate alternative self-supervision strategics to
differentiability

---

further enhance the segmentation performance, leveraging the vast amount of available unlabeled data, within the proposed framework.
The approximate posterior for the topmost latent žt, can now be expressed as
go (z(
Σ qo - (a(c=1=))) (
where (c(()))) is the approximate posterior probability of the GMM component set label / given input æ and qs (zy (z, c) is the topmost approximate posterior conditioned on æ and component c. We model (= x, c) over all possible labels itself with a Gaussian
(z æt, c) = (V(z juz (x)) (z))),
(10)
by predicting fsz (a:) and oz (x) (see boxes labeled with "posterior" in Figure 1). In practice, the parameters fzz (a:) and (x)) are computed once from the FiLMM-coditioneed encoder output and are shared across all components l. As result, the mixture in Equation 9 reduces to
4 (2r, x)) (2 (zn. or.)))
(11)
as depicted in Figure 1 In order to predict (x) and (x)), we must compute the conditional posterior.
Computing the Conditional Posterior. In this section, we describe the main backbone of our method
leading from a given input patch æ E X the computed posteriors l = =N(((x), o² (x)). Figure illustrates the overall pipeline of e-Seg.
The encoder, parametrized by d, processes æ. leading to intermediate features h in the topmost hierarchy level L. These features are then passed through an MLP classifier (rouge box in Figure I. producing vector of logits (h) with dimensionality C, coinciding with the number of classes e-Seg is tasked to distinguish.
Instead of directly using h posterior distribution parameters, done in our Vanilla HVAE baseline, we are using fh), fed through two additional MLPs, By and ga (see violet boxes in Figure D. to compute parameters, 7 and B such that 7 9- (fh)) and ,3 g)))))))-
Those MI.Ps are mapping logits (h) into feature wise scaling and shifting factors. In this way, the encoded features f are modulated via these FiLM [23] parameters and 3 into At' via computing h' = 7 C h + 3, where O denotes the Hadamard product (element-wir multiplication). The modulated feature representation h is then chunked into two parts, (x) and o o (x), and used to parameterize the conditional Gaussian posterior in Equation IT
The Latent Sematic Segmentation Head. To avoid computationally costly downstream latent space clustering to perform the semantic segmentation task (as done in Xie et al. [34) and Han et al. [14] using K-Means clustering), we are introducing segmentation head tasked to perform the semantic pixel classification tasks directly from the computed logits s(h).
To compute (c(z) of Equation we use categorical reparameterizatioo trick via Gumbel- Softmax [19]
The standard Gumbel-Sofftax formula using the class probabilities n; is
exp( ,, + 9:)/T)
v/
(12)
E exwppllggppppppp. + 9;)/T)*
where 9s Gumbel 1) are Gumbel noise samples. Instead of probabilities स,, we work with logits f(h (raw scores before softmax). The equivalent formula becomes
v exp((I(M)+ + g:)/T)
(13)
Ef=1 (x)(()(()) + 9;)/T)
The temperature parameter T in the Gumbel-Softmax distribution plays a crucial role in controlling the degree of discreteness in the sampled values. During training, is often annealed from higher value to a lower one, gradually transitioning from a smooth approximation to a discrete categorical distribution.
5

---

In e-Seg, we use a typical annealing schedule r = max (Tmii- exp(-rt)), where r = 0.999 is the de- cay rate, Tmin 0.5, and is the training step. Therefore. Gumbel enables the differentiable sampling of categorical variables, improving gradient estimation, and semi-supervised classification [19]
Next, we draw vector y', representing the class assignment (segmentation prediction) for input patch x+() in the batch X, by sampling from the Gumbel-Softmax distribution parameterized by logits f(h) with temperature T.
For input patches (+) X for which we know the class label 13. we want to ensure that 1/² E y the largest entry. We do so using the cross-entropy loss
LOE =-
Σ (f
log V1 (1)
(14)
Computing the Kullback Leibler Divergence. As it is commonly done in VAES 20 the KL- divergence term is regularizing the parameters of our encoder, é, such that the approximate posterior will be close our prior (=))). In HVAEs, KL is computed at each hierarchy level. Changing from a standard Gaussian prior at the highest hierarchy level L to using GMM prior, as described earlier in this section, requires to define strategy to compute the KL-divergence appropriately.
Hershey and Olsen 77 address the challenge of efficiently approximating the KL divergence between two GMMs, and Durrieu et al. [13 propose lower and upper bounds to estimate this divergence. While these approaches can be needed practical setups [10 8. we only need to compute the KL divergence between the posterior (= (x) (Equation [1] and the I-th GMM component, where I is either the known class label for an input patch (+)
s/() for patch a²()) for y(ily/
arg max
which we do not have a ground truth class label.
Hence, Equation 8 becomes Pa, (=,) = (< μr, of), and LKL is therefore still computed as the divergence between two normal distributions. The KL loss over all hierarchy levels therefore
CxL (KL(au(z](<) pe(2)]=a))
Di KL(ga(=/)==)) pe(=(z=11)
KL(au(z/(zL-1..) po.((z)))). (15)
Contrastive Loss. The contrastive loss consists of two terms, positive pair loss L+, which encourages proximity between samples belonging the same class, and negative pair loss L_, that penalizes proximity between samples of different classes, ensuring inter-class separation. We define boolean ma-
trices and N for positive pairs and negative pairs, respectively, as P.j = {
if 4 = ly and i + j. 0 otherwise
and Noj = [1 14 * lj.
0 otherwise, with 4 and being the labels of patches i and j. respectively. These loss
En, Paj D(p (()) and L_ E,, (D(x)) ())))) with (*) being the predicted means of the posterior distribution over all hierarchy levels for a patch i in batch X, and (f) أقام distance function. In our experiments, we used the Euclidean
terms then become -+ Ei
if m, with m distance. Note that for L_ we define the penalty function (d) (m d)² otherwise" being the so-called margin, a hyperparameter that must be set appropriately, egg. using grid-search. The full contrastive loss term is finally defined as
Ler XC+ (1-
(16)
with À being hyperparameter that balances the positive and negative pair loss with each other.
Readers might wonder why a contrastive loss useful when a GMM prior used, where for each structure to be classified (i.e. for each label) we have defined a Gaussian component in its own right. The main reason is that the GMM prior only takes effect at the uppermost hierarchy level L. At all levels i < L,, LCL is taking care of the desired label-wise segregation of latent encodings.
The Overall Loss of c-Seg. Taken all together, the overall loss of e-Seg is
C Li + agLoE + azCx1 + agLoL,
(17)

---

Learning Paradigm
Model
N
G
M
Avg DSC
Vanilla HVAE [24
0.44
0.55
0.34
0.13
0.37
Self-Supervised
Han et al." [4
0.66
MAESTER*
34
0.84
0.95
0.56
0.79
0.79
Labkit 22
0.85
0.44
0.68
0.61
0.65
Sparsely Supervised
U-Net
0.90
0.96
0.78
0.66
0.83
<-Seg (ours)
0.91
0.96
0.82
0.86
0.89
Vanilla ViT
0.91
0.98
0.77
0.87
0.88
Fully Supervised
Segmenter 29
0.91
0.99
0.86
0.90
0.92
U-Net 26
0.94
0.99
0.90
0.87
0.93
Table 1: Dice similarity coefficient per class and average across all classes on the "BetaSeg" dataset (22). Methods marked with an asterisk use K-Means clustering on latent features to conduct semantic segmentation (see Section 3). U: Unrecognized, N: Nucleus, G: Granules, M: Mitochondria.
Image
MAESTER
Labkit
nilee grancles aitechend-la
irrecognized
Labels
é-Seg (ours)
U-Net
Figure 2: Qualitative segmentation result on part of the test image stack (here we show section 627 of high_ch e4 of the "BetaSeg" dataset (22)
where a;' are hyperparameters to adjust the contribution of each loss to one another. We tuned those hyperparameters using grid search and manual tuning.
Next, we present empirical results obtained using e-Seg and comparisons to several baseline methods on two dense EM datasets and one fluorescence microscopy dataset.
Experiments and Results
Datasets. We used the "BetaSeg" [22 datase which made publicly available by the authors. This Focused lon Beam Scanning Electron Microscopy (EIB-EEM) dataset captured primary mouse pancreatic islet B cells at 16 isotropic resolution. The final dataset consists of groups of high and low glucose cells and also provides human curated binary segmentation masks for seven subcellular structures, Lee. centrioles, nucleus, plasma membrane, microtubules, golgi body, granules, and mitochondria. Consistent with (34), we also chose the 4 high glucose cells for this work. For evaluation, cells 1, 2, and from four cell volumes of high glucose were used for training, while cell 4 served as an independent test set.
Next, we used the "liver FIBSEM" dataset, which consists of samples that were fresh needle biopsies fixed with 4%PFA and 2%GA in phosphate buffer. High contrast staining was performed with reduced osmium and Waltons lead aspartate stain 33 and embedded in Epon. Sample preparation and imaging were done on a ZEISS GeminiSEM according to prior reports [35]. The final dataset consists of one cell volume with 11 crops that have been extracted from cell volume, annotated manually, and used for training, validation, and testing. The segmentation masks consist of six subcellular structures, mitochondria, peroxisomes, lipofuscin, basolateral membrane, open bile canaliculi, and closed bile canaliculi, along with an additional "background" category. caterorr...
http://bbetsee_githubb

---

Trained
Per-Class Dice Coefficient
Avg
DSC
Per-Class Dice Coeflicint
highLel
0.85
0.38
0.68
0.61
0.63
Background
Cytoplasm
Nuclei
DSC
high_c2
0.80
0.33
0.58
0.56
0.57
0.90
high c3
0.63
0.42
0.58
Table 2:
Dice similarity coefficient per class and average for "aisslbbbioimaging"" datasets.
Table 3: Labkit results. Due to different im-
age sizes, Labkit was trained individual volumes. U: Unrecognized, N: Nucleus, G: Granules, M: Mitochondria.
Model
B
M
P
L
BM
OBC
CBC
Avg DSC
U-net-Fully Supervised
0.97
0,95
0.85
0.79
0.52
0.87
0.90
0.84
U-net- U-nne-Sparsely Supervised
0.94
0.81
0.68
0.81
0.49
0.39
0.00
0.59
--Seg-Sparsely Supervised
0.91
0.82
0.63
0.81
0.39
0.70
0,46
0.67
Table 4: Dice similarity coefficient per class and average across all classes comparing our model with baselines for "liver FIBSEM" dataset. B: Background, M: Mitochondria, P. Peroxisomes, L: Lipofuscin, BM: Basolateral Membrane, OBC: Open Bile Canaliculus, CBC: Closed Bile Canaliculus.
RLF
Per-Class Dice Coefficient
Avg
G
M
DSC
20
0.89
0.98
0.81
0.83
0.88
0.88
0.98
0.81
0.78
0.86
Entropy Loss
Per-Cluss Dice Coefficient
Avg
0.86
0.98
0.80
0.75
0.85
G
M
DSC
0.85
0.96
0.77
0.76
0.84
0.78
0.86
0.98
0.80
0.75
0.85
Table 5: DSC per class and average across all classes. The "RLF" column (Relative La- beling Factor) specify a scaling factor where 20 corresponds to 0.05% and as small as 0.0025% of the total labeles available. U: Unrecognized, N:Nucleus, G:Granules, M:Mitochondria.
Table Effect of entropy loss: The best checkpoint of a sparsely supervised model was further trained using batches with 50%
unlabeled data. U: Unrecognized, N:Nucleus, G:Granules, M:Mitochondria.
pe presiser
caulinals
cooot bilo
(b)
Image
GT Labels
E-Seg (ours)
U-Net
Figure 3: Qualitative segmentation result on two crops of the whole 3D volume. (a) and (b) are section 80 and 26 of crop00 and crop10 in "liver FIBSEM" dataset respectively. The U-Net is sparsely-supervisee (for the fully-supervised U-Net result, see Figure S4h.

---

Cytoplasm
Nuclei
GT Labels
c-Seg
Figure 4: Qualitative results representative 2-channel image from the overlapping subset of the I 1" and AAislab-bioimagingg"" datasets. The first two panels show the fluorescence microscopy channels: EGFP SGFF--alectin---labee cytoplasm (left) and Hoechst 33342-stained nuclei (center-left). The center-right panel (GT) displays the ground truth semantic segmentation with nuclei (cyan) and eytoplasm (magenta). The rightmost panel (e-Seg) shows the prediction from our method.
While it is true that IIB-EEM datasets like "BetaSeg" [22] offer isotropic resolution suitable for 3D processing, this is not always the case in EM imaging, where data often come in 2D slices (especially in gegher-throughpur screens).
Furthermore, conducted experiment the overlapping subset of two datasets Aitslab- bioimaging1 I and miaslabbbboimagingg2 [25). The Aitslab-bioimagnng dataset is benchmarking fluorescence microscopy dataset containing 50 images of Hoechst 33342-stained U2OS osteosarcoma cell nuclei, including annotations for nuclei, nuclear fragments, and micronuclei, designed for training and evaluating neural networks for instance and semantic segmentation and the Attsaan-biommaging2 dataset is a fluorescence microscopy dataset containing 60 images of EGFP-Galectin-3 labeled U2OS osteosarcoma cells with aund-annotated cell outlines, designed for training and benchmarking neural networks for instance and semantic segmentation, with over 2200 annotated cell objects and compatibility with object detection tasks. The overlapping subset of them contains 30, 2-channel images for training and 10 for testing.
Evaluation Metrics. We used the Dice Similarity Coefficient (DSC) evaluate the segmentation performance. DSC is widely used metric in image segmentation and measures the similarity between the predicted and actual segmentation masks.
Let A and B be two sets representing the binary segmentation masks of the ground truth and the
predicted segmentation. The Dice coefficient is defined as Dice(A, B) = 2|An +1 where |A IAn B), the number of overlapping pixels between the predicted and ground truth masks, [A], the number of pixels in the ground truth mask, and [B]. the number of pixels in the predicted mask.
Experiments. We use an architecture similar the one used in the HDN work [24]. For all hyperpa- rameters we have introduced, we used grid searches to find good balance between performance and stability. We first evaluate our method the "BetaSeg" dataset (22] and compare its performance against baseline methods, as shown in Table [ They demonstrate that our approach outperforms existing baselines in terms DSC (Fl-score). For the Labkit baseline, we trained per cell and show the results in Table 3 and report the best class-wise performance in Table Quantitative segmentation
results are shown Figure 2
(complete Figure S33
To further validate the robustness of our method, we conduct experiments on the "liver FIBSEM" dataset, comparing with U-Net baselines (fully and pparsely-supervieedd Quantitative and qualita- tive results are shown Table 4 and Figure respectively (complete Figure (S4). Additionally, we show e-Seg results on fluorescent microscopy dataset (see Table and Figure
Model Ablations. We strip our model down to a vanilla HVAE and then re-introduce one component at time, showing how each of the modules we have introduced above contributes to the overall performance we report. These results on the "BetaSeg" dataset are shown in Table
Additionally, we evaluate how the quality of the results depends on the amount of available training labels. To this end, we are starting from 0.05% of the total image data available in the "BetaSeg" dataset and gradually decreasing the number of used training labels down 0.0025%. The results of

---

these experiments can be found in T As discussed in Section Cf helps us gain additional performance also from the unlabeled data, which we measure and report in Table Finally, we measured the effect of differently sized masking regions in Table
Limitations. While e-Seg achieves competitive segmentation results using only sparse supervision. several limitations remain. First, all experiments we present are conducted on 2D images. Extending the presented framework to operate full 3D is an important next step, especially for volume EM data analysis. Second, we noticed that the effectiveness of our entropy-based loss must be improved, e.g. by replacing with a more adaptive or data-driven strategy. Finally, in the presented form, hyperparameters such as the contrastive loss margin still require manual tuning, which is not ideal for ease of use by biological experts.
RL
Lois
CL
UE
Distribution
Prior
U
K
Her Cnciliiint
U
XI
DSC
Avg
Size
"urerogaieed
A
0:44
0.55
0.34
0.13
0.37
0.53
0.69
0.76
0.31
0.51
0.30
0.75
0.53
0.81
0.33
097
0.30
GMM
0.82
0.72
0.75
1.52
098
GMM
0,56
0.30
035
Table 8: Label consistency ablation on "Be- taSeg" dataset. The "Mask Size" column indicates the size of the center-region mask, within which the pixel-wise ground truth la- bels are consistent.
Table Loss components and prior distribu- tion ablation on "BetaSeg" dataset. U: Unrec- ognized, N: Nucleus, G: Granules, M: Mito- chondria.
5 Discussion
Here we presented e-Seg, novel semantic segmentation approach that leverages the variational latent representation of hierarchical variational autoencoders (HVAEs) trained on a limited amount of pixel-labels in an inpainting setup. We used GMM prior instead of the traditionally employed Gaussian prior and introduced a novel segmentation head that incorporates both a cross-entropy loss and entropy loss to leverage available data for which ground truth (GT) class-labels are available. The integration of contrastive loss, combined with the structural advantages the GMM prior, provides means effectively distinguish biological structures directly from the latent space encoding.
Transformer-based architectures, as used in MAESTER [4 usually have rather large number of trainable parameters (i.e. 328, 452, 352 trainable parameters in MAESTER). This makes such approaches less applicable to life-scientists since they require rather powerful computing setups. Even our biggest network, in contrast, only employs 3, 800, 869 trainable parameters (see Tables SS2 S3), making it fast to train and easy to use. Our experiments also highlight an interesting fact, namely that smaller mask sizes with consistent labels emerged as the best strategy. This stands in contrast to Transformer- basoo approaches, where relatively large fraction of the input images is masked during training 34
By combining hierarchical representations with advanced regularization techniques such as contrastive learning, we have shown that we can achieve competitive segmentation performance on complex microscopy data, even with relatively small models and limited training data. The proposed approach tackles the challenge of label scarcity, enhances latent space representations tailored to structured biological data, and lays the groundwork for future exploration of semi-supervised learning techniques and adaptive latent priors.
Overall, this work bridges the gap between fully supervised and unsupervised methods by offering a scalable approach for large-scale biomedical semantic image data segmentation.
References
[1] Malou Arvidsson, Salma Kazemi Rashed, and Sonja Aits. An annotated high high-content fluorescence microscopy dataset with hoechst 33342- stained nuclei and manually labelled outlines. Data Brief, 46: 108769, 2023.
10

---

[2] Matthias Arzt, Joran Deschamps, Christopher Schmied, Tobias Pietzsch, Deborah Schmidt, Pavel Toman- cak. Robert Haase, and Florian Jug. Labkit: Labeling and segmentation toolkit for big image data. Frontiers in Computer Science, 2022.
[3] Anusha Aswath, Ahmad Alsahaf, Ben N G Giepmans, and George Azzopardi. Segmentation in large-scale cellular electron microscopy with deep learning: literature survey. Medical Image Analysis, 89: 89:102920, 2023.
[4] Róger Bermúdez- Bermidez-Chacón, Chacón, Okan Altingövde, Carlos Becker, Mathicu Salzmann, and Pascal Fua. Visual correspondences for unsupervised domain adaptation on electron microscopy images. IEEE Trans. Med. Imaging, 3964):1556-1267, 2020.
[5] Ahcène Boubekki, Michael Kampffimeyer, Robert Jenssen, and Ulf Brefeld. Joint optimization of an autoencoder for clustering and embedding. Machine Learning, 1101))1901111933 2021.
[6] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 1597-1607. PMLR, 2020.
[7] Rewon Child. Very deep VAEs generalize autoregressive modcls and can outperform them on images. In International Conference on Learning Representations (ICLR), 2021.
[8] Mark Collier and Hector Urdiales. Scalable deep unsupervised clustering with concrete GMVAEs. In Ist Workshop on Deep Continuous Dontinuous-Discrete Machine Learning, ECML, 2019.
[9] Ryan Conrad and Kedar Narayan. CEMS00K, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning. eLife, 10:e65894, 2021.
[10] Nat Dilokthanakul, Pedro A. M. Mediano, Marta Gareelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumarnn, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXtv preprint ar Xiv: /6v2666 2016. Under review at ICLR 2017.
[11] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at seale. In International Conference on Learning Representations (ICLR), 2021.
[12] Damjana Drobne. 3D imaging cells and tissues by focused ion beam/scanning electron microscopy (FIB/SEM), Methods in Molecular Biology, 950:275-292, 2013.
[13] Jean-L Durrieu, Jean-Philippe Thiran, and Finnian Kelly. Lower and upper bounds for approximation the kullback-leibler divergence between gaussian mixture models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4833 4836, 2012.
[14] Hongqing Han, Mariia Dmitrieva, Alexander Sauer, Ka Chun Tam, and Jens Rittscher. Self cuff-supervised vexel-level representation rediscovers subcellular structures in volume electron microscopy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2276-2285, 2022.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020.
[16] Larissa Heinrich, Davis Bennett, David Ackermana, Waohyun Park, John Bogovic, Nils Eckstein, Alyson Petruncio, Jody Clements, Song Pang, C. Shan Xu, Jan Funke, Wyatt Korff, Harald F. Hess, Jennifer Schwartt,,Schwarz,,, Stephan Saalfeld. Aubrey V. Weigel, and COSEM Project Team. Whole-cell organelle segmentation in volume electron microscopy. Nature, 599(7883):141-B5 2021.
[17] John R. Hershey and Peder A. Olsen. Approximating the kullback-leibler divergence between gaussian mixture models. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages IV 317-111717 -V-30, 2007.
[18] Zhicheng Huang, Xiaojic Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng, Contrastive masked autoencoders are stronger vision learners. IEEE Trans. Pattern Anal. Mach. Intell., 2023.
[19] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
11

---

[20] Diederik P. Kingma, Danilo J. Rezende. Shakir Mohamed, and Max Welling, Semi-supervised learning with deep generative models. In Conference on Neural Information Processing Systems (NeurIPS), 2014.
[21] Lars Maaloe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA; very deep hierarchy of latent variables for generative modeling. In Advances in Neural Information Processing Systems (NeurlPS). Curran Associates, Inc., 2019.
pages 1544865597
[22] Andreas Müller, Deborah Schmidt, C Shan Xu, Song Pang. Joyson Verner D' DoCssa, Susanne Kretschmar, Carla Münster, Thomas Kurth, Florian Jug, Martin Weigert, Harald F Hess, and Michele Solimena. 3D FIB-SEM reconstruction of microtubule orgganeellllee interaction in whole primary mouse 3 cells. Journal of Cell Biology, 220(2):e000110039 2021.
[23] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual rea- soning with general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 3942-3951 2018.
[24] Mangal Prakash, Mauricio Delbracio, Peyman Milanfar, and Florian Jug. Interpretable unsupervised diversity denoising and artefact removal. In International Conference on Learning Representations (ICLR), 2022.
[25] Salma Kazemi Rashed, Malou Arvidsson, Rafsan Ahmed, and Sonja Aits. An annotated high high-content fluorescence microscopy dataset with GGFP-galectin- saleetin-3-stainee 3-stained cells and manually labelled outlines. Data Brief, 58:1 148, 2025.
[26] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234-241. Springer, 2015.
[27] Marek Smieja, Maciej Wolczyk, Jacek Tabor, and Bernhard C. Geiger. SeGMA: Semi superviuee gaussian mixture autoencoder. IEEE Trans. Neural Netw. Learn. Syst., 32(9):3930- 394439999-39 2021.
[28] Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Spren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems (NeurIPS), pages 3745 3753, Curran Associates, Inc., 2016.
[29] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7272-7272, 2021.
[30] Eichi Takaya, Yusuke Takeichi, Mamiko Ozaki, and Satoshi Kurihara. Sequential supe-supe segmentation for serial electron microscopy image with small number of labels. J. Neurosci. Methods, 351: 109066, 2021.
[31] Kevin P Treder, Chen Huang. Judy S Kim, and Angus Kirkland. Applications of deep learning in electron microscopy. Microscopy (Oxford), 7 710Supplement 11:1000 i115, 2022.
[32] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NewrIPS), pages 19667-19679, 2020.
[33] J. Walton. Lead aspartate, an en bloc contrast stain particularly useful for ultrastructural enzymology. J. Histochem. Cytochem., 27(10): 1337 1342, 1979.
[34] Ronald Xie, Kuan Pang, Gary D. Bader, and Bo Wang. MAESTER: Masked autoencoder guided segmenta- tion at pixel resolution for accurate, self-supervised subcellular structure recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 17521-17531 2023.
[35] C. Shan Xu, Kenneth J. Hayworth, Zhiyuan Lu, Peter Grob, Ahmed M. Hassan, José G. Garefa- Cerdán, Krishna K. Niyogi, Eva Nogales, Richard J. Weinberg, and Harald F. Hess. Enhanced FIB-SEM systems for large-volume 3D imaging, eliife, 6:c25916, 2017.
[36] C. Shan Xu, Song Pang, Gleb Shtengel, Andreas Müller, Alex T. Ritter, Huxley K. Hoffman, Shin ya Takemura, Zhiyuan Lu, H. Amalia Pasolli, Nirmala Iyer, Jeeyun Chung, Davis Bennett, Aubrey Weigel, Melanie Freeman, Schuyler B. van Engelenburg. Tobias C. Walther, Robert V. Farese Jr., Jennifer Sppincott-Schwart Ira Mellman, Michele Solimena, and Harald F. Hess. An open-access volume electron microscopy atlas of whole cells and tissues. Nature. (788)))177771111 2021. Erratum in: Nature, vol. 599, 7885, p. ES, 2021, 10:::0..133//4115555..1. 1033//411860001-41111111
12

---

E-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data
Supplementary Material
batch
1
(L
(())
FKL
Snescognized
IA toteboodria
peior
X
Figure $1: The overall pipeline of Vanilla HVAE in Table (first row in Table 7 which is trained on an inpainting task (of the center-region masked inputs). e and are encoder and decoder of the network, respectively. Dotted arrows show sampling from distribution. is intermediate feature embedding of input coming from the encoder & and it is posterior distribution's 's parameters which divided into chunks shown f.t, and oz. zr, is sample from (x), of (x)). For L inpainting loss and LK refer to Equations) and respectively.
batch
L1
COL
posterior
(L(x)
((x)
h
CKL
Unrecognized
Musteas
Grazules
tochoodria
2rior
X
X
Figure S2: The overall pipeline of Vanilla HVAE with only CL added in the pipeline in the second row in Table 7 which is trained arrows are showing positive and negative pair respectively, a batch. e and are encoder and decoder of the network. respectively. Dotted lines show sampling from distribution. h is intermediate feature embedding of input æ coming from the encoder @ and is posterior distribution's 's parameters which is divided into two chunks shown μa and o 2 is sample from A M(u (x:), af (x))) For C, inpainting loss, Lor, contrastive loss and L refer to Equations]
inpainting task (of the center-region masked inputs). Green and red
T6 and resppectivll.
13

---

Model
Learning Paradigm
M
Avg DSC
Vanilla HVAE" [24
Sell-Supervieed
0.44
0.55
0.34
0.13
0.37
Labkit [2
Sparsely Supervised
0.85
0.44
0.68
0.61
0.65
U-net 226
Fully Supervised
0.94
0.99
0.90
0.87
0.93
U-net
Sparsely Supervised
0.90
0.96
0.78
0.66
0.83
Vanilla ViT II
Fully Supervised
0.91
0.98
0.77
0.87
0.88
Segmenter 22
Fully Supervised
0.91
0.99
0.86
0.90
0.92
MAESTER* 134
Seff-Supervied
0.84
0.95
0.56
0.79
0.79
Han et al" [14
Self-Supervised
0.66
E-Seg (+L#)
Sparsely Supervised
0.89
0.98
0.81
0.83
0.88
Table S1:
Dice similarity coefficient per class and average across classes comparing our model with baselines on the "BetaSeg" dataset [22]. Methods marked with asterisk use K-Means clustering on latent features to conduct semantic segmentation (more explanation can be found in Section 3). U: Unrecognized, N:Nucleus, G:Granules, M:Mitochondria.
# res. blocks
Per-Class Dice Coefficient
Avg
U
N
G
M
DSC
5
0.86
0.98
0.80
0.75
0.85
4
0.85
0.97
0.80
0.74
0.84
3
0.88
0.96
0.81
0.80
0.86
2
0.87
0.97
0.81
0.77
0.86
0.85
0.97
0.80
0.72
0.84
Table S2: Residual blocks ablation (3 latent variables). U: Unrecognized, N: Nucleus, G: Granules, M: Mitochondria.
Entropy-based Loss. When the sample y' of the Gumbel-Softmax distribution is uniform, the network maximally unsure about which class to predict for the current input patch. We noticed that this is commonly the case early during training, where the network has not seen lot of patches for which ground truth labels are available.
To encourage the network not to predict uniform y', we introduced an entropy loss for all patches (()) E for which we do not have a ground truth class label.
Σ xUE
y (oggyy
(18)
# latent
Per-Class Dice Coefficient
Avg
N
G
M
DSC
2
0.87
0.98
0.81
0.76
0.86
3
0.86
0.98
0.80
0.75
0.85
Table S3: Latent variables ablation res. blocks/layer). Unrecognized, N:Nucleus, G:rrnlless M-Mitochondria.
14

---

Image
MAESTER
GT Labels
e-Seg (+Lr)
Labkit
U-Net
Vanilla HVAE
Han et al.
nucleus
granules tochondria unrecognized
Figure S3: Qualitative segmentation result on part of the test image stack (section 627 of hiigh_ch in "BetaSeg" dataset).
15

---

Image mi tochondria peroxisomes
GT Labels
e-Seg (+C#)
lipofuscin
open bile
canaliculus
closed bile
canaliculus
basolateral
membrane
background
U-Net (sparse)
U-Net (full)
(b)
Image
Labels
e-Seg (+CH)
mi tochonndiaa
peroxisomes
lipafuscin
open bile
canaliculus
closed bile
canaliculus
basolateral
membrane
background
U-Net (sparse)
U-Net (full)
Figure S4: Qualitative segmentation result on two crops of the whole 3D volume. (a) and (b) are section 80 and 26 of crop00 and crop10 in "liver FIBSEM" dataset respectively. U-Net (sparse) and (full) is sparsely-supervise and fully-supervised respectively.
16

---

RLF
Model
U
N
G
M
Avg DSC
20
U-net
0.63
0.75
0.51
0.12
0,50
6-Seg
0.89
0.98
0.81
0.83
0.88
U-net 0.53
0.64
0.41
0.14
0.43
6-Seg
0.88
0.98
0.81
0.78
0.86
10
U-net
0.30
0.20
0,42
0.34
0.31
e-Seg
0.86
0.98
0.80
0.75
0.85
U-net
0.71
0.00
0.00
0.03
0.18
6-Seg 0.85
0.96
0.77
0.76
0.84
U-net 0.17
0.00
0.37
0.02
0.14
6-Seg
0.79
0.95
0.69
0.69
0.78
Table S4: Comparison between U-Net and e-Seg on the "BetaSeg" dataset under varying label sparsity levels. "RLF" (Relative Labeling Factor) specifies the fraction of available labels, where 20 corresponds to 0.05% and to 0.0025% of total labels. U: Unrecognized, N: Nucleus, G: Granules, M: Mitochondria. Although both models were trained with balanced supervision, using patches selected to include all classes, the U-Net still fails to segment the nucleus at very low labeling levels (RLF and 5). This illustrates key limitation of discriminative models such U-Net, under extreme supervision sparsity, even balanced examples may not suffice to generalize fine-grained or context-sensitive structures like the nucleus. In contrast, e-Seg benefits from its class-aware latent modeling via the GMM prior, which enables it to extract meaningful representations for different structures and distinguish them semantically. We note that the sparse U-Net reported earlier was trained on slice numbers 800, 600, and 500 of the "high_e cl", "high_co c2", and "high _ath__"3""_____ volumes of the "BetaSeg" dataset. For selecting the same amount of data used in e-Seg, train the 2D U-Net as reported in the table above, we extracted 64x64 patches where except background, different classes are approximately well balanced.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions, including the design of e-Seg, an HVAE-based segmentation framework with GMM prior, center- region inpainting, contrastive learning, and dedicated semantic segmentation head. These claims are appropriately scoped and supported by the methodology and experiments pre- sented in the rest of the paper. The text also specifies that the method works with extremely limited supervision and addresses common practical challenges EM segmentation, which are demonstrated through empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We included a dedicated Limitations mini-headline at the end of the Exper- iments and Result (Section 4. There, we discuss that the current method is restricted to 2D data and would likely benefit from a 3D extension. We also note that the entropy- based loss could be further optimized, and that dataset-specific tuning is required for some hyperparameters, such as the contrastive loss margin.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
18

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include formal theoretical results or proofs (e.g.. theorems
lemmas). However, it provides detailed derivations and explanations of the model components and loss functions (see Section 3 including the use of a GMM prior in the HVAE framework and KL divergence formulation.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper provides all necessary implementation details, including model architecture (Figure training settings, dataset descriptions and evaluation metrics (Sec- tion 4). Loss terms, and component configurations are also disclosed to allow faithful reproduction of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
19

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Two of the datasets used in our experiments are publicly available and ref- erenced in the paper. The third dataset private and cannot be shared due data access restrictions. We will publicly release the code GitHub along with detailed instructions to reproduce all experiments based on the public datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
cablic/guides Btaii///guddes/ooeeuummi...e.e
(oodeSuuiisiiiooooooiioooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https http://ooooSuuisiioiooooo
/ cttpps..ccpubbicec http:///ggiiiee//eeuuubbbiibaai
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all relevant training and evaluation details, including data splits, optimizer type, learning rate, batch size, and other key hyperparameters. Where appropriate, we explain how hyperparameters were chosen, either based on prior work or grid search.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
20

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While our main experiment (Table 1 includes 5-fold cross-validation to mitigate variability due data splits, did not report error bars perform statistical significance tests. Given the limited size of our dataset and the exploratory nature of our work, our focus was on assessing the feasibility of the proposed method rather than establishing statistically significant performance differences.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this work, mentioned the number of parameters in largest model and the efficiency of our approach. Our method improves upon previous techniques by eliminating the need for K-Means clustering, allowing the model directly generate segmentation labels from the segmentation head. This change significantly accelerates the inference process, resulting faster segmentation without sacrificing accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
21

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::://marrpg...cc/oaal////aca
Answer: [Yes]
Justification: Yes, the research conducted in this paper conforms with the NeurIPS Code of Ethics. We have adhered to all relevant ethical guidelines, ensuring transparency, fairness, and respect for privacy in work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The focus this paper primarily on technical advancements in segmentation, and while it does not explicitly address societal impacts, the method may have positive implications in fields like medical imaging. However, the societal implications are, if at all, only indirect and we believe the answer 'NA' is most appropriate.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper does not involve models or data with a high risk for misuse, and thus does not describe any specific safeguards related their release.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Yes, all creators and original owners of assets used in this paper, including datasets, code, and models, have been properly credited. Additionally, the licenses and terms of use associated with these assets have been explicitly mentioned and respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawerssi.ppwwiiie........ thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Yes, the private dataset used in this paper well documented, including details on its structure, size, and usage. However, due to privacy and confidentiality constraints, the dataset is not publicly available. Access the dataset is restricted, but interested parties can contact the authors to be connected to the dataset owners.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve research with human subjects, and therefore, no IRB or equivalent approvals were required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: No large language models (LLMs) were used as part of the core methods in this research.
Guidelines:
The answer NA involve LLMs as any important, original, or non-standard components.
that the core method development in this research does
Please refer to our LLM policy hettgs//hearrpps.cc/o/eeeeeeeeee for what should or should be described.
24


Reviews:
[{"review_id": "AnwFoU66bL", "reviewer": "Reviewer_sMMT", "summary": "This paper considers the problem of semantic segmentation for EM volumes, particularly from the perspective of low amounts of available training data.  The proposed method for tackling this problem consists of a hierarchical VAE, using a Gaussian mixture model prior with one component per semantic class to be labeled.  The HVAE is applied to an inpainting task of reconstructing masked center pixels.  Additional components to the model consist of using a segmentation head to directly perform semantic classification using Gumbel-Softmax, adding an entropy-based loss for patches without a ground-truth label, and adding a contrastive loss.  The proposed method is then evaluated on three data sets, two from volume EM and one from fluorescence microscopy.", "questions": "see the weaknesses raised above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "strengths:\n\n- the paper tackles an important problem in EM segmentation, and the proposed method seems to do well on the evaluated data sets relative to the considered baselines\n\nweaknesses:\n\n- it seems like a potentially large limitation that the proposed method operates on 2d data, considering that FIB-SEM allows for isotropic 3d volumes like the studied \"BetaSeg\" data.  it would have been good to know the performance of a 3D unet model on this data\n\n- the u-net performance on sparsely supervised data is generally quite close to the proposed $\\epsilon$-Seg.  it would be informative to know its performance in the relative labeling factor table 4 to get a better sense of how it compares.\n\n- u-net also seems to generally do comparable or better on most classes, except giving poor performance on one or two.  for instance, in Table 6 on the liver FIBSEM data, the listed DSC for the CBC class is 0 - is that correct or a typo?  it seems strange given full supervision performance is high at 0.9.  How was the data sampling done with training u-net on sparse supervision?  was it still given balanced samples/weighting from all classes to avoid under-predicting low-frequency classes?\n\n- the u-net is not making use of unlabeled data, but $\\epsilon$-Seg is via the entropy loss.  it would be good to give a direct comparison of the two methods without the additional unlabeled data, since in theory the same component could also be added to u-net.\n\n- in the table 7 ablation study, it seems like the contrastive loss hurts performance when looking at row three versus row 4, when using the standard normal prior distribution.  it would be good to report performance with the same settings but with the GMM prior, to validate whether the contrastive loss is actually helping.", "was_revised": true, "final_justification": "The authors' detailed response addressed my primary concerns, so although I think the paper is still somewhat borderline, I have raised my rating to borderline accept.", "rebuttal": "We thank this reviewer for the time to familiarize themselves with our paper and are happy to address the criticism below point by point (italic text is from the submitted reviewer feedback):\n\n- *it seems like a potentially large limitation that the proposed method operates on 2d data, considering that FIB-SEM allows for isotropic 3d volumes like the studied \"BetaSeg\" data. it would have been good to know the performance of a 3D unet model on this data*\n    \n    Regarding the use of 2D vs. 3D models, while it is true that FIB-SEM datasets like BetaSeg offer isotropic resolution suitable for 3D processing, this is not always the case in EM imaging, where data often comes in 2D slices (especially in high-throughput screens).\n    Our method was designed to be applicable in such scenarios, where 3D context is not reliably available.\n    That said, we acknowledge the value of a direct comparison.\n    \n- *the u-net performance on sparsely supervised data is generally quite close to the proposed $\\epsilon$-Seg. it would be informative to know its performance in the relative labeling factor table 4 to get a better sense of how it compares.*\n    \n    For the different label sparsity ratios (Table 4, Relative Labeling Factor), we will add a corresponding 2D U-Net baseline.\n    We balanced the data and below you can see the result of sparsely supervised U-Net on BetaSeg dataset with different relative labeling factors.\n    We expect this will further highlight our method’s robustness in the low-label regime.\n    \n    | RLF | Model    | Unrecog.| Nucleus | Granule | Mito.   |Avg DSC|\n    |-----|----------|---------|---------|---------|---------|-------|\n    | 20  | U-Net    | 0.63    | 0.75    | 0.51    | 0.12    | 0.50  |\n    |     | ϵ‑Seg    | 0.89    | 0.98    | 0.81    | 0.83    | 0.88  |\n    | 15  | U-Net    | 0.53    | 0.64    | 0.41    | 0.14    | 0.43  |\n    |     | ϵ‑Seg    | 0.88    | 0.98    | 0.81    | 0.78    | 0.86  |\n    | 10  | U-Net    | 0.30    | 0.20    | 0.42    | 0.34    | 0.31  |\n    |     | ϵ‑Seg    | 0.86    | 0.98    | 0.80    | 0.75    | 0.85  |\n    | 5   | U-Net    | 0.71    | 0.00    | 0.00    | 0.03    | 0.18  |\n    |     | ϵ‑Seg    | 0.85    | 0.96    | 0.77    | 0.76    | 0.84  |\n    | 1   | U-Net    | 0.17    | 0.00    | 0.37    | 0.02    | 0.14  |\n    |     | ϵ‑Seg    | 0.79    | 0.95    | 0.69    | 0.69    | 0.78  |\n    \n    Although both models were trained with balanced supervision, using patches selected to include all classes, U-Net still fails to segment the nucleus at very low labeling levels (RLF 1 and 5).\n    This illustrates a key limitation of discriminative models like U-Net that under extreme supervision sparsity, even balanced examples may not suffice to generalize fine-grained or context-sensitive structures like the nucleus.\n    In contrast, $\\epsilon$-Seg benefits from its class-aware latent modeling via the GMM prior that allows it to extract meaningful patterns from different structures to distinguish them semantically.\n    We would like to briefly mention that the sparse U-Net reported earlier was trained on slice numbers 800, 600 and 500 of “high_c1”, ”high_c2” and ”high_c3” of Betaseg. Then, we calculate the Inverse-Frequency Weighted Dice Score (IFWDS) of the sparsely supervised U-Net.\n     $\\text{IFWDS} = \\sum_{c=1}^{C}\\frac{\\frac{1}{n_c}}{\\sum_{k=1}^{C}\\frac{1}{n_k}}\\cdot DSC_c = 0.86$.\n\n    For selecting the same amount of data used in $\\epsilon$-Seg, to train the 2D U-Net on, as reported in the table above, we extracted 64x64 patches where except background, different classes are approximately well balanced. \n- *u-net also seems to generally do comparable or better on most classes, except giving poor performance on one or two. for instance, in Table 6 on the liver FIBSEM data, the listed DSC for the CBC class is 0 - is that correct or a typo? it seems strange given full supervision performance is high at 0.9. How was the data sampling done with training u-net on sparse supervision? was it still given balanced samples/weighting from all classes to avoid under-predicting low-frequency classes?*\n    \n    Regarding the segmentation of CBC in the liver FIBSEM dataset, yes, the DSC of 0 for U-Net is correct and not a typo.\n    The U-Net struggles significantly with this rare class under sparse supervision.\n    Below you can see the dice scores of sparsely-supervised U-Net vs $\\epsilon$-Seg with same amount of balanced GT.\n    Table 6 further highlights the above-mentioned point, under the same sparse label budget, our method significantly improves the segmentation of rare classes, contributing to a higher average DSC.\n    This supports our claim that $\\epsilon$-Seg is better suited for identifying low-frequency classes, thanks to its class-aware latent structure.\n    \n- *the u-net is not making use of unlabeled data, but $\\epsilon$-Seg is via the entropy loss. it would be good to give a direct comparison of the two methods without the additional unlabeled data, since in theory the same component could also be added to u-net.*\n    \n    We also appreciate the suggestion to compare with and without unlabeled data. As the reviewer noted, the U-Net is a fully supervised method and cannot leverage unlabeled samples. In contrast, $\\epsilon$-Seg benefits from its entropy loss and generative regularization to extract structure from unlabeled regions. To isolate this effect, an additional ablation where $\\epsilon$-Seg is trained without the entropy loss, showing that the improvement stems from better uncertainty reduction, a property not easily transferred to U-Net architecture.\n    \n    | Class  | Sparse U-Net (Balanced)  | ϵ-Seg (Sparsely Supervised; with-without Entropy)|\n    |--------|--------------------------|----------------------------|\n    | B      | 0.6522                   | 0.91  -  0.87              |\n    | M      | 0.5629                   | 0.82  -  0.74              |\n    | P      | 0.2888                   | 0.63  -  0.44              |\n    | L      | 0.5772                   | 0.81  -  0.75              |\n    | BM     | 0.1075                   | 0.39  -  0.31              |\n    | OBC    | 0.0003                   | 0.70  -  0.68              |\n    | CBC    | 0.0004                   | 0.46  -  0.36              |\n    | Avg DSC| 0.3128                   | 0.67  -  0.59              |\n    \n- *in the table 7 ablation study, it seems like the contrastive loss hurts performance when looking at row three versus row 4, when using the standard normal prior distribution. it would be good to report performance with the same settings but with the GMM prior, to validate whether the contrastive loss is actually helping.*\n    \n    Regarding the contrastive loss, the reviewer makes an excellent observation.\n    The mixed performance of CL in Table 7 (rows 3–4 vs. rows 1–2) motivated our introduction of the GMM prior, which aligns more naturally with the class-aware contrastive loss.\n    We agree that the missing “GMM + no CL” condition should be reported, and we will include that experiment in the camera-ready version for completeness.\n    Below you can see the updated Table 7, where one can see that adding CL only helps when a GMM prior is used. Thanks for spotting this, we believe the new table is indeed much better.\n    \n    | Row    | Prior  | Losses Used      |Unrecognized | Nucleus | Granule | Mito.   | Mean DSC |\n    |-----   |--------|------------------|-------------|---------|---------|---------|----------|\n    | 1      | Normal | KL                  | 0.44        | 0.55    | 0.34    | 0.13    | 0.37     |\n    | 2      | Normal | KL + CL         | 0.83        | 0.95    | 0.69    | 0.76    | 0.81     |\n    | 3      | Normal | KL +  CE        | 0.81        | 0.97    | 0.80    | 0.75    | 0.77     |\n    | 4      | Normal | KL + CL + CE     | 0.81        | 0.97    | 0.73    | 0.72    | 0.75     |\n    |*new line* | GMM    | KL +  CE     | 0.82        | 0.97    | 0.72    | 0.75    | **0.82** |\n    | 5      | GMM    | KL + CL + CE     | 0.86        | 0.98    | 0.80    | 0.75    | 0.85     |"}, {"review_id": "r87tDULSun", "reviewer": "Reviewer_LW2g", "summary": "The authors propose a method for semantic segmentation of biological microscopy data given only very sparse labels. They achieve this through a hierarchical VAE architecture with GMM prior, masked region inpainting task, classification head, and auxiliary losses including contrastive learning and label entropy minimization. They demonstrate their method on two volume EM datasets and one fluorescence microscopy dataset, producing results nearly on par with fully supervised baselines.", "questions": "1. Can you clarify the inference of the topmost latent $z_L$? In equation 10, do the parameters of the variational distribution not depend on the class $c$? In other words, I am unclear why the class-conditional posterior in equation 10 and class-marginal posterior in equation 11 are the same.\n2. How do you suggest mitigating the apparent over-segmentation observed in the images?\n3. In Table 6, it seems that the gain in average DSC comes primarily from the failure of the baseline to segment OBC and CBC. Are these the least common classes? If so, could you have achieved the same result through a simpler model with some sort of class-weighting of the training distribution or loss?\n4. What is being learned from the unlabeled data? The inpainting, classification, and contrastive learning objectives all require labels. This leaves the entropy and KL losses that can be applied on unlabeled patches, but it is not clear why this would help the model. How does the performance change when you ablate the entropy loss?\n5. In Table 7, for the cross-entropy loss ablation, how are classes predicted without the classification head? Is it via clustering?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "_Strengths:_\n\nThe paper addresses the important problem of segmenting noisy biological data given extremely sparse labels (0.05% of the training data). \nThe authors demonstrate that sparsely-supervised methods can achieve segmentations nearly as good as fully supervised methods on real datasets of different imaging modalities.\n\n_Weaknesses:_\n\nThe improvement over the U-Net baseline is not convincing. In both Figures 3 and 4, the proposed method tends to over-segment the images. In Figure 3, I would argue that the U-Net segmentation could be preferable, since it maintains the short sheet-like topology of the basolateral membrane, as opposed to the proposed method, which produces segmentations that look blobby. Likewise in Figure 4, the boundaries between cells can be seen visually in the first image, but the proposed method merges all the cytoplasms into one continuous region. Similarly, in Table 6, the proposed method underperforms the U-Net baseline on several classes. \n\nAdditionally, the method combines many different learning techniques, without complete ablations to justify their use. In Table 7, if removing the CE loss, how are the predictions being made? An ablation of the entropy loss could also be performed, to show that learning on unlabeled data is useful. You might even consider ablating the inpainting loss itself, and train only the encoder and segmentation head.", "was_revised": true, "final_justification": "In light of the new comparisons to the U-Net under different levels of data sparsity, I can better appreciate the results of the proposed method. The additional ablations and explanations of the model components also help in understanding the methodological novelty. I maintain concerns that the increased performance on rare classes comes at the cost of over-segmentation of other classes, as seen in Figure 3. While the authors argue that this is a desirable tradeoff for their experimental collaborators, such segmentation behavior may not be desirable in other areas of microscopy or semantic segmentation more broadly. I will raise my score to a 4.", "rebuttal": "We thank this reviewer and are happy to address the criticism below point by point (italic text is from the submitted reviewer feedback):\n- *The improvement over the U-Net baseline is not convincing. In both Figures 3 and 4, ... In Figure 3, I would argue that the U-Net segmentation could be preferable, ...*\n\n   Regarding the visual examples in Figures 3 and 4, we agree that the U-Net produces ok quality segmentations that may appear preferable on first sight. However, for our life science collaborators, the most critical requirement typically is the ability to detect and distinguish regions of all classes, particularly the CBC, for which the U-Net shows a very high false negative rate. Our method leads to more faithful detections of both OBC and CBC regions, which can be seen in the qualitative figures in our manuscript but is also clearly visible in the quantitative evaulation we provide.\n- *Likewise in Figure 4, ...*\n\n   In Figure 4, regions are indeed merging, which is a common property of pixel-classification methods.\n\n- *Similarly, in Table 6, the proposed method underperforms the U-Net baseline on several classes.\nIn Table 6, it seems that the gain in average DSC comes primarily from the failure of the baseline to segment OBC and CBC. Are these the least common classes? If so, could you have achieved the same result through a simpler model with some sort of class-weighting of the training distribution or loss?*\n\n   We must respectfully disagree, but also have to admit that the tables in our submitted manuscript can (and will) be improved. In the current Table 6, the U-Net only really outperforms our method when trained fully supervised, on a impractical amount of training data. When compared to the sparsely trained U-Net, our method is already much preferable. One thing we did not communicate well in our manuscript is the fact that we did not perform class-balancing during training of the U-Net, and that the amount of (sparse) supervision data diverges dramatically between tables. For example, all entries in Table 4, use manyfold less label data during training than we use in Table 1 (row 4) and same between rows 2 and 3 of Table 6. In liver FIBSEM dataset, 10 volumetric crops were used for training (one slice per volume, which was representative of all classes available) fully annotated per volume. OBC and CBC are the least common classes, therefore the performance dropped.\n   For this reason, and also in response to reviewer sMMT, we have now added the class-balanced U-Net baseline into Table 4 (both trained on the exact same amount of labels now): Please find it in response to reviewer sMMT.\n   Table 6 further highlights the above-mentioned point, under the same sparse label budget, our method significantly improves the segmentation of rare classes, contributing to a higher average DSC.\n   While a class-weighted loss could help, our architecture is designed specifically to separate class distributions in the latent space with minimal supervision, which class weights alone cannot achieve.\n\n   |.| Sparse U-Net | ϵ-Seg|\n   |--------|-----------------|-------|\n   | Avg DSC | 0.3128 | **0.67**|\n\n- *Additionally, the method combines many different learning techniques, without complete ablations to justify their use. In Table 7, if removing the CE loss, how are the predictions being made?\nIn Table 7, for the cross-entropy loss ablation, how are classes predicted without the classification head? Is it via clustering?*\n\n   Regarding the use of multiple components, we believe their individual roles and interactions are well-justified throughout the paper. Nonetheless, we will expand the explanation in Section 3 and Table 7 with additional ablations, including the GMM prior without contrastive loss (as requested by reviewer fGHF), and a separate ablation of the entropy loss, as requested by the reviewer. As mentioned in our manuscript, in the absence of cross-entropy supervision (Table 7, rows 3–4), predictions were made by applying Kmeans clustering on the latent space, similar to MAESTER-style methods. As other reviewers pointed out, our ablations illustrate the importance of each part of our method and how they stack to achieve the overall results we are reporting on different datasets. Please check the response to fGHF.\n\n- *An ablation of the entropy loss could also be performed, ...\nYou might even consider ablating the inpainting loss itself, and train only the encoder and segmentation head.*\n\n   Regarding the reviewer’s suggestion to ablate the inpainting loss, we appreciate this comment.\n    However, we would like to clarify the motivation behind our inpainting strategy.\n    Rather than masking arbitrary regions, we deliberately mask only the centre pixel of each patch.\n    This design choice is rooted in the observation that EM patches often contain multiple adjacent structures, and masking a larger region would likely mix signals from different classes.\n    By focusing the prediction on a single centre pixel, we encourage the model to learn discriminative features that are class-consistent and spatially precise.\n    Indeed, Table 8 supports this design choice, as the mask size increases, segmentation performance degrades, confirming the benefit of our localized inpainting approach.\n    Moreover, this centre-pixel masking aligns naturally with our inference strategy, where we generate segmentation labels per pixel by extracting the patch centered on that pixel.\n    This symmetry between training and inference is important for model consistency and accuracy, especially under sparse supervision.\n    The entropy loss plays a key role in reducing uncertainty on unlabeled inputs, and we observed measurable performance improvements when it is applied.\n    Below you see the result of all models that were also trained with unlabeled data (Entropy Loss), reported without applying it (without $\\mathcal{L}_{H}$).\n    Table 5 was given only for one of the configurations, below you see for the rest, if applicable. We add the full table in the camera-ready version. Table 2 the Avg DSC without Entropy Loss is 0.88. Table 4 and 8 below.\n    |RLF|Entropy Loss|Unrecog.|Nucleus|Granules|Mito.| Avg DSC |\n    |--------------|--------------|---------------|---------|----------|--------------|-----------|\n    |20|With-Without|0.89-0.85|0.98-0.97|0.81-0.75|0.83-0.77|0.88-0.83|\n    |15|With-Without|0.88-0.84|0.98-0.97|0.81-0.75|0.78-0.73|0.86-0.82|\n    |10|With-Without|0.86-0.81|0.98-0.97|0.80-0.74|0.75-0.71|0.85-0.81|\n    |5|With-Without|0.85-0.80|0.96-0.96|0.77-0.71|0.76-0.72|0.84-0.80|\n    |1|With-Without|0.79-0.69|0.95-0.94|0.69-0.58|0.69-0.62|0.78-0.71|\n\n    | Mask Size|Entropy Loss|Unrecog.|Nucleus|Granules|Mito.|Avg DSC|\n    |----------|-------------|--------|--------|---------|------|--------|\n    |1×1|With-Without|0.86-0.81|0.98-0.97|0.80-0.74|0.75-0.71|0.85-0.81|\n    |3×3|With-Without|0.88-0.85|0.97-0.96|0.81-0.76|0.80-0.76|0.87-0.83|\n    |5×5|With-Without|0.87-0.84|0.94-0.93|0.78-0.74|0.80-0.76|0.85-0.82|\n    |7×7|With-Without|0.84-0.79|0.97-0.96|0.72-0.66|0.75-0.70|0.82-0.77|\n    |9×9|With-Without|0.83-0.79|0.95-0.94|0.65-0.58|0.73-0.68|0.79-0.75|\n- *Can you clarify the inference of the topmost latent $z_L$? In equation 10, do the parameters of the variational distribution not depend on the class $c$? ...*\n\n    To answer the reviewer’s question on Equation 10 vs. 11, we clarify that while the class-conditional posterior $q(z_{L} | x, c)$ is written explicitly, in practice we share the same encoder parameters for all classes (via FiLM modulation), so the marginal posterior simplifies to a single Gaussian, as explained in our derivation of Equation 11.\n    Below, we will expand on the connection between the latent FiLM-conditioned segmentation head and the GMM prior.\n    Regarding the marginal and conditional posterior (Eq. 10 and 11), we hope the following lines will help. We would like to mention that the class-conditioning modulates the output of the encoder in the highest level of hierarchy in order to keep the semantic features of only the centre region.\n    Derivation of Equation 11, we model the approximate posterior over the top-level latent variable $z_L$ as a mixture of Gaussians, conditioned on the predicted categorical distribution over components. Starting from Equation 9, $$q_\\phi(z_L \\mid x) = \\sum_{l=1}^{C} q_\\phi(c = l \\mid x) \\cdot q_\\phi(z_L \\mid x, c = l)$$, each conditional posterior $q_\\phi(z_L \\mid x, c = l)$ is modeled as a Gaussian distribution $$q_\\phi(z_L \\mid x, c = l) = \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x)).$$\n    In our implementation, the parameters $\\mu_L(x)$ and $\\sigma_L(x)$ are computed once from the FiLM-conditioned encoder output and are shared across all components $l$.\n    As a result, the mixture in Equation 9 reduces to $$q_\\phi(z_L \\mid x) = \\sum_{l=1}^{C} q_\\phi(c = l \\mid x) \\cdot \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x)) = \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x))$$\n    This simplification is valid under the assumption that the conditional Gaussians share the same parameters across all components.\n    Equation 11 is thus used as the effective posterior in our model.\n    While we believe our manuscript does already point this out at some level of clarity, if the reviewer thinks that the explanation above adds value to readers more generally, we are happy to include this arument in the appendix.\n\n- *How do you suggest mitigating the apparent over-segmentation observed in the images?*\n    \n    On the concern about over-segmentation, we agree that our method tends toward confident, full-mask predictions, which may lead to larger object regions.\n    However, we believe that this behavior directly stems from our entropy loss, which suppresses ambiguous predictions, a necessary bias in the low-label regime.\n    While this can visually appear as oversegmentation, we note that it yields consistently higher DSCs, particularly on rare classes, which we consider a desirable tradeoff."}, {"review_id": "uVWa1q8zK1", "reviewer": "Reviewer_fGHF", "summary": "The paper introduce ϵ-Seg, a method based on hierarchical variational autoencoders, employing center-region masking, sparse label contrastive learning, a Gaussian Mixture Model prior, and clustering-free label prediction to handle electron microscopy  images of biological samples. The paper claims that center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, and could deal with cases when training labels are sparse. The paper proposes an entropy-based loss that can improve segmentation quality when fewer training labels are available, and employs contrastive learning and a Gaussian Mixture Model prior to shape the latent space of the hierarchical variational autoencoders. The paper also proposes a semantic segmentation head composed of MLP and FiLM layers to directly predict class labels from latent embeddings. Experimental results of the proposed method demonstrate the applicability on fluorescence microscopy data and show that ϵ-Seg is capable of achieving competitive semi-supervised segmentation results on complex biological image data when fewer training labels are available.", "questions": "1.Is there a unified theoretical connection between the various modules proposed?\n2.In Table 7, it can be seen from rows 3 and 4 that the addition of CL loss does not bring any improvement, while rows 1 and 2 show that CL loss brings a significant improvement. Should the experimental results under the condition of without CL loss but with GMM be reported?\n3.Is there a visualization showing the different segmentation effects brought by adding different modules proposed?\n4.The segmentation results of the proposed method as visualized in Figure 2 and Figure 3 do not appear to have smooth segmentation edges. Is there any analysis or explanation for this?", "limitations": "There are too many hyperparameters to tune, and there is no analysis of the model's complexity.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper has good quality and clarity.But there are so many hyperparameters in the proposed method,as shown in Eq.17,Eq.18, and there is no overall theoretical discussion on the various modules proposed,such as HVAEs, sparse label contrastive learning and  the Gaussian Mixture Model prior.", "was_revised": true, "final_justification": "I have decided to maintain my rating at 4. Solid theoretical basis is better for the proposed method.", "rebuttal": "We thank this reviewer for the time to familiarize themselves with our paper and are happy to address the criticism below point by point (italic text is from the submitted reviewer feedback):\n\n- *The paper has good quality and clarity.But there are so many hyperparameters in the proposed method,as shown in Eq.17,Eq.18, and there is no overall theoretical discussion on the various modules proposed,such as HVAEs, sparse label contrastive learning and the Gaussian Mixture Model prior.*\n    \n    *1.Is there a unified theoretical connection between the various modules proposed?*\n    \n    We thank the reviewer for the thoughtful evaluation and for highlighting the clarity and technical quality of our work.\n    We are not sure what a “unified theoretical connection” could look like, but are afraid the answer is likely: “no”. Of course the parts of our proposed method make sense together and jointly lead to the results we report, as also shown by our ablation studies, but we are afraid that this is not the “connection” the reviewer was refering to.\n    \n    After discussing this point among ourselves, we concluded that adding a related work section to the appendix might be the right way to better express how things play together and are related to other related ground-work that has been conducted by our community. For example, we would like to relate our work to:\n\n    [1] Berkhahn et al., *Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un)supervised, and Supervised Learning*, arXiv, 2019  \n    [2] Liévin et al., *Towards Hierarchical Discrete Variational Autoencoders*, PMLR (Approximate Bayesian Inference Symposium), 2020  \n    [3] Śmieja et al., *SeGMA: Semi-Supervised Gaussian Mixture Auto-Encoder*, arXiv, 2020  \n    [4] Boubekki et al., *Joint Optimization of an Autoencoder for Clustering and Embedding*, Machine Learning (Springer), 2021\n\n    Below, we will expand on the connection between the latent FiLM-conditioned segmentation head and the GMM prior.\n    Regarding the marginal and conditional posterior (Eq. 10 and 11), we hope the following lines will help. We would like to mention that the class-conditioning modulates the output of the encoder in the highest level of hierarchy in order to keep the semantic features of only the centre region.\n    Derivation of Equation 11, we model the approximate posterior over the top-level latent variable $z_L$ as a mixture of Gaussians, conditioned on the predicted categorical distribution over components. Starting from Equation 9,\n    $$q_\\phi(z_L \\mid x) = \\sum_{l=1}^{C} q_\\phi(c = l \\mid x) \\cdot q_\\phi(z_L \\mid x, c = l),$$\n    each conditional posterior $q_\\phi(z_L \\mid x, c = l)$ is modeled as a Gaussian distribution\n    $$q_\\phi(z_L \\mid x, c = l) = \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x)).$$\n    In our implementation, the parameters $\\mu_L(x)$ and $\\sigma_L(x)$ are computed once from the FiLM-conditioned encoder output and are shared across all components $l$.\n    As a result, the mixture in Equation 9 reduces to $$q_\\phi(z_L \\mid x) = \\sum_{l=1}^{C} q_\\phi(c = l \\mid x) \\cdot \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x)) = \\mathcal{N}(z_L; \\mu_L(x), \\sigma_L(x))$$\n    This simplification is valid under the assumption that the conditional Gaussians share the same parameters across all components.\n     Equation 11 is thus used as the effective posterior in our model.\n    While we believe our manuscript does already point this out at some level of clarity, if the reviewer thinks that the explanation above adds value to readers more generally, we are happy to include this arument in the appendix.\n    \n- *2.In Table 7, it can be seen from rows 3 and 4 that the addition of CL loss does not bring any improvement, while rows 1 and 2 show that CL loss brings a significant improvement. Should the experimental results under the condition of without CL loss but with GMM be reported?*\n    \n    Regarding the contrastive loss, the reviewer makes an excellent observation.\n    The mixed performance of CL in Table 7 (rows 3–4 vs. rows 1–2) motivated our introduction of the GMM prior, which aligns more naturally with the class-aware contrastive loss.\n    We agree that the missing “GMM + no CL” condition should be reported, and we will include that experiment in the camera-ready version for completeness.\n    Below you can see the updated Table 7, where one can see that adding CL only helps when a GMM prior is used. Thanks for spotting this, we believe the new table is indeed much better.\n    | Row    | Prior  | Losses Used      |Unrecognized | Nucleus | Granule | Mito.   | Mean DSC |\n    |-----   |--------|------------------|-------------|---------|---------|---------|----------|\n    | 1      | Normal | KL        | 0.44        | 0.55    | 0.34    | 0.13    | 0.37     |\n    | 2      | Normal | KL + CL      | 0.83        | 0.95    | 0.69    | 0.76    | 0.81     |\n    | 3      | Normal | KL +     CE     | 0.81        | 0.97    | 0.80    | 0.75    | 0.77     |\n    | 4      | Normal | KL + CL + CE     | 0.81        | 0.97    | 0.73    | 0.72    | 0.75     |\n    |*new line* | GMM    | KL   + CE     | 0.82        | 0.97    | 0.72    | 0.75    | **0.82** |\n    | 5      | GMM    | KL + CL + CE     | 0.86        | 0.98    | 0.80    | 0.75    | 0.85     |\n    \n- *3.Is there a visualization showing the different segmentation effects brought by adding different modules proposed?*\n    \n    Concerning visualization, we have indeed generated qualitative outputs for all experimental variants, which were omitted from the main text due to space constraints.\n    Since all models already have DSC metrics reported, we propose to add selected visual comparisons (e.g., for models with vs. without CL, GMM, or entropy loss) to highlight their structural effects in the appendix.\n    If space permits, we can also add a t-SNE projection of the latent space, which would show clear semantic separation with GMM + CL, either in the main manuscript or otherwise in the appendix.\n    \n- *4.The segmentation results of the proposed method as visualized in Figure 2 and Figure 3 do not appear to have smooth segmentation edges. Is there any analysis or explanation for this?*\n    \n    Our method is a pixel-classification method that makes a decision for a central patch area (typically a single pixel) given the entire patch. Whenever a patch is at the very boundary of a foreground region, or at the interface between two foreground classes, the precise point for the class-switch is to some degree uncertain and the segmentation head will make inconsistent decisions due to the lack of any memory between neighboring patches.\n    \n    This can obviously be fixed by a whole plethora of possible postpreocessing methods, as also mentioned by other reviewers. While this is true, we believe that going into such postprocessing options is well beyond the scope and value of our work, and we hope the reviewer agrees with our assessment and opinion.\n    \n- *There are too many hyperparameters to tune, and there is no analysis of the model's complexity.*\n    \n    We do, indeed, have multiple hyperparameters to set. Still, all experiments we conducted use, on purpose, the same hyperparameter settings (alphas, lambda, contrastive margin, etc.). We did find those values by a simple grid-search, which we propose to add to the methods section (or appendix) of the camera-ready manuscript. Additionally, we propose to add the requested sensitivity analysis to the supplement (see also the example table below).\n\n    | (w_CL) \\ (w_KLD) | -4.0  | -3.0  | -2.0  | -1.0  | 0.0   |\n    |------------------|-------|-------|-------|-------|-------|\n    |  0.0             | -0.20 |  0.00 |  0.06 | -0.20 | -0.18 |\n    | -1.0             |  0.91 | -0.12 |  0.12 |  0.05 | -0.06 |\n    | -2.0             |  0.27 |  0.84 |  0.86 |  0.88 |  0.89 |\n    | -3.0             |  0.21 |  0.35 |  0.36 |  0.41 |  0.54 |\n    | -4.0             |  0.01 |  0.02 |  0.01 |  0.03 |  0.01 |\n    \n    We performed a grid search over the log-scale weights of the CL and KL divergence, using the Silhouette Score to evaluate the resulting latent space structure.\n    The Silhouette Score ranges from -1 to 1 and measures how well individual latent embeddings are separated according to their class identity, balancing intra-class compactness and inter-class separability.\n    A high positive score indicates well-separated, compact clusters, a score near 0 means overlapping clusters and a negative score suggests incorrect clustering (points are closer to another cluster than their own).\n    This was particularly relevant to our setup, the contrastive loss encourages class-wise separation, while the KL loss pushes the posterior closer to the prior distribution (KL is only applied on the highest level of hierarchy while CL for all layers), which can reduce separability.\n    The grid search allowed us to identify regions where both terms work in synergy, resulting in well-structured, class-aligned latent representations.\n    These results also validate our use of the GMM prior."}, {"review_id": "96g7jbsVUP", "reviewer": "Reviewer_H473", "summary": "This paper focuses on the task of sparsely supervised semantic segmentation of electron microscope images and proposes a method based on hierarchical variational autoencoders, called $\\epsilon$-Seg. The proposed method involves several improvements including center-region masking, sparse label contrastive learning, gaussian mixture model prior, and clustering-free label prediction, and achieved a certain performance improvement over baselines such as U-Net and Labkit in the sparsely supervised setting.", "questions": "see Weaknesses.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The proposed method was experimented on several datasets including BetaSeg, liver FIBSEM, fluorescent microscopy dataset.\n- There are sufficient ablation experiments and equations to illustrate the results and ideas of the paper.\n\nWeaknesses:\n- Lack of novelty. This paper combines several existing techniques rather than presenting an original method, making it somewhat lacking in novelty.\n- Absence of a related work section. The absence of the discussion of related work makes the paper incomplete, which should be added (at least in the appendix).\n- The motivation is unclear. The paper combines several techniques and lacks a clear motivation for the necessity of using them together.\n- The baselines compared are weak. Table 1 compares various self-supervised and fully supervised methods while lacking sufficient comparisons with sparsely supervised work which is more related (e.g., [1][2]).\n\n[1] Learning Cross-Representation Affinity Consistency for Sparsely Supervised Biomedical Instance Segmentation. ICCV, 2023.\n[2] Sparse Object-level Supervision for Instance Segmentation with Pixel Embeddings. CVPR, 2022.", "was_revised": true, "final_justification": "Thanks to the author's detailed rebuttal, I've slightly increased my score while maintaining my concerns about novelty.", "rebuttal": "We thank this reviewer and are happy to address the criticism below point by point (italic text is from the submitted reviewer feedback):\n\n- *Lack of novelty. This paper combines several existing techniques rather than presenting an original method, making it somewhat lacking in novelty.*\n    \n    While it’s true that our approach integrates existing components, we respectfully argue that the adaptation of a GMM prior to a hierarchical VAE, combined with the contrastive loss and segmentation head, followed by a FiLM layer, should be considered a non-trivial combination that does require rather deep theoretical and practical knowledge and merits publication. Additionally, we have invested considerable efforts to explain the overall setup and how components work together in the manuscript, such that all parts are now also theoretically motivated.\n    Especially, in the sparse supervision setting, combining GMM priors with contrastive alignment required careful formulation, which we derived and implemented successfully to enable the model to learn a semantically meaningful, class-aware latent space.\n    To the best of our knowledge, our work is unique, opens the door for many use-cases beyond pixel classification, and, already today, offers the possibility to learn semantically meaningful latent representation that are of high enough quality to distinguish biological structures in real biomedical image data with moderate computational cost and low training label requirements. \n    In the light of this, and the arguments below, we hope the reviewer can see the value of our work.\n    \n- *Absence of a related work section. The absence of the discussion of related work makes the paper incomplete, which should be added (at least in the appendix).*\n    \n    We attempted to give credit to the related work throughout the manuscript, which we believe worked well for many if not most the relevant related work.\n    Still, we understand the value of community conventions, and like the idea of adding a dedicated related work section in the appendix.\n    Doing so, we will not only re-cite the work we had already pointed at throughout the current manuscript, but will use the extra space in the appendix to also mention some additional papers that embed our work into broader context.\n    \n    Here a subset of the papers we will additionally include:\n     \n    [1] Berkhahn et al., *Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un)supervised, and Supervised Learning*, arXiv, 2019  \n    [2] Liévin et al., *Towards Hierarchical Discrete Variational Autoencoders*, PMLR (Approximate Bayesian Inference Symposium), 2020  \n    [3] Śmieja et al., *SeGMA: Semi-Supervised Gaussian Mixture Auto-Encoder*, arXiv, 2020  \n    [4] Boubekki et al., *Joint Optimization of an Autoencoder for Clustering and Embedding*, Machine Learning (Springer), 2021\n- *The motivation is unclear. The paper combines several techniques and lacks a clear motivation for the necessity of using them together.*\n    \n    We would like to respectfully challenge the notion of our method being an arbitrary combination of unrelated components. In fact, each component plays a targeted role within our disentangled latent (semantic segmentation) objective. As other reviewers pointed out, our ablations illustrate the importance of each part of our method and how they stack to achieve the overall results we are reporeting on different datasets.\n    \n    Here a short list that summarizes what each component’s role in the overall workings of $\\epsilon$-Seg is:\n    * The centre pixel masking and inpainting loss help the encoder to learn robust structure-aware features (each input patch contains more than one structure).\n    * The CE loss enforces alignment between high-level semantic features and class labels.\n    While at this point we could in theory go on without the FiLM layer, we made sure by adding it, to have a closer approximation of the conditional posterior on the class label (derivation of Equation 11 from Equation 9).\n    * The GMM prior and contrastive loss work together to ensure disentanglement and separation between latent clusters.\n    * CL is applied through all levels of latent encodings (from coarse to fine), while KL in the highest level of latent keeps the class aware features disentangled.\n    \n    Hence, these components are not just stacked heuristically, they are integrated to support a coherent and sound pipeline that learns class-aligned, interpretable representations from very limited amounts of supervision labels.\n    \n- *The baselines compared are weak. Table 1 compares various self-supervised and fully supervised methods while lacking sufficient comparisons with sparsely supervised work which is more related (e.g., [1][2]).*\n    \n    We appreciate the suggestion to include additional baselines. Since the mentioned methods are instance segmentation approaches and our method performs semantic segmentation, such a comparision is not straight forward (please see the fruitful discussion with reviewer LwaP.\n    Additionally, while our method is indeed presented as a pixel classification method, the value of the latent representations we learn will go much beyond segmentation use-cases alone, and a comparision with instance segmentation approaches will distract from this more than we believe it will help.\n    \n\nLast but not least, we will of course ensure that all reference formatting issues are fixed in the final version. Thank you for pointing this out."}, {"review_id": "eHU9gSsCoo", "reviewer": "Reviewer_LwaP", "summary": "The authors introduce $\\epsilon$-Seg, a method for semantic segmentation of EM data. The method modifies a hierarchical VAE approach with several changes, including an entropy-based loss, and a contrastive loss (+modify the standard Gaussion prior to instead use GMM) in addition to an inpainting loss and the standard KL-term. The authors auditionally  introduce some hyperparameters for these loss terms and a semantic segmentation head that directly predicts labels from latent embeddings. Evaluations on 2 EM datasets shows strong results compare to other methods that use limited supervised training data.", "questions": "Can the authors add sprase supervision to the self-supervised methods they compare against (e.g. MAESTER)?\n\nCan the authors compare against different forms of postprocessing such as watershed/multi-cut rather than k-means?", "limitations": "The authors do a good job discussing their limitations throughout, but should detail more carefully how they tune their hyperparameters and their effects.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- S1: The authors study an important problem\n- S2: The paper is clearly written\n- S3: The author's approach for introducing the GMM prior seems well-motivated and well-implemented\n- S4: The authors show 4 ablations on their introduced loss terms in Table 7\n- Major weakness: The proposed methods' main benefits are from learning from limited amount of training labels and as such, should compare to supervised/self-supervised methods which are generally more sample-specific (and postprocess using a watershed/multicut). For supervised methods, this would include methods such as [MALA](https://pubmed.ncbi.nlm.nih.gov/29993708/) or [PEA](https://ojs.aaai.org/index.php/AAAI/article/view/19984) and for self-supervised, this would include bethods such as [BYOL](https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html), [dbMIM](https://arxiv.org/abs/2310.04148), or [MS-Con-EM](https://arxiv.org/abs/2308.09917). \n- W2: The authors should additionally show sensitivity to their introduced hyperparameters", "was_revised": true, "final_justification": "I thank the authors for their clarifications but maintain my concerns with baselines / hyperparameter complexity+sensitivity, so I maintain my score.", "rebuttal": "We thank this reviewer for the time to familiarize themselves with our paper and are happy to address the criticism below point by point (italic text is from the submitted reviewer feedback):\n\n- *Can the authors add sprase supervision to the self-supervised methods they compare against (e.g. MAESTER)? Can the authors compare against different forms of postprocessing such as watershed/multi-cut rather than k-means?*\n    \n    Regarding postprocessing, we used MAESTER-like k-means as one of our baselines/ablations (see Table 7). We have then proposed to replace this time-consuming postprocessing with our newly introduced segmentation-head, which improves the speed+performance, and therefore the applicability of our method quite dramatically.\n    \n    $\\epsilon$-Seg takes ~280seconds for giving the full segmentation of one slice of “high_c4” volume from BetaSeg vs segmentation-head free which takes ~3250 (!!!) seconds.\n    \n    Regarding the reviewer’s question about post-processing other than k-means, there is obviously no reason for future users not to do so. In our work, we have been focussing on learning the most expressive latent space with respect to the classes users want to distinguish, and it stands to reason if the paper would gain in readability and value if we dilute the detailed description of those achievements down by also adding an evaluation of possible ways for how to benefit from the learned latent encoding downstream. We value the reviewer’s focus on best possible final segmentation results, but would like to argue that our approach is novel in various ways and shows great potential for various use-cases in the future. Hence, we believe there is value of sharing our achievements with a broader readership who can then benefit from the ideas we present in the context of their own work.\n    \n- *W2: The authors should additionally show sensitivity to their introduced hyperparameters. The authors do a good job discussing their limitations throughout, but should detail more carefully how they tune their hyperparameters and their effects.*\n    \n    We do, indeed, have multiple hyperparameters to set. Still, all experiments we conducted use, on purpose, the same hyperparameter settings (alphas, lambda, contrastive margin, etc.). We did find those values by a simple grid-search, which we propose to add to the methods section (or appendix) of the camera-ready manuscript. Additionally, we propose to add the requested sensitivity analysis to the supplement (see also the example table below).\n\n    | (w_CL) \\ (w_KLD) | -4.0  | -3.0  | -2.0  | -1.0  | 0.0   |\n    |------------------|-------|-------|-------|-------|-------|\n    |  0.0             | -0.20 |  0.00 |  0.06 | -0.20 | -0.18 |\n    | -1.0             |  0.91 | -0.12 |  0.12 |  0.05 | -0.06 |\n    | -2.0             |  0.27 |  0.84 |  0.86 |  0.88 |  0.89 |\n    | -3.0             |  0.21 |  0.35 |  0.36 |  0.41 |  0.54 |\n    | -4.0             |  0.01 |  0.02 |  0.01 |  0.03 |  0.01 |\n\n    We performed a grid search over the log-scale weights of the CL and KL divergence, using the Silhouette Score to evaluate the resulting latent space structure.\n    The Silhouette Score ranges from -1 to 1 and measures how well individual latent embeddings are separated according to their class identity, balancing intra-class compactness and inter-class separability.\n    A high positive score indicates well-separated, compact clusters, a score near 0 means overlapping clusters and a negative score suggests incorrect clustering (points are closer to another cluster than their own).\n    This was particularly relevant to our setup, the contrastive loss encourages class-wise separation, while the KL loss pushes the posterior closer to the prior distribution (KL is only applied on the highest level of hierarchy while CL for all layers), which can reduce separability.\n    The grid search allowed us to identify regions where both terms work in synergy, resulting in well-structured, class-aligned latent representations.\n    These results also validate our use of the GMM prior.\n    \n- *Major weakness: The proposed methods' main benefits are from learning from limited amount of training labels and as such, should compare to supervised/self-supervised methods which are generally more sample-specific (and postprocess using a watershed/multicut). For supervised methods, this would include methods such as [MALA](https://pubmed.ncbi.nlm.nih.gov/29993708/) or [PEA](https://ojs.aaai.org/index.php/AAAI/article/view/19984) and for self-supervised, this would include bethods such as [BYOL](https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html), [dbMIM](https://arxiv.org/abs/2310.04148), or [MS-Con-EM](https://arxiv.org/abs/2308.09917).*\n    \n    We agree with the reviewer, the mentioned instance segmentation approaches (and other well known methods, some mentioned for example in [1]) are interesting alternatives for practical applications. We intended in our work to present an interesting new approach that leads to a powerful latent-space organization that is aware of user-defined classes without requiring an impossible amount of supervision signal. As mentioned before, we believe in the value of sharing our achievements with a broad readership of experts who can benefit from the presented ideas, even without including a comparison to the mentioned instance segmentation approaches. It is, in fact, rather uncommon to compare a pixel-classification approach (like ours) against instance segmentation methods, since such a comparison is in itself not straight forward and would require the pixel-classifier to be paired with a suitable and optimized post-processing routine. This would, in our humble opinion, make this already quite dense manuscript less readable and clear.\n\n    Naturally, MAESTER could be modified to also have a supervised component. Still, we did not even consider this options since MAESTER has **328 million parameters** (our deepest model, in comparison, has only **3.8 million parameters** (see line258 and Tables S1, S2)). Future users of our method can easily train a model for their data using our approach.\n\n    [1] Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey 2022"}]

Author rebuttal:

Meta-review: The authors present an interesting new approach to semantic segmentation of microscopy data, combining several different ideas (contrastive learning, HVAE's, and GMMs) into a single segmentation model. The problem itself is of considerable importance in the analysis of microscopy data, and the results compared to competitive u-net baselines are sufficiently compelling, particularly in the low-labeled data regime. Given the novelty of the overall method and its performance on several different benchmark tasks, the paper is a valauble addition to the ML for 2d segmentation literature. The latent space learned by this type of model may also be useful for other tasks. 

During the discussion, the authors successfully clarified baseline experiments, including novel results on sparse data performance of u-nets and other aspects. Some weakness remains in that the method seems relatively complicated, and sensitivity to numerous hyperparameters is a potential issue.
Decision: Accept (poster)