Title: $\epsilon$-Seg: Sparsely Supervised Semantic Segmentation of  Microscopy Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: semantic segmentation

microscopy

representation learning

variational networks
Abstract: Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences.
EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming.
We introduce $\epsilon$-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering-free label prediction.
Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse ($0.05$\% of the total image data or less).
For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster w.r.t. the semantic classes we wish to distinguish. 
Finally, instead of clustering latent embeddings for semantic segmentation, we propose a MLP semantic segmentation head to directly predict class labels from latent embeddings.
We show empirical results of $\epsilon$-Seg and baseline methods on $2$ dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. 
Our results show that $\epsilon$-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.
Code available at https://github.com/juglab/eps-Seg.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce $\\epsilon$-Seg, a method for semantic segmentation of EM data. The method modifies a hierarchical VAE approach with several changes, including an entropy-based loss, and a contrastive loss (+modify the standard Gaussion prior to instead use GMM) in addition to an inpainting loss and the standard KL-term. The authors auditionally  introduce some hyperparameters for these loss terms and a semantic segmentation head that directly predicts labels from latent embeddings. Evaluations on 2 EM datasets shows strong results compare to other methods that use limited supervised training data.\n\nSTRENGTHS AND WEAKNESSES:\n- S1: The authors study an important problem\n- S2: The paper is clearly written\n- S3: The author's approach for introducing the GMM prior seems well-motivated and well-implemented\n- S4: The authors show 4 ablations on their introduced loss terms in Table 7\n- Major weakness: The proposed methods' main benefits are from learning from limited amount of training labels and as such, should compare to supervised/self-supervised methods which are generally more sample-specific (and postprocess using a watershed/multicut). For supervised methods, this would include methods such as [MALA](https://pubmed.ncbi.nlm.nih.gov/29993708/) or [PEA](https://ojs.aaai.org/index.php/AAAI/article/view/19984) and for self-supervised, this would include bethods such as [BYOL](https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html), [dbMIM](https://arxiv.org/abs/2310.04148), or [MS-Con-EM](https://arxiv.org/abs/2308.09917). \n- W2: The authors should additionally show sensitivity to their introduced hyperparameters\n\nQUESTIONS:\nCan the authors add sprase supervision to the self-supervised methods they compare against (e.g. MAESTER)?\n\nCan the authors compare against different forms of postprocessing such as watershed/multi-cut rather than k-means?\n\nLIMITATIONS:\nThe authors do a good job discussing their limitations throughout, but should detail more carefully how they tune their hyperparameters and their effects.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-08"}

comments: SUMMARY:
This paper focuses on the task of sparsely supervised semantic segmentation of electron microscope images and proposes a method based on hierarchical variational autoencoders, called $\epsilon$-Seg. The proposed method involves several improvements including center-region masking, sparse label contrastive learning, gaussian mixture model prior, and clustering-free label prediction, and achieved a certain performance improvement over baselines such as U-Net and Labkit in the sparsely supervised setting.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed method was experimented on several datasets including BetaSeg, liver FIBSEM, fluorescent microscopy dataset.
- There are sufficient ablation experiments and equations to illustrate the results and ideas of the paper.

Weaknesses:
- Lack of novelty. This paper combines several existing techniques rather than presenting an original method, making it somewhat lacking in novelty.
- Absence of a related work section. The absence of the discussion of related work makes the paper incomplete, which should be added (at least in the appendix).
- The motivation is unclear. The paper combines several techniques and lacks a clear motivation for the necessity of using them together.
- The baselines compared are weak. Table 1 compares various self-supervised and fully supervised methods while lacking sufficient comparisons with sparsely supervised work which is more related (e.g., [1][2]).

[1] Learning Cross-Representation Affinity Consistency for Sparsely Supervised Biomedical Instance Segmentation. ICCV, 2023.
[2] Sparse Object-level Supervision for Instance Segmentation with Pixel Embeddings. CVPR, 2022.

QUESTIONS:
see Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduce ϵ-Seg, a method based on hierarchical variational autoencoders, employing center-region masking, sparse label contrastive learning, a Gaussian Mixture Model prior, and clustering-free label prediction to handle electron microscopy  images of biological samples. The paper claims that center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, and could deal with cases when training labels are sparse. The paper proposes an entropy-based loss that can improve segmentation quality when fewer training labels are available, and employs contrastive learning and a Gaussian Mixture Model prior to shape the latent space of the hierarchical variational autoencoders. The paper also proposes a semantic segmentation head composed of MLP and FiLM layers to directly predict class labels from latent embeddings. Experimental results of the proposed method demonstrate the applicability on fluorescence microscopy data and show that ϵ-Seg is capable of achieving competitive semi-supervised segmentation results on complex biological image data when fewer training labels are available.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has good quality and clarity.But there are so many hyperparameters in the proposed method,as shown in Eq.17,Eq.18, and there is no overall theoretical discussion on the various modules proposed,such as HVAEs, sparse label contrastive learning and  the Gaussian Mixture Model prior.\n\nQUESTIONS:\n1.Is there a unified theoretical connection between the various modules proposed?\n2.In Table 7, it can be seen from rows 3 and 4 that the addition of CL loss does not bring any improvement, while rows 1 and 2 show that CL loss brings a significant improvement. Should the experimental results under the condition of without CL loss but with GMM be reported?\n3.Is there a visualization showing the different segmentation effects brought by adding different modules proposed?\n4.The segmentation results of the proposed method as visualized in Figure 2 and Figure 3 do not appear to have smooth segmentation edges. Is there any analysis or explanation for this?\n\nLIMITATIONS:\nThere are too many hyperparameters to tune, and there is no analysis of the model's complexity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for semantic segmentation of biological microscopy data given only very sparse labels. They achieve this through a hierarchical VAE architecture with GMM prior, masked region inpainting task, classification head, and auxiliary losses including contrastive learning and label entropy minimization. They demonstrate their method on two volume EM datasets and one fluorescence microscopy dataset, producing results nearly on par with fully supervised baselines.\n\nSTRENGTHS AND WEAKNESSES:\n_Strengths:_\n\nThe paper addresses the important problem of segmenting noisy biological data given extremely sparse labels (0.05% of the training data). \nThe authors demonstrate that sparsely-supervised methods can achieve segmentations nearly as good as fully supervised methods on real datasets of different imaging modalities.\n\n_Weaknesses:_\n\nThe improvement over the U-Net baseline is not convincing. In both Figures 3 and 4, the proposed method tends to over-segment the images. In Figure 3, I would argue that the U-Net segmentation could be preferable, since it maintains the short sheet-like topology of the basolateral membrane, as opposed to the proposed method, which produces segmentations that look blobby. Likewise in Figure 4, the boundaries between cells can be seen visually in the first image, but the proposed method merges all the cytoplasms into one continuous region. Similarly, in Table 6, the proposed method underperforms the U-Net baseline on several classes. \n\nAdditionally, the method combines many different learning techniques, without complete ablations to justify their use. In Table 7, if removing the CE loss, how are the predictions being made? An ablation of the entropy loss could also be performed, to show that learning on unlabeled data is useful. You might even consider ablating the inpainting loss itself, and train only the encoder and segmentation head.\n\nQUESTIONS:\n1. Can you clarify the inference of the topmost latent $z_L$? In equation 10, do the parameters of the variational distribution not depend on the class $c$? In other words, I am unclear why the class-conditional posterior in equation 10 and class-marginal posterior in equation 11 are the same.\n2. How do you suggest mitigating the apparent over-segmentation observed in the images?\n3. In Table 6, it seems that the gain in average DSC comes primarily from the failure of the baseline to segment OBC and CBC. Are these the least common classes? If so, could you have achieved the same result through a simpler model with some sort of class-weighting of the training distribution or loss?\n4. What is being learned from the unlabeled data? The inpainting, classification, and contrastive learning objectives all require labels. This leaves the entropy and KL losses that can be applied on unlabeled patches, but it is not clear why this would help the model. How does the performance change when you ablate the entropy loss?\n5. In Table 7, for the cross-entropy loss ablation, how are classes predicted without the classification head? Is it via clustering?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of semantic segmentation for EM volumes, particularly from the perspective of low amounts of available training data.  The proposed method for tackling this problem consists of a hierarchical VAE, using a Gaussian mixture model prior with one component per semantic class to be labeled.  The HVAE is applied to an inpainting task of reconstructing masked center pixels.  Additional components to the model consist of using a segmentation head to directly perform semantic classification using Gumbel-Softmax, adding an entropy-based loss for patches without a ground-truth label, and adding a contrastive loss.  The proposed method is then evaluated on three data sets, two from volume EM and one from fluorescence microscopy.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n\n- the paper tackles an important problem in EM segmentation, and the proposed method seems to do well on the evaluated data sets relative to the considered baselines\n\nweaknesses:\n\n- it seems like a potentially large limitation that the proposed method operates on 2d data, considering that FIB-SEM allows for isotropic 3d volumes like the studied \"BetaSeg\" data.  it would have been good to know the performance of a 3D unet model on this data\n\n- the u-net performance on sparsely supervised data is generally quite close to the proposed $\\epsilon$-Seg.  it would be informative to know its performance in the relative labeling factor table 4 to get a better sense of how it compares.\n\n- u-net also seems to generally do comparable or better on most classes, except giving poor performance on one or two.  for instance, in Table 6 on the liver FIBSEM data, the listed DSC for the CBC class is 0 - is that correct or a typo?  it seems strange given full supervision performance is high at 0.9.  How was the data sampling done with training u-net on sparse supervision?  was it still given balanced samples/weighting from all classes to avoid under-predicting low-frequency classes?\n\n- the u-net is not making use of unlabeled data, but $\\epsilon$-Seg is via the entropy loss.  it would be good to give a direct comparison of the two methods without the additional unlabeled data, since in theory the same component could also be added to u-net.\n\n- in the table 7 ablation study, it seems like the contrastive loss hurts performance when looking at row three versus row 4, when using the standard normal prior distribution.  it would be good to report performance with the same settings but with the GMM prior, to validate whether the contrastive loss is actually helping.\n\nQUESTIONS:\nsee the weaknesses raised above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}