Title: Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-Budget Active Learning

Semantic Segmentation

Diffusion Models
Abstract: Semantic segmentation demands dense pixel-level annotations, which can be prohibitively expensive -- especially under extremely constrained labeling budgets. In this paper, we address the problem of low-budget active learning for semantic segmentation by proposing a novel two-stage selection pipeline. Our approach leverages a pre-trained diffusion model to extract rich multi-scale features that capture both global structure and fine details. In the first stage, we perform a hierarchical, representation-based candidate selection by first choosing a small subset of representative pixels per image using MaxHerding, and then refining these into a diverse global pool. In the second stage, we compute an entropy‐augmented disagreement score (eDALD) over noisy multi‐scale diffusion features to capture both epistemic uncertainty and prediction confidence, selecting the most informative pixels for annotation. This decoupling of diversity and uncertainty lets us achieve high segmentation accuracy with only a tiny fraction of labeled pixels. Extensive experiments on four benchmarks (CamVid, ADE-Bed, Cityscapes, and Pascal-Context) demonstrate that our method significantly outperforms existing baselines under extreme pixel‐budget regimes. Our code is available at https://github.com/jn-kim/two-stage-edald.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an active learning algorithm for semantic segmentation for pixel selection. Specifically, it proposes using a diffusion model to estimate pixel-level uncertainty, augmented with entropy. It also leverages a hybrid pixel acquisition function that combines representation diversity and uncertainty for subset selection.\n\nSTRENGTHS AND WEAKNESSES:\n-- Strengths\n\nUsing diffusion models and mutual information for uncertainty estimation is interesting and intuitively motivated.\n\nCombining representation-based and uncertainty-based strategies is an effective choice for selecting pixels, overcoming the limitations of each method individually.\n\nThe experimental results are promising, showing a consistent advantage over baselines, especially in low-labeling-budget scenarios.\n\n-- Weaknesses\n\nThis paper may lack of deeper analysis of the uncertainty mechanism. It would be helpful if the paper could provide an empirical analysis of the mutual information distribution across pixels to validate its effectiveness as an uncertainty metric.\n\nIn addition, it is recommended to provide at least some discussion on how to determine the weighting factor between the diffusion-based term and entropy term in Equation 8 (currently set to 1).\n\nOverfitting could pose a significant challenge when training the model with limited labeled data. It would be helpful if the paper included a discussion on mechanisms employed to mitigate overfitting.\n\nThe representation-based sample selection strategy with RBF kernel primarily builds upon established kernel herding methods, with additional top-K selection at both the image and global levels. Therefore, the methodological novelty of the approach appears limited from this standpoint.\n\nQUESTIONS:\nPlease refer to the items listed in the weakness section. It would be helpful if the authors could provide an explanation to improve the clarity of the paper.\n\nLIMITATIONS:\nLimitations are discussed at the end of the paper. This paper does not introduce potential negative socialtal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel two-stage diffusion-based pixel-level active learning method for semantic segmentation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Overall, a well-motivated diffusion-based active learning method\n- Overall, good metrics, ablation results, and curves \n- The paper is well written and easy to follow\n\nWeaknesses:\n- What is the practical benefit of pixel-level active learning? The cost of having a human annotate particular pixels is potentially higher than region-level methods.\n- The diffusion-based approach seems to be dependent on having a diffusion model trained on imaging similar to that which you would want to segment. Thus, if your dataset lacks a relevant diffusion model, your approach would not be feasible. \n- It would be nice to see what the overall performance of MaxHerding is in the ablation study as a baseline\n- It is very weird to me that DALD performs so poorly but just adding Entropy to it produces the best result\n- It seems like all the entropy-based methods are utilizing the diffusion model backbone? This assumes that the diffusion model-based approach is better than other approaches. It would be good to see the baseline algorithms with other backbones, at least compared to the novel diffusion model approach. Also it seems like there are no diversity-based algorithms compared against, like Coreset.\n- There is no comparison against other diffusion model-based active learning approaches.\n\nQUESTIONS:\n- What is the overall performance of MaxHerding for the ablation study?\n- Can you explain why DALD performs so poorly, but eDALD (just adding an entropy term) performs so much better?\n- Is the diffusion-based approach better than other approaches? Please provide comparisons with different backbones for entropy-based methods and results on diversity-based methods.\n- How does the novelty of your approach compare against other diffusion-based AL approaches?\n\nAddressing these concerns would move my score to borderline accept.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies active learning for semantic segmentation using features from a diffusion model to select pixels of an especially high content of information. In a first step, the authors let a diffusion model noise the images in the data set. From this, the inner features of the score vector field  defining the drift in the diffusion process are read out at the respective position of the noised image. As this is realized with a fully convolutional architecture, this is possible. Using these features, a classifier network is trained on the data set of pixels that have been uncovered so far. This classifier is then used to select the point that improves the coverage of the unlabeled features (in the sense of maximum similarity in feature space) the most. (representation based selection) or the point that improves the mutual information between the label of the new point and its feature vector the most. This expression can be approximately evaluated using the classifier, see eq. (5)--(7). This is first done via  a preselection of pixels within one image followed by second round of competition between all selected pixels. From the 2nd competition, the number of pixels is chosen that corresponds to the round budget. Thereafter the model is retrained and a second, third... round is started until the procedure finally is terminated.\nThe authors provide numerical tests that operate on an extremely low budget of pixels, in average 1px per image in total over 10 rounds. They compare their two methods with simple and more elaborate baselines and find significant improvement over naive baselines and limited but consistent improvements over more elaborate baselines (mesured in mIoU). The tests are repeated fora number of data sets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n* The paper impresses with a method that operates on an extremely low pixel budget of in average 1px/image\n* To invoke diffusion features for AL is an original idea. \n* The authors consistently gain performance over baselines\n* The paper is well written and carefully prepared\n\nWeaknesses\n* The technical relevance of this paper is somewhat limited, as segmentation click-work  labels are produced by polytopes, not by labeling single, where each edge of a polytope affords one click plus one click for the class label. These clicks nowadays are significantly reduced using models like SAM, where the polytope contours are suggested and only need to be slightly improved. Hence a practical comparison would be on click basis, here one pixel would only require one class click. It would have been of interest to see click based comparisons with non pixel based AL for SemSeg methods as well.\n* The mIoUs achieved <= 40% seem to be too low for most practical purposes.        \n\nMinor remarks:\n*l. 68: which outliers,describe precisely.\n* l. 158 the set notation at the end of the line is inprecise\n* l. 185 is M the same as b?\n* Table 1: mention the interpretation of numbers (mIoU?!) in the caption.\n\nQUESTIONS:\nQ1: AL is often evaluated at budget for 95% performance of the model trained on the full data set. Would this AL procedure be able to go there?\nQ2: Did you test the assumption that Y is conditionally independent of X given Z, e.g. by a classification experiment with and without X and a likelihood ratio test? (see lines 201->202) What consequences do you see for (5)-(7) if the effect of X is significant?\nQ3: What is the outcome of a click-cost based comparison of your methods with region based methods?\nQ4: in l. 152 - on which data sets is the frozen diffusion model trained - are these disjoint from your AL set?\nQ5: l. 224 Why there is no weighting factor between I and H?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of low-budget active learning for semantic segmentation, where only a tiny fraction of pixels can be labeled per round. It introduces a two-stage selection pipeline that leverages a pre-trained diffusion model to obtain rich multi-scale pixel representations.\n\nStage 1 (Representation Filtering) uses MaxHerding to pick a small set of diverse candidate pixels per image, then refines these across the dataset to form a global pool.\n\nStage 2 (Uncertainty Scoring) computes an entropy-augmented disagreement score (eDALD) over noisy diffusion features-combining mutual information (BALD-style) with a single-sample entropy term-to select the most informative pixels.\n\nExtensive experiments on CamVid, ADE-Bed, Cityscapes, and Pascal-Context under extreme pixel-budget regimes demonstrate substantial mIoU gains (e.g., +10.94 pp over DALD alone) and outperform prior pixel-level baselines by margins up to 21 mIoU points.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\n\n**Strength**: solid methodological design combining diversity and uncertainty; comprehensive experimental evaluation across four benchmarks.\n\n**Weakness**: no statistical significance (e.g., error bars or multiple seeds) reported for mIoU improvements; computational cost of extracting multi-scale diffusion features and MaxHerding at scale is not analyzed.\n\n# Clarity\n\n**Strength**: clear exposition of the two-stage algorithm with pseudocode (Alg. 1) and illustrative figures; well-structured related work and motivation.\n\n**Weakness**: some implementation details (e.g., choice of kernel bandwidth $\\sigma$ in RBF, hyperparameters for MaxHerding) are only briefly mentioned; description of how batch formation for adapted baselines is carried out could be clearer.\n\n# Significance\n\n**Strength**: practical framework for extremely low-budget pixel-level active learning in segmentation; demonstrates consistent, large gains.\n\n**Weakness**: strict reliance on a diffusion backbone limits applicability to settings where such models are unavailable.\n\n# Originality\n\n**Strength**: novel integration of diffusion-model features into active learning; introduction of entropy-augmented DALD (eDALD).\n\n**Weakness**: the two-stage idea echoes previous coverage-then-uncertainty approaches; the main novelty is the use of diffusion features rather than core AL strategy.\n\nQUESTIONS:\n1. Could you report standard-error bars (over multiple random seeds) for mIoU to confirm that the observed gains are statistically significant?\n\n2. What are the wall-clock times and memory costs for diffusion-feature extraction and MaxHerding on large datasets?\n\n3. How sensitive is performance to choices of K (candidates per image), M (global pool size), and the number of diffusion timesteps T? Some crucial ablation studies are missing.\n\n4. Can the two-stage eDALD approach be applied with non-diffusion backbones (e.g., ViT, CNN) by replacing diffusion features?\n\n5. Since segmentation quality often hinges on boundary accuracy, have you analyzed whether selected pixels concentrate on object edges or uniform regions, and how this affects final segmentation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-14"}