Potts Relaxations and Soft Self-labeling for Weakly-Supervised Segmentation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:02:35

Abstract: We consider weakly supervised segmentation where only a fraction of pixels have ground truth labels (scribbles) and focus on a self-labeling approach where soft pseudo-labels on unlabeled pixels optimize some relaxation of the standard unsupervised CRF/Potts loss. While WSSS methods can directly optimize CRF losses via gradient descent, prior work suggests that higher-order optimization can lead to better network training by jointly estimating pseudo-labels,
e.g. using discrete graph cut sub-problems. The inability of hard pseudo-labels to represent class uncertainty motivates the relaxed pseudo-labeling. We systematically evaluate standard and new CRF relaxations, neighborhood systems, and losses connecting network predictions with soft pseudo-labels.
We also propose a general continuous sub-problem solver for such pseudo-labels.
Soft self-labeling loss combining the log-quadratic Potts relaxation and collision cross-entropy achieves state-of-the-art and can outperform full pixel-precise supervision on PASCAL.

Main paper: 
 introduction: 
 Full supervision for semantic segmentation requires thousands of training images with complete pixelaccurate ground truth masks. Their high costs explain the interest in weakly-supervised approaches based on image-level class tags [21, 4] , pixel-level scribbles [26, 36, 35] , or boxes [23] . This paper is focused on weak supervision with scribbles, which we also call seeds or partial masks. While only slightly more expensive than image-level class tags, scribbles on less than 3% of pixels were previously shown to achieve accuracy approaching full supervision without any modifications of the segmentation models. In contrast, tag supervision typically requires highly specialized systems and complex multi-stage training procedures, which are hard to reproduce. Our interest in the scribblebased approach is motivated by its practical simplicity and mathematical clarity. The corresponding methodologies are focused on the design of unsupervised or self-supervised loss functions and stronger optimization algorithms. The corresponding solutions are often general and can be used in different weakly-supervised applications.Assume that a set of image pixels is denoted by Ω and a subset of pixels with ground truth labels is S ⊂ Ω, which we call seeds or scribbles as subset S is typically marked by mouse-controlled UI for image annotations, e.g. see seeds over an image in Fig. 7 (a). The ground truth label at any given pixel i ∈ S is an integer ȳi ∈ {1, . . . , K}Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.where K is the number of classes including the background. Without much ambiguity, it is convenient to use the same notation ȳi for the equivalent one-hot distribution ȳi ≡ (ȳ1 i , . . . , ȳK i ) ∈ ∆ K 0,1EQUATIONwhere [ • ] is the True operator for the condition inside the brackets. Set ∆ K 0,1 represents K possible one-hot distributions, which are vertices of the K-class probability simplex∆ K := {p = (p 1 , . . . , p K ) | p k ≥ 0, K k=1 p k = 1}representing all K-categorical distributions. The context of specific expressions should make it obvious if ȳi is a class index (1) or the corresponding one-hot distribution (2) .Loss functions for weakly supervised segmentation with scribbles typically use negative loglikelihoods (NLL) over scribbles i ∈ S ⊂ Ω with ground truth labels ȳi- i∈S ln σ ȳi i (3)where σ i = (σ 1 i , . . . , σ K i ) ∈ ∆ K is the model prediction at pixel i. This loss is a standard in full supervision where the only difference is that S = Ω and usually, no other losses are needed for training. However, in a weakly supervised setting the majority of pixels are unlabeled, and unsupervised losses are needed for i ̸ ∈ S.The most common unsupervised loss in image segmentation is the Potts model and its relaxations. It is a pairwise loss defined on pairs of neighboring pixels {i, j} ∈ N for a given neighborhood system N ⊂ Ω × Ω, typically corresponding to the nearest-neighbor grid (NN) [6, 17] , or other sparse (SN) [38] and dense neighborhoods (DN) [22] . The original Potts model is defined for discrete segmentation variables, e.g. as in {i,j}∈N P (σ i , σ j )whereP (σ i , σ j ) = [σ i ̸ = σ j ]assuming integer-valued one-hot predictions σ i ∈ ∆ K 0,1 . This regularization loss encourages smoothness between the pixels. Its popular self-supervised variant isP (σ i , σ j ) = w i,j • [σ i ̸ = σ j ]where pairwise affinities w ij are based on local intensity edges [6, 17, 22] . Of course, in the context of network training, one should use relaxations of P applicable to (soft) predictions σ i ∈ ∆ K . Many types of its relaxation [33, 42] were studied in segmentation, e.g. quadratic [17] , bi-linear [36] , total variation [32, 8] , and others [14] .Another unsupervised loss highly relevant for training segmentation networks is the entropy of predictions, which is also known as decisiveness [7, 18] iH(σ i )where H is the Shannon's entropy function. This loss can improve generalization and the quality of representation by moving (deep) features away from the decision boundaries. Widely known in the context of unsupervised or semi-supervised classification, this loss also matters in weakly-supervised segmentation where it is used explicitly or implicitly 1 .Other unsupervised losses (e.g. contrastive), clustering criteria (e.g. K-means), or specialized architectures can be found in weakly-supervised segmentation [39, 31, 20, 9] . However, a lot can be achieved simply by combining the basic losses discussed aboveEQUATIONwhich can be optimized directly by gradient descent [36, 38] or using self-labeling techniques [26, 28, 27] incorporating optimization of auxiliary pseudo-labels as sub-problems.We observe that self-labeling with hard pseudo-labels y i , which is discussed in the Appendix A, is inherently limited as such labels can not represent the uncertainty of class estimates at unlabeled pixels i ∈ Ω\S. Instead, we focus on soft pseudo-labelsEQUATIONwhich are general categorical distributions p over K-classes. It is possible that the estimated pseudolabel y i in ( 5) could be a one-hot distribution, which is a vertex of ∆ K . In such a case, one can treat y i as a class index, but we avoid this in the main part of our paper starting Section 2. However, the ground truth labels ȳi are always hard and we use them either as indices (1) or one-hot distributions(2), as convenient.Soft pseudo-labels can be found in prior work on weakly-supervised segmentation [25, 41] using the "soft proposal generation". In contrast, we formulate soft self-labeling as a principled optimization methodology where network predictions and soft pseudo-labels are variables in a joint loss, which guarantees convergence of the training procedure. Our pseudo-labels are auxiliary variables for ADM-based [5] splitting of the loss (4) into two simpler optimization sub-problems: one focused on the Potts model over unlabeled pixels, and the other on the network training. While similar to [28] ,instead of hard, we use soft auxiliary variables for the Potts sub-problem. Our work can be seen as a study of the relaxed Potts sub-problem in the context of weakly-supervised semantic segmentation.The related prior work is focused on discrete solvers fundamentally unable to represent class estimate uncertainty. Our contributions can be summarized as follows:• convergent soft self-labeling framework based on a simple joint self-labeling loss• systematic evaluation of Potts relaxations and (cross-) entropy terms in our loss• state-of-the-art in scribble-based semantic segmentation that does not require any modifications of semantic segmentation models and is easy to reproduce• using the same segmentation model, our self-labeling loss with 3% scribbles may outperform standard supervised cross-entropy loss with full ground truth masks. 
our soft self-labeling approach: 
 First, we apply ADM splitting [5] to weakly supervised loss (4) to formulate our self-labeling loss (6) incorporating additional soft auxiliary variables, i.e. pseudo-labels (5) . It is convenient to introduce pseudo-labels y i on all pixels in Ω even though a subset of pixels (seeds) S ⊂ Ω have ground truth labels ȳi . We will simply impose a constraint that pseudo-labels and ground truth labels agree on S. Thus, we assume the following set of pseudo-labelsY Ω := {y i ∈ ∆ K | i ∈ Ω, s.t. y i = ȳi for i ∈ S}.We split the terms in (4) into two groups: one includes NLL and entropy H terms keeping the original prediction variables σ i and the other includes the Potts relaxation P replacing σ i with auxiliary variables y i . This transforms loss (4) into expression- i∈S ln σ ȳi i + η i̸ ∈S H(σ i ) + λ ij∈N P (y i , y j )equivalent to (4) assuming equality σ i = y i . The standard approximation is to incorporate constraint σ i ≈ y i directly into the loss, e.g. using KL-divergence. For simplicity, we use weight η for KL(σ i , y i ) to combine it with H(σ i ) into a single cross-entropy term- i∈S ln σ ȳi i + η i̸ ∈S H(σ i ) + η i̸ ∈S KL(σ i , y i ) + λ ij∈N P (y i , y j ) η i̸ ∈S H(σ i , y i )defining joint self-labeling loss for both predictions σ i and pseudo-labelsEQUATIONbi-linear ∼ "graph cut" quadratic ∼ "random walker" P BL (p, q) := 1 -p ⊤ q P Q (p, q) := 1 2 ∥p -q∥ 2 normalized quadraticP NQ (p, q) := 1 -p ⊤ q ∥p∥∥q∥ ≡ 1 2 p ∥p∥ -q ∥q∥ 2Table 1 : Second-order Potts relaxations, see Fig. 1(a, b, c ) approximating the original weakly supervised loss (4) .Iterative minimization of this loss w.r.t. predictions σ i (model parameters training) and pseudolabels y i effectively breaks the original optimization problem for (4) into two simpler sub-problems, assuming there is a good solver for optimal pseudo-labels. The latter seems plausible since the unary term H(σ i , y i ) is convex for y i and the Potts relaxations were widely studied in image segmentation for decades. Section 2.1 discusses standard and new relaxations of the Potts model P . Section 2.2 discusses several robust variants of cross-entropy H for connecting predictions with uncertain (soft) pseudo-labels y i estimated for unlabeled points i ∈ Ω\S. Appendix B proposes an efficient general solver for the corresponding pseudo-labeling sub-problems.We focus on second-order relaxations for two reasons. First, to manage the scope of this study.Second, this includes several important baseline cases (see Table 1 ): quadratic, the simplest convex relaxation popularized by the random walker algorithm [17] , and bi-linear, which is non-convex but tight [33] w.r.t. the original discrete Potts model. The latter implies that optimizing it over relaxed variables will lead to a solution consistent with a discrete Potts solver, e.g. graph cut [6] . On the contrary, the quadratic relaxation will produce a significantly different soft solution. We investigate such soft solutions.Figure 2 shows two examples illustrating local minima for (a) the bi-linear and (b) quadratic relaxations of the Potts loss. In (a) two neighboring pixels attempt to jointly change the common soft label from y i = y j = (1, 0, 0) to y ′′ i = y ′′ j = (0, 1, 0), which corresponds to a "move" where the whole object is reclassified from A to B. This move does not violate smoothness within the region represented by the Potts model. But, the soft intermediate state y ′ i = y ′ j = ( 1 2 , 1 2 , 0) will prevent this move in bi-linear caseP BL (y ′ i , y ′ j ) = 1 2 > 0 = P BL (y i , y j ) = P BL (y ′′ i , y ′′ j )while quadratic relaxation assigns zero loss for all states during this move. On the other hand, the example in Figure 2 (b) shows a move problematic for the quadratic relaxation. Two neighboring pixels have labels y i = (1, 0, 0) and y j = (0, 0, 1) corresponding to the boundary of objects A and C. The second object attempts to change from C to B. This move does not affect the discontinuity between two pixels, but quadratic relaxation prefers that the second object is stuck in the intermediate state y′ j = (0, 1 2 , 1 2 ) P Q (y i , y ′ j ) = 3 4 < 1 = P Q (y i , y j ) = P Q (y i , y ′′ j ) while bi-linear relaxation P BL (y i , y j ) = 1 remains constant as y j changes.We propose a new relaxation, normalized quadratic in Table 1 . Normalization leads to equivalence between quadratic and bi-linear formulations combining their benefits. As easy to check, normalized collision cross entropy log-quadratic P CCE (p, q) := -ln p ⊤ q P LQ (p, q) := -ln 1 -∥p-q∥ 2 2 collision divergenceP CD (p, q) := -ln p ⊤ q ∥p∥∥q∥ ≡ -ln 1 -1 2 p ∥p∥ -q ∥q∥ 2Table 2 : Log-based Potts relaxations, see Fig. 1(d, e, f ) 1 and 2 : interaction potentials P for pairs of predictions (σ i , σ j ) in ( 4) or pseudo-labels (y i , y j ) in ( 6) are illustrated for K = 2 when each prediction σ i or label y i , i.e. distribution in ∆ 2 , can be represented by a single scalar as (x, 1 -x).The contour maps are iso-levels of P ((x i , 1 -x i ), (x j , 1 -x j )) over domain (x i , x j ) ∈ [0, 1] 2 .The 3D plots above illustrate the potentials P as functions over pairs of "logits" (l i , l j ) ∈ R 2 where each scalar logit l i defines binary distribution(x i , 1 -x i ) for x i = 1 1+e -2l i ∈ [0, 1].quadratic relaxation P NQ does not have local minima in both examples of Figure 2 . Table 2 also proposes "logarithmic" versions of the relaxations in Table 1 composing them with function -ln(1 -x). As illustrated by Figure 1 , the logarithmic versions in (d-f) addresses the "vanishing gradients" evident in (a-c). Shannon's cross-entropy H(y, σ) is the most common loss for training network predictions σ from ground truth labels y in the context of classification, semantic segmentation, etc. However, this loss may not be ideal for applications where the targets y are soft categorical distributions representing various forms of class uncertainty. For example, this paper is focused on scribble-based segmentation where the ground truth is not known for most of the pixels, and the network training is done jointly with estimating pseudo-labels y for the unlabeled pixels. In this case, soft labels y are distributions representing class uncertainty. We observe that if such y is used as a target in H(y, σ), the network is trained to reproduce the uncertainty, see Figure 3 7), (b) reverse (8) , and (c) collision (9) . (d) shows the empirical comparison on the robustness to label uncertainty. The test uses ResNet-18 architecture on fully-supervised Natural Scene dataset [30] where we corrupted some labels. The horizontal axis shows the percentage η of training images where the correct ground truth labels were replaced by a random label. All losses trained the model using soft target distributions ŷ = η * u+(1-η) * y representing the mixture of one-hot distribution y for the observed corrupt label and the uniform distribution u, following [29] . The vertical axis shows the test accuracy. Training with the reverse and collision cross-entropy is robust to much higher levels of label uncertainty.(a) standard H CE (y, σ) (b) reverse H RCE (y, σ) (c) collision H CCE (y, σ) (d) empirical comparisonbetween the predictions and pseudo-labels, but avoid mimicking the uncertainty in both directions: from soft pseudo-labels to predictions and vice-versa. We show that the last property can be achieved in a probabilistically principled manner. The following three paragraphs discuss different crossentropy functions that we study in the context of our self-labeling loss (6) .Standard cross-entropy provides the obvious baseline for evaluating two alternative versions that follow. For completeness, we include its mathematical definitionEQUATIONand remind the reader that this loss is primarily used with hard or one-hot labels, in which case it is also equivalent to NLL loss -ln σ yi i previously discussed for ground truth labels (3). As mentioned earlier, Figure 3 (a) shows that for soft pseudo-labels like y = (0.5, 0.5), it forces predictions to mimic or replicate the uncertainty σ ≈ y. In fact, label y = (0.5, 0.5) just tells that the class is unknown and the network should not be supervised by this point. This problem manifests itself in the poor performance of the standard cross-entropy (7) in our experiment discussed in Figure 3 (d) (red curve).Reverse cross-entropy switches the order of the label and prediction in (7)EQUATION)which is not too common. Indeed, Shannon's cross-entropy is not symmetric and the first argument is normally the target distribution and the second is the estimated distribution. However, in our case, both distributions are estimated and there is no reason not to try the reverse order. It is worth noting that our self-labeling formulation (6) suggests that reverse cross-entropy naturally appears when the ADM approach splits the decisiveness and fairness into separate sub-problems. Moreover, as Figure 3 (b) shows, in this case, the network does not mimic uncertain pseudo-labels, e.g. the gradient of the blue line is zero. The results for the reverse cross-entropy in Figure 3 (d) (green) are significantly better than for the standard (red). Unfortunately, now pseudo-labels y mimic the uncertainty in predictions σ.Collision cross-entropy resolves the problem in a principled way. We define it asEQUATIONwhich is symmetric w.r.t. pseudo-labels and predictions. The dot product σ ⊤ y can be seen as a probability that random variables represented by the distribution σ, the prediction class C, and the distribution y, the unknown true class T , are equal. Indeed,Pr(C = T ) = k P r(C = k) Pr(T = k) = σ ⊤ y.Loss (9) maximizes this "collision" probability rather than the constraint σ = y. Figure 3 (c) shows no mimicking of uncertainty (blue line). However, unlike reverse cross-entropy, this is also valid when y is estimated from uncertain predictions σ since ( 9) is symmetric. This leads to the best performance in Figure 3 (d) (blue). Our extensive experiments are conclusive that collision cross-entropy is the best option for H in self-labeling loss (6). 
experiments: 
 We conducted comprehensive experiments to demonstrate the choice of each element (cross-entropy, pairwise term, and neighborhood) in the loss and compare our method to the state-of-the-art. In Besides, we also compare our soft self-labeling approach on the nearest and dense neighborhood systems in Section 3.3. We summarized the results in Section 3.4. In the last section, we show that our method achieves the SOTA and even can outperform the fully-supervised method. More details on the dataset, implementation, and additional experiments are given in Appendix C.To 7 in the appendix.). Note that it is reasonable to have higher uncertainty around the boundaries.Second, the results prefer the normalized version, which confirms the points made in Figure 2 . Third, the simplest quadratic formulation P Q can be a fairly good starting point to obtain decent results. Additionally, we specifically test H Q + P Q due to the existing closed-form solution [1, 17] . Since the pseudo-labels generated from this formula tend to be overly soft, we explicitly add entropy terms during the training of network parameters and the mIoU goes up to 68.97% from 67.8%.In this section, we compare different cross-entropy terms while fixing the pairwise term to P Q due to its simplicity and using the nearest neighborhood system. The results are shown in Figure 4 . One can see that H CCE performs the best consistently across different supervision levels, i.e. scribble lengths. Both H CCE and H RCE are consistently better than standard H CE with a noticeable margin because they are more robust, as explained in Section 2.2, to the uncertainty in soft pseudo-labels when optimizing network parameters. We also test the performance of using H CCE + P Q with hard pseudolabels obtained via the argmax operation on the soft ones. The mIoU on the validation set is 69.8% under the full scribble-length supervision.Until now, we only used the four nearest neighbors for the pairwise term. In this section, we also use the dense neighborhood and compare the results under the self-labeling framework. Firstly, to optimize the pseudo-labels for the dense neighborhood, we still use the gradient descent technique as detailed in Appendix B. The gradient computation employs the bilateral filtering technique following [35] . For the pairwise term, we use P Q . The cross-entropy term is H CCE . Note that the bilateral filtering technique only supports quadratic pairwise terms, i.e. P BL and P Q . Since P BL leads to hard solutions, P Q is the only practical choice for soft self-labeling. We obtained 71.1% mIoU on nearest neighbors while only getting 67.9% on dense neighborhoods (bandwidth is 100).Some qualitative results are shown in Figure 5 . Clearly from this figure one can see that a larger neighborhood size induces lower-quality pseudo-labels. A possible explanation is that the Potts model gets closer to cardinality/volume potentials when the neighborhood size becomes larger [37] .The nearest neighborhood is better for edge alignment and thus produces cleaner results.In this section, we give a summary in Table 4 6 , we also compare these approaches across different scribble lengths. In this section, we use a different network architecture, ResNet101, to fairly compare our method with the current state-of-the-art. We only compare the results before applying any post-processing steps.The results are shown in Table 5 . Note that our results can outperform the fully-supervised method when using 12 as the batch size. We also observe that a larger batch size usually improves the results quite a lot.  
conclusions: 
 This paper proposed a convergent soft self-labeling framework based on a simple well-motivated loss (6) for joint optimization of network predictions and soft pseudo-labels. The latter were motivated as auxiliary optimization variables simplifying optimization of weakly-supervised loss (4). Our systematic evaluation of the cross-entropy and the Potts terms in self-labeling loss (6) provides clear recommendations based on the discussed conceptual advantages empirically confirmed by our experiments. Specifically, our work recommends the collision cross-entropy, log-quadratic Potts relaxations, and the earest-neighbor neighborhood. They achieve the best result that may even outperform the fully-supervised method with full pixel-precise masks. Our method does not require any modifications of the semantic segmentation models and it is easy to reproduce. Our general framework and empirical findings can be useful for other weakly-supervised segmentation problems (boxes, class tags, etc.). uses a special quadratic formulation where closed-form solution and efficient solver [1, 17] exist.We compute the standard soft Jaccard index for the pseudo-labels between the solutions given by our solver and the global optima. The soft Jaccard index is 99.2% on average over 100 images.Furthermore, our experimental results for all other formulations in Figure 7 , 5, and Section 3 confirm the effectiveness of our optimization solver. In all experiments, the number of gradient descent steps for solving y is 200 and the corresponding learning rate is 0.075. To test the robustness of the number of steps here, we decreased 200 to 100 and the mIoU on the validation set just dropped from 71.05 by 0.72. This indicates that we can significantly accelerate the training without much sacrifice of accuracy. When using 200 steps, the total time for the training will be about 3 times longer than the SGD with dense Potts [36] . 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code will be released upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We reported the best following everyone else.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See end of Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Dataset and evaluation We mainly use the standard PASCAL VOC 2012 dataset [16] and scribblebased annotations for supervision [26] . The dataset contains 21 classes including background.Following the common practice [10, 35, 36] , we use the augmented version which has 10,582 training images and 1449 images for validation. We employ the standard mean Intersection-over-Union (mIoU) on validation set as the evaluation metric. We also test our method on two additional datasets in Section 3.5. One is Cityscapes [13] which is built for urban scenes and consists of 2975 and 500 fine-labeled images for training and validation. There are 19 out of 30 annotated classes for semantic segmentation. The other one is ADE20k [43] which has 150 fine-grained classes. There are 20210 and 2000, images for training and validation. Instead of scribble-based supervision, we followed [25] to use the block-wise annotation as a form of weak supervision.Implementation details We adpoted DeepLabv3+ [12] framework with two backbones, ResNet101 [19] and MobileNetV2 [34] . We use ResNet101 in Section 3.5, and use MobileNetV2 in other sections for efficiency. All backbone networks (ResNet-101 and MobileNetV2) are pre-trained on Imagenet [15] . Unless stated explicitly, we use batch 12 as the default across all the experiments.Following [35] , we adopt two-stage training, unless otherwise stated, where only the cross-entropy loss on scribbles is used in the first stage. The optimizer for network parameters is SGD. The learning rate is scheduled by a polynomial decay with a power of 0.9. Initial learning is set to 0.007 in the first stage and 0.0007 in the second phase. 60 epochs are used to train the model with different losses where hyperparameters are tuned separately for them. For our best result, we use η = 0.3, λ = 6,• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators.Interestingly, a unary decisiveness-like term is the difference between convex quadratic and tight, but non-convex, bi-linear relaxations[33,27] of the discrete pairwise Potts model. 


Decision: This submission proposes a method for weakly-supervised segmentation using scribbles. The submission proposes to use a soft pseudo-labelling approach that jointly optimizes pseudo-labels and predictions.
On the positive side, several reviewers mention that:
1. The paper is well written and easy to understand
2. It proposes a well-motivated pseudo-labelling method, with limitations that are reasonably well-discussed.

At the same time, after the discussion period, all reviewers continued to have concerns about the marginal results and the lack of statistical significance testing. 

In addition, there were concerns about the quantitative evaluation and the authors’ stance that “we leave the comprehensive evaluation on more different tasks and more datasets to the future work”. Reviewers would have liked to see a more comprehensive analysis, and to see some of the experiments in the Appendix moved to the main paper, but the authors were reluctant to do this.

Ultimately, the submission received mixed recommendations from the reviewers. The AC believes that the concern about the statistical significance of the approach is an important one. Given that, as the authors mention, their “work is focused on studying the properties of different relaxations of the standard Potts model [...], as well as different cross-entropy terms”, the experimental validation is key. The AC recommends rejection at this time, and encourages the authors to implement the suggestions of the reviewers to extend the evaluation of this work in its next iteration.

Reviews: review: paper_summary: paper_summary: This paper proposes a new framework for Weakly-Supervised image Segmentation. The main contribution is to use a soft-labelling approach which is considered superior to classic hard labelling because it can keep track of the centanty of the label. Then different forms of second order potts relaxation and cross-entropy are evaluated.
Results show that the proposed approach with its best setting is able to perform better than previous approaches and comparably to a fully supervised approach with only 3% of annotated pixels.
strength_weakness: strength_weakness: \- It is not clear if results are significant. For instance is 1% a meaningful improvement or it can be generated by just noise? It is important to see results on multiple runs with also standard deviation.

\- The motivation about soft-labelling because it keeps information about the label certainty is not clear to me

\- In tab. 5 it is not clear what is the base model you start from. Could you report also the base model with standard cross-entropy and hard potts model? It seems that results are very good because the baseline is already quite good.

\- Authors did not say much about the computational cost of the proposed approach compared to common models.
questions: questions: I wrote some questions in the weaknesses part.
limitations: limitations: Authors did not mention the possible limitations of the proposed approach.
One possible limitation is the need of additional hyper-parameters to tune which can take time. Authors should comment on that.
paper_summary: paper_summary: The work proposes a soft self-labeling framework for weakly supervised semantic segmentation using scribbles. This model-agnostic framework requires only the joint optimization of network predictions and pseudo labels, guided by specific loss functions: collision cross-entropy and log-quadratic Potts relaxations. The design choices are supported by theoretical concepts and experimental results.
strength_weakness: strength_weakness: Method
- The proposed method is tested on DeepLab. Although the theoretical concept should hold for any model, its effectiveness on other segmentation models remains unknown. 
- The work explains the rationale behind design choices from a theoretical perspective, but it does not clarify why these choices lead to specific pseudo-labels from a vision perspective. For instance, in Figure 5, NN successfully segments the bicycle while DN does not. Is this because the bicycle is a minor class? 
- The work does not provide mIoU per category, leaving it unclear whether the method is effective for all categories or just a few major ones.

Minor Writing Issue:
- The legends of the figures, such as Figures 1, 4, and 6, are small.
- Figure 1 (b) contains an extra icon (house).

Related works:
- There are more image-level WSSS works than just [4, 21]. The author should consider including additional relevant works in lines 16-18.

- Weakly-Supervised Image Semantic Segmentation Using Graph Convolutional Networks
- Weakly supervised learning of instance segmentation with interpixel relations.
- Extracting class activation maps from on-discriminative features as well.
- Boundary-enhanced co-training for weakly supervised semantic segmentation.
- Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation
- Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation
- Group-wise semantic mining for weakly supervised semantic segmentation
questions: questions: - Does the same conclusion hold for other segmentation models from an experimental perspective?
- What is the relationship between the visualized pseudo labels/predictions and the specific design choice? In Figure 5, NN successfully segments the bicycle while DN does not. Is this because the bicycle is a minor class? In Figure 7, bilinear tends to over-segment the object instead of under-segmenting it. What is the reason?
- Lines 225-227 mention that the proposed framework can outperform a fully-supervised method with a batch size of 12. Is it also better from a visualization perspective?
- It appears that the performance gap between the proposed framework is much larger on the Cityscapes and ADE20K datasets. Why?
limitations: limitations: I agree with the author that the training time is one of the limitations.
I hope the author can consider my suggestion in the above section to build a link between the design choice and the pseudo labels from the computer vision perspective.
paper_summary: paper_summary: This paper considers semantic segmentation under scribble supervision. The paper studies relaxations of the Potts model and proposes a framework for generating soft pseudo-labels, which benefit over hard pseudo-labels in that they can represent uncertainty. The paper highlights problem cases with two standard relaxations, the quadratic and bilinear, and proposes a normalized quadratic relaxation. Moreover, the paper proposes to use a collision cross-entropy loss between the prediction and the pseudo-labels. Different settings are evaluated experimentally, and the proposed approach is compared to the state-of-the-art.
strength_weakness: strength_weakness: Some details are missing or unclear in the main paper. Considering that the soft self-labeling loss in (6) is a key contribution, it would be useful to include some details regarding the optimization of the pseudo-labels in in the main paper from Appendix B. Additionally, pairwise affinities based on intensity edges are mention at line 42, but it is unclear whether they are used in the proposed approach, see questions.

The proposed approach is only evaluated on a single dataset in the main paper. The results on additional datasets in the appendix should be moved to the main paper to better communicate the empirical findings, as they are easy to miss otherwise. This also raises some confusion about Appendix C which says that three datasets are used in Section 3.5, but results are only reported on PASCAL. What was the reason to exclude these from the main paper?

No error bars. Considering that some settings have fairly similar performance, e.g. in Table 3, standard deviation or similar over multiple runs would be useful.

Some figures have excessively small fonts, e.g. Figures 3, 4, 6.
questions: questions: Are pairwise affinities based on intensity edges used in the proposed approach? If yes, how is w on line 42 defined? If no, how can the model learn to draw the segmentation contours at object borders, which most often correspond to color edges? 

How is the bandwidth of the dense neighborhood defined?

Considering that a larger dense neighborhood size reduces pseudo-label quality in Figure 5, did you try values lower than 25? E.g. 1, 2, 5, 10?
limitations: limitations: The paper does not discuss limitations.

Societal impacts are not discussed.
paper_summary: paper_summary: This paper proposes a method to improve weakly-supervised semantic segmentation using sparse annotations. The authors introduce a Potts relaxation method, which is an extension of the traditional CRF-like methods. The experiments are conducted on the PASCAL dataset.
strength_weakness: strength_weakness: Weakness:

1.	The comparisons with other methods are not comprehensive. Recent advances [1,2,3] in sparsely annotated semantic segmentation are not discussed or compared. Point-supervised is more challenging but it is ignored. The survey in the paper is limited, making it difficult to understand the contributions.

2.	The use of a small dataset like PASCAL may not demonstrate the superiority of the proposed method. Results from larger datasets like Cityscapes and ADE20K (Table 6) should be emphasized and compared with state-of-the-art methods.

3.	Self-labeling is also well employed in many weakly-supervised methods that use image-tags labels. It seems that your method can also applied to them, am I right? If so, experiments on the COCO dataset would be important.

4.	DeepLab is an old-fashioned network architecture. The authors should prove that Potts relaxation can also work on Vision Transformer since ViT has demonstrated the SOTA performances in both fully- and weakly-supervised semantic segmentation. If using ViT without Potts can already get good performances, the relaxation may be not important.

5.	In recent fully- and weakly-supervised semantic segmentation works, CRF-like methods have been discarded due to their computation cost. The efficiency of Potts relaxation (FLOPs) is not evaluated in your paper, which is very important. 

6.	With complex formulation, on the very toy dataset PASCAL, the mIoU only increases by 1% in Table 5 (77.1 to 78.1), while the increased computation cost has not yet been evaluated. 

7.	As stated in the abstract “ … can outperform full pixel-precise supervision on PASCAL”, which is not convincing to me. The fully-supervised performance should be considered as the upper bound of weakly-supervised learning. Such results may be caused by unfair settings. 

8.	The importance of relaxation should be introduced in the abstract since it is your main contribution. In Section 2.1, the two claimed reasons are not intuitive to me. “Manage the scope of this study” seems not a strong motivation. 

9.	With the development of ViT and vision pre-training, the improvements of CRF-based self-labeling become marginal. Thus, I think it is important to evaluate ViT and vision-pretraining (DINO[4]) backbone with your relaxation. I am wondering whether Potts can improve performances beyond these strong backbones.

Overall,  I think it would be better if the authors could conduct more comprehensive experiment comparisons and related work discussions. The motivation for relaxation should be introduced in the beginning.

Refers:

1.	Sparsely Annotated Semantic Segmentation With Adaptive Gaussian Mixtures. CVPR 23

2.	Label-efficient Segmentation via Affinity Propagation. NIPS 23

3.	Modeling the Label Distributions for Weakly-Supervised Semantic Segmentation. Arxiv 24

4.	DINOv2: Learning Robust Visual Features without Supervision. TMLR 24

5. CC4S: Encouraging Certainty and Consistency in Scribble-Supervised Semantic Segmentation. TPAMI 24
questions: questions: Questions:

Please refer to the weakness.

Suggestion:

1. Add more comprehensive comparisons and discussions of related works.
2. Some important experiments and evaluations could be considered
3. The motivation for relaxation could be introduced in the abstract.
limitations: limitations: Yes.

Comments: title: Response from reviewer dJES
Authors' response to the additional comments by Reviewer jiv8
Author's response to the reviewer point 5 (the missed point 8 of the original review)
Author's response to the reviewer points 3 and 4
Author's response to the reviewer points 1 and 2
Response to Authors from Reviewer dJES
comment: We thank the reviewer for taking the time to consider our responses. This is highly appreciated. We also would like to share some more thoughts addressing the last three points.

>In the response, the 1% improvement appears to lack significance, with minimal variation or none at all. This indicates that the novelty compared to previous works is not substantial. Additionally, a 1% enhancement on the Pascal VOC dataset does not represent a significant advancement, especially considering that it is not a particularly challenging dataset.

Regarding novelty... This is harder to argue about since novelty evaluation is naturally subjective. We can only say that in our own opinion (also subjective, of course) the novelty is the property of ideas and concepts, not the results. On the other hand, the significance of our results is that they demonstrate that **complex systems can be outperformed by simple general ideas** that are easy to understand. The "deep" community desperately needs more understanding and our paper contributes in this direction. The value in general unsupervised segmentation ideas studies in our paper is that they can be easily used to design (complex) systems going after SOTA in any practical application. We view our experimental results as a "proof-of-concept" motivating such ideas in general. 

We also believe that Pascal is sufficient for a proof-of-concept in WSSS and most related prior work is focused on it. Its "simplicity" and wide use make it harder to achieve any improvement, particularly because (unlike many SOTA methods) we do not use any tricks. Improving over full supervision (even by 1%) also speaks strongly of the power of our simple general ideas. Moreover, we have not seen many examples where the relative performance on PASCAL is not repeated on more complex datasets. Our results on COCO also confirm that simple ideas typically generalize well (better than over-designed systems).

Of course, this is just our subjective opinion.

>The limitation of Potts not functioning effectively on points is a notable drawback.

We agree that point supervision is a drawback for the standard Potts model (and its tight non-convex relaxations, e.g. bilinear). This was a motivation to study convex relaxations, e.g. quadratic. While such relaxations depart from the geometric properties of the boundary motivating the Potts model, consequently they stop suffering from the corresponding boundary shrinking bias that ruins the point supervision. Such relaxations also have their own "probabilistic" motivation (as detailed in "random walker"). These relaxations also have biases, but they are better suited for point supervision.

These were the very motivations for our systematic study of Potts relaxations. We also tried to identify a sweet spot. Our "normalized quadratic" variant came up as such. It is well-motivated conceptually and works best in practice. It is a recommendation in our conclusions. 

> scribbles may not be as "weak" as initially perceived.

We agree. However, this conclusion does not diminish the significance of all the hard work made by WSSS subcommunity over the last 10 years allowing one to make this conclusion now. It is amazing to know that labeling only 3% of the pixels works as well as labeling 100% of the pixels and that this quality can be achieved using only simple unsupervised ideas like the Potts model. The main practical significance of our study is that we show that Potts model is sufficient (at least for full scribbles) as prior WSSS work (even if comparable in quality) had to use more complex combinations of losses, system modifications, or post processing steps.
We now see that we can improve the clarity about the Potts model definition around line 42 and to emphasize that we use this standard "weighted" Potts model. We also totally agree that we should define the function we use for kernel w. While we use standard w based on image contrast [6,17,22] that improves edge alignment in an unsupervised manner, we have to provide this formula for completeness. That would certainly help the readers.

Thanks for your feedback!
We thank the reviewer for taking the time to carefully consider our response. 

We also agree that the tone about the ground truth on Pascal should be carefully weighted. We will revisit this part.
I appreciate the patience of the authors.

I understand that the authors may feel it is unnecessary to include per-category mIoU. They may also feel it is unfair because many other works do not include it. While this reasoning is acceptable to me, it is just fine. I have decided to give a borderline accept.

Regarding the issue of the ground truth in the Pascal dataset, I think the explanation is helpful for readers. However, since there is no evidence provided in the paper to support this claim, I suggest using a less strong tone when mentioning this explanation.
Apologies for the delay in my response. I truly appreciate the efforts made by the authors, and I am pleased to see that they have incorporated some significant experiments to address the concerns raised. Finally, I would like to improve my rating to 'borderline reject" due to some remaining concerns:

- In the response, the 1% improvement appears to lack significance, with minimal variation or none at all. This indicates that the novelty compared to previous works is not substantial. Additionally, a 1% enhancement on the Pascal VOC dataset does not represent a significant advancement, especially considering that it is not a particularly challenging dataset.
- Upon revisiting TEL [25*], I noted that they have included point-supervised experiments (Table 1). The limitation of Potts not functioning effectively on points is a notable drawback.
- Furthermore, as highlighted by the authors, the superior performance of scribble-supervised learning compared to fully-supervised learning implies that scribbles may not be as "weak" as initially perceived.
Thank you for the answers.

Regarding CRF, I am familiar with the methodology but I do not know all the related work. The source of my confusion was that the formulation between lines 41-42 suggests that there are multiple options in the literature. The equation above line 42 was presented as one possible variant. Thus, my question whether it was used in the proposed approach, which it is now clear that it was. Also, if it is central to the work, the precise definition of w could still be reiterated for completeness, and to capture a broader audience, even if it is standard. Nevertheless, these details are minor.

I have read the other reviews and rebuttals. Overall I retain my original score.
I thank the authors for providing new results. Please give me some time to reconsider the per-category mIoU part.
> I have read the other reviews and feel that some of my concerns are not fully addressed.

We are sorry to hear this and will try to do better in this second iteration. As you noted, some of dJES concerns overlap with yours, but we did not realize you might not see our response to dJES. We will rectify this below, though our screen now looks like the same discussion is copied around. Anyways...

> I am not satisfied with the answer to the concern, "Does the same conclusion hold for other segmentation models from an experimental perspective?" After reading the answer, the impact and improvement on other models is still unknown. Additionally, it looks like other reviewers (dJES) have the same question (q.9).

We did not have any concerns about ViT in our paper as Potts is a general unsupervised loss for segmentation and its properties for training are independent of architecture. We see no technical reason to assume that Potts would not matter for ViT. While it might be a better architecture, it is not smart by itself without training. 

However, your original review and dJES commens motivated us to add ViT experiments. By now we obtained the following results for ViT (vit_base_patch16_224) on PASCAL with full scribbles. We hope it helps to confirm that Potts matters for WSSS with any backbones, including ViT. The results below show that Potts matters for our approach using it during training ViT, and it matters for [3] (arXiv from Marcvh 2024, reference from dJES's review) using dense CRF (a version of Potts) to post-process ViT output.

**batch size 12**:

only partial CE  =>  **74.61%** mIOU (**the baseline [3] suggested by  dJES**)

partial CE + Log Div Potts +  CCE => **80.80%** mIOU   (**our**)

**batch size 16**: 

only partial CE   =>  **75.10%** mIoU  (**the baseline [3] suggested by  dJES**)
			
partial CE + Log Div Potts + CCE => **80.94%** mIOU   (**our**)

**arXiv [3], March 2024 (no batch size reported)**:     

=>  **78.7%** mIoU  (**without CRF postprocessing**)

=>   **80.3%** mIoU  (**with CRF post-processing**)

We will be happy to add these numbers to the final version of Table 5 and discuss the importance of the Potts model across architectures, including ViT. Our final numbers for ViT (with batches 12 and 16) might be higher than the above as we might find better tuning (learning rate, number of epochs, etc) when we have a bit more time.

Another related general observation about the numbers above. It is natural to expect that direct integration of the Potts model as a loss for standard end-to-end training may be a simpler and more stable approach than using the Potts model for postprocessing, as in [3]. The latter requires independent tuning of multiple stages. 

>I believe the concern, "The work does not provide mIoU per category, leaving it unclear whether the method is effective for all categories or just a few major ones," is important. Unfortunately, the authors do not respond to this.

While this is an interesting suggestion, class-specific mIOU is not too common in related WSSS literature. It did not occur to us to collect such statistics primarily because we will have very little to compare against as most of the relevant prior WSSS works (cited in our submission) report only the average over all classes. We hope you will find this excuse acceptable.

>The concern, "Lines 225-227 mention that the proposed framework can outperform a fully-supervised method with a batch size of 12. Is it also better from a visualization perspective?" is also not addressed. dJES has a similar concern in q.7.

About "visualization perspective". If you mean qualitative results, of course, it is easy to "cherry-pick", as common. It is possible even with 1% improvement over full supervision. However, we did not think it was important (given the 1% difference) and preferred to save this space for numerical results. We can reconsider this. 

There is a natural explanation for why WSSS could beat full supervision. We observed that there are sufficiently many wrong labels in the original Pascal ground truth masks. WSSS may avoid overfitting to such errors. This point may be interesting enough to squeeze into the paper, but it is highly speculative, and (similarly to the qualitative improvements above) we are not sure it is worth the space it would require to illustrate all of these. Let us know what you think.

> The authors don't promise to address the minor writing issues, which QTqT also mentions. It is okay to see this in a first draft, but for the camera-ready version, these minor writing issues are terrible.

Sorry. We assumed that the focus of the discussion phase was to answer questions and concerns posted by reviewers. Of course, we always use all the corrections/typos and other minor writing issues found by the reviewers to correct the final version. Sorry if the lack of acknowledgment made it look like we plan to ignore them. Thanks a lot for finding all of these!
>5. Missing response to earlier point 8: “the importance of relaxation should be introduced in the abstract since it is your main contribution. In Section 2.1, the two claimed reasons are not intuitive to me. “Manage the scope of this study” seems not a strong motivation.”

Indeed, we might have overlooked this somehow.

The systematic study of CRF relaxations is stated in the abstract (the sentence stating our contributions, see line 8). We are happy to stress their importance even more.

We can clarify the meaning of “managing the scope of this study.” Note that there are infinitely (uncountably) many ways to relax the standard (discrete) Potts model. Even within the domain of polynomial relaxations, one can choose a polynomial order corresponding to any natural number (we use 2). There are also many different relaxations using the same order polynomials. Moreover, relaxations do not have to be polynomial. Any (convex) combination of different relaxations is also a valid relaxation. 

We had to focus on something and second-order polynomials are the simplest form of relaxation (there are no linear relaxations since one can not fit a plane to 4 values P(0.0), P(0,1), P(1,0) and P(1,1) defining the discrete Potts model). Moreover, bilinear and quadratic relaxations are probably the most standard relaxations in the optimization of pairwise CRF terms, but were not systematically evaluated/compared in WSSS. We discuss their properties in Sec 2.1, many of which are well-known. In Sec 2.1 we use this discussion of limitations and biases to motivate new "second-order" relaxations addressing these limitations.
>3. The essence of my sixth comment lies in the marginal improvements, yet the response shifts the focus to formulation complexity and overlooks this aspect. I maintain that the enhancements remain marginal, as indicated by your response stating that the improvements are less than 1%.

We did not realize that you might not be able to see our response to a similar point by Pwad: “Is 1% a meaningful improvement or just noise?"  Below is a copy of our response to this reviewer. We hope it helps.

**About variation**: While we did not properly collect the variations over multiple runs (the tests are expensive even without this), our informal observations are that this variation is very low (below 1%). Also, it is standard in WSSS literature (and prior works we cite) to repost the best run.

**About improvements wrt SOTA**: Table 5 includes many prior scribble segmentation methods, including those that design specialized complex architectures, see the "architectural modifications" block. It makes the most sense to directly compare our results only with methods using standard architectural backbones (the last block in Table 5) since this constitutes a fair comparison of different loss functions for WSSS, which is the focus of our study. For example, one can easily use such general losses, including ours, to build complex systems or specialized architectures, but this is not the focus of our work studying the basic conceptual properties of a large general class of Potts relaxations.

Indeed, according to Table 5, our method with standard V3+ backbone (16 batches) outperforms the method in [25] modifying V3+ architecture (also 16 batches) only by 1%, which may or may not be significant. However, it may not be a fair comparison since [25] designs a more complex 2-branch architecture. Moreover, their approach has some technical flaws as their training is not guaranteed to converge (their procedurally-defined iterative method does not have a clearly defined self-labeling loss). Such ad-hoc methods typically do not generalize well to datasets other than those for which they were designed (e.g. Pascal in this case).

In any case, it makes more sense to compare our results mainly within the last block where we collected many 12-batch results on V3+ from comparable prior work studying loss functions on standard architectures. We consistently outperform those by at least 2% or more only by using new loss functions and a stronger well-defined optimization algorithm. In this 12-batch scenario we even **outperform the full supervision by 1%**. One of such experiments may or may not be significant, but the consistency of our improvements matters particularly because they come only from simple general loss functions that anyone can use in any system or architecture (simple or complex).


>4. Your explanation regarding the point-supervised experiments does not fully convince me. Identifying your point-supervised experiments in Fig. 6 is challenging, and I am uncertain why Potts does not perform well on points. Considering that TEL conducted experiments on both points and scribbles, which serve as the primary comparison objects in the paper.

The point supervision in Fig 6 and Fig 4 is the leftmost point that is marked 0%, but it means points only (which mathematically corresponds to scribble area of measure zero). We can emphasize this important information. We missed it because this is standard in scribble supervision, but we should state this. 

Points (or very short scribbles in general) are a problem for the original (discrete) Potts due to the implicit “shrinkage” bias (as it minimizes boundary length). This may explain why the 100% scribbles in the standard database are pretty long. In the context of Potts relaxation, our paper discusses that this is the issue for the **tight** relaxations, but non-tight relaxations (e.g. quadratic) do not have this bias - they have other biases. These various biases are discussed in Sec .2.1 and supported by Figs 1 and 2.

TEL experiments in Table 5 are only for 100% scribbles, which does not show the full picture as the results for varying scribble lengths in Fig 6 and Fig 4. We do not see a similar evaluation in [25].
> 1. There are several works in the field of WSSS that have utilized ViT (not just [3]). In reference to my fourth comment, I suggest using pure ViT as a baseline and then comparing it with (ViT + Potts) to assess whether Potts can enhance ViT. **If Potts does not lead to improvements in ViT, its contribution may be marginal**, considering ViT's superior strength compared to deeplab.

Based on your original review, as well as comments from jiv8 and other reviewers, we started running some ViT experiments. By now we obtained the following results on the ViT backbone (vit_base_patch16_224) on PASCAL with full scribbles. We hope it helps to confirm that Potts matters for WSSS with any backbones, including ViT. The results below show that the Potts model matters with our approach using this loss directly during training, and it matters for [3] using dense CRF (also a version of Potts) as post-processing.

**batch size 12**:

only partial CE  =>  **74.61%** mIOU (**the baseline you suggested**)

partial CE + Log Div Potts +  CCE => **80.80%** mIOU   (**our**)

**batch size 16**: 

only partial CE   =>  **75.10%** mIoU  (**the baseline you suggested**)
			
partial CE + Log Div Potts + CCE => **80.94%** mIOU   (**our**)

**arXiv [3], March 2024 (no batch size reported)**:     

=>  **78.7%** mIoU  (**without CRF postprocessing**)

=>   **80.3%** mIoU  (**with CRF post-processing**)

We will be happy to add these numbers to the final version of Table 5 and discuss the importance of the Potts model across architectures, including ViT. Our final numbers for ViT (with batches 12 and 16) might be higher than the above as we might find better tuning (learning rate, number of epochs, etc) when we have a bit more time.

Another related general observation about the numbers above. It is natural to expect that direct integration of the Potts model as a loss for standard end-to-end training may be a simpler and more stable approach than using the Potts model for postprocessing, as in [3]. The latter requires independent tuning of multiple stages. 

> 2. Line 428 only mentions the training time, which can be influenced by various factors. In my fifth comment, I emphasize efficiency, and evaluating FLOPs should only take a few minutes.

We find that FLOPS are common for papers focused on architecture. But, based on our experience, it is not common for WSSS papers, perhaps because they typically focus more on unsupervised or weakly supervised losses. Thus, there is a very limited base for comparison in prior work. One exception for WSSS is a recent arXiv [3] from March 2024 brought up by the reviewer. 

Moreover, it is unclear how useful such a comparison would be since one can report FLOPs only for one iteration, while the number of iterations may very significantly between the algorithms. For some algorithms there are also post-processing steps, e.g.  [3],  that adds different FLOPs for an unknown number of such steps. Thus, we doubt the usefulness of reporting FLOPs. 

In any case, in a good faith effort to address the reviewer’s question, we did compute FLOPS for one iteration of our self-labeling method that combines a forward pass for the network (for deeplab from calflops library) and 200 steps of gradience descent for our pseudo-labeling estimation (we use at each iteration). We could not find FLOPs for the deeplab backpropagation step (for ViT we do not have even the forward numbers).

**185.96** + **59.77** GFLOPs (513 * 513 input size, resnet101 backbone, deeplabV3+)

One useful thing these two numbers show is that the complexity is dominated by network training, not pseudo-label estimation. In any case, why do you think it is important to include these numbers in the paper and what should we compare them with? The numbers in [3] are for a different architecture, making them hard to compare.
Thank the authors for the rebuttal and the comments from other reviewers.

I have read the other reviews and feel that some of my concerns are not fully addressed.

I am not satisfied with the answer to the concern, "Does the same conclusion hold for other segmentation models from an experimental perspective?" After reading the answer, the impact and improvement on other models is still unknown. Additionally, it looks like other reviewers (dJES) have the same question (q.9).

I believe the concern, "The work does not provide mIoU per category, leaving it unclear whether the method is effective for all categories or just a few major ones," is important. Unfortunately, the authors do not respond to this.

The concern, "Lines 225-227 mention that the proposed framework can outperform a fully-supervised method with a batch size of 12. Is it also better from a visualization perspective?" is also not addressed. dJES has a similar concern in q.7.

The authors don't promise to address the minor writing issues, which QTqT also mentions. It is okay to see this in a first draft, but for the camera-ready version, these minor writing issues are terrible.

I decide to lower my score at this moment, but I may adjust accordingly after seeing more discussion.
Thank you for your response. I have thoroughly reviewed all of your responses and the comments from other reviewers. While I can accept some of your responses, there are still some concerns that have not been adequately addressed.

1. There are several works in the field of WSSS that have utilized ViT (not just [3]). In reference to my fourth comment, I suggest using pure ViT as a baseline and then comparing it with (ViT + Potts) to assess whether Potts can enhance ViT. If Potts does not lead to improvements in ViT, its contribution may be marginal, considering ViT's superior strength compared to deeplab.

2.  Line 428 only mentions the training time, which can be influenced by various factors. In my fifth comment, I emphasize efficiency, and evaluating FLOPs should only take a few minutes.

3. The essence of my sixth comment lies in the marginal improvements, yet the response shifts the focus to formulation complexity and overlooks this aspect. I maintain that the enhancements remain marginal, as indicated by your response stating that the improvements are less than 1%.

4. Your explanation regarding the point-supervised experiments does not fully convince me. Identifying your point-supervised experiments in Fig. 6 is challenging, and I am uncertain why Potts does not perform well on points. Considering that TEL conducted experiments on both points and scribbles, which serve as the primary comparison objects in the paper.

5. I did not find a response to point 8; could this be an oversight?

I am inclined to accept responses 2, 3, 7, and 9 partially. I sincerely appreciate your detailed responses. In consideration of the comments from other reviewers, I am willing to adjust my ratings if the remaining concerns are addressed satisfactorily.