Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the phenomenal that WSSS performance will degrade with dataset biases whee a specific target object frequently appears in the background. To resolve it, the paper proposed an augmentation method that disentangles the target object and background-related features, such that the localization map will more focus on the real foreground. Experimental results show that the proposed module can largely improve the mIoUs on top of various existing approaches.
strength_and_weaknesses: Strength:
+ The author provide a detailed explanation to the unresolved issue for data augmentation of current WSSS approaches (based on CAM).
+ The proposed idea is simple and reasonable. Improvements are obvious and consistent.
+ The two-way swapping idea is also interesting and reasonably improved the performance.

Weaknesses:
- The overall idea is similar to Lee et al. (2021d) despite with different tasks.
- I am not 100% convinced by the paper being motivated as "biased data leading to degraded WSSS performance". My understanding is that the CAM will attend to the background since classification also utilize the background context, so this paper proposed an approach to avoid such "leaking" and to have more accurate foreground seeds. The goal here is very different as Lee et al. (2021d) that de-bias to increase the diversity and improve the generalization. I am not sure if describing it as de-bias is a good choice here.
- There are many unclear presentation that need to be improved.
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: 
There are many unclear presentations:
1. "bias-aligned" is not defined. I have to resort to Lee et al. 2021d for a clearer definition.
2. I am not sure what Figure 1 (b) trying to show even with explanation in the text.
3. How’s the AFA/ABA value calculated in detail and why it shows two regions are correlated?
4. Where is f_{fg} and f_{bg} in Figure 2?

Novelty: many ideas are similar to Lee et al. (2021d), i.e., feature swapping for augmentation.
summary_of_the_review: The paper proposed an augmentation method that disentangles the target object and background-related features, such that the localization map will more focus on the real foreground. The work demonstrates an interesting idea and consistent improvements on existing WSSS approaches. Still, many questions/concerns exists. I am currently in the borderline position, and would like to render a final decision upon the author's response. 

Post rebuttal/Discussion:
The authors' rebuttal/revision has partially address several of my points, including the AFA/ABA descriptions, the formulations of equations. However, there are serval concerns remaining: 1. For the idea, CAM is based for classification which explores both fg and bg. Thus, I am not sure whether the CAM itself can be well-trained if they are randomized with this augmentation.  2. As pointed by other reviewers, the presentation is not perfect and still has potential to be significantly improved. 3. Performance concern, as pointed by sGvK. Thus, I'd lower my rating to 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a feature augmentation method to improve the performance of the WSSS task. The problem with the existing methods is that the pseudo groundtruth for the specific foreground (fg) class also includes information of a highly correlated background (bg). In order to completely separate this background information, the authors divide the features and classifiers for fg and bg and make a loss for each combination. As an augmentation method, these separated fg/bg features are swapped between images in the mini-batch. The proposed method was applied to existing WSSS methods to improve performance.
strength_and_weaknesses: ### Strengths
1. The paper is clearly written and easy to follow with detailed equations and figures.
2. Many analyzes were performed to explain motivation and results.
\
&nbsp;

### Weaknesses

**1. Motivation** \
It is hard to understand the motivation (in Section 3.1) by Figure 1. The authors claim that backgrounds highly correlated with objects are included in pseudo groundtruth. However, it seems that the pseudo groundtruth is not just fine-grained and contains the surrounding area of the object. When we see the CAM results, the values are not high in the background that is far away from the object. For Figure 1a, I guess the reason ABA and AFA have similar values is that the surrounding pixels are included in proportion to the object's size. Also, for the qualitative results of the baseline method in Figure 3, it is difficult to find a case where the background far away from the object is recognized as a foreground class.

**2. Disentanglement loss (Eq. 3)** \
Basically, it is hard to understand why this loss can disentangle the fg and bg features well. As with existing methods, even if a highly correlated background (for a specific class) is included in the fg feature, this loss seems to work properly. Of course, I agree that the proposed architecture has the potential to operate in the ideal direction (as the authors claimed). However, the training method seems insufficient to make this possible.

Additionally, the following questions exist.
- Why is (1-y) the target for the result of f_fg(z_bg)? I think there should be no information for the foreground objects in the background feature, so it is hard to understand why f_fg should predict as 1 for classes that are not in the image.
- Also, why do y_zero or y_one have to be vectors with the same shape as y? f_bg is a classifier that simply predicts whether the input feature is background or not, so it seems the scalar value is sufficient as an output.

**3. Performance comparison** \
Performance comparison with the recent methods is lacking. The mIoU values for the actual WSSS task are summarized in Table 2b, but only two outdated methods are listed. (Also, only IRN (2019) is compared in Table 4.) The performance of the proposed method (68.7) is far behind the SOTA methods below.
- RCA + EPS: 72.8 [A]
- Puzzle- CAM: 72.2 [B]
- SPML: 71.6 [C]

[A] Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation, CVPR 2022. \
[B] Puzzle-CAM: Improved localization via matching partial and full features, ICIP 2021. \
[C] Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning, ICLR 2021.
\
&nbsp;

### Other Comments
“For the evaluation and testing, 1,449 and 1,456 pixel-level labels were used.”: evaluation &rarr; validation
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, so it is not difficult to implement the proposed method even if the code is not provided. The novelty exists to some extent, but performance comparison is insufficient.
summary_of_the_review: There is weak evidence on the motivation and method claimed by the authors. Also, the performance comparison was not made sufficiently. As a result, my initial rating is weak reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a data augmentation technique for weakly supervised semantic segmentation. It is well known that previous weakly supervised semantic segmentation works have an issue of spurious correlation between foreground and background. This paper proposes the augmentation technique by swapping the disentangling features. The proposed techniques improve several previous techniques.
strength_and_weaknesses: Strength: 

+ The paper is overall well-written and easy to understand. 

+ The authors address the important problem in weakly supervised semantic segmentation.

+ The proposed technique improves several previous works. 

Weakness:

- Motivation in Section 3.1 is not convincing for me. 

1) The correlation between AFA and ABA seems trivial, because both values are determined by the scale of network output. The affine parameters of batchnorm determine the scale of activation scale, so AFA and ABA may be determined by those affine parameters. For example, if we split the image into two regions (left-half and right-half) and plot the average activation score of each region, the two scores will be also correlated.

2) In my opinion, the problem in Figure 1(b) is not spurious correlation, but the smoothness of CAM. 

3) If the authors would like to check if the augmentation for the classification task brings misleading correlation, the baseline without any augmentation should be compared.

- The overall structure is too similar to Zhu et al. (2021). In addition, it is not clear the disentangling can be achieved as we intended. For example, if the network thinks the grass is a part of cow, z_fg can be {cow, grass} and z_bg  can be {other patterns}. 

- Following recent works, experiments on the MS COCO dataset should be conducted.

- Minor points

1) If I understand correctly, in Eq (3), f_fg(z_bg) and f_bg(z_fg) should be changed.

2) In the last part of Section 3.2, the sentence "an unintended representation can be learned" is ambiguous. Can you present more details?

3) In Table 4, I recommend the authors to include improvements for all classes. In particular, I want to see the performance change for 'train' class, because 'train' has a significant correlation with 'rail'. 

4) Table 3 seems strange. What is the meaning of (a, b, c, d)? 

5) In Section 3.2, why M channels should be introduced? Can you present the effectiveness of values of M?

6) I want to see the results of AMN+DEFT in Table 2(a).

7) Please consider to include the T-SNE of IRN in Figure 4. 
clarity,_quality,_novelty_and_reproducibility: Please see above
summary_of_the_review: The paper is easy to understand and straightforward. However, my major concerns are 1) whether the disentangling can be achieved as we intended and 2) the marginal contribution over previous work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To alleviate the problem that the classifier confuses the background with the target objects, this paper proposes to disentangle the feature representation between the target object and the background,  and swaps the background representation while the foreground representation is fixed and vice versa. The classifier trained using this augmented representation can generate more accurate localization maps. 
strength_and_weaknesses: Strength:
1. The paper is clear and well-written.
2. The paper is well-motivated.
3. Extensive experimental results showed that when the proposed augmentation method was used in various weakly supervised semantic segmentation methods, the performance of the localization maps and pseudo-mask and the segmentation results improved.

Weaknesses:
1. The description of the dataset is not clear. For example, it is better to clarify whether the results in Table 2 (a) is evaluated on the test set or validation set.
2. The novelty is somewhat weak. There are some works[1,2,3] that use saliency-based mixup data augmentation. Please discuss the differences from these works.

[1] Kim J H, Choo W, Song H O. Puzzle mix: Exploiting saliency and local statistics for optimal mixup[C]//International Conference on Machine Learning. PMLR, 2020: 5275-5285.
[2] Kim J H, Choo W, Jeong H, et al. Co-mixup: Saliency guided joint mixup with supermodular diversity[J]. arXiv preprint arXiv:2102.03065, 2021.
[3] Dabouei A, Soleymani S, Taherkhani F, et al. Supermix: Supervising the mixing data augmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 13794-13803.

clarity,_quality,_novelty_and_reproducibility: The authors should clarify the dataset used for evaluation in Table 2 (a) in detail.
summary_of_the_review: The paper is well-motivated and clear, and the experimental results verify the effectiveness of the proposed augmentation method. However, the authors should clarify the differences from the works.
After a discussion with other reviewers, I decide to change my score to 5 since the idea of this paper is similar to Lee et al.(2021) and saliency-based argumentation methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper initially received borderline scores, and was therefore additionally discussed in a virtual meeting with reviewers. All reviewers agreed that the work addresses an important problem and overall presents a method that is easy to understand. However, multiple reviewers felt that the work does not make significant contribution compared to previous papers pointed out by the reviewers. Some concerns were also raised about the details and reasoning behind the approach, and the experimental validation. Due to these reasons, reviewers converged on a reject recommendation during the meeting and some reviewers also lowered their score. I also concur with the reviewer assessments and recommend rejection at this time.
justification_for_why_not_higher_score: The limited contribution compared to previous work, as well as methods and experiments concerns, were the main factors why acceptance is not recommended.
Decision: Reject