Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a data augmentation method that separates the content and style of images to solve the OOD (out-of-distribution) problem. To this end, the content and style features of the image are causally separated. By keeping one of the two features similar to the original image and widening the distance of the other features, benign OOD and malign ODD data are generated, respectively. Experimental results show that the proposed algorithm is applicable to various OOD problems.
strength_and_weaknesses: This paper is well-written and interesting. The formulations used in this paper are novel. Various organized experiments show that the proposed algorithm works practically well in various situations and datasets and, in most cases, shows SOTA performance. It was good to show the effect of using benign and malign samples in an ablation study.

Despite the many advantages of this paper mentioned earlier, I am puzzled by the qualitative results of data augmentation. According to Figure 4, despite the data augmentation, the image content does not seem to change significantly between positive and negative. If more qualitative results were provided, it would be better to understand the operational consequences of data augmentation. Also, I wonder if the augmentation algorithm will work well for higher-resolution images (e.g., ImageNet image examples).

Comparing the effects of two data augmentation methods not covered in this paper (maintaining both content and style or moving them away) would also be helpful in an ablation study.

Simple error: In chapter 3.1, on page 3, the operator for a conditional independent is duplicated twice.
clarity,_quality,_novelty_and_reproducibility: his paper is good enough in all four respects.
summary_of_the_review: Overall, I read this paper as interesting and positive for acceptance. Adding some qualitative and quantitative experiments may further improve the quality of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new approach called HOOD to disentangling the content and style features, and then basing on them, design a data augmentation method to construct benign and malign OOD data. They validate the proposed approach on the tasks of OOD detection, open-set SSL, and open-set DA.
strength_and_weaknesses: **Strength:**

The problem studied in this paper is quite important to the community. The motivation is convincing and the logic is easy to follow. The paper is generally well written. 

 **Weaknesses:**

+ My main concern is about the causal diagrams presented in Fig. 1. 
   - In Fig. 1a, the authors assume that the style $S$ causes the environment/domain $D$. Apparently, it contradicts the common assumption widely used in the literature that the environment $D$ causes the style $S$, *e.g., Magliacane et al., 2018; Suter et al., 2019; Wang and Jordan, 2021; Sun et al., 2021; Lu et al., 2022; Quinzan et al., 2022*. Why do the authors make this assumption? Is there any particular reason for this? How practical is it in comparison with the common assumption?
   - Also, in Fig. 1a, the authors do not assume any dependence between $S$ and $C$, which is quite limited in real world scenarios. One commonly spurious correlation between $S$ and $C$ is due to the existence of the environment $D$, see *Sun et al., 2021; Liu et al., 2021; von Kugelgen et al., 2021*. In other words, we usually have the spurious path $S \leftarrow D \rightarrow C$, which also contradicts the assumed causal diagram in Fig. 1b.
   -  In fact, the caption of Fig. 1a is incorrect. Note that, a SCM is defined as a set of functional assignments accompanying with its corresponding causal diagram. Hence, Fig. 1a is only a causal diagram, no a SCM. 
   - In Fig. 1c, both causal edges and non-causal edges are mixed up, which could be misleading to readers. 

+ Another main concern is about the identifiability. That is, how is it guaranteed, both theoretically and practically, that the content can be distinguished from the style only from the observed data? This is one of the key problems in the thread of work on causal disentanglement, *e.g., Arjovsky et al., 2019; Sun et al., 2021; von Kugelgen et al., 2021*.

+ From the experimental results, it seems that the performance improvement is small, most only having around 1% improvement.

+ notation abusing, e.g., the capitals are used in Fig. 1, whilst the letters are used in the text, etc. 
+ unexplained terms, e.g., 
  - what do you mean by ID data? There exists no explanation before it first appears in the paper. 
  - no explanation on the open-set methods in the introduction.
+ grammar errors, e.g., 
  - the first sentence in the abstract contains two independent clauses, but without a conjunction, etc.
  - on line 6 of the abstract, the indefinite article "a" is missed before "HOOD method".
  - on line 4, "deep models meet with domain shift ..." should be replaced with "deep models meet domain shift"?

**References**

- Suter et al., *Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.* ICML 2019.
- Wang and Jordan. *Desiderata for representation learning: A causal perspective.* 2021.
- Quinzan et al., *Learning counterfactually invariant predictors.* 2022.
- Magliacane et al., *Domain adaptation by using causal inference to predict invariant conditional distributions.* NeurIPS 2018.
- Lu et al., *Invariant causal representation learning for out-of-distribution generalization.* ICLR 2022.
- Sun et al., *Recovering Latent Causal Factor for Generalization to Distributional Shifts.* NeurIPS 2021.
- Liu et al., *Learning causal semantic representation for out-of-distribution prediction.* NeurIPS 2021.
- Arjovsky et al., *Invariant risk minimization.* 2019.
- von Kugelgen et al., *Self-supervised learning with data augmentations provably isolates content from style.* NeurIPS 2021. 
clarity,_quality,_novelty_and_reproducibility: + The paper is easy to follow
+ The idea of disentangling the content and the style is not new, and has been widely discussed, see the above references. 
+ Also, using variational inference to estimate them is a common way in the community. 
+ The submission seemingly does not contain enough details to reproduce the results.

summary_of_the_review: My main concerns are on the assumptions over the causal diagrams and on the identifiability, and the experimental results also seems not quite impressive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework that identifies and utilizes benign and malign OOD based on the disentanglement of content and style features. The benign OOD data may contain new styles but hold contents, and so they can help improve style-invariant model training. The malign OODs are vice versa and can confuse OOD applications. The proposed method is designed as a variational inference framework based on a structural causal model. Data augmentation based on this decomposition improves performance in three representative OOD applications. 
strength_and_weaknesses: [+] The idea of defining and discriminating benign and malign OOD data based on the disentanglement of content and style is novel and interesting. 

[+] The paper presents a clear explanation of how to harness the ODD data with the proposed framework in several OOD problems. Its effectiveness is validated through various experiments. 

[-] The authors need to provide a more specific model architecture for HOOD for reproducibility. 

[-] It is not clearly presented how well the proposed method disentangles content and style, although its effectiveness is shown in terms of performance at OOD applications. It will help if the authors can provide at least brief validation results on this. Figure 4 is not very informative and quite hard to recognize. 

[-] According to Figure 3, the results seem sensitive to the number of augmented samples (benign OOD data). It is not in line with the assumption that benign OOD data can improve model performance.

[Q1] In Algorithm 1, the usage of unlabeled input data is not well represented. Is it a required input or optional, or depends on the target task? Which part of the pseudo-code does it affect? 

[Q2] The authors explain that the pseudo-labels can be utilized in the case of unlabeled data, but I am curious about how sensitive the result is depending on how the pseudo-labels are obtained. 

[Q3] It will be helpful if the authors can show examples of benign and malign OOD samples using images. 

[Q4] The authors chose to use adversarial data augmentation that adds a learnable perturbation. I wonder if there is any other option for this, and whether it’s optimal. 

[Q5] I wonder if there are more recent models for comparison (e.g., ODIN was introduced in 2017, and other methods shown in Table 1 are quite traditional, except OpenGAN) and how this one performs in comparison with those. 

[Minor]

- The term ID is used without being introduced. 
- p.2:  by the blue and red lines in Fig. 1 -> by the green line and ... 
- p.9 (right above Figure 4) can effective eliminate 

clarity,_quality,_novelty_and_reproducibility: The paper is well organized overall. However, while it uses a lot of notations (e.g., for parameters, or functions), they are not clearly defined before being used and some details are not provided sufficiently for reproducibility. Algorithm 1 could also be improved by directly referring to each input and output component in the main lines. 
summary_of_the_review: This paper introduces a novel view and framework for handling OOD instances, which seems interesting and reasonable. However, more comparisons with recent models and a clearer description of the model details would improve the paper further.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To address three applications of OOD: OOD detection, Open-set SSL and Open-set DA, this work proposes a novel data augmentation method, named HOOD. They analyze the data generation process from a causal perspective, and uses a variational inference network to disentangle the content and style from the input data. Then, they perform data augmentation on the content and style, respectively, to obtain benign and malign OOD samples. Finally, the authors use these augmented samples to improve the generalization of the model. Extensive experiments on three OOD applications demonstrate the effectiveness of this method.
strength_and_weaknesses: Strength:
1. The proposed method can improve performance on all 3 OOD applications.
2. The causal disentanglement method proposed in this paper can separate content and style, which is novel.

Weakness:
1. According to Algorithm 1, each iteration needs to pre-train the VAE, which consumes a lot of computational cost and running time cost. The author needs to give the time complexity of running the algorithm, and the running time of the experiment, and compare it with the baseline methods.
2. The author only carried out three kinds of OOD applications through experiments, but what is the proposed methods in the traditional OOD setting with supervised learning, i.e., the training set and the test set do not belong to the same distribution. And how does this data augmentation method perform compared to other traditional generalization algorithms, such as ERM, DRO [1] or IRM [2].
3. Why use adversarial data augmentation, the author did not give sufficient reasons and proofs. For example, after we can decouple the content and style, why not use random data augmentation, or other data augmentation methods.
4. Whether the adversarial data augmentation can hurt the performance. For example, adversarial training will degrade the performance of the model on clean image.

[1] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

[2] Invariant Risk Minimization

clarity,_quality,_novelty_and_reproducibility: The idea of causal disentanglement is novel.
But authors do not conduct experiment on OOD setting or compare with other generalization algorithms.

summary_of_the_review: Please refer to the “Weakness”.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Based on the collected information from all reviewers and my personal judgment, I can make the recommendation on this paper, **accept**. Here are the comments that I summarized, which include my opinion and evidence.

**Research Problem**

This paper studies the well-defined Out-of-Distribution (OOD) problem. The authors argue that the OOD samples can be further divided into two categories. Based on that, the authors aim to explore the mechanisms of benign and malign OOD data. Specifically, the authors claim that the benign and malign OOD data can be identified from the content and style, where the content means the categorical information and the style includes other influential factors such as brightness, orientation, and color (I believe the authors put all non-content information into this category). In light of this, the authors propose an algorithm named HOOD to achieve this. Furthermore, the authors conduct data augmentation to generate the benign and malign OOD data, where the benign OOD data contain novel styles but hold our interested contents, in contrast, the malign OOD data inherit unknown contents but carry familiar styles.

**Presentation**

The presentation is good.

**Motivation**

The motivation of this paper sounds reasonable to me. The OOD data are intuitively harmful to the learning task. The authors provide some literature to demonstrate the existence of benign OOD samples. Putting the above together, it is reasonable to categorize the ODD data into two categories. 
Further, the authors conduct causal disentanglement from the content and style aspects. The major concern from several reviewers is the causal graph. One reviewer provides some references to support that the style determines the domain, which conflicts with the assumption in this paper that the domain determines the style. In my opinion, I support both. There is no ground truth in this causal graph. Both sides have some supporting evidence (The authors also provide some in their responses). Moreover, it is beneficial to speak different voices to thrive in the community.  

**Philosophy**

The philosophy of this paper can be further strengthened. For example, why disentangle the image from the content and style aspects? It seems that the authors put all non-content information into this category. Do you have the third aspect? Some discussions and explanations are needed. 

Moreover, the authors assume that the benign OOD data contain novel styles but hold our interested contents, in contrast, the malign OOD data inherit unknown contents but carry familiar styles. This point also needs discussion and empirical verification (Please refer to my points in the experiment section). 


**Technique**

The technique of this paper consists of two parts. One is to disentangle the samples from the content and style aspects. Another is to conduct data augmentation to generate benign and malign OOD data. The authors use both the generated benign and malign samples to boost the performance. Specifically, the benign samples increase the training volume, while the malign samples work as a constraint via a one-vs-all classifier. 

**Experiments**

The experimental results are rich and solid in various settings. The authors provide fair comparisons and in-depth analysis of their proposed method as well.

The authors provide the execution time in the response. I encourage the authors to put it in the appendix. 

The authors need to verify whether the samples with unknown contents but carrying familiar styles are malign OOD. It is suggested to include these malign samples in training and expect performance degradation. In the current version, the authors only verify the impact of benign samples.

**Minor**

1. In ‘OOD data$^1$.’, the superscript comes after the period. 

2. Some figures occupy too many top margins. 
justification_for_why_not_higher_score: I launched the internal discussion among reviewers. Only one reviewer had deep discussion with me, who increased the score from 3 to 5. Other reviewers did not reply to my comments.
justification_for_why_not_lower_score: One reviewer has a major concern on the causal graph, who provides some references to support that the style determines the domain, which conflicts with the assumption in this paper that the domain determines the style. In my opinion, I support both. There is no ground truth in this causal graph. Both sides have some supporting evidence (The authors also provide some in their responses). Moreover, it is beneficial to speak different voices to thrive in the community.  

This paper is solid in the motivation, philosophy, technique and experiments. Based on my experience and criteria, this paper deserves the acceptance.
Decision: Accept: poster