Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new task: open-set single domain generalization, where only one source domain is available and unknown classes and unseen target domains increase the difficulty of the task. A new method CrossMatch is proposed to solve this new problem. Firstly, auxiliary examples are generated for unknown classes out of the source classes. Then multi-binary classifiers are used to deal with unknown class identification in domain adaptation. Then, the paper proposes a cross-classifier consistency regularization that minimizes the multi-binary classifier's output and one-vs-all multi-class classifier's output. The experiments show the proposed method could largely improve the accuracy for unknown classes in the target domain.
main_review: Strength
+ OS-SDG is an interesting and realistic problem, and the paper clearly described its difference from existing methods.
+ The proposed method makes sense and is technically sound.
+ The experimental results show the effectiveness of the proposed methods.

Weakness
- One concern is that the author maybe could analyze which parts of their approach decrease the known class's accuracy. This could be interesting to know since in some settings like Table 3, adding CM decreases the accuracy for known classes by about 3%. 
- Another thing is that some results in the appendix could move forward to replace the converging figures in Figure 4. 

summary_of_the_review: The paper is well-written and is technically sound and the proposed problem makes sense.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In domain generalization (DG), label set of target domain is that of source domains. However, we might meet the unknown classes in the target domain, which will cause significantly prediction error on such unknown-class data points in the target domain. To avoid this issue, this paper formulates a new problem setting: open-set DG, where the label set of target domain contains the label set of source domains. This paper actually considers a more challenging problem: open-set single DG (OS-SDG) that extends the problem setting of DG to a more general situation.

To address this very challenging problem, this paper designs a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. This paper also adopts a consistency regularization on generated auxiliary samples between multibinary classifiers and the model trained by SDG methods, to improve the model’s capability on unknown class identification. Experimental results on benchmark datasets prove the effectiveness of CrossMatch on enhancing the performance of SDG methods in the OS-SDG setting.

In general, this paper contributes a novel problem setting and a validate solution to this setting. Although some motivations are unclear, the contributions of this paper are enough.

main_review: Pros:

+ If there are unknown classes in the target domain, existing DG methods cannot handle this situation. As a result, existing methods will cause significantly prediction errors on such unknown-class data points in the target domain, making the prediction of a network unreliable. In this paper, they propose a new problem setting and a new method to handle this challenging problem.

+ Introducing multi-binary classifier to the OS-SDG problem seems very interesting, since it may identify the unknown-class region well.

+ This paper is easy to follow. Experiments can partially support the claims made in this paper. A plus should be that the authors also design some baselines to their problem setting, which provides solid baselines to see if the gains obtained by CM are significant.

Cons:

- The presentation should be improved. There are many modules introduced in this paper, however, they are not well-motivated. It is better to explain their intuitions why they can help improve the performance.

- I am not sure if it is necessary to list the contributions in the introduction. Such contributions have been described clearly in intro and abs. It seems that you do not need to restate them.

- Key related works are missing. In the literature, open-set domain adaptation and open-set learning are very relevant topics to your proposed problem setting. They should be carefully reviewed and discussed with your method/setting. In some open-set learning papers, they also consider to generate unknown-class data, which is also a key module of your method.

- In the theory of DG, researchers need to assume the relations between source domains and target domain to ensure that DG can be solved. However, this paper does try any theoretical analysis to their problem setting, which raises my concerns regarding the performance of CM on other datasets. It would be great if some theoretical analysis can be concluded or analysed. I would not like to make this problem can only be addressed by some heuristic methods. 
 
- How many times do you repeat your experiments? I did not see error bar/STD values of your methods. This should be provided to verify that the experimental results are stable.

- One key experiment is missing. Openness experiments should be done to show the performance of your method when unknown classes change.

- Although hs is a new criterion for open-set DA, it is better to include the known acc as well. Besides, I did not see acc_u in Table 3, which should be provided.

- I did not see how the threshold affects the performance of your method. The threshold \mu is a very important hyperparameter. How do you choose \mu in the DG problem? 


summary_of_the_review: In general, considering the significance of the researched problem, this paper might be accepted by the ICLR2022. However, some points should be clarified and strengthened in the revision.


-----POST REBUTTAL-----

The authors have addressed my concerns. Thus, I increase my score from 6 to 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This study tackles a novel domain generalization task, called open-set single domain generalization, in which a model is to be trained with single source domain but needs to generalize well at unseen target domains that may include unseen classes. To solve this problem, the authors extend single domain generalization methods to be able to learn detection of unseen classes by adopting adversarial data augmentation. Experimental results show that the proposed scheme can facilitates the capability of existing methods on detecting unknown-class data.
main_review: Strength

- Open-set single domain generalization is a novel, challenging, but practically important problem setting. Figure 1 is a good summarization on relationship between this problem setting and related ones.

- Experimental results with several benchmark datasets show that the proposed scheme makes it possible to extend existing domain generalization methods to detect unknown-class data in the inference phase.


Weakness

- I have several concerns regarding the design of CrossMatch.

  - Do we need to assume that all target domains share the same label space?

  - In Eq. (7), since D(D_u, D_s) > \rho is the only constraint on D_u, it seems that D_u can freely go far away from the source data distribution, if we take supreme with respect to D_u.

  - Although the objective function for a whole training process is defined in Eq. (7), the loss functions in each stage do not follow this definition.

    - Eq. (8) adopts an additional hyperparameter \alpha.

    - Eq. (9) and (10) adopt totally different loss functions, which are L_sdg and L_unk.

  - Considering the motivation of cross-classifier consistency regularization, it would be better to stop gradient for F_b. Is there any reason to minimize L_ccr with respect to F_b?

- Although open-set single domain generalization can be seen as SDG + open-set classification task, the authors only use SDG methods as baselines in the experiments. Naive open-set classification (or out-of-distribution detection) methods should perform good for unknown classes but not for known classes due to domain shift, which highlights the advantage of the proposed method more clearly. I could not judge the significancy of the proposed method from the current experimental results.

- How did the authors tune \mu in the experiments? And is it also used for baseline methods (ERM, ADA, and MEADA)? The accuracy on unknown classes should heavily depend on the setting of \mu.

- Several notations are confusing. I list some in the following.

  - In general, "domain" means a pair of representation and distribution of data [R1], not a dataset. 

    [R1] "A Survey on Transfer Learning," IEEE Trans. on KDE, 2009.

  - The arguments of the loss functions sometimes change in the manuscript.

  - In Eq. (7), it should be better to explicitly describe which data distribution is used to take expectation.

  - In Eq. (13), p_b^i (t=1|x) should be equal to p_b^i.

Minor concerns that do not affect my score

- In Table 4, L_unk seems to have more impact on performance than L_ccr, though classifier consistency regularization is only included in the title.


summary_of_the_review: Open-set single domain generalization is an interesting and important problem setting. However, the proposed method is not approapriately designed for this problem setting. In addition, the experiments and discussion lack an important related work, which is open-set classification (or out-of-distribution detection). I vote for "weak reject."
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method called CrossMatch for open-set single domain generalization where only one source domain is available to train the model. 
This problem studied here is interesting and sounds reasonable, which is also closely related to open-set domain adaptation and single domain generalization.
In particular, CrossMatch designs a new strategy to generate auxiliary samples for unknown classes and develops a novel consistency regularization to help identify samples from unknown classes in target domains.
Results on several datasets verify that the proposed method achieves impressive results on three widely-used datasets.
main_review: strengths:
1. this paper proposes to study a new and interesting problem, open-set domain generalization

2. experiments show that CrossMatch improves previous domain generalization methods like ADA (NeurIPS, 2018) and MEADA (NeurIPS, 2020) and obtains impressive results on three object recognition datasets

weaknesses:
1. novelty of the proposed method sounds somewhat incremental compared with prior works in DG (Volpi et al., 2018; Zhao et al., 2020a) and UDA (Liu et al., 2019; Saito & Saenko, 2021)

2. the intuition behind L_{ccr} in Eq.(14) is hard to understand, how does matching the outputs of the one-vs-all classifier and the multi-class classifier benefit the generalization ability? The results of the variant "L_unk^'+L_unk" are missing in Table 4.

3. Also, the results of open-set domain adaptation are missing in the experiments, which are vital and could be considered as the upper bound of this new studied problem setting.


Typos:
1. around Eq.(12), L_{consis} should be L_{ccr}
summary_of_the_review: This paper proposes to study a new and interesting transfer learning setting called open-set domain generalization and develops a new method (CrossMatch) for this problem. However, CrossMatch is mainly built on several previous methods, e.g., the multi-binary classifier (Liu et al., 2019; Saito & Saenko, 2021), adversarial data augmentation (Volpi et al., 2018; Zhao et al., 2020a), making the overall novelty incremental for ICLR. Thus, I tend to give a "borderline reject" score.
review_title: null

Meta-review: The paper presents a new problem: open-set single domain generalization, where only one source domain is available and unknown classes and unseen target domains increase the difficulty of the task. To tackle this challenging problem, this paper designs a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier. CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy. Then, the paper proposes a cross-classifier consistency regularization that minimizes the multi-binary classifier's output and one-vs-all multi-class classifier's output. 

The proposed OS-SDG is an interesting and realistic problem. However, since it is way more challenging, the optimal solution to it remains elusive. Some reviewers think the method might be heuristic and lack theoretical guarantees. Nevertheless, the results are promising and the paper makes a first step toward the challenging OS-SDG problem. Another concern is that the CCR loss needs more ablation studies to further analyze its role. Though the authors have added more explanation of this part, I suggest the authors put more ablation studies in the final supplementary document. 

Overall, the paper is novel and interesting.  I would recommend acceptance of this paper given its novelty and impressive performance, but I highly suggest the authors add more ablation studies in the final supplementary, as suggested by the reviewers.
Decision: Accept (Poster)