Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a loss for domain generalization. The loss is learned by meta-learning on based on RotatedMNIST. The paper evaluate the loss's generalization ability in several other datasets.
main_review: + The idea is interesting.
+ Lots of experiments and ablation is done.
---------------
- The paper lacks an ablation over multi-label binary sigmoid cross-entropy loss. The proposed loss is actually multi-label loss which regards the multi-class problem as a multi-label problem although the label vectors are always in the form of one-hot vectors. Multi-class cross-entropy is a multi-class loss. So it's more fair to compare the proposed loss with multi-label loss, instead of the multi-class cross-entropy loss, as the multi-label loss might be a good baseline for DG.
- The performances in DG is not SOTA and the paper lacks important references:
Self-Challenging Improves Cross-Domain Generalization (ECCV 2020) provides better performance in PACS while authors did not mention this paper in their paper. I doubt this paper can achieve SOTA performances in all the benchmarks.
- The writing and clarity can be improved in the following aspects:
a. the texts in fig. 1, 3, 4 are extremely small. It's better to enlarge them
b. the authors should add more descriptions about how they conducted analysis and experiments. For example, what is "+90%, +80%, - 90%" in table 5. 


summary_of_the_review: This paper lacks evidences to show its SOTA performances and lacks important references and ablation study. I suggest authors to improve from the above points.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors design a scheme for meta-learning loss functions for domain generalization: Based on the RotatedMNIST problem, a new implicit gradient algorithm is used to select a loss function from a chosen parametric family, and then applied to other domain generalization problems. The authors demonstrate improvements over the Mixstyle method from Zhou et al (ICLR'21) on PACS, and improvements over ERM and other methods on DomainBed.
main_review: The paper is well written and well structured. The idea of meta-learning a loss function based on a given domain generalization task is interesting and (to the best of my knowledge) novel. The experimental setup (especially on DomainBed) appears rigorous (but see my comments below).

### Weaknesses:

- The final loss function used is actually not printed in in the paper---or did I miss it? Given that the whole result section is meant to show improvements of using this loss function, it should be added to the main paper along with information about any implementational details needed for re-implementing it.
- Table 2 is missing confidence intervals or standard deviations / standard errors. Otherwise the differences (sometimes as small as .1 percentage points) are hard to judge. For example, the original paper reported 41.8% for CORAL on DomainNet, vs. 41.5% reported here, so there seems to be some variantion. As an alternative suggestion, what about showing the average rank of the method across all domains in a dataset in brackets behind the performance?
- §4.3, Repeatability analysis: ColoredMNIST seems to be a bad task choice for this claim, as all results in the paper essentially fail on the task (which is expected---it was specifically designed to show an advantage of the Invariant Risk Minimization algorithm). I suggest to re-run the experiment in Table 5 on a different dataset.
- The presentation of the results is generally misleading. At many places, section 4 reads more like an advertisement of the method, rather than a neutral assessment and fair comparision of the results. It is good that the full tables are in the appendix to give a better overview of where the method works, and where it fails. For example, on VLCS, CORAL outperforms the method on 3/4 domains, still the average performance is a bit better --- communicating the results in this form is misleading, and needs to be improved (it is simply conditional on the baseline numbers of the different domains). This is not the only example: E.g. in Table 7, the wrong number is bolded for VLCS.
- The conclusion does not match the paper story and should be improved. I highly recommend to write an extra paragraph on the limitations of the presented experiments.
- The paper lacks a good "Figure 1" that outlines the approach. I suggest one of the less interesting analysis figures (E.g. the current Figure 1) to the supplement, in case space is needed for such a figure.

### Minor points

- The plots need work: When printed, labels and legend entries are not readable.
- Tables should not have vertical lines according to the formatting instructions.
- The Table 2 caption should highlight that these results are for a ResNet-50 model.

### Additional questions

- What informed the choice of 0.025 as the signficance threshold?
- Table 2: How is it possible the the p-value for all comparisions is exactly 0.016? Could you clarify what was actually compared? Was the test performed on average performances, or by considering the ranking on individual domains? If you check all individual domains in DomainBed, what is the ranking of the considered methods? Looking at Table 6 suggestions that e.g. CORAL is often close in performance (despite being a quite old algorithm), the only clear improvements are observed on TerraIncognita and to some extend on OfficeHome. 
- Abstract: "enables simple ERM to outperform signficiantly more complicated prior DG methods" --> I disagree with this statement --- the amount of work needed to come up with the loss function is arguably larger than e.g. the CORAL technique. 
summary_of_the_review: The paper proposes a new interesting setting for discovering new loss functions based on a predefined task, and extensively evaluates one "proof-of-concept" loss function derived with this technique.

My main concern is that the authors occassionally oversell their results, along with some issues related to this, e.g. a potential inconsistency in the hypothesis testing done for Table 2 (see above). Realistically, it seems that the method works for a few of the settings, but not all of them---which I think is totally fine, especially in the light of the results in the DomainBed paper, that showed that a lot of methods proposed over the past years actually fail to meaningfully outperform ERM.

The authors should also critically examine their full results again, adapt the language in the paper to be more realistic about the actual merit of the method and the improvements that can be expected. This should be easy to fix by revising the language in the result and conclusion section a bit, and adding a short discussion of the failure cases.

I assign a score of (5) for now, but would be very happy to increase it --- based on the results already in the paper, I think the paper could become an interesting contribution to ICLR, if presented correctly.

---

**Post-Rebuttal comments:**

The authors sufficiently addressed my concerns, and I decided to increase my score to (6), as I think that this work might be an interesting contribution in the space of domain generalization, first and foremost for putting forward the idea of learning loss functions based on an auxilary domain generalization task itself, which to the best of my knowledge this paper provides the first proof-of-concept for.

For the camera ready, I would however urge the authors to carefully re-check the paper again, and tune down claims that are not sufficiently supported by the presented data. I would also consider to remove the statistical analysis (but keep the average ranks--these are informative), cf. our ongoing discussion, I think in the current form it does not add much information to the paper. If the authors include it, the performed test should be well justified in the supplement, which is currently not the case. I would also recommend to run a test applicable to multiple comparisons, and then perform a suitable post-hoc test, instead of the current practice of testing all models with a Wilcoxon test. It should be highlighted that especially for CORAL/SagNet vs. the proposed method, the gains are rather small in some cases.

If the authors add a bit more critical discussion on their method to the paper, I would hence recommend to accept the paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a novel viewpoint in domain generalization, i.e, the loss function search. Specifically, the search procedure is decomposed as a bi-level optimization and solved through implicit function. The author later adopted the Neumann series for memory saving. The extensive empirical results validate the correctness of the proposed approach.

====== Final Update after rolling discussions

I would appreciate the authors for the detailed responses and I have read the rebuttal. Unfortunately, the concern of rigor is still not addressed after discussion and I will maintain the same rating. Below are my *final* remarks (I fully understand your high-level idea), I hope the author will carefully rethink the paper and improve it in the future version.

- In the rebuttal and paper, many terms are not mathematically/rigorously defined, which made me rather confused. These terms even made me doubtful about the generalization properties since they should be rigorously defined and discussed. (see Note 1 for the detailed examples)

- The motivation for choosing the specific loss parameter families is not properly addressed in the paper. From the rebuttal, I finally got the logic/motivation but it was too late. I do think a major revision for fully justifying the motivation can greatly improve the paper. (see Note 2 for the detailed discussions)

---------------------

Note 1.  Examples of unclear terms, most of them are not formally formulated/defined. The descriptive words rather than the math formula made the reviewer confused and a bit doubtful about the generalization property since generalization should be rigorously defined.

  - "Scale of the domain shift is similar enough between meta-train and deployment datasets." What is the formal definition of the scale of domain shift?

 -  “the cardinality of the classification problem is not too different between meta-train and meta-test.” What is the formal definition of the cardinality of classification?

  - “But please note that something like MNIST->SVHN transfer is not exactly the problem setting we address.” It is not domain generalization (trained on digit A and deployed on digits B)? What is the formal (or mathematic) definition of a single source setting? It is best to use equations rather than descriptive words to depict the problem settings. In fact, the single source is a confusing word (since it is not mathematically defined.)

 -  “large search space is expected to lead to meta-overfitting.” What is the formal definition of meta-overfitting? Overfitting w.r.t. which distribution? The author should be cautious to use unclear words since meta-overfitting has not been mathematically defined. 


Note 2. The motivation of choosing the specific loss parameter.

Thanks for the rebuttal. It makes me clear about the motivation. But it requires a major revision of your paper to fully justify it. For example, I would suggest testing the linear combination and showing its results. Also exploring other loss families to fully justify the value of the loss search. In the current version, it is more or less we proposed loss family A (without strong motivation or benefits) and it works in some classification benchmark.  A *comprehensive and systematic* empirical understanding can significantly improve it.  

-------------------------------------------------------

main_review: Overall the paper is interesting since the author provides an alternative perspective in domain generalization: searching the proper loss function.  However, this reviewer still found a couple of important concerns (including theory, method, and paper writing) that prevent me from accepting the current version. Based on these, I believe a major revision and rethinking about the cons can significantly improve the paper.

Pros:

- This paper proposes a novel direction in domain generalization: searching the proper loss. Although this idea is not completely novel in the related domains such as NAS or AutoML, applying them to the domain generalization seems quite interesting and novel.
- The empirical studies and analysis are extensive.

Cons (see detailed reviews for explanations):

- The main concern lies in the theoretical aspects: why and when it works or fails? The proposed idea currently seems a bit ad-hoc. 
- The paper is not self-contained. Some details are quite difficult to follow or lack motivation.

---------------Detailed reviews

**About theoretical understanding.**

I totally agree with the author’s viewpoint about the problem revealed in domainBed. However, in this paper, the author fails to clearly justify why the proposed parametric loss $w$ can effectively solve this problem. Specifically,

1. Does the bi-level optimization indeed search for the correct parametric loss rather than over-fitting? Note in the domain generalization, the source number is generally quite limited (different from NAS), supposing you only have 3 sources, it is quite difficult to show the bi-level indeed finding the best configuration. (It is more likely overfitting). To address this, the author should provide a clear theoretical analysis to show when the proposed method can work. 

2. The selected Taylor Polynomial representation is wired and lacks motivation. Why do we choose this specific loss? What is the convergence behavior in the bi-level optimization if this loss is adopted? Why not MSE loss? Triple loss? This paper lacks a clear motivation in illustrating why it is essential.

3. The experimental settings in the single DG are also wired. Clearly, one source could not generalize to other related targets. Therefore I am a bit doubtful about the proposed method...

**Other technique details**

- Some parts are not self-contained and unclear. For example, 

1. Algo 1, Line 15 there is no description of the grad-surgery approach.

2. Algo1, Line 16. h{update...} what does it mean?

3. In algorithm 2, the description seems inconsistent with the sum of the Neumann series. The role of j is missing in the loop. 

4. Computing jacobian-vector product in PyTorch seems a bit difficult if we directly adopt PyTorch since PyTorch currently does not support the form (algorithm 2) Jacobian-vector-product(parameter of neural-network,\theta,v). I am wondering how you addressed this.

- The convergence bound of using Neumann series or Implicit gradient is highly expected to provide. (I think it is feasible under proper assumptions.)

- There may exist a memory complex concern since the $H$ in the algorithm depends on the task number. If n is large, the memory complexity can be quite high (although the implicit function addressed the computational complexity).








summary_of_the_review: This paper adopts the idea of NAS or autoML to search for the proper loss in the domain generalization. Overall the idea sounds novel in the domain generalization while the current versions lack clear justifications (in terms of theory). Based on these, I do not recommend accepting the current version but encourage a major revision for the resubmission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors introduce a bi-level optimization procedure aiming to learn parametric loss functions that result in predictors able to better generalize to new data sources, unseen during training. Specifically, an outer optimization loop iteratively updates a parametric loss function to minimize the empirical risk (under a standard loss) on a left-out data source, not presented to the model during the inner optimization loop, which trains the model on a set of domains using the current version of the parametric loss. The proposed approach is computationally expensive, but authors empirically showed that losses learned in low-scale tasks directly transfer to other cases, rendering the proposal practical.
main_review: Strengths:

+ The paper is well written and easy to follow, the idea is presented clearly.

+ The idea is novel and interesting; the empirical observation that learned losses transfer to new tasks results in a practical framework.

+ An extensive empirical evaluation is carried out, comparing the proposal against a diverse set of alternative methods and using a number of benchmarks, clearly supporting authors' claims.

+ The proposed loss can be directly combined with other mechanisms to further improve performance; e.g., it could be used along with some invariance-inducing strategy. 


Weaknesses/questions/suggestions:

- It's possible that the proposal requires assumptions that are not discussed in the manuscript. In particular, empirical evaluation is carried out on the more common cases where data marginals shift, but data-conditional label distributions P(y|x) are fixed across domains; i.e., observing x suffices in order to determine y. It's unclear whether the learned losses would still yield improvements in the more general cases where P(y|x) is domain-dependent, i.e. the same x could have different labels depending on the domain it was observed from; C.f. [1] for an in-depth discussion on the effects of conditional shifts. In particular, if the proposal induces some type of domain-invariance, be it in the feature or prediction level, it would likely damage performance relative to standard ERM in such cases.

- The evaluation/discussion lacks in determining how learned losses improve models' robustness; e.g., are resulting models more domain-invariant in some sense? If not, what is it that makes them more robust than models training with standard ERM?

- The fact that learned losses transfer to new tasks is clearly supported by the reported evaluation. However, it would be interesting to get an idea as to whether one could benefit from training a loss tailored to a particular set of domains. In that case, it would be also interesting to get an idea of the computational cost involved in running Algorithm 1 in a larger-scale task such as PACS.

- In section 3, the text makes it a bit confusing to determine what are the authors' original contributions and what are the bits that were introduced previously and re-used in the proposal.

- Please include in the manuscript a brief description of the "gradient surgery" scheme used to aggregate gradients in Algorithm 1. Is there any reason why this particular approach would work better than simply averaging?

- Authors follow the common practice within domain generalization literature and perform evaluations using models pre-trained on ImageNet. However, in my opinion, doing so renders comparisons noisy and it's unclear how much pre-training helps in generalization to new domains. As a suggestion, I recommend repeating a subset of the evaluations without pre-training and determining whether conclusions hold.

- What is the cross-validation criterion used for experiments on PACS?

- Please report the sample size used for the significance test reported in Table 2.

References:

[1] Zhao, Han, et al. "On learning invariant representations for domain adaptation." International Conference on Machine Learning. PMLR, 2019.
summary_of_the_review: The proposal is novel to my knowledge and interesting. It can also be directly combined with other approaches to further improve performance. The execution is robust and covers a number of well-known datasets; comparisons include recent alternative methods. Results include significance tests.
review_title: null

Meta-review: This paper considers the idea of meta-learning the loss function for domain generalization. It's a simple idea, that seems to work reasonably well. Although, as pointed out by the reviewers, the margin is actually quite modest when compared to the strongest baselines (not ERM).

On a positive note, many reviewers agree that the idea was simple, novel, and interesting. The insight that cross-entropy can be improved for domain generalization is interesting. On the other hand, many reviewers pointed out that the, despite some careful empirical work, it's not clear why this idea works. I read the paper myself, and I agree that the paper could use a bit more work before it is ready for publication. Specifically, I agree with Reviewer eZ71, who asked for a clear justification of the proposed idea. The idea seems sensible, but there is some burden on the paper to provide insight, and not simply present an idea.  Here are some specific suggestions that came up during discussion, which could strengthen the paper:
- A more comprehensive discussion of the limitations of this approach.
- It would be good to understand how critical was the specific choice of parametric loss family. Here are some questions that would be good to address: does the parametric family interact with the type of domain shift in the datasets? Why are Taylor polynomials preferable or beneficial for domain generalization compared to, e.g., a linear combination of standard loss functions?
- Is the dataset on which you learn your ITL loss critical? I.e., how critical was the choice of rotated MNIST for learning the ITL loss? Does it generalize to very different and more diverse domain shift tasks, like those in the WILDS benchmark? It would be particularly interesting to see if loss functions meta-trained on distinct datasets learn similar parameters.
- More broadly, evaluation on larger and more diverse domain shift tasks, like those in the WILDS benchmark, would further strengthen the conclusions in the paper.
Decision: Reject