Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce the combination shift problem towards generalization - especially when the goal is to learn with as few combinations are observed in data. Towards this goal, the author extend the definition of disentaglement beyond of invariances to group actions - highlighting the necessity to exploit the equivariance structure. The authors then provide a strategy to effectively learn data augmentations towards the combination shift problem and improve generalization and subsequently provide proof of concept experiments.
strength_and_weaknesses: **Strengths:**
1. The combination shift problem is very well motivated, appropriately formulated and most importantly is an important problem in real world settings 
2. Most of the paper (and appendix) is very well written and clear
3. The proposed algorithm (sub section 4.3) is simple, without losing the majorly desired properties
4.  The proof of concept experiments are well designed and show the use case of the proposed approach

**Weaknesses**:
1. While in definition 1, the authors assume that the product algebra in not necessarily finite (via an indexing set), it is unclear  - (the ramifications) of what happens when product decomposition is not countable. The algebraic regularization subsection - maybe is indicative of this - but currently the clarity of that is on the lower side.
2. It is unclear if there is a strategy to learn the product decomposition (without underlying knowledge about the data - and especially in relation to 1 above). Please include synthetic proof of concept experiments to showcase the same.

**Minor:**
1. Is there a citation for the 1st line of the introduction? From cognitive studies/ neuroscience ?
2.  Work [1] which assumes invariance to all groups initially (unless evidence shows otherwise) and then narrows over the subspace lattice  to identify the groups to be not invariant to is relevant work


**References**
1. Mouli, S. Chandra, and Bruno Ribeiro. "Neural Networks for Learning Counterfactual G-Invariances from Single Environments." ICLR 2021
clarity,_quality,_novelty_and_reproducibility: Clarity - Please see strength 2

Reproducibility - Currently code is not provided

Novelty and Quality - Please see strengths 1,3, 4
summary_of_the_review: In my opinion, the strengths of the paper out weigh the weaknesses - and therefore recommend border line accept (initial review)
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 1. The authors propose to study a problem that they call combination shift: a simplified version of domain generalization in which all environments and labels are available at test time, but not necessarily all combinations.
2. Higgins et al [2018] define disentanglement based on equivariance wrt to the action of a product groups. This paper slightly extends that discussion from groups to more general algebraic structures, in particular semi-groups and monoids.
3. Starting from this definition, the authors argue that one could solve the combination shift problem if the action of domain shifts and label shifts on the input data is known.
4. They point out that this action can be learned from data pairs showing a sample before and after the domain or label has been shifted, but all other aspects are kept constant. They also and discuss learning these actions from unpaired datasets.
5. They demonstrate the approach in experiments.
strength_and_weaknesses: Strengths:
1. Domain generalization is a crucial and largely unsolved problem. Any progress on this question could be very impactful.
2. It is sensible to extend Higgins et al's definition of disentanglements from groups to more general algebras.
3. Phrasing invariance algebraically makes sense. The commuting diagrams illustrate the structure very clearly.
4. The proposed regularizers allow for the practical operationalization of the algebraic structures.

Weaknesses:
1. The authors introduce the combination shift problem clearly. But it would be great if they could shed a bit more light on why this is a relevant problem. Are there any real-world problems that fall into this category? Or do they primarily consider this as a stepping stone to solving the full domain generalization problem? In the latter case, could they argue why progress on the combination shift problem should translate into progress on other domain generalization problems?
2. The proposed method requires either knowing the action of domain and label shifts on the data space (act_X), or learning it from data. The authors correctly point out that such data can be learned from paired data of a sample before and after changing the corresponding factors without affecting any other sample details. But the existence of such data pairs is a strong requirement, and this requirement is not part of the problem setup (nor the introduction of the paper). Learning such augmentations from unpaired data is a much harder problem. The authors gloss over this difference around Eq. (7) and in Remark 3, but it is a crucial point. Without strong assumptions, the actions will in general not be identifiable from unpaired data. Therefore I believe that the entire approach hinges on what we could either call inductive bias or wishful thinking: ML algorithms somehow being able to identify act_X even in the absence of theoretical guarantees. Algorithms without theoretical guarantees are of course fine, but the authors could be more explicit about in which sense the algorithm is expected to work.
3. I don't fully understand the experiment in section 5.1. How are the augmentations (act_X) learned here? Could the authors show the effect of the learned act_X somewhere in the appendix?
4. The writing could at times be clearer.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: Formulating disentanglement and domain generalization through algebraic structure is an excellent approach, and there are some good ideas in this paper. However, in its current form I don't think the paper meets the bar. The proposed approach hinges on identifying the (isolated) effect of domain and label shift on data representations from CycleGANs, and there is neither a theoretical guarantee for this to work nor an appropriate discussion of this step. I encourage the authors to discuss this further and to be clearer about their claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors present a new problem of combination shift (slightly different from the domain generalization problem) where examples from certain domain and label combination pairs are missing in the training data and the hope is to generalize to these examples. The authors discuss this problem with algebraic formulation and present a method that they call equivariant disentangled transformation (EDT). Experimental results highlight issues with some existing techniques and highlight the efficacy of the proposed approach in semi-synthetic settings on several datasets. 
strength_and_weaknesses: **Strength** 
- The proposed variant of the domain generalization problem is well-motivated, highly relevant and to the best of my knowledge seems original 
- Authors have taken a good effort in setting up preliminaries for algebra used in the paper 


**Weakness** 
- There are three major weaknesses of the paper: 
  -  The writing of the paper is complex and hard to follow for someone who is not well-versed in algebraic formulations. Starting in the abstract and introduction authors use terms like "equivariance", and "disentanglement requirements" without defining what they mean. 
  - It is also very unclear if the proposed EDT algorithm follows naturally as claimed in the abstract. While Figure 2 is clear, the cited equations do not naturally seem to follow to me. 
  - The experimental validation of EDT is limited to MNIST-like datasets. Authors are encouraged to simulate combination shift settings on Domainbed datasets [1]. Moreover, authors observe modest empirical improvements over MixStyle making it unclear how the proposed augmentation technique will work in other vision datasets. 
- Due to the algebraic notation used in the paper, most of the paper is spent on setting up the notation and authors come to the combination shift problem very late in the paper (page 6) which is still mixed with a lot of notations that at times gets hard to keep track of. 


[1] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.  
clarity,_quality,_novelty_and_reproducibility: **Clarity and writing suggestions** 
- As mentioned before, the current draft of the paper is hard to parse even after the preliminaries discussed by the authors 
- In my opinion, it is also unclear if the problem of combination shift and the proposed EDT algorithm can not be explained without algebraic notation. 

**Reproducibility** 
- Since the paper presents a novel algorithm and experimental results, it would be great if the authors can add a reproducibility statement. 
summary_of_the_review: Overall, in its current form, the paper is very hard to read even with the preliminaries and setup in Section 3. In particular, it is unclear how EDT is simply followed by their algebraic requirements as claimed in the abstract. Experimental results are limited to small-scale (MNIST-like) datasets with marginal improvements over MixStyle. Authors are encouraged to include results on semi-synthetic in Domainbed datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies domain generalization, but not in its standard setting, in a new "combination shift" setting instead. The paper also proposes an algebraic formulation for the combination shift problem and proposes some augmentation methods. The paper also has some empirical results. 
strength_and_weaknesses: - strength
    - the setting "combination shift" is reasonably new
    - the algebraic formulation is interesting and might indicate some powerful methods in the future. 
- weakness
    - whether the "combination shift" is important needs more discussion. It seems the entire motivation behind this new setting is that this setting is more feasible, which does not seem to be a good reason to study this problem. 
    - the empirical scope of this paper is quite limited. 
clarity,_quality,_novelty_and_reproducibility: - clarity: good
- quality
    - the empirical scope of this paper is probably too limited. 
       - Domain generalization has been studied as a popular topic for multiple years recently, and the authors propose a new setting with no particular reason that the existing methods will not work on it, but the empirical scope discards most of those methods. 
       - the datasets used in the study seems quite small, thus it's unclear about the true performances of the methods. 
   - data selection seems a critical step, it seems unclear how that is done. 
- novelty
    - data augmentation operations have Compositionality and Commutativity seems quite intuitive to the community, although this might be an early time of these topics to be formally discussed. 
    - another highly relevant work for the prediction phase eq. 10
        - Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation
summary_of_the_review: an interesting discussion, but probably needs further discussions (especially on the empirical end) to be considered for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The reviewers agree that the problem of combination shift is well motivated and important, but cited a number of limitations of the work in its present form: the empirical validation is limited to relatively simple datasets and the method is not compared to all relevant previous works, the writing was hard to follow for some of the reviewers, and the method requires knowing the action of domain and label shift, or paired data, which may not be realistic. For this reason I recommend to reject the paper in its current form, but encourage the authors to continue working on this important problem and improving the paper.
justification_for_why_not_higher_score: The empirical validation is limited to relatively simple datasets and the method is not compared to all relevant previous works, the writing was hard to follow for some of the reviewers, and the method requires knowing the action of domain and label shift, or paired data, which may not be realistic.
justification_for_why_not_lower_score: n/a
Decision: Reject