Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on multi-domain long-tail classification, where each domain has its own long-tailed distribution. To solve this problem, the authors propose a novel balanced augment method TALLY, which separates and reassembles class-relevant semantic information and domain-associated nuisance information. Specifically, first, the representation of each instance is divided into semantic representation and nuisances. Then the semantic information and domain-associated factors are separately enhanced using argument techniques to improve stability and reduce noise. Finally, the semantic information and nuisances are reassembled to generate new instances for training. Several experiments on four synthetic multi-domain long-tailed datasets and two real-world datasets indicate that the proposed methods improve the overall performance for subpopulation shift and domain shift compared to two types of baselines.

strength_and_weaknesses: This paper focuses on a less-studied problem and defines the problem formally. With an example on the real-world dataset and some discussion, the authors illustrate why it is not trivial to explore the problem. Unlike single-domain long-tail classification, the authors figure out that multi-domains exist with their own long-tail distributions, and previous approaches may exacerbate class imbalance in this scenario.
For the implementation details of the proposed method, the authors give good motivation. For example, in proposing selective balanced sampling, the authors illustrate by figure the drawbacks of balanced sampling from all domain-class groups and how selective balanced sampling solves this problem.
The proposed approach is relatively novel. Although some of the techniques are derived from existing work, such as the representation disentanglement and reassembly section, the authors provide some new technical insights.
The experimental results broadly validate the authors' claim that TALLY is more effective than the existing single-domain imbalanced classification and domain-invariant learning methods under subpopulation shift and domain shift tasks. In addition, sufficient experiments are carried out to demonstrate the importance of each component in TALLY.
However, my main concerns are some details in the model and experimental part of this paper. Here are the detailed comments:
[Presentation] The paper's writing and statements are easy to follow. However, I have some minor suggestions. (1) Since one of the main contributions of the paper is to formalize the multi-domain long-tailed learning problem, clearer notation should be given to help future works better understand the problem and better use these notations. For example, the meanings of some symbols are explained but remain complex, considering the different superscripts and subscripts. (2) Also, some definitions should be better given in more obvious places, such as class-agnostic statistics u_d^{(0)} and v_d^{(0)}.
Experimental results demonstrate the effectiveness of the proposed method, but some theoretical analysis can be given to shed light on the algorithm designs of TALLY and inspire future work.
[Experiments] On domain generalization benchmarks, overall performance is used to measure the performance of subpopulation shift and domain shift; on naturally imbalanced multi-domain data, macro F1 score and average accuracy are used. However, I am not sure of the reason for using these measures. Considering that we are exploring the multi-domain long-tailed problem, should some evaluation metrics that measure tail performance be considered?
[Reproducibility] Although the critical parts of the method are carefully clarified with the pseudo algorithm, the code of this paper is not given, which limits the reproducibility of this paper.

clarity,_quality,_novelty_and_reproducibility: The writing and statements of this paper are easy to follow. The authors focus on an essential problem of multi-domain long-tail learning with good motivation and further propose a novel TALLY algorithm with balanced representation reassembly to solve this problem. Nevertheless, the code of the proposed method is not given, which limits reproducibility. 

summary_of_the_review: This paper identifies a critical problem, multi-domain long-tail learning, and proposes a novel TALLY method to address this problem with reasonable motivation and some novelty. The experimental results on synthetic and real-world datasets illustrate the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-domain long-tailed recognition method to simultaneously address the two types of distribution shift: subpopulation shift and domain shift. The basic idea is sample augmentation where new samples are generated by representation disentanglement for tailed class. The experimental results on synthesis and real-world multi-domain long-tailed data show its effectiveness.
strength_and_weaknesses: Strengths: 
--the paper focus on a general problem: the multi-domain long-tailed learning.

--The motivation is clear. The solution for multi-domain long-tailed learning is data augmentation by representation disentanglement, where any sample is synthesized with class-relevant semantic information and domain-associated nuisances. 

--This paper is well written.

Weaknesses:

--The novelty is limited. Representation disentangle and date augmentation are the common approaches for long-tailed recognition, and no novel approach is proposed for the specific issue of multi-domain long-tailed recognition.

--Some details are questioned. For example, why the representation of each examples can be decoupled into semantic information and nuisances, and the domain information is contained in nuisances? 

--The construction of prototype is too simple to ineffective. I think the updating of prototypes is needed.
clarity,_quality,_novelty_and_reproducibility: the originality is there, but novelty are limited.
summary_of_the_review: the proposed method is effective and interesting, but some details are missed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a balanced sampling data augmentation technique, for addressing the multi-domain and long-tailed learning problems in which the number of samples in different classes is imbalanced and potential domain shift. First, the class-specific and domain-specific representations are decoupled by instance-normalization (Ulyanov et al., 2016),  thus new domain-class pairs can be formulated by a swap operation similar to style transfer (Huang and Belongie, 2017). Moreover, prototype vectors are computed and mixup techniques are used to make the class and domain representations more stable and diverse. The proposed approach is compared against single-domain long-tailed learning methods and domain adaptation or generalization methods and evaluated under two settings. (1) when all classes in test data are balanced (2) when test data is from an unseen domain. Performance improvements are demonstrated especially on synthetic datasets.  The ablation study demonstrated that the proposed approach works better on small-size classes and can learn more domain-invariant features.  
strength_and_weaknesses: The paper studies the long-tailed learning problem in the multi-domain setting. Unlike traditional methods that can only upsample or downsample using data from the same domain, the proposed approach is able to borrow representations cross-domain, so the model can see data of more variety with data augmentation. To achieve this, this paper combines a few effective techniques, including instance normalization, moment matching, prototype, and mixup. Experiment results show improvements against many baseline methods on both synthetic and real-world datasets. 

The difference between the selective balanced sampling strategy and balanced sampling is unclear. I read Section 3.2 and “Analysis of sampling strategies” but still confusing. Figure 3 does not do a good job on explain the difference either.  What is the difference between y, d ~ Uniform(C, D) vs y~ Uniform(C) and d ~Uniform(D)? Can you explain more about the transition probabilities in Figure 3?

For the evaluation protocol, it is unclear what exactly is held-out? It seems in the long-tailed learning setting, all the classes and domains (except for one) are seen. It is very important to make sure there is no information leakage from training to test sets, and that the comparisons are fair. 

The paper considered a wide range of baselines, including both domain adaptation and domain generalization methods. It would be helpful to explain the underlying assumptions of different methods, such as whether labels or data from the target domain is needed. Is equation (4) possible for a new unseen domain?
clarity,_quality,_novelty_and_reproducibility: The concept of subpopulation shift is confusing. Conventionally, it refers to the setting that the supclass remains the same but the subclasses comprising each supclass are different across domains. For example, 

Santurkar, S., Tsipras, D., & Madry, A. (2020). Breeds: Benchmarks for subpopulation shift. *arXiv preprint arXiv:2008.04859*.

It seems here it means training domains are imbalanced but the classes in the test domain are balanced. This seems to be a convention in long-tailed learning, although may not be realistic. It is more related to the long-tailed setting, rather than the subpopulation shift in domain adaptation. Would it be better to report performance vs class size such as in Figure 4?  

How does the presented work compare to a simpler approach: train on imbalanced and fine-tune on balanced datasets? For example, domain-specific transfer learning? 

Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S. (2018). Large scale fine-grained categorization and domain-specific transfer learning. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 4109-4118).

What does TALLY stand for?
summary_of_the_review: This paper presents an interesting combination of ideas for the multi-domain long-tailed setting and achieves improved results over many baselines. However, the sampling scheme is unclear, and I am not sure whether the evaluation protocol is fair. I’ll vote for borderline reject. I will re-evaluate based on the authors' responses.

######
After rebuttal: Thank you for the response and additional experiments during the rebuttal. Most of my questions have been clarified. I have updated my score to reflect this. Two additional comments: 

- Selective balanced sampling: it seems the main difference to balanced sampling is whether the sampling of class and domain representations are done separately and in proportion to the sample size. The proposed method can avoid oversampling from the minority class, but the current text description and the transition probabilities make things overcomplicated.  

- Domain adaptation baselines: Fig 5, CORAL should be an unsupervised domain adaptation method since it uses unlabeled data from the target domain. This should be clarified - as the proposed method does not even need to use any data or labels from the target domain.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the multi-domain long-tailed recognition problem.         
(1) An augmentation strategy is proposed for this goal:     
I. For each time, uniformly sample an image across all classes and uniformly sample an image across all domains.     
II. Each image representation is disentangled by class-relevant features and domain-relevant features.     
III. Combining the semantic representation of one with the domain-related nuisance factors of the other to form the augmented representation.    

(2) Experiments are conducted on several long-tailed datasets with subpopulation shift and domain shift. Clear improvements are observed.

strength_and_weaknesses: Strength:    
(1) The paper writes clear and is easy to follow.    
(2) The method is simple and elegant.    
(3) Experiments on different benchmarks show good improvements over baselines.    
(4) Sound ablations are conducted.  

Weakness:     
(1) Some long-tailed methods are selected as baselines, like CRT, and MiSLAS. However, state-of-the-art long-tailed methods can be transferred to this multi-domain long-tailed recognition task, like PaCo [1], and RIDE [2].    

(2) Would the representation augmentation module be removed at inference?    

(3)  Regular augmentations like RandAug can also improve performance. Can the proposed TALLY work with them?

(4) Can the proposed TALLY  work with long-tailed methods like PaCo [1], and RIDE[2]?


[1] Parametric Contrastive Learning. ICCV 2021.
[2] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts. ICLR 2021.


clarity,_quality,_novelty_and_reproducibility: Code is not provided.
summary_of_the_review: This paper proposes an augmentation strategy to address multi-domain long-tailed recognition. 
If more advanced long-tailed methods, like PaCo and RIDE, are explored (as comparison baselines or whether the proposed method is complementary to them), the experimental results will be more convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper proposes a data-augmentation method TALLY for multi-domain long-tailed learning. TALLY provides balanced augmentation by disentangling and resembling class-relevant semantic information and domain-associated nuisance information in the latent domain. The experimental results on synthesis and real-world data show its effectiveness.

Strengths:
- The paper has addressed an important and less studied problem in machine learning -- multi-domain long-tailed learning.
- The experiments are extensive and valid.
- The paper is well-written and easy to follow.

Weaknesses:
- It has limited novelty because the proposed method is a combination of existing techniques.
- There are missing details on why the representation of each examples can be decoupled into semantic information and nuisances.



justification_for_why_not_higher_score: This is a borderline paper. Reviewers acknowledge that this investigation is worthwhile. However, there are shared concerns on the limited novelty, missing details on why the representation of each examples can be decoupled into semantic information and nuisances, the reproducibility and the evaluation of the proposed algorithm. The authors address some of these, providing code and revised paper, but these were not enough to sway reviewers. I think this paper is not ready for publication at the current stage due to limited novelty.
Decision: Reject