Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers supervised learning in the setting where the data is drawn in a 2 stage process: first, a domain is selected, second, a batch of data is drawn from this domain. The observations include the batch index, but not the underlying domain the data was collected from. The learning problem considered here is to find 1. a predictor that is optional for each domain, and 2. a rule for assigning a batch to a domain. The paper proposes a straightforward method for doing this.
strength_and_weaknesses: For context, it may help to consider two closely related cases:
1. (latent variable modeling) We know that each data point is drawn according a hierarchical scheme that first draws a distribution, then draws the data point from this distribution. In this case, the obvious thing to do would be to set up learning via a latent variable model. This corresponds to the case where batch_size=1 in the present paper.
2. (unsupervised adaptation) We observe batches drawn from a number of domains, but we don't know a priori the relationship between domains. In particular, we don't know that there are only K distinct kinds of possible relationship. Then, the natural approach is to build a model that, given a new set of inputs, self-modifies to improve predictions on these inputs; see https://arxiv.org/abs/2102.12206 for an example of this kind of approach, and https://arxiv.org/abs/2112.05090 for a benchmark of such methods. In the present paper, this is done by guessing which of the K classifiers should be used for the test batch.

With this context, the main strength of the paper is that the particular setting it considers---data is observed as batches, and we know each batch is from a fixed domain---seems to add in extra structural assumptions in a manner that is realistic for some applications, and which may allow for better methods or analysis. The particular approach they take here (use EM to alternate assigning batches to classifiers and updating the classifiers according to batch assignment) is straightforward and reasonable.

However, in this view, the main weakness of the paper is that doesn't really focus on what's interesting about their data setup. Specifically, the natural baselines for comparison are 1 (latent variable modeling) and 2 (unsupervised domain adaptation). But it seems to me that the discussion, theory, and experiments all comparatively neglect these comparitor cases. That makes it both hard to assess the importance of the method, and hard to know what the important conceptual takeaways of the paper are. Generally, I think the paper needs to be substantially reworked to make all this clear.

As particular instances,
1. How does knowing that some pairs share the same domain help with identification (relative to straight latent variable modelling)? 
2. How does the assumption that that there are only K domains help relative to general unsupervised domain adaptation? E.g., in terms of the PAC rate and in terms of empirical performance

Some further issues I see:
1. Theorems 1 and 2 seem trivial. 

2. It's actually not obvious that ERM should fail in this data setup. The optimal classifier is P(Y| x) = \sum_k p(k | x) P(Y | k, x), where k denotes the underlying domain. If p(k|x) is learnable from the data, then a suitable ERM procedure should succeed. The claim here is presumably something like p(k|x) is not learnable given only single sample batches, but is learnable using multi sample batches. But, if that is the claim, I'm not able to find it spelled out clearly anywhere in the paper.

3. In the experiments, it's not clear to me what "ERM" means. In particular, is the capacity of the ERM model the same as the LEAF model? A fair comparison would be to run the EM algorithm, but eliminate the information about which datapoints come from the same batch---is that what you're doing? If so, it wasn't clear to me
 
clarity,_quality,_novelty_and_reproducibility: See above.

A plus of the paper is that the writing is unusually clear. Though, I do think the focus of comparisons is not quite right to really clearly communicate the significance of the results. 
summary_of_the_review: Although I think the data setup is interesting and potentially fruitful, I don't think the results of the paper in its current form really communicate this. Minimally, the paper would require making it precisely clear how the batch size > 1 case helps vs simple latent variable modeling.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper first formally defines a new and practically important problem setting where data are generated from multiple domains and the generation process is completely hidden. It then propose a concrete EM-based algorithm with reasonable theoretical guarantees. Empirical evaluations are conducted on various tasks and high quality python code is provided.

strength_and_weaknesses: Strength
- This paper considers the new problem setting of learning from hiddenly conflicting data, and proposes a reasonable rigorous definition based on domain generalization.
- It then proposes a pratical EM-based algorithm for the new problem, and analyzed its learnability and generalization ability compared with existing general supervised learning case.
- Empirical evaluation is carefully conducted and experiment python files are provided with detailed README.

Weakness
- As the proposed algorithm is based EM, it is nice to have dicussions on its convergence conditions.
- Hidden conflicts defined in this paper resembles graphical models with a latent variable, although graphical models cast stronger assumptions on the data generation process. Some dicussion on this topic is desired.

clarity,_quality,_novelty_and_reproducibility: - The paper is of high quality, overall rigorously written and easy to follow.
- The problem setting has reasonably high novelty and pratical importance.
- Reproducibility is guaranteed by the detailed description and provided sample python code.

summary_of_the_review: I recommend to accept the paper considerting the overall contribution of problem setting, algorithm, theoretical analysis and empirical evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authros proposed the method that can deal with conflicting contexts scenario. Authors proposed a LEAF architecture for solving the problem. Authors also showed the effectiveness of the proposed method using toy and real-world datasets. Also, they offer the theoretical bounds for their method.
strength_and_weaknesses: - I think the problem of conflicing contexts seems novel and important for solving the real-world problems.
- The examples for context-conflicting cases are well exampled.
- The LEAF method looks sound and novel enough.
- Experimental results are impressive, showing superior performance to several competitive baselines (ie. meta-learning, MoE, etc)
- Theoretical bounds also look sound.
clarity,_quality,_novelty_and_reproducibility: The method is clearly described in the draft.
summary_of_the_review: I think the draft contains the solid method for solving unique aspect of the supervised learning task. The problem looks important and the proposed method looks also sound. Experimental and theoretical analyses are also thorough. I recommend accept for this draft.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for learning from data following a mixture of distributions whose input-output relationships are different. The proposed method maintains multiple hypothesis functions accounting for those different distributions and minimizes the loss between the label and the closest output among those given by the hypothesis functions. The authors explain a connection between the proposed method and the EM algorithm. They also show an identifiability result and a generalization error bound between the proposed training objective and an oracle risk. The experiments show the superiority of the proposed method against other methods that are not designed for this setup.
strength_and_weaknesses: # Strengths
- The paper seems to tackle an important and interesting problem.
- The method performs excellently according to the experiments.

# Weaknesses
- The problem setup is not very clear. In particular, the evaluation metric, or the test error, is not clearly defined, and the paper does not explain what information is given to the model in prediction.
- It does not seem easy to choose $K$. The authors mention some connection to $R(D)$, but how do we estimate this number?
- In Eq. (3), the method allows $\hat{g}$ to look at $y^{ij}$, which feels strange to me because we don't have access to it in the test phase.
- There is no guarantee that the proposed training minimizes the 
- Knowing each set of $m$ samples comes from one distribution seems to be a strong assumption. The theory requires $m$ tends to infinity to ensure the convergence of $\hat{\operatorname{err}}(H)$ to $\operatorname{err}(H)$. The paper briefly mentions this point, but the argument is not very convincing to me.
clarity,_quality,_novelty_and_reproducibility: I think the writing should be improved. Even the problem setup is not totally clear to me. According to the PAC learnability analysis in the appendix, the authors consider something similar to the top-k error, but there is no discussion about related work on the top-k error.
summary_of_the_review: I like the direction of the work, but I think the quality of the paper should be improved before being published. There are many major issues as I mentioned in the weakness section. I suggest rejecting the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: After the authors' responses, there was an agreement between reviewers that the paper presents a valuable contribution. Even though the reviewers pointed out some weaknesses of the paper, overall they agreed that the main strength of the paper is the novelty or practical value of the framework.

After further discussion however, more attention was given to the prior work by Su et al. "Task Understanding from Confusing Multi-task Data" (ICML 2020). Looking more closely at this prior work, it seemed that:

1- The setup, solutions and experiments are very similar. Both papers use a task assignment process that is performed during the training of task-specific classifiers with an EM-style approach. It seems that the main differences are that the current submission works with batches and performs per-batch task assignment while the previous paper works on the basis on individual samples and learns a network to predict a soft task assignment. Even though there are new tasks in the experimental evaluation, the datasets are the same and tasks are similar.

2- The way the Su et al is cited in the submission mislead the reviewers to believe that this prior work was not closely connected to the present submission. However, there is a consensus among reviewers that Su et al. is very close to the current work. 

In  the end, during the disucussion phase the reviewers revised their opinion regarding the novelty of the paper, which is now assessed as weak by all of them. Also, before any publication, it seems necessary to thoroughly discuss Su et al.'s paper in the future iterations of this paper and better explain the novelty, differences and the practical relevance of these differences.
justification_for_why_not_higher_score: the novelty of the paper that motivated relatively high scores by the reviewers was questioned by a more detailed analysis of prior work and a consensus was reached to reject the submission.
Decision: Reject