Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a learning setting involving a sequence of domains
and a collection of hypotheses, During learning, each domain should be
assigned to the most appropriate hypothesis by iteratively grouping
together non-conflicting domains and keeping apart conflicting ones,
so as to minimize the number of hypotheses learned and leverage
cross-domain information when appropriate.
main_review: While the intuition behind the problem and the proposed solution is
clear, the method is not clearly explained. In section 2.3, the
connection between the M and E steps is not clearly stated. More
importantly, the difference between conflicting and non-conflicting
domains is unclear. From the overall algorithm, it seems to me that
domains that do not share labels will be assigned to different
hypotheses (even if, in principle, they could be mutually-exclusive
labels), because a hypothesis previously trained on a different set of
labels will not predict the new labels. If this is the case, the only
non-conflicting domains are those that share the same label set but
maybe have differences in the input (like in a domain-adaptation
setting).  In this case the method seems an overkill, as one can
easily match the domains in the episodes by matching their labels (and
the subjective error should be minimized this way). The experiments
seems to suggest that this is indeed the setting, because different
domains correspond to different label sets.

Another unclear part is how the method is applied in testing. Assuming
that testing is made on an episode corresponding to a single domain,
how is/are the appropriate hypothesis/hypotheses selected? If the
predictions of all of them are collected, this seems to boil down to a
simple collection of multiclass classifiers (not sure what happens in
the regression case).

Depending on the specific setting (to be clarified), there could be
other (possibly simpler) competitors that are more appropriate. The
cross-domain fertilization is not evaluated in the experiments as far
as I can tell (apart from the matching of domains with the same
labels).
summary_of_the_review: The paper is unclear in the description of the method, so much that
some trivial solutions seem even more appropriate for the setting.

Substantial clarifications are needed to properly evaluate the quality
of the contribution.

AFTER REBUTTAL:

The authors did manage to address many of my concerns in their detailed rebuttal and extended experimental evaluation. The difference between the proposed setting and the existing alternatives is now clearer and the advantage of the method better highlighted thanks to the novel experiments. Some aspects are still a bit unclear (e.g. how the method is applied at the test phase), but I am now more inclined towards acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a framework for open-ended data by learning a subjective functions that allows to represent multiple domains without interference. The framework works in two stages: 1) evaluating a set of candidate hypotheses for each domain using a batch of data for that domain, 2) training the hypothesis with the smallest error  for each domain. The paper also proposes two new evaluation metrics, one to determine the error of the subjective function and one for calculating the model error in domain prediction. The authors provide both theoretical and experimental analyses of the proposed framework. In terms of theoretical analyses, the authors provide some guarantees in terms of learnability and generalisation error. In terms of experimental results, the authors test both regression and classification tasks with data of multiple labels and hierarchical labels. Comparisons to other approaches such as ProbCon, Pseudo-L and LabelProp in Colored MNIST, CIFAR-100 and Fashion Product Images datasets demonstrate gains of the proposed approach versus counterparts. 
main_review: Strengths of the paper are:
-  The paper studies the important problem of learning on multiple domains with open-ended data. The proposed approach is novel, simple yet effective, as clearly demonstrated in both the theoretical and experimental results
- The paper is very well contextualised in the existing literature 
- The paper is very well-written, organised and the claims are sufficiently justified

Weaknesses of the paper are:
- A few aspect are not entirely clear in the writing of the paper. For example, in page 3 it is mentioned that "In the above setting, an episodic sample number parameter n is introduced to maintain the local consistency of data, implicitly assuming that we are able to sample a size-n data batch at a time from each domain." - does this mean that at each time all domains should have corresponding data batches or that each domain is sampled sequentially one after the other? Please clarify
- There are some typos in the paper, e.g. Section 2.4 page 4
summary_of_the_review: I find this paper very interesting. The proposed approach is novel, yet simple, and clearly works well for the problem of learning from open-ended data. I would recommend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces "Open-ended Supervised Learning (OSL)", a method that handles data form multiple domains (open-ended data). OSL leverages an Expectation-Maximization (EM) algorithm to train the "subjective function" that allocates models to data domains. The paper presents the theoretical foundations and empirical results in which it outperforms existing baselines for (generated) regression and classification tasks.
main_review: **List strong and weak points of the paper. Be as comprehensive as possible.**

**Pros:**

* Generally, the paper is well written and easy to grasp the underlying ideas and concepts except how the data sampling works (see questions).
* The intorduction makes it very clear what the problem is this paper is trying to solve. (see additional feedback)
* A strength of Section 2+3 is that it bundles the theoretical formulas with intuitions in natural language, which makes it very clear at any time what the author trying to achieve or show.

**Cons:**

* While the generated regression example is great to gain intuitions on how OSL works and the influence of its hyperparameters, it's difficult to draw conclusions in regard to "real" regression problems. (see question)
* The proposed concept of the mapping rank seems to not account for label noise explicitly - but I would guess that in the presence of label noise there could be a rank > 1 even though there is only one sensible target function? 
* The experimental evaluation appears to be rather specific and the baselines appear not directly related to other state of the art approaches. It would probably be a much stronger paper, if the authors showed that the proposed approach improves on some metric that is not specific to this submission
* The paper do not show training and prediction times for OSL and the other baselines. 
* Minor/Typos:
  * in the introduction: would prefer to change "(image, label) pairs" -> "image-label pairs"
  * in the introduction: "can correlates" -> "can correlate" or "correlates"
  * last paragraph of the introduction: define PAC before use the abbreviation
  * first paragraph of Section 6: "facilicate" -> "facilitate"
  * page 4 last line: "objectives 2 3" -> "objectives 2 and 3"

**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**

If the authors can clarify the points below, I vote for accept. The introduced approach OSL solves the problem of allocating data from different domains to their model. Currently, we often see very over-parametrized models that try to solve this implicitly. OSL could present a path to more efficient ML.

**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**

* Did you try at least a multidimensional generated regression? I think it would be interesting to see the method comparison in at least >1-D generated regression, better for a datasets, e.g., form OpenML.
* Could you clarify how the data sampling mechanism works?
  * It is still not clear to me whether the dataset X is sampled once or iteratively. 

**Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.**

* The first paragraph after Definition 1 (introduction) is a bit lengthy. Shortening to its main points (difference between R=1 and R>1, metadata difficult to use for data allocation) could improve clarity
* Providing more description for Figure 2 and 3, especially, what the three lines show would add clarity
* You should consider to move the description of the DNNs from appendix to the paper.
* Providing an overview of training and prediction times of OSL and the baselines could be interesting for some readers.
summary_of_the_review: Interesting approach and novel concept of mapping rank, the experimental evaluation could provide more direct evidence for the value of the contribution. 

Update after authors‘ response:
The authors addressed all points and provided convincing additional experiments, I’m hence increasing my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a novel framework for learning different target concepts. They argue that so far, a single classifier is supposed to learn mixed concepts but usually fail. Hence, they propose a framework for sampling instances according to only a single target concept and solving this problem with classical ERM and repeating for other concepts. They provide theoretical analyses and empirical insights. 
main_review: I like the general idea of the paper and appreciate the framework that I consider as novel and well suited for publication in general. I also find the technical part correct but did not check everything in detail. Concerns are caused by the baselines in the empirical comparison. The proposition of the framework clearly needs empirical support as the main argument is that a single classifier fails to learn all concepts properly. There need to be more information provided on the baselines to better understand whether they are appropriate or not. I am under the impression that the baselines may not be suitable to support the claims but this may be part of the rebuttal. 

I would appreciate a better differentiation to other ensemble learning approaches as well as learning with many classes/labels as well as transfer learning approaches in the discussion. 
summary_of_the_review: Conceptually and theoretically strong but line of argument needs empirical evidence to work out. Though evidence is provided in form of empirical results on three data sets, the baselines remain unclear. This problem need to be solved before publication would be OK (could be straight forward to do so)
review_title: null

Meta-review: Two reviewers increased their scores after considering the responses from the authors, and all reviewers are somewhat positive. However, the increased scores are still 6 only, and as the area chair, I have concerns about the foundations of this research.

The authors write "there is no off-the-shelf baseline that can automatically disentangle the data from different domains in the open-ended regression setting." This is not true for the standard situation of a mixture of regression lines, as in Section 4.1. Completely standard EM (not necessarily hard EM) will solve this problem, as long as the individual lines (sinusoids etc.) can be represented easily by the EM components. Another standard method that should be another baseline is a mixture-of-experts neural network. 

One thing that EM cannot handle is learning the number of components in a mixture model, as in learning the _k_ in _k_-means. To the extent that "open-ended" here refers to a new approach for this problem, with mathematical guarantees, it is interesting. But this point of view needs more explanation.

The paper begins "A hallmark of general intelligence is the ability of handling open-ended environments, which roughly means complex, diverse environments with no manual task specification." If there is one aspect of natural environments that is crucial and fundamental, it is the presence of noise. However, starting theoretically with Definition 1 and empirically with Section 4.1, the authors work in a world of deterministic functions. This mismatch undermines the conceptual significance of the paper.

Perhaps because of the universality of noise, the authors do not present a real-world dataset or task for which the OSL method is directly natural or applicable. Rather, they impose restricted specifications on datasets such as MNIST and introduce metrics that are novel, hence hardly natural, undermining the empirical significance of the paper.
Decision: Reject