Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of discovering the generalizable tasks over the models.
In this context, the image label assignment of labels is considered a different task. 
Then, the method proposed in the paper seeks to find different assignments of label sets over images such that desired generalizability is ensured. In a way, the labels are merged to create new labels to simulate new tasks. Such fusion may or may not be semantically meaningful. In this work, the authors wonder if a meaningful fusion (thus the derivation of a new task) can be performed by seeking the generalizability of the derived task over the test set. The authors propose a framework for task discovery that automatically finds generalizable tasks using the proposed agreement score. Furthermore, a case of “adversarial train-test splits' ' where a split of datapoints between training and testing leads to the failure of the model is also studied.


strengths_and_weaknesses: 
Strengths:

* This paper studied an interesting problem to uncover the nature of the deep learning models by means of agreement in the test set, in the context of the multi-task learning setup for task discovery for generalization.

* The proposed hypothesis and developed framework are intuitive and convincing.

* The paper is well written and easy to follow. The unconventional writing of the abstract and the discussion-rich paper make it an interesting read.

* Experimental results, both quantitative and qualitative, validate the significance of the proposed method.

* The studied case of adversarial train-test splits for the setup of the proposed setup is particularly interesting (although not very surprising).

* Experimental results in Fig 7 in the supp. Material is meaningful. I suggest that to be included in the main paper.

* Supp. Material provided with the submission (document, code, and video) are helpful.


Weakness: 

* Conclusion: The conclusion of the paper is not really helpful. It reads as few comments of discussion and largely limitation in rather a short paragraph. The conclusion does not summarize the discovery of the paper in terms of theoretical or experimental results. In fact, the conclusion provided in the supp. Video is rather more meaningful.

* Experimental setup: The experimental setup used in this paper is rather limiting. Although already highlighted in the limitation section, this becomes rather a serious concern to gauge the effectiveness of the proposed method in more practical task discovery.

* Future work: Authors' thought processes behind formulating the problem, beside understanding the neural networks, is not clear. In particular, how the proposed framework can potentially be utilized beyond the experimental setup of this paper.


questions: * Trivial solution: to avoid the trivial solution the proposed formulation of (3) makes a trade off between tasks and agreement. This trade-off makes it unclear what is indeed being searched, as the hypaerparameter \lambda plays the role. Obviously, the presence of hypaerparameter in itself is not a problem. In this context of this paper however, the discovery of the generalizable tasks which is unknown, becomes the function of the hyperparameter that needs to be tuned. I wonder how this hypaerparameter can be selected meaningfully as there is no ground truth (or any interpretable measure as far as I can see). 

* Regulated discovery: In line 236, the figure Fig.5.2 (bottom right) is incorrectly referred (also in line 225). The regularization of simCLR is as a matter of fact simply a better visual extractor.  Such pre-training learns the some desired invariances thereby leading to similar invariances in the discovered tasks. Such regularization however needs to be task dependent. For example, if one wishes to perform rotation estimation, features learned for rotation invariance are not desired. This particular case is justified for the image classification by object class (in the used case here). However, a broader discussion on what kind of regulated discovery is meaningful needs to be provided. Here again, this question arises as one is unaware beforehand what task is being searched, which is instead searched merely based on some objective which is effectively only vaguely interpretable. 

* Embedding space: Monte-Carlo gradient estimator technique resulting uniformity in dissimilar tasks, and rest of the discussion is unclear. This may either require references or more details. Is the presented claim coupled with the choice of embedding space for the task?

* Training size in the context of utility: discovery of the generalizable tasks can potentially be used for knowledge transfer across tasks. This however is meaningful in practice when only a limited amount of data from the target task is available. The proposed method appears to be heavily biased from the pretraining in the case of limited data.


limitations: There is no obvious severe limitation of this paper besides what the authors have already mentioned in the paper. I can say nothing regarding the negative social impact of this paper from my reading.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors formulate an agreement score that scores how well two networks from different initializations, etc converge to a common solution on a task. They claim that the agreement score is a proxy measure for how generalizable a task is. They use this agreement score to find tasks that networks can generalize on. 
strengths_and_weaknesses: Strengths:
- The problem formulation is unique and probably interesting. The agreement score is also interesting since it shows how robust a network is on a task. 

Concerns/weaknesses:
- I am mostly concerned about the practicality of such a problem definition. Why would one want to fix a network and find a task? Tasks are defined by practical demand and we find a solution to them, not the other way around. I believe this would be interesting if we were to find the set of tasks a network is good at, but that is rarely the use case. Searching in the opposite way makes more practical sense - which networks are good for my task? 
- I am not convinced why the agreement score is a proxy measure for generalizability. It seems like it is a proxy measure for robustness of a network for a task. If a network can converge a good solution from different initializations, it means that the task space is well defined and convex and easy for the network. Doesn't necessarily mean generalization - which in my knowledge is defined as performance on unseen and novel combinations of data. 
- The authors say that the agreement score based search doesn't discover any real tasks. This further convinces me that the searching in the task space is probably unnecessary. 
- Discovering the adversarial splits seems like an interesting direction to me here, but I didn't quite understand how the AS is being used here. To me, it seems like we can do this with any dataset and split it in a way where test labels are starkly different or opposite to train labels for a certain feature or feature combinations.
questions: - How exactly is the AS being used to discover adversarial splits? 
- Can the authors provide some qualitative examples for the tasks being discovered? Are they useful tasks? 
limitations: The authors did include a paragraph on limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new paradigm in which given a fixed model architecture (with learnable parameters and different initializations) the goal is to automatically discover a "task", or rather a binary label for inputs in a given dataset. This is achieved by learning a task network via optimizing a proposed metric termed the agreement score. This metric assesses on average how often two different models of the same class trained on the same task will agree with one another on unseen data, with the premise being that if the class of models generalize well then they should be in agreement more often on unseen data. This metric is proposed for both automatically learning tasks as mentioned before as well as being a investigatory tool for analyzing different biases and settings.
strengths_and_weaknesses: The proposal is novel and interesting as both a tool for analysis as well as for automatic task discovery. The paper has a good mix of technical contributions, ablation studies / investigations, as well as qualitative demonstrations. The methodology developed is well supported and motivated. I found the premise, proposed techniques, and experiments to all be clear and concise. 

One slight correction should be made to the equations throughout the paper. Often within an expected value there is something like $\tau_1(x)=\tau_2(x)$ such as Eq. 2. These all need to be inputs to an indicator function to be proper.
questions: I was wondering if the authors could speak more to their decisions regarding the approach taken for learning multiple tasks simultaneously. In your findings, was having a shared encoder with task-specific linear discriminators sufficiently flexible? Did you happen to notice the embeddings to be particularly sparse in high-dimensions? I am curious if a more flexible task-specific operation was considered, such as a network that accepts the shared encoding and a learned task-specific embedding or even just learning a task-specific shallow network that accepts the shared encoding? 

Finally, just to confirm my own intuition, when discovering a new task an entire dataset is essentially being partitioned into two classes ($t_\theta(x)\approx 1$ and $t_\theta(x)\approx 0$). In your findings, was is common that this process had a preference for discovering tasks with high rates of separability or decisiveness (meaning for a random $x\sim X$ it is highly likely for $t_\theta(x)$ to be close to 0 or 1) or did it also allow for instances where there are larger subsets of the data where the class is unsure ($t_\theta(x)\approx 0.5$)? Additionally, would you say that this process tended to prefer discovering tasks with roughly equal class balance? It would be interesting to be able to control or target the degrees of class imbalance exhibited within a class as I could see that leading to uncovering some uncommon but distinct traits or features within a dataset.
limitations: The authors have adequately discussed the limitations of their approach and findings.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces task discovery, which aims to find tasks that the model can generalize on.

strengths_and_weaknesses: **Strengths**

1. The task is interesting. Compared to existing works that mostly focus on optimizing models to achieve SOTA performance on several benchmarks, this task is more interesting and can help us understand NN more deeply.

2. I think the construction of  ``adversarial train-test splits’’ is valuable for understanding current out-of-distribution generalization or domain generalization problems.

3. The experiments are pretty good and basically answer most of my questions about the proposed task.

**Weaknesses**

1. Just as the author mentioned, one of my main concerns of me is the expensive meta-optimization.
2. I think the proposed task discovery task is relevant to domain generalization without demographics [1], optimal experimental design [2], and data-centric machine learning. Maybe more discussion should be included.

[1] Environment Inference for Invariant Learning, ICML, 2021

[2] Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design, ICML, 2021
questions: see Weaknesses
limitations: see Weaknesses
review_title: null

Meta-review: metareview: This paper contributes an interesting point of view: given a particular model, can we find tasks for which it will be good?  For this end, the paper proposes an agreement score whose applications include finding generalizable sub-tasks and adversarial train-test splits.  All reviewers were in favor of acceptance, generally agreeing that the paper provides theoretical and practical contributions.
confidence: Certain
Decision: Accept