Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: When no task specific heads are available sample labels cannot be predicted and the dependency between two tasks and thus task transferability cannot me measured. The paper uses a Gaussian process to model the task space and comes up with a closed solution for the mutual information gain. Proposes a greedy selection algorithm for selecting checkpoints to maximize the information gain. Experiments with text and image datasets are performed to show that selecting checkpoint based on this criterion outperforms random selection.
main_review: Strengths:
-	Using unsupervised probing to get around the problem of missing true and predicted labels and showing that kernel alignment can be achieved reasonably well at the level of feature extractors without having access to task specific heads. 
-	Good leveraging of the analogy between sensor replacement and checkpoint selection for a more generalizable task transfer from seen tasks to unseen ones.
Weaknesses: 
-	Imaging experiments are less compelling


Each checkpoint is added in a greedy way to maximize the gain, where the gain is formulated as the information gained by adding the current checkpoint to the current set of checkpoints minus the information lost by removing the current checkpoint from the remaining part of the checkpoint zoo. A good checkpoint generalizable to unseen tasks is the one that is representative of the remaining part of the checkpoint zoo (the information loss minimum when removed) but at the same that surprises the current set of checkpoints the most. 

In the CIFAR 100 experiments classes within seen and unseen tasks can overlap as 30 classes per task are chosen randomly. What would happen if unseen tasks do not contain any classes used in seen tasks?

Imaging experiments select n=100 seen tasks. How does the accuracy change with respect to n?

It is not clear why choosing too many checkpoints hurts the performance in such a significant way? Overfitting explanation is not very compelling

Typos:
From the abstract: “This objective is submoludar”. Is this supposed to be submodular?
An extra “the” in line 225
Is missing in line 232.
Senor placement lince 340

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: provide a method of how to (adaptably) combine  existing task models for new unseen task with aiming to relieve practitioners'  selection difficulty.  Through analogizing sensor placement problem, the authors give the above method and conduct effective verification on language and CV tasks.
main_review: The idea of the submission is interesting and it uses the sensor placement as checkpoints to formulate the problem in hand for intuitive understanding. Concretely, using GP to model task space to capture the task relation and while using checkpoints and unlabeled probing data to estimate covariance of the GP involved, then using Max mutual information criterion to identify the representative checkpoints for new tasks. To my best knowledge, this is relatively new. the experiments also show effectiveness of the proposed method. My comments are
1. a task corresponds to its data distribition, then what differences are between the distribution/task-oriented inductive learning and the proposed method? 
2. when a task is viewed a domain, again what differences are between the domain generalization and the proposed method? 
3. the checkpoint selection here is quite similar to the coreset selection, thus in some sense this point is not new.
4. if the selected checkpoint diversity is needed, or can the diversity be useful? 
limitations_and_societal_impact: Considering both data biases and task biases, the proposed method inherits naturally these biases to some extent.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
The paper considers the question of how to select an ensemble of existing pretrained models to be used as feature extractors, when the task for which they are to be used is not known a priori. It seems to aim less to generalize to a given fixed target task distribution, but rather at generalizing over an expectation over target task distributions.

To achieve this, the authors propose essentially two steps:

Firstly, considering the feature representation of each model from the zoo, and computing a within-model similarity matrix between the features extracted for one fixed model over a set of samples. Then to use the cosine angle between similarity matrices of different models as a similarity between models.

Secondly, to select a fixed number of models (corresponding to a fixed budget) based on mutual information estimates between the set of models and the remaining models. This is intuitively appealing.
main_review: Originality: the approach seems to be novel. The reviewer finds the amount of originality good enough for NeurIPS.

Quality: 

there are limitations in the experiments:

1 lack of comparison against existing baselines for ensemble selection, in particular A-Map seems to be within the scope of the question.

They compare only against random selection for NLP and random and top-k performance selection for CIFAR-100. At least for CIFAR computing gradients for A-MAP as another baseline should be not an issue. 

possibly they could also use 

2a evaluation using a small number of single point tasks in 5.1. While they seem to optimize without having a fixed task in mind, they evaluate on four standard tasks, for which one may suspect that some of the NLP models were implicitly optimized for and thus possibly leaking onto the 4 tasks. 

2b As a consequence, when evaluating against 4 fixed tasks as in 5.1, one should also show how the best single model selected for one task using a task-dependent selection algorithm as baseline performs ... on all 4 tasks (including also the three tasks for which it was not optimized).

The reviewer disagrees on the statement that backprop is expensive when discussing A-MAP. Each model was trained using backprop, thus it seems to be feasible to use backprop also after training, at least until the feature representation if the computations below were a black box.


Significance: In terms of setup there is one doubt: While theoretically interesting, what is the practical applicability of optimizing for an expectation over tasks and not for a given single target task ? When this can be helpful?

Question: 5.2 - how often this experiment has been repeated?


Clarity, the paper is written such that it can be read easily.
However, a few things can be improved:

3.2 the interpretation of the kernel alignment in terms of cosine-angle between similarities matrices within a model can be added. This interpretation also holds without any assumptions on the tasks

Sec 2: generalizing to unseen tasks seems to amount to optimizing over an expectation of a statistic over tasks. This can be worked out somewhat. 

Is there a connection between optimizing MI over feature similarities between the chosen set and the complement set of tasks, and under certain assumptions, between the expected prediction performance over many tasks ? It might be helpful to assume that each model and its feature representation is optimal for the task it was trained for, among the set of all models used for MI optimization. 
The reviewer would be surprised if such a link could not be discussed (whereas the submodularity does not give a clear result, this would provide some theoretical link to the set of a priori unknown target tasks).


typos:

This objective is submoludar. 

how many tasks has to be seen
--> have

 picking checkpoints amount to identifying useful 
--> amounts

In optimal senor placement [15]


limitations_and_societal_impact: As stated in the previous box:

While theoretically interesting, what is the practical applicability of optimizing for an expectation over tasks and not for a given single target task ? When this can be helpful (as opposed to optimizing for a given target task which one usually has in practice)?
The reviewer sees this as some level of weakness and feels that motivating this would make the paper stronger.

The discussions on submodularity yielded no conclusive result and are maybe not the most important.

The reviewer sees no issues regarding societal impact. Reusing checkpoints is positive, as it does  waste less GPU training time and energy.
review_title: null

Meta-review: PRELIMINARY: The reviewers generally agree that the paper is interesting and can be accepted for publication at NeurIPS.
Decision: Accept (Poster)