Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The main contribution is to tune differentiable “hyperparameters” by splitting the trainable parameters and dataset in a way that separates the data seen for tuning hyperparameters, ensuring that the set of hyperparameters are generally good for a wide range of parameters. The splitting technique is inspired by the marginal likelihood.
strength_and_weaknesses: The paper mainly proposes a technique of splitting the training/validation dataset in an iterative manner, inspired by the marginal likelihood decomposition for cross validation, which is claimed to be an optimization objective that requires no validation data. 

While marginal likelihood is standardly used for Bayesian model selection, the choice of Bayesian prior/function class strongly influences the generalization capabilities of these methods since almost all loss functions, such as the L2/cross entropy can be derived by using Bayesian priors. In fact, a recent paper [https://arxiv.org/abs/2202.11678] demonstrated the pitfalls of using marginal likelihood without validation. Since the marginal likelihood is the main theoretical justification for the method design, I find the justification extremely weak so giving a theoretical example with concretely define Bayesian priors/assumptions would be better.

The novelty of the paper is also limited in the practical algorithm. The method of partitioning the model and parameters are reminiscent of cross validation and drop out techniques, which have both been readily used in the ML community. Adding a baseline of using dropout + cross validation would be better, as well as spending more time of using the toy example (of 1/2 spurious variables) to illustrate WHY your method does better than all the "standard" baselines. 
clarity,_quality,_novelty_and_reproducibility: Novelty: Mentioned in main review, but it is limited. It is extremely unclear if the main benefit of the method is data efficiency, or improved generalization, or better optimization runtime, or all tree.

Clarity/Quality: Generally good but there are some concerns:

It is mentioned that: Although a subset of the parameters in each w (k) s is fixed and hence would likely be a poor approximation to the posterior over the weights, it can be a reasonable approximation in function space. Why would this be true? 

It is mentioned that: "There is no need to compute nor invert a Hessian with respect to the weights, as in the Laplace approximation". Why would this hold since the posterior for most Bayesian networks are still intractable to compute?

In Figure 1: Updated the caption to explain why hat(w_i) is fixed. 

In Figure 2: Why do the different line colors mean? Why are there no other baselines compared here?

summary_of_the_review: The main method proposed in the paper is limited in its novelty and shaky in its theoretical groundings. The paper does not clearly state the main benefits of using this method and such empirical benefits are not convincingly displayed in the experiments, especially in the "toy model" which are designed to clearly demonstrate the superiority of the new method over baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm that partitions both the data and model parameters to efficiently approximate the log marginal likelihood. The key idea is to define the out-of-training-sample loss with the partitioned network and utilize this objective for hyperparameter optimization. Empirically, the authors show that the proposed method can adapt various hyperparameters in neural networks without a validation set. Moreover, the proposed methods can be utilized in federated learning, where cross-validation is difficult.

strength_and_weaknesses: Strength
- The paper is well-written and clearly motivated. I believe that the idea of explicit network partitioning is interesting and novel and the work is relevant to the ICLR community.
- The technical details of the paper look correct. 
- The experiment setup is detailed making it easy to implement and reproduce the results.

Weakness
- While the paper argues that the proposed method is more efficient compared to previous methods such as Laplace approximation, it is not empirically shown in Section 5. I am happy to increase the score if this concern is addressed.
- The proposed method is not compared with a competitive hyperparameter optimization baseline (that uses a validation set). While I understand that the proposed framework does not require partitioning the train and validation set, it would be helpful to understand how the method compares to other strong hyperparameter optimization baselines. 
- The authors do not sufficiently describe the limitation of the proposed approach. What are some additional hyperparameters and how sensitive are they? Can the method be more scalable (in terms of both parameter and hyperparameter) compared to the baseline methods?
clarity,_quality,_novelty_and_reproducibility: 1. Originality

Although I am not very familiar with the relevant literature, I believe that the idea of explicit network partitioning is interesting and novel.

2. Clarity

The paper is generally clear and well-written. However, there is some ambiguity in the notation which I comment on below. 

3. Reproducibility

While the authors did not provide the code, the paper describes the experimental setup in the Appendix and I believe that it is possible to reproduce the results.

4. Minor Comments
- On page 1, “... capabilities enables researchers to train powerful models …” → “... capabilities enable researchers to train powerful models …”
- On page 1, there seems to be an empty space in front of 10%.



summary_of_the_review: Overall, I believe that the idea presented in this paper is interesting and novel and the paper is well-written. While I have concerns about the diversity of the baseline in the experiment section and the lack of computation and timing experiments. At the current state, I recommend 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work starts with using marginal likelihood as an alternative objective to validation metric for HPO, which has advantages in the small data region and potentially better generalization performance. It then talked about how to compute marginal likelihood and proposed an efficient method that only requires a single training run and no validation set. The experiments on synthetic and a diverse set of real world experiments (data argumentation, classical HPO setting and Federated Learning) demonstrated its usefulness.
strength_and_weaknesses: On the strength of the work, using marginal likelihood as a HPO objective is very interesting and I can see practical settings that marginal likelihood can be useful, such as in small data regime, federated learning or potentially a better objective for general HPO. On the methodology, it is a bold exploration that partitioning the networks into subnetworks and training them separately over data chunks. On the empirical evaluations, it covers a wide range of tasks from different areas. It is also a well written paper, with clear structures and abundant details for reproducibility.

On the weakness, my first concern is motivation. The author mentioned partitioning the parameters, in the beginning of page 4, “can be a reasonable approximation in function space”. Why is that? Also, one thing I am not sure, if the goal is to approximate eq(2) and avoid the model sees every datapoint in the training set after 1 epoch, why one can’t create $C$ copies of the whole parameters and assign each copy to one $D_k$, without even partitioning the parameter (update all the parameter with related data chunk in each copy)? 

Then it is the support for the claims. The authors argued the proposed method “is more scalable than previous works that rely on marginal likelihood and Laplace approximations”. I can see why it is more scalable (avoid inverting a Hessian (Immer et al., 2021)), but I don’t see any empirical results supporting it. 

Since the main contribution is a new method to approximate the marginal likelihood, the comparison to relevant baselines, for example, Schwobel et al. (2021) and Immer et al., (2022), will be critical to demonstrate its usefulness. But they are not all the times included in all the experiments. For example, in the experiments of learning invariances through data-augmentations Schwobel et al. (2021) is not in Table 1 and Immer et al., (2022) not in Fig 3(a)? There are no baselines (Schwobel et al. (2021) and  Immer et al., (2022)) in the experiments for “Comparisons to traditional training / validation split”. It is hard to attribute the improvement to the partitioned network (a particular way to approximate learning speed) or the learning speed objective in general. Similarly, no other HPO baselines are included in the Federated Learning experiments. Are there some particular reasons why these results are missing? If so, it would be nice to explain them in the paper.

In the end, I am wondering about the impact of partitioning? How sensitive are the results to a partition?


=== After reading authors' response ===

The authors' response answered all my questions and I increased the score to 6.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and it contains many details on the methods, experiments setup, hyperparameters and training details, hence also good reproducibility. The quality is overall good, but I think the experiments could be more targeted towards the claim, as mentioned in the weakness section. The proposed method is novel and has a potential of big impact. 
summary_of_the_review: The paper has many merits and they are mentioned already in the strength section. But the  weaknesses, so far, are also clear from my perspective. Thus, without more clarifications or justifications for the proposed method, I am not confident to accept the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to tune hyperparameters which requires only a single training run and no validation data. This is achieved by partitioning the weights of a neural network and train each partition with respect to only its own training partition. Then, the remainder of the training data can be used for validation. The authors demonstrate the effectiveness of this approach for hyperparameter optimization in several scenarios including federated learning.
strength_and_weaknesses: **Strengths**
- Good discussion of related work
- Diverse set of experiments
- Relevant problem, great motivation, well-written paper

**Weaknesses**
- Missing results to support the claim that the proposed method is computationally cheaper
clarity,_quality,_novelty_and_reproducibility: The authors are very clear about how their method works and motivate its use very well. There is many prior work on the same topic but the authors discuss it in detail and clearly point out the difference. The idea itself reminds of the out-of-bag error used in bagging. However, its application to a single neural network and the theoretical discussion are novel.
I understand that the authors mostly use benchmarks considered in the most related work. I appreciate the interesting case study on federated learning. However, it would have been great if the authors went deeper into one particular problem and had a more extensive experiment. Many experiments seem to be toy examples. I would have loved to see a challenging hyperparameter optimization problem (jointly optimize augmentation, learning rate, etc.) including a comparison against state-of-the-art methods such as ASHA. Currently, it is unclear how good the hyperparameters actually are.

What is not fully clear to me is the difference to differentiable Laplace. Is the proposed method faster? If yes, how much faster. There is a claim that the proposed method is faster but I couldn't find any reported run times.
summary_of_the_review: This is a well-written paper on an important scientific problem with important practical applications. The authors discuss related work in detail and demonstrate the effectiveness of the method on a couple of problems. Overall, I think the paper could be a great addition for the conference.

**After Rebuttal Phase**
Overall the paper was interesting and seemed novel to me, and therefore I was positive about it. However, based on the comments of some other reviewers, it might not have been very novel (I am not familiar with that related work) and therefore I've updated my confidence and recommendation accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.
The rebuttal notably strengthens some aspects of the experimental validation of the work (quality of the found hyperparameters, sensitivity analysis w.r.t. partitioning, measure of training overhead, more detailed related work) and confirms the interesting, original and sound contributions of the paper, which should inform future research (as acknowledged by the reviewers).

However, there are some important aspects that have not been fully addressed during the rebuttal. _As a result, the reviewers and the meta reviewer are weakly inclined to accept the paper._
In particular, the authors are urged to carefully update their final manuscript with the following points:

(i) The discussion about the connection with the marginal likelihood should be further developed and should be made more formal. Right now, several arguments are still intuitive. Similarly, the notation of the sections where the discussion takes place (Sec. 2 and 3) needs to be improved. For example, the prior over function (f) is not never really discussed, and the mix between w and f is confusing, especially because we only deal with parametric models (w) in the paper.
It would be important to add a complete example of what happens when the model is very simple (say linear) and all the derivations (e.g., assuming Gaussian distributions) can be exactly computed and easily presented.

(ii) Right now, the experiments operate at an arguably small/moderate scale, for both the model and the datasets. It would be important to probe the behavior of the proposed approach on larger scales (e.g., ImageNet). Indeed, this is at a larger scale that hyperparameter tuning becomes a real bottleneck. 

(iii) To fully prove its generalizability, it would be important to complementarity report the performance of the proposed approach for the tuning of more “standard” hyperparameters such as L2 regularization and dropout rate.

(iv)  Several claims in the paper, e.g., “Compared to other non-Bayesian methods like Benton et al. (2020), our approach is broadly applicable to any hierarchical modelling setup” or the fact that the proposed approach is a mature alternative to HPO, need to be clarified and/or toned down. For the latter example claim, there are only a few HPO baselines/settings considered in the experiments.

If the paper was submitted to a journal, it would be accepted conditioned on those key changes, the meta-reviewer thus expects all those changes to be carefully implemented.

justification_for_why_not_higher_score: * Missing an experimental validation at a larger-scale to increase its practical significance.
* Missing more HPO-related comparisons (techniques & settings) to demonstrate the method is an actual alternative for HPO problems 
* Connection with marginal likelihood could be improved and beefed up.
justification_for_why_not_lower_score: * Original idea and partitioning scheme.
* Well-written and well-presented manuscript.
* Relevant/well-motivated applications to federated learning.
Decision: Accept: poster