Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel batch active learning algorithm for multi-fidelity tasks. While prior work only queries one example at a time, the authors propose novel mutual information objectives that also take into account of the correlation within a batch. Moreover, the authors provide a greedy relaxation algorithm in solving the optimization, which is further shown theoretically to be an (1 - 1/e)-approximation of the optimal objective. Extensive experiments have been conducted on four physics applications while outperforming the existing baseline significantly.
strengths_and_weaknesses: Strengths:
1. The paper targets an important issue not addressed by prior work, which is the diversity among collected examples. The objectives are straight-forward and very intuitive.
2. The algorithm appears to be rather efficient and also enjoys theoretical worst case guarantees.
3. The experiments clearly shows the proposed algorithm's advantage in the batch setting.

Weaknesses:
1. The high labeling cost incurred by physical simulations motivated this work on applying active learning. However, when addressing accumulated cost, it does not seem to take into account the extra computation cost of the active learning algorithm. In the time complexity section, it is also lacking a comparison with existing work.
2. Moreover, a comparison with uniform random sampling strategies may be informative, especially when taking into account the extra computation cost incurred by the active learning algorithm. If they significantly underperforms, I think it's worth addressing in the text at least.


Minor Suggestion:
1. On the last line of equation (6), it's better to put sum_{hat{m}} lambda_{hat{m}} into parentheses. 

questions: 1. Is knowing a-priori the costs lambda_m for each fidelity m a reasonable assumption? Does it make sense to assume they are fixed costs?
2. Other issues mentioned in the weakness section.
limitations: 1. An advantage of batch active learning is the fact we could parallelize annotation. Now, consider if the cost of annotation comes only from time constraints, then the annotation cost of a batch actually comes from the example that takes the longest to annotate (highest fidelity) rather than the sum of the costs from each fidelity. It is not immediate to me that the method proposed in this paper is sufficient to solve such settings. Therefore, I recommend the authors to properly address as a limitation in their paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
The paper proposes BMFAL an extension of DMFAL for batch multi-fidelity active learning with budget constraints. The proposed method is a greedy sequential algorithm that selects datapoints which maximise the information gain. The experiments show that BMFAL outperform a naive extension of DMFAL in the batch multi-fidelity active learning with budget constraints. 
strengths_and_weaknesses: 
Strengths:
 - To my knowledge, this is the first method proposed for Batch Multi-Fidelity Active Learning with Budget Constraints. The setting is very specific but it's an important setting. Although not mentioned in the paper, multi-fidelity methods are also very useful for biological sequence design or drug design. In these settings, wet-lab experiments are very expensive, so there are methods to approximate them at various fidelities.
 - The experiments are conclusive and show a clear benefit of BMFAL over DMFAL in the batch setting. 

Weaknesses:
 - The experiments are limited to up to 3 fidelities. However, I would imagine that in certain settings we would want a larger number of fidelities. It would be interesting to see how this scales to a larger number of fidelities.
 - No code is included but the details appear sufficient to reproduce the experiments.


Overall, I would rate the paper:

Novelty: Medium

Clarity: High

Significance: Medium-High

questions: 
 - I think Section 3.3 could be improved. The algorithm complexity in 3.3 ignores the cost of searching (i.e., performing the argmax) $x \in \Omega$. In addition, it ignores the cost of computing the acquisition function which is not negligible. The space complexity can also be simplified to $O(\frac{B}{\lambda_1}(r+d))$ since the constant is negligible. 
 - Figure 2 is unnecessarily large. The figure can be made smaller and further discussion regarding the limitations of the method should be included.
limitations: The authors could include further discussion regarding in what specific areas batch multi-fidelity active learning with budget constraints is used. The discussion should include the potential negative societal impact of those areas.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Multi-fidelity learning is applied whenever observations generated from a cheap but less precise surrogate model are combined with limited observations from the actual process, such that a mapping can be learned between low and higher fidelity outputs. This is particularly challenging when the output dimensionality becomes greater at higher fidelities. In this work, the authors focus on the problem of deciding at which fidelity and location of the input domain the next sample points should be acquired. In order to mitigate the risk of successive sequential acquisitions focusing on the same limited region, the authors propose a batch active learning scheme which jointly selects different input points to sample across different fidelities, subject to a computational budget. Via a range of experiments, the authors show how the selection procedure results in more efficient learning than other competing methods when given with the same sampling budget.
strengths_and_weaknesses: **Strengths**

_[Significance]_ - As evidenced by the variety of application domains featured in the experiments section, multi-fidelity modelling is an emerging topic in many practical areas. This paper tackles an interesting problem within this space that does not appear to have been directly investigated for multi-fidelity regression prior to this work.

*[Clarity]* - The paper is well-written overall and easy to follow. There are a few typos here and there, but these aren’t distracting and should be easily fixed in a second revision.

*[Evaluation]* - I appreciated how the experiments were varied by leveraging datasets sourced from different domains. The choice of methods used for comparison is also appropriate, and the plots effectively illustrate both the overall performance and variance of the methods. The results indicate that the proposed approach consistently performs better than competing methods, which adds credence to the usefulness of the contributions presented in this work. 

**Weaknesses**

*[Novelty]* - The paper bears similarities to another recent paper, _“Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks”_, which shares a lot of the motivation and set-up for this work, with the exception that it is centred on the single-output Bayesian optimisation case (rather than the high-dimensional outputs considered here). While I appreciate that the two set-ups are different, there is considerable overlap which makes the lack of any reference to that paper particularly puzzling. I would like to see this featured more prominently in a future revision, as it is currently conspicuous by its absence.

*[Evaluation]* - One disappointing aspect about the evaluation section is that most of the text is devoted to explaining the datasets and the parameters used for the different methods, whereas the insights and findings themselves are limited to one or two brief sentences at the end of each experiment. While the results of the graphs are fairly self-evident, further insights or anecdotal examples of where the batch selection is particularly effective would be welcome.

*[Completeness]* - One interesting limitation highlighted in the presentation of the method is the potential degradation of the greedy optimisation mechanism - however, this isn’t revisited in the experimental evaluation, whereas it would add to the paper’s completeness if the authors could include an example of how/where such behaviour manifests itself.
questions: My largest reservations around this work lie around the lack of citation and connection to _“Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks”_, which shares a lot of the motivation and approach of the work presented here. Again, while I appreciate that the two use-cases have different requirements, the overall significance and novelty of this paper come across as slighter in light of this other work.

limitations: I appreciated the discussion on the quality of the weighted greedy optimisation procedure, which was also appropriately hinted at in the introduction of this work. As mentioned earlier, providing an anecdotal example of where this could occur (even if synthetic) would be helpful.

There should not be any ethical or societal issues tied to the publication of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presentes BMFAL-BC, a new batch multi-fidelity active learning approach that takes into account budget constraints. Differently from prior work, BMFAL-BC selects query points at different fidelities ensuring both that the queried points have low correlation among themselves and that the budget is constraint is complied with. The intuition behind ensuring low correlation among the queried points is that higher diversity will contribute to a better learning process, i.e., higher learning efficiency, hence decreasing the required number of queries and the cost of the learning process. The authors prove bounds for their algorithm and compare their approach against state-of the art multi fidelity active learning approaches, showing that BMFAL-BC finds better solutions at cheaper learning costs. 
strengths_and_weaknesses: **Originality:** The authors propose a new method that extends existing work in order to account for budget constraints and thus decrease the cost of the learning process. This is a novel combination and application of existing techniques to the context of batch multi-fidelity active learning. As far as I know, active learning related work is adequately cited however there is significant (and recent) work in the context of multi-fidelity hyper-parameter tuning optimisation which is not mentioned. Specifically, the authors may find works such as [1] and [2] relevant for their research. [1] seems a particularly good citation for line 123.

**Quality:** The claims seem to be sound, however I did not carefully check the math and proofs. The evaluation methods are appropriate and consistent with those used by related work. The weaknesses and limitations could be discussed more explicitly. One possible limitation that is not discussed is the use of one NN for each fidelity. It would be interesting to explore research directions that address this overhead and use only one NN, for example.

**Clarity:** The active learning setting could be better described. It is not clearly explained how each batch of samples actively contributes to the learning. There could be more information on the experiments and parameters tested (in table format) in the appendix. This would make reproducibility easier. Also, some guidance on possible ways to set the budget could aid potential users of this approach.

**Significance:** The problem tackled in the paper is a relevant one and having approaches to optimize the search and learning process at lower costs is valuable for the community. The proposed method is better than existing ones on the evaluated scenarios.


[1] Mendes, Pedro, et al. "Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling." 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 2020.

[2] Klein, Aaron, et al. "Fast bayesian optimization of machine learning hyperparameters on large datasets." Artificial intelligence and statistics. PMLR, 2017.

questions: **a)** I don't understand the meaning of "the cost of computing examples at each fidelity, was set to the normalized average running time". Does this mean you gathered queries from each fidelity multiple times, measured how long each query took, and set that as the cost? Also, "the normalized average running time" of what?

**b)** In the sentence "For active learning, we randomly generated {10, 2} and {10, 5, 2} training examples", what does the 10 and 2 stand for in {10, 2}? Same question for {10, 5, 2}.

**c)** When BMFAL-BC selects a batch, does it test all points in that batch? Also, do all points in a batch belong to the same fidelity?

**d)** If my understanding is correct, BMFAL-BC has a budget B per batch, which it can spend to evaluate several queries. In your implementation of DMFAL-BC do you allow it to select as many queries/points as it needs to reach budget B before you consider the batch as complete? Or do you terminate	 the run the first time DMFAL-BC is complete?

**e)** Is the accumulated cost equal to number of evaluated batches x B?

**f)** How many more queries does BMFAL-BC execute compared to DMFAL-BC?

**g)** What is the time complexity/overhead of the proposed method compared to existing methods?

**Comments:**
- PDE is introduced in line 224 however it is used already in line 25. Maybe introduce it the first time you use it for clarity.
- Lines 38-42 lack supporting references
- The sentence in lines 226-227 is not complete.
- The legend of Figures 2 and 3 could be made easier to read: all methods have BC in their names, so BC is redundant. Also, maybe you could align the methods with the same name, i.e., MFAL-BC would be above MFAL-BC-RF and DMFAL-BC above DMFAL-BC-RF
- typo in line 485? when --> well

limitations: No limitations were discussed. Under which conditions can the method be applicable? Do the underlying problems need to adhere to any specific format? Are there any underlying assumptions? For instance with respect to the shape of the function under optimisation?
review_title: null

Meta-review: metareview: The paper proposes a new batch active learning algorithm for multi-fidelity tasks. The proposal is a sequential greedy algorithmwith approximation guarantees and authors provide extensive experimental comparison.

The reviewers generally agree that the paper should be accepted.

In the camera-ready version, the authors are strongly encouraged to include:
- additional discussions that they promised in their rebuttal
- additional experimental results with more than 3 fidelities
- a pointer to their open-sourced code and datasets
confidence: Certain
Decision: Accept