Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents an active learning framework for a distributed learning system, i.e., a fleet of autonomous vehicles, collecting data for a perception system on the roads. Ideally, the data collected is limited in size, but at the same time, the most informative and diverse for a neural network to learn from. The paper builds upon the previous work of submodularity for active learning, and provide a solution called INTERACTIVE, which minimizes communication between the agents, while distributed maximizing the information gain. Experimental results show that the proposed solution yields similar results to the completely centralized approach, while outperforming the purely distributed one.
review: Pros:
- the paper grounds itself in practical scenario of data gathering for autonomous cars. Perhaps in control literature, there has been similar set ups with distributed systems/sensing, e.g., sensor network placement, etc, the paper brings these ideas for practical robotic scenarios.
- the paper is generally well written in terms of explanations. Especially, the paper clearly outlines the assumptions and settings.


Cons:
- in the experiments, the choice of baselines should be motivated.
The paper chooses a completely distributed approach and a centralized approach as baselines. Why not add a random baselines, which picks from the complete data pool (like centralized) and the data pool of each robots (like decentralized)? While experiments do validate the design of the approach to certain degrees, readers may appreciate more, when a thorough comparison to basic baselines (random uniform sampling) and other pool-based active learning methods for distributed systems, e.g, uncertainty sampling.  

- the claimed contributions and related work require clarifications.
The paper states that one of its contribution is to frame the data collection problem of multiple robots as submodular maximization problem. Does not reference 46 and 47 also use submodular maximization approach for similar problem, where the goal is to maximize the information gain? It would help to comprehend the contributions of the paper. 

I also think that some of the statements in the abstract can be misleading, The statements on the performance in particular, is with respect to one baseline, which is completely decentralized, while the paper utilizes certain communication with some of the robots. I am not sure, if that would be a fair comparison, and I would appreciate for the clarifications.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I generally find the paper interesting and relevant for many problems in robotics. The paper is also well written, and has rigours for the presented method. I however would like to see some of the clarifications to the raised issues above, before making the final recommendations.

------------------------------------ Post rebuttal ----------------------------------

I have increased the score to weak accept, since the authors have addressed most of my comments through experiments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper studies how autonomous vehicles (AV) can improve their datasets for perception and planning by actively selecting new datapoints for training. When deploying a fleet of AVs it is impractical to collect and process all available data due to bandwidth constraints and computational cost. Active learning methods allow for choosing only the most informative datapoints, yet distributed execution on multiple vehicles can lead to redundant data and thus poor performance. The authors propose a hybrid method where all AVs iteratively select their next data set. Here, each vehicle builds on the data collected by all AVs in previous rounds. The problem is formulated as a submodular maximation, and the authors show that their approach is a 2-factor approximation. Simulation results suggest that the method performs equally well as a centralized approach and clearly outperforms a purely distributed approach where robot exchange no information.

review: Overall, the paper is well written, easy to follow and studies a relevant and active problem. The proposed approach is technically sound and comes with a thorough theoretical analysis. The simulations are using real-world data sets with a variety of different configurations and showcase advantages of the proposed method. However, I have a few comments that should be clarified, please see below.


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, this is a well-written and original paper with a solid contribution. However, my two main concerns are the assumptions about monotonicity and submodularity that need further discussion, and some clarification on the simulation results since it is quite surprising that there is no gap between centralized and the proposed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper tackles the problem of intelligently selecting useful data collected by a multi-robot system under constraints. Each robot collects data, but only some of them are useful as other robots might have collected similar data already. The collected data is sent to a cloud server, where the underlying machine learning model is updated. 
The paper proposes to allow communication between the robots to effectively select a subset of their newly collected data sets based on a submodular objective that involves similarity measures between data points. The whole data set can not be sent to the cloud due to bandwidth constraints. 
The proposed method is shown to outperform the baseline and achieve the same performance as a centralized method which assumes to have access to the whole data set.
review: Review:

Strengths:
- Nice overview of related work, especially explicitly mentioning the differences between the proposed method to related works is very valuable 
- Nice writing. The paper is easy to read such that the reader can easily follow the story. 
- Promising results


Weaknesses:
- No major weaknesses, however, additional comparisons would be nice (see Questions for Rebuttal)
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents an interesting approach to Fleet robotics with promising results. I believe it is relevant to the research community.
review_title: null

Meta-review: The authors study a fleet active learning approach where autonomous agents can select training data samples to improve local models. The chosen approach in particular utilizes submodular maximization in the sample selection process. The review process resulted in a uniform recommendation of "weak accept" among the reviewers. Among the strengths of the work highlighted by the reviewers were
in particular that the work is well written and readable, that the grounding of the problem was well motivated and potentially relevant to many problems in robotics. The questions raised by one reviewer regarding the submodulariy assumptions are I believe an area that may warrant additional future investigation in particular and a careful discussion of potential edge or failure-cases could further strengthen the work.
Decision: Accept (Poster)