Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a robotic benchmark with focus on manipulation tasks. It addresses the longstanding problem of reproducibility of real-world robotic experiments. The proposed contribution is a framework that relies on a set of baselines to be implemented by multiple research labs to create a global ranking. This partially addresses the problem of variability in the setup that leads to different outcomes in different research labs.
strengths: The paper addresses a serious problem in robotics research that is still largely unsolved and hinders progress in the field. Although the proposed framework only addressed the task of manipulation, it contributes ideas and findings to address these issues.
weaknesses: The implementation and practicality of such an implementation remain unclear.

Scalability and missing experiments:
a) From the current description (line 143-144), the researcher that is testing their new algorithm must locally reproduce the existing baselines. As the number of baseline grows this becomes more and more tedious and impractical. How is the case of missing local baselines handled?

b) As the number of baselines grows, it becomes impractical that N labs maintain all M baselines. Some labs might only be able to implement a subset of the baselines. How is this handled in the current rankink scheme?

Using this benchmark:
Let's assume I am a researcher who would like to publish a new method and compare against the baselines of R2B. Let's also assume that I implemented all the existing baselines of the benchmark locally. No global ranking of my method exists yet, so I can only report my local ranking. How does this local ranking help the reviewer of my research paper to assess the proposed method? To my understanding, the proposed benchmark only provides additional information through its global ranking. As such, my method will only be compared against if all the labs contributing to the benchmark also implement my method.

Malicious intent:
It cannot be ruled out that research groups deliberately submit weak implementations of competing approaches/groups. How can this be addressed?

Experiments:
a) Section 5.4 introduces a vision front-end that is very simplistic together with the following sentence: "We want to focus RB2 on measuring manipulation performance". However, vision is used in closed-loop in many manipulation approaches and thus an integral part of solving the problem. What is the reasoning behind trying to exclude the vision pipeline from the benchmark? Are the researchers contributing to RB2 tied to using the simple model proposed in this paper? If not, please clarify this statement.

b) It would have been insightful to add a third research group that only partially contributes to the benchmark. This way it would be clear how this case is handled.
correctness: The evaluation methodology, as described in the paper, is constructed in a sound way. The experiments seem appropriate.
clarity: See weakness section. Missing clarity in how the benchmark implements corner cases of missing experiments and how the benchmark helps in the review process.
relation to prior work: Yes, the difference to prior work is clearly discussed
additional feedback: The title suggests a generic robotics benchmark while the paper is specifically about manipulation. This must be adapted to avoid confusion.

My rating is based on the fact that there are several open questions about implementation, utility, and documentation. I am open to raising the rating if my concerns are properly addressed. Overall, the proposed approach is a step in the right direction.
review_title: A step towards standardizing manipulation experiments but questions remain

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work proposes a way to benchmark robotic algorithms so that the findings with every new algorithm are more or less consistent across different evaluation settings, and add real value to robotics research. The contributions of this work are two-fold:

1. It introduces a method to evaluate a proposed robotic algorithm in settings that rigidly enforce experimentation protocols but could differ in system/intrinsic parameters. Following the evaluations, the models in consideration are assigned a global rank by combining the local ranks across different settings. This helps in testing the generalization capability of an algorithm and identify models that are better than others in the majority of the settings.

2. It also introduces 4 tasks of varying difficulty that could be replicated across multiple laboratories with variations only in the system/intrinsic parameters of the environment, agent, and the object(s) involved, and 5 state-of-the-art baselines on these tasks that have been tested in 2 different lab environments. This repository of baselines and their results on the benchmark could offer useful trends and insights needed in designing even stronger models, and also allow for making systematic progress in the tasks.
strengths: 1. Interesting idea: This is a simple but interesting idea to ensure that new models and algorithms that are claimed to be state-of-the-art actually offer benefits over other pre-existing techniques in settings that follow some standardized experimentation and task instantiation protocols. That helps in providing a level playing field for most researchers and reduces the chance of optimizing the evaluation setting(s) to suit one's own model/algorithm.  

2. Model zoo: the paper tests 5 strong baselines in 2 different settings with very similar experimentation protocols and task instantiation parameters and find trends about the relative strengths of the algorithms. This not only reveals which algorithms actually work better but also allows a new researcher in the area to efficiently run experiments because they know which model is the one to beat. 

weaknesses: 1. L155, 'Note that individual ... their results to a central repository': If a newly proposed model isn't uploaded to the model zoo, how would the global ranking be generated? As far as I understand, each new model should be tested across some standard lab environments and the results from these trials should be aggregated to finally rank all the models.

2. Global ranking algorithm: The paper doesn't clearly mention how the rankings from the 2 labs are aggregated to generate the final global ranking (table 1).  


Edit on 9/29:
My concerns have been adequately addressed. I vote for accepting the submission. I am changing my rating for the paper accordingly.
correctness: The claims made in the paper and the supporting experiments/evaluation methods look correct to me.
clarity: The paper is written clearly.
relation to prior work: The work compares with prior work and tries to identify issues with existing benchmarks.
additional feedback: Minor typos:
1. L309, 'Thus, Our final': 'Thus, our final'? 

2. L255, 'and tend the models tend': 'and the models tend'?
review_title: Intersting and useful technique for robotic benchmarking

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a method of allowing for benchmarks across a variety of hardware and environments. In the proposed RB2 benchmark new algorithms are tested alongside current baselines over four manipulation tasks. These relative local results are then passed to a centralized system where multiple local rankings are aggregated into  global rankings.  
strengths: Evaluating the performance of robots is challenging due to the fact that they operate in the real world, in environments that cannot be fully controlled and that no two robots are identical, even those of the same model. This paper presents a method to mitigate these challenges by requiring the use of a set of baselines and then using the relative local rankings of a proposed method versus the baseline as the evaluation metric.
weaknesses: To get accurate results requires significant buy in by the community. Adding in a different set of items might be an issue. Using the same objects as YCB objects might mitigate this.
Limiting the tasks to manipulation with objects already in the robot’s hand might be too restrictive.
correctness: The training/test splits should be universal, comparing policies trained on scooping nuts and tested on coffee might not be similar enough to one trained on coffee and nuts and tested on nuts.
clarity: The paper was clear and easy to follow.

relation to prior work: The paper discusses other robotics benchmarks, which are focused on specific tasks or hardware, while this work proposes a general approach across diverse hardware and multiple tasks.
additional feedback: It would be interesting to see if there was a way to integrate previous benchmarks if they include a local comparison to a baseline.
review_title: A robotic manipulation benchmark to that works over with heterogeneous hardware and environments 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a robot benchmark that includes 4 tasks (scooping, pouring, zipping, and insertion) that can be performed by a robot under different physical settings. The paper focuses on defining the benchmarking process for robotics that accounts for real-world uncertainties. The presented tasks are evaluated on existing SOTA models. The paper concludes by showing some SOTA models are indeed robust to changing conditions. Their presented methodology can also be adopted by other researchers for their environments or tasks, and models.
strengths: 1. The paper considers real-world uncertainties when constructing benchmarking results
2. The framework is expandable by adding more task and lab settings and can be adopted by other research groups
3. The benchmarking process is comprehensively explained
weaknesses: I would like to ask the authors few clarification questions:
1. In the proposed system of global ranking between different research groups with different robotic setups, would it make it easy for someone or a group of people to game the system by evaluating their models in an easier setting and influencing global ranking ?
2. How easy would it be to expand this system to include more tasks and benchmarking models? More specifically are you proposing to create and maintain this system and accept submissions or is it more of a guideline for others to build their ranking systems. 

correctness: Everything appears to be correct (to me).
clarity: The benchmarking process is comprehensively explained. The results on all 4 tasks with all the models are clearly shown in the graphs and table. 

relation to prior work: The paper doesn't claim to be first of a kind benchmark but discusses how it differs from previous contributions

additional feedback: I don't have any additional feedback
review_title: Review for RB2: Robotics Benchmarking with a Twist

Meta-review: This paper presents a method of benchmarking robotic manipulation tasks across a variety of hardware and environments. The core idea is that multiple different groups implement a set of baselines, and the local performances on each baseline are combined into a global ranking against which work can be compared. The paper introduces four baseline tasks, which are tested in two different research settings.

The approach proposed in the paper addresses a real existing problem in robotics, where different platforms and environments tend to make comparison of research results difficult. The approach is clearly and thoroughly explained. As conceived, the ranking approach is amenable to being extended with additional baselines and is constructed to have additional research groups join. However, some concerns remain. The problem explicitly omits the vision and grasping components of the manipulation pipeline, which may not be appropriate in all cases. More seriously, when a new method/model is proposed, it must be implemented by multiple groups in order to receive a global ranking, and it's not clear how the purely local ranking should be considered; both for this reason and in general, the proposed benchmark is of most use when there is significant buy-in from multiple research groups, which may be difficult to bootstrap. Reviewers also have concerns about the potential for bad actors to affect the system by maliciously manipulating their claimed local results; however, these concerns are consistent with the current state of the art in robotics. Authors have engaged constructively with reviewers, and are encouraged to thoroughly incorporate improvements from these discussions in subsequent versions of the paper.
Decision: Accept