Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a framework for generating and simulating multiple scenarios that involve contact data with human demonstrations. The work leverages the pybullet simulation environment and develops robot-centric components for control and sensor measurement. In addition, the work integrates ROS into the the pybullet interface which readily enables the use of human feedback for collecting demonstrations. 
strengths_and_weaknesses: The paper presents several use-cases for their proposed framework. This is presented by the plenty ways to interface and use the presented framework. The breadth of robots and interfaces is very useful when prototyping and developing robotic experiments. 

The paper can relate and compare itself more closely with existing simulators that offer similar interfaces with ROS. One example is Gazebo which not only has similar interfaces, but can also choose between contact simulators. Ideally, this work could benefit from allowing other simulators to interface with the ROS environment and the utilities created. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The work provides a necessary tool for robotics that can facilitate research progress, especially with human demonstrations. However, there are other tools that claim similar libraries that attain the same contributions. It would strengthen the paper to illustrate where their library stands-out and why this is a preferred way of developing robot simulation utilities. 
issues: A comparison with Gazebo (and other similar libraries).
Comprehensive overview of other robotics toolbox-libraries, what they provide, and the gap that this fills in.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a ROS-PyBullet Interface bridging between the popular physics simulator PyBullet with Robot Operating System (ROS). Different tools are supported in addition to the native PyBullet and ROS functions such as Inverse Kinematics interface and interfaces for human interaction.  
strengths_and_weaknesses: **Strengths:**
* The paper clearly outlines the features in ROS-PyBullet and made it very clear what features are supported in the current frame work. 
* The framework presented can be a very useful tool for the robotics community. There’s definitely value in developing systems like this.
* Demonstration of hardware integration is helpful. 

**Weaknesses:**
* It is not entirely clear what the contribution of this work is after removing the pre-existing modules. In the contribution section, it is claimed that one contribution of the work is the proposed framework that contains different pieces of software, which demonstrates the engineering effort went into the work, but does not speak for the research contribution. 
* Related to the previous point, seems like section 2.4, 2.5 and 2.6 are not related to integrating ROS with PyBullet, they are more like standalone ROS nodes (they can be very useful nonetheless). 
* It is also not clear how the proposed system helps different projects (Does it achieve things that was not possible before, or does it make things so much easier?) Section 3 listed different use cases for the ROS-PyBullet interface. While it definitely helps to see the example usage, what might be even more helpful is to see how these examples are done differently in the proposed system compared to other systems. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: ROS and PyBullet are both tools commonly used in the robotics community, and integrating a physics simulation engine with ROS could potentially benefit future research. While the authors clearly achieved the goal of integrating these two tools and even provided a few other tools to make it more user friendly, it is not entirely clear how this integration is fundamentally better than other existing frameworks (or a combination of frameworks). The engineering effort and quality of this work are clear but the research contribution is not well presented.
issues: * This work seems like an integration of multiple pre-existing tools and the contribution section also more or less listed the features of the framework rather than stating the contributions.  
* It might be helpful to clearly separate what parts of in this framework are pre-existing and what parts are developed in this work. 
* As a paper proposing a system consisting of existing resources (which could be okay), it is very important to demonstrate why it is better than everything else that’s out there. This can be done through experiments comparing with other software, or a dedicated subsection clearly stating the difference and improvement.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors present a bridge between PyBullet and ROS. This empowers researchers to connect an easy-to-work with simulator that’s suitable for simulating contact-rich tasks directly into robotics contexts, where ROS is a dominant middleware. The authors describe the components of their bridge and then present four use cases that highlight the value of the system.
strengths_and_weaknesses: Strengths

* Serves a real need in the community. I am aware of several researchers that have independently built subsets of this bridge to suit their needs. 

* Well motivated use cases. While simulation solutions oriented toward data generation for learning are the predominant use case in this community, the author’s use cases illustrate some of the main scenarios where simulation is a key part during execution on a real robot system.

Weaknesses

* Limited comparison or contextualization in the landscape of comparable solutions. The paper does not have a related works section, nor does it allude to alternative solutions. The framework tackles a smattering of technical needs (see list on 48-54), which means that readers will come with a variety of comparisons in mind. A table that highlights comparable features and limitations of different options would help.
    * OSRF recently worked with NVIDIA to bridge Gazebo and Isaac Sim (which is PhysX-based). How does this work relate?
    * Readers of CoRL papers may be looking for a contextualization wrt simulators for learning vision-based policies. As the authors call out in 137-140, lots of research is being done that leverages realistic simulated environments. PyBullet doesn’t include a good renderer out of the box. Even discounting the vision use case, nicer renderers are an enabler for work that asks users to interact with the simulation. 
    * You may also get readers who are primarily ROS users. They will find the comparisons to Gazebo provided unilluminating, inaccurate, and incomplete. You should highlight why Gazebo can't easily address the needs of certain use cases. In my assessment, it has much more to do with Gazebo requiring the user to know C++ than simply that it's default backend isn't good enough. These readers will also pause at the description of an independent set of kinematics solvers and motion planners, when they are likely most familiar with MoveIt as an integrated solution here. Highlight the benefit you get, be it in ease of use or in capability, compared to simply loading the same URDF into MoveIt and using it's solvers.
    * Necessarily, every effort to enhance robotics simulation platforms will be lacking in some feature, but the onus is on the authors to help prospective users understand how their solution fits in the landscape.

* The motivations for some features may be niche to many readers. There are a couple places (169, 176) where the authors motivate a feature by citing a use case from their own work, indicated by the anonymization of the reference. If there aren’t independent works that have a similar need or use case, it’s an indication that the audience for a particular feature is limited. The authors should consider whether it is worth providing more exposition on the value of these features, or to frame them in an alternate way that more readers will connect with.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors provide a set of tools which will be valuable either as a reference or even as an immediate aide to researchers. The specific implementation path they took is pragmatic and maintains the strengths of using a common middleware like ROS while enabling some tasks that wouldn’t be possible without a well-behaved simulator in the loop. I think there is room to improve in regards to identifying the key selling points of the system and highlighting contrasts with other options that readers may be considering.

**Update after rebuttal:**

The authors have provided a reasonable qualitative characterization of the landscape of alternatives. I believe this will greatly help readers understand this work and how it fits into the broader landscape. I have raised my recommendation to accept.
issues: * Contextualize the work. Provide qualitative comparisons to alternatives. Think about the people you would most like to use your system, and the people who are most likely to come across it, and highlight the value for them.

* Gazebo supported four simulation backends from its earliest days. In fact, in its recent incarnations, DART has become the default. If the other backends are deprecated, or lack a specific feature, point to a notice of the deprecation or documentation of the limitation. You should clarify specifically which version you are referring to, as they have muddied the branding considerably while flip-flopping on their Ignition rename.
review_title: null

Meta-review: metareview: Summary:

This paper proposes the ROS-PyBullet interface, a framework that provides a bridge between the reliable contact/impact simulator PyBullet and ROS.
It also provides additional utilities to facilitate human-robot interaction in simulated environments.

Strength:

The paper clearly outlines the features in ROS-PyBullet and made it very clear what features are supported in the current frame work.
Since the simulation solutions oriented toward data generation for learning are the predominant use case in this community, the framework presented can be a very useful tool for the robotics community.

Weakness:

The paper does not have a related works section, nor does it allude to alternative solutions.
It is not entirely clear what the contribution of this work is after removing the pre-existing modules.
Engineering effort is easy to understand; however, it is unclear what the research/academic contributions are.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)