Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work proposes a BRI system that allows users to control a robot to perform a diverse set of tasks using only non-invasive EEG. Users command the robot to complete tasks by first selecting the object to interact with, then selecting the type of interaction (corresponding to a particular robot action primitive), and finally selecting where to interact with the object. The system leverages robot learning to reduce human effort when selecting both how and where to interact with objects, attempting to predict users' intent based on the object they are manipulating. In a human study with 3 participants, the authors demonstrate their method on a set of 20 complex tasks.
review: # Strengths
- The paper is very presented very clearly and thoughtfully. In particular, the authors have done a really good job of making the text digestible to an audience who may not have much familiarity with BRI/BCI techniques (e.g. Question and Answer section provided in Appendix, etc.)
- NOIR is demonstrated on a truly challenging set of long-horizon tasks. The activities selected for evaluating the system are convincingly difficult and well-motivated.

# Weaknesses
- The human study presented is very small, making it difficult to extrapolate much from the results.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is clear and its central ideas are thoughtfully communicated. While the authors could benefit from more strongly differentiating this work from the existing literature from a technical perspective, their contributions are convincing overall. Further, the authors showcase their method on the most challenging set of long-horizon tasks that this reviewer is personally familiar with, which is impressive in and of itself. This paper is limited most by its extremely small human study size, which makes it difficult to accept the results at face value. While n < 5 studies are not uncommon in BCI/BRI literature, this study presumably did not face the same types of participant recruitment challenges since this work was performed exclusively with able-bodied people. This paper appears sound in every other aspect, however, so I will recommend that the paper be accepted on the condition that other reviewers do not also view the limited study size as a severe limitation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a framework for codifying robots from human EEG data called NOIR.
NOIR uses machine learning methods to analyze human EEG data acquired from EEG and performs human goal decoding.
The robot then performs primitive skills along with human goal prediction.
A benchmark dataset was used in the experiment. The performance of the proposed system was tested on various skills.
review: This paper deals with the very challenging topic of controlling a robot from a person's brain waves.
Therefore, the importance of this research is very high.
However, extensive expertise is required to understand this paper.
For example, brain, BIM, robotics, and machine learning.

The experiments were primarily a detailed analysis of the proposed system. Due to the nature of the research, comparative validation with other similar methods is expected to be complicated.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: This paper is challenging to read due to the field's broad scope.
However, it is interesting because it makes use of recent machine-learning techniques.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The main idea and contributions of the paper are the following points: 
* The authors developed a brain-robot interface (BRI) system that is capable of performing a wide variety of tasks while adapting to user preferences.
* The authors successfully demonstrated that the proposed BRI system could perform a wide variety of 20 household activities, such as making Sukiyaki, cleaning a book and watering a plant, using a robotic agent through brain signals.

review: Strengths:
* The paper is well organized and seems to provide sufficient information to the reader. Therefore, the clarity of the paper is high.

Weaknesses:
* The main contribution of the paper, the versatile task execution using a robotic agent through brain signals, appears to be performed without leveraging robot learning methods. The object and skill selection and parameter learning methods were only applied to a few tasks to show their effectiveness. Thus, the quality and significance of the paper in terms of robot learning is not high.
* Related work on BRI systems using the object and skill selection and parameter learning methods are not mentioned in the paper. Thus, it is difficult to assess the originality of the paper with regard to robot learning.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The contribution of the paper is not sufficient: weakness in both quality and significance of the paper. The main contribution of the paper seems to be less related to robot learning.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors present an approach where an operator can teleoperate a robot via an EEG-based Brain-Computer-Interface. The main contribution in my opinion is a learning-based method which allows the system to preselect the intended action of the operator (e.g., pick mug from the left side) which reduces the need of the operator to interact with the system through EEG and therefore speeds up task execution. The authors decompose the "human goal decoding" into three parts. 1) What object 2) How to interact (what task) and 3) Where to interact (task parameters). For each part they developed a different Brain-Computer-Interface suited for selecting choices or choosing parameters.
review: The paper in my opinion is well written and straightforward to understand. The line of argument behind the approach is reasonable and the design decisions made by the authors are well explained and reasonable as well. The approach is evaluated in various experiments and the results seem to be convincing. Citations seem to be adequate, however I am not an expert in this field and therefore can't really tell if important related work has not been cited.

For the skill classification part, If I understood it correctly, the operator can choose from 4 different classes: left hand, right hand, legs, rest. However, I don't see how those 4 classes are connected to all the 20 tasks which are performed by the subjects. There are tasks where for instance multiple/different sub-tasks can be performed with the left hand. Do you show the operator a list of subtasks and how they are associated to the 4 skill classes? This should be clarified in the paper.

I was wondering how the object selection method would perform against a simpler but maybe more robust method like highlighting the objects one after the other and letting the operator select an object based on a simple detected signal pattern (e.g., highlight the objects one after the other until the operator selects the current object by emitting a certain signal). This would basically require a sequence of binary classifications instead of a single N-class classification. In general, a comparison against one or multiple baseline methods for the individual what/how/where parts would have made the paper stronger.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I chose my rating mainly because I did not find any major flaws in the paper.
review_title: null

Meta-review: The reviewers commend the paper for being well-written and organized. The proposed system to teleoperate a robot via an EEG-based Brain-Computer-Interface with a real human-study evaluation was well received. I encourage the authors to further address any remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Poster)