NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
Authors: Ruohan Zhang
Sharon Lee
Minjune Hwang
Ayano Hiranaka
Chen Wang
Wensi Ai
Jin Jie Ryan Tan
Shreya Gupta
Yilun Hao
Gabrael Levine
Ruohan Gao
Anthony Norcia
Li Fei-Fei
Jiajun Wu
Venue: CoRL 2023 Poster (2023)
Keywords: Brain-Robot Interface
Human-Robot Interaction

Abstract: We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals. Through this interface, humans communicate their intended objects of interest and actions to the robots using electroencephalography (EEG). Our novel system demonstrates success in an expansive array of 20 challenging, everyday household activities, including cooking, cleaning, personal care, and entertainment. The effectiveness of the system is improved by its synergistic integration of robot learning algorithms, allowing for NOIR to adapt to individual users and predict their intentions. Our work enhances the way humans interact with robots, replacing traditional channels of interaction with direct, neural communication.


Reviews:
[{"review_id": "zUB1TskpgS", "reviewer": "Reviewer_SXKL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present an approach where an operator can teleoperate a robot via an EEG-based Brain-Computer-Interface. The main contribution in my opinion is a learning-based method which allows the system to preselect the intended action of the operator (e.g., pick mug from the left side) which reduces the need of the operator to interact with the system through EEG and therefore speeds up task execution. The authors decompose the \"human goal decoding\" into three parts. 1) What object 2) How to interact (what task) and 3) Where to interact (task parameters). For each part they developed a different Brain-Computer-Interface suited for selecting choices or choosing parameters.", "summary_of_recommendation": "I chose my rating mainly because I did not find any major flaws in the paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As written above, could the authors please clarify how the 4 skill classes are mapped to a subtask? If multiple actions can be performed with the same hand on an object, how do you choose between the corresponding actions?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper in my opinion is well written and straightforward to understand. The line of argument behind the approach is reasonable and the design decisions made by the authors are well explained and reasonable as well. The approach is evaluated in various experiments and the results seem to be convincing. Citations seem to be adequate, however I am not an expert in this field and therefore can't really tell if important related work has not been cited.\n\nFor the skill classification part, If I understood it correctly, the operator can choose from 4 different classes: left hand, right hand, legs, rest. However, I don't see how those 4 classes are connected to all the 20 tasks which are performed by the subjects. There are tasks where for instance multiple/different sub-tasks can be performed with the left hand. Do you show the operator a list of subtasks and how they are associated to the 4 skill classes? This should be clarified in the paper.\n\nI was wondering how the object selection method would perform against a simpler but maybe more robust method like highlighting the objects one after the other and letting the operator select an object based on a simple detected signal pattern (e.g., highlight the objects one after the other until the operator selects the current object by emitting a certain signal). This would basically require a sequence of binary classifications instead of a single N-class classification. In general, a comparison against one or multiple baseline methods for the individual what/how/where parts would have made the paper stronger.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "pD5_wY9rRH", "reviewer": "Reviewer_beZJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a framework for codifying robots from human EEG data called NOIR.\nNOIR uses machine learning methods to analyze human EEG data acquired from EEG and performs human goal decoding.\nThe robot then performs primitive skills along with human goal prediction.\nA benchmark dataset was used in the experiment. The performance of the proposed system was tested on various skills.", "summary_of_recommendation": "This paper is challenging to read due to the field's broad scope.\nHowever, it is interesting because it makes use of recent machine-learning techniques.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "My concern in controlling robots from brain waves is safety.\nSafety is an essential aspect for both people and robots.\nDoes the person always need to wear the EEG to control the robot?\nIs there any danger of the robot destroying objects around it?\nAnother aspect that seems essential is versatility. Are there any individual differences in this system?\nDoes a person need to be trained in using this system to pass away the robot?\nThe above may be slightly different from the purpose and contribution of this paper.\nHowever, if there is a direction for a solution to these questions, they should be coordinated.\n\nIn order to control a robot from brain waves, a mechanism will be needed better to capture the characteristics of the brain wave data.\nSeveral machine learning methods have been selected for the proposed method, but it is unclear why they are considered suitable.\nIn addition, the novelty of machine learning techniques such as robot learning is unclear.\nThe proposed system appears only to use existing models.\n\nThe text in figures in the text is very small and difficult to read.\n\nMany references to the Appendix. Therefore, it is difficult to understand this paper alone fully.\nThe study is rich in content, and the omission of many details may be a disadvantage.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the very challenging topic of controlling a robot from a person's brain waves.\nTherefore, the importance of this research is very high.\nHowever, extensive expertise is required to understand this paper.\nFor example, brain, BIM, robotics, and machine learning.\n\nThe experiments were primarily a detailed analysis of the proposed system. Due to the nature of the research, comparative validation with other similar methods is expected to be complicated.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "97eGHrL83t", "reviewer": "Reviewer_KKsW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The main idea and contributions of the paper are the following points: \n* The authors developed a brain-robot interface (BRI) system that is capable of performing a wide variety of tasks while adapting to user preferences.\n* The authors successfully demonstrated that the proposed BRI system could perform a wide variety of 20 household activities, such as making Sukiyaki, cleaning a book and watering a plant, using a robotic agent through brain signals.\n", "summary_of_recommendation": "The contribution of the paper is not sufficient: weakness in both quality and significance of the paper. The main contribution of the paper seems to be less related to robot learning.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* I would like to know the difference between the proposed system and existing BRI systems from the robot learning point of view.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The paper is well organized and seems to provide sufficient information to the reader. Therefore, the clarity of the paper is high.\n\nWeaknesses:\n* The main contribution of the paper, the versatile task execution using a robotic agent through brain signals, appears to be performed without leveraging robot learning methods. The object and skill selection and parameter learning methods were only applied to a few tasks to show their effectiveness. Thus, the quality and significance of the paper in terms of robot learning is not high.\n* Related work on BRI systems using the object and skill selection and parameter learning methods are not mentioned in the paper. Thus, it is difficult to assess the originality of the paper with regard to robot learning.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "1j1c6f2Rdy", "reviewer": "Reviewer_r85X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a BRI system that allows users to control a robot to perform a diverse set of tasks using only non-invasive EEG. Users command the robot to complete tasks by first selecting the object to interact with, then selecting the type of interaction (corresponding to a particular robot action primitive), and finally selecting where to interact with the object. The system leverages robot learning to reduce human effort when selecting both how and where to interact with objects, attempting to predict users' intent based on the object they are manipulating. In a human study with 3 participants, the authors demonstrate their method on a set of 20 complex tasks.", "summary_of_recommendation": "The paper is clear and its central ideas are thoughtfully communicated. While the authors could benefit from more strongly differentiating this work from the existing literature from a technical perspective, their contributions are convincing overall. Further, the authors showcase their method on the most challenging set of long-horizon tasks that this reviewer is personally familiar with, which is impressive in and of itself. This paper is limited most by its extremely small human study size, which makes it difficult to accept the results at face value. While n < 5 studies are not uncommon in BCI/BRI literature, this study presumably did not face the same types of participant recruitment challenges since this work was performed exclusively with able-bodied people. This paper appears sound in every other aspect, however, so I will recommend that the paper be accepted on the condition that other reviewers do not also view the limited study size as a severe limitation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Can the authors expand on how a task attempt is defined? Does this mean that a participant reached an unrecoverable state at some point in task execution and the task had to be reset from the beginning? Or does this indicate that they had to reattempt a particular skill if it did not yield the expected outcome? Some clarity here would help with the interpretation of Table 1.\n\n2. Can the authors provide some insights into why some tasks seem to require more attempts? Are participants encountering decoding errors (which may be mitigated by the safety mechanism but maybe not avoided completely?) or do they require more challenging/creative manipulation that caused the participant to need to try different strategies? I'm hoping to get a more intuitive understanding of how difficult these tasks are relative to one another. How do I interpret long task times and few attempts on one task vs. the same task time and many attempts on another?\n\n3. I don't see human time, presented as a percentage of total time in Table 1, as an intuitive metric. Human time is ~80 for most tasks, except for TrashDisposal and CovidCare - why is it lower for these two tasks? There seems to be some ambiguity as to whether human time (as a percentage of total time) is lower because decision-making/decoding occurred faster or if the robot skill execution actually took longer. Instead, I think it would be helpful to understand how much raw time humans are spending on decision-making/decoding for each manipulation step (for selecting how and where the robot will interact with an object). That way, we can better compare how long this process takes between tasks.\n\n4. The human study sample size is quite small, which significantly limits the conclusions that can be made about the effectiveness of the system. The paper would benefit from additional experiments with more participants, though it is understandable if this is not possible to execute within a short rebuttal period. At the very least, without these experiments, it would be helpful for the authors to more explicitly address this as a limitation in the text of the paper. While a small statement about the potential universality of the system is provided in the appendix, I would argue that a statement addressing the size of the study should more materially address how the reader can/should understand the actual evaluation results and should appear in the actual body of the main paper.\n\n5. I think the paper would benefit from more clearly differentiating itself from existing BRI literature. While it is clear that NOIR enables control for a wide variety of complex tasks, it would be helpful to illustrate just how challenging the demonstrated tasks are than those in previous literature (require more variety of skills? require more skills to be used in succession? are harder to plan for? etc.). I think it would also be interesting to compare/contrast this work to non-invasive BRI that enable continuous real-time (or hybrid) control, especially since there is evidence that users prefer to be actively involved in task execution rather than simply selecting goals before the robot moves.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n- The paper is very presented very clearly and thoughtfully. In particular, the authors have done a really good job of making the text digestible to an audience who may not have much familiarity with BRI/BCI techniques (e.g. Question and Answer section provided in Appendix, etc.)\n- NOIR is demonstrated on a truly challenging set of long-horizon tasks. The activities selected for evaluating the system are convincingly difficult and well-motivated.\n\n# Weaknesses\n- The human study presented is very small, making it difficult to extrapolate much from the results.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers commend the paper for being well-written and organized. The proposed system to teleoperate a robot via an EEG-based Brain-Computer-Interface with a real human-study evaluation was well received. I encourage the authors to further address any remaining points raised in the reviews for the final version of the paper.
Decision: Accept (Poster)