Designing a Human-Robot Collaborative Interface for Nursing Robots
Failure Recovery
Lorena Maria Genua1 and Zhi Li1
I. I NTRODUCTION
In recent years, many mobile manipulators and humanoid
robots have been deployed all over the world in hospitals,
nursing facilities, and homes [1]. The introduction of robotics
in healthcare addresses the significant challenge posed bya rising aging population, alongside a reduction in available nursing staff. These nursing robots are expected toperform some general-purpose assistance tasks to alleviate
nurses’ workload, such as fetching and delivering medicalsupplies, preparing and cleaning nursing workspace andpatient rooms, and taking vital sign measurements. However,robots operating in unstructured environments face frequent
failures and must rely on nurses to understand and resolvethese issues. For this reason, in our prior work, we havedeveloped an innovative multilateral multimodal humanrobot (MMHR) collaboration prototype system that enables
effective collaboration and communication between the robot
and both remote and local nurses (Fig. 1). In this paper,
we develop a user-centered recovery procedure, allowing therobot to receive guidance and learn from professional nurses
in case of failure. Therefore, we propose a human-robot
collaborative interface to facilitate robot recovery fromfailures. Through the interface, nurses can utilize high-level
control commands to leverage the robot’s primitive skills
for manipulation to recover from the failure states. We alsoestablish communication between remote and on-site nurses,
using Augmented Reality visual cues to exchange action
information. By combining these elements, our approach offers a comprehensive solution to effectively support nursing
robots in overcoming failure scenarios and learn from nurses’correction.
II. R ELATED W ORK
Nursing robots operating within unstructured environments pose significant challenges due to the unpredictable
and complex nature of these settings, which might lead tofrequent failures in robot autonomy. This underscores thenecessity for an effective human-robot collaboration framework to facilitate recovery efforts [2]. While some robots andmulti-robot systems have the capabilities to autonomously
resolve failures through pre-programmed corrective actions,these solutions are often inadequate when encountering
complex or unexpected errors, and therefore necessitate
human intervention [3]. Methods such as teleoperation andcorrective shared-autonomy allow for direct and real-time
1 Robotics Engineering Department, Worcester Polytechnic Institute(WPI), Worcester, MA 01609, USA {lgenua,zli11}@wpi.edu

Fig. 1: MMHR system includes a remote user employing a screen-based
interface and a local user wearing a mixed-reality headset to interact witheach other and with the robotic system.

human corrections [4]. These approaches can increase theoperator’s workload and might not leverage the full potentialof human guidance, impacting the operator’s trust in thesystem [5]. In contrast, high-level teleoperation interfaces
have been shown to improve task performance and perceived
mental and physical workload compared to low-level control
interfaces [6]. However, a gap exists in utilizing manipulation
actions for failure recovery within a collaborative frameworkthat also enables robots to learn from human inputs. Our
proposed approach is to develop a human-robot collaborative
failure recovery system that leverages the robot’s primitiveskills for manipulation to recover from failure states. By
incorporating high-level human commands during failures,the robot can learn to autonomously execute these skills infuture similar situations, thus improving its adaptability andability to recover from failures independently.
III. S YSTEM OVERVIEW
Our system integration enables the multilateral collaboration among nursing robots, local nurses sharing the robot’s
workspace, and remote nurses operating the robot froma different location. The nursing robot IONA (IntelligentrObotic Nursing Assistant), a mobile humanoid nursing robot
developed in our recent work [7], is equipped with multiple
Kinova Gen3 manipulator arms and RGB+D cameras forautonomous manipulation and perception in cluttered environments. The remote nurses utilize a screen-based graphicalinterface, while the local nurses use a mixed-reality headset(Microsoft HoloLens 2).
The robotic system comprises essential modules for task
execution and communication with remote and local nurses(See Fig. 2 (b)). The Task State Manager tracks theFig. 2: a) The User Interface for failure recovery includes two camera views and control panels to command high-level actions to the robot. b) Systemarchitecture and communication framework to command target pose to the robot and to visualize AR cues.

task progress and sends information related to the current
target to the Object Tracker for object detection. The
Pose Estimation module computes the 3D object positionof the detected object in the camera view and handles all
frame transformations. It uses Azure Spatial Anchor to set
a common coordinate reference system between the robot
and the local operator. Given the target pose, the Motion
Planner plans robot arm motions.
IV. I NTERFACE D ESIGN
Our proposed Graphical User Interface (GUI), shown in
Fig. 2, incorporates two camera views: a primary view froma camera mounted on the robot’s chest, and a secondary
view from a standalone workspace camera. Any information
pertinent to the robot’s current status, such as operational
states (e.g., in action, paused, fault, homing) and ongoingactions (e.g., grasping, placing), is displayed in an Information panel. Beneath this panel, toggle buttons offer options
to enable or disable the secondary camera view, as well as
to reveal the task list and progress status. In the event ofa detected failure, the information panel displays an error
message describing the issue. The remote operator can use
the provided buttons in the Control panel to engage with therobot and issue high-level action commands. The interface
provides users with a flexible method for selecting target
objects or placement locations. Specifically, the user canclick and drag the cursor across the screen to define arectangular area around a specific point of interest (See Fig. 2(a)). Subsequently, the user can utilize the buttons in theObject Manipulation panel to command the robot to:• Grasp: Grasp the selected object on the screen;• Place: Place in the selected location on the screen;• Set Aside: Move the object to a designated disposal area;• Try again: Re-attempt the previous action (i.e., graspingor placing).
The Robot Control section includes buttons to adjust thecamera view by Moving the camera up and down, andbutton to Home the robot to a predefined initial configuration. Should the robot fail to recover with human input fromthe remote user interface, the Local Operator Interaction
panel can be activated to establish communication with anon-site operator sharing the robot’s workspace. When thispanel is enabled, any command initiated through the Object
Manipulation panel will be directed to the local operator andwill not be sent to the robot. The available functions are:• Call: Send a notification to the local operator with astandardized message with a request for assistance;• Input text: Write a custom message for the local operator;• Define actions: Select a sequence of actions for thelocal operator to execute. For instance, they can selecta target object on the screen for grasping, then press”Grasp”. Subsequently, they may designate a placement
location for the object and press ”Place”. This process
sends augmented reality (AR) visual cues to guide the local
operator in task execution;• Send: Sends the input message and/or the AR visual cues
to the local operator;
The Task control section allows the user to Pause the robot,
Continue its action, or Cancel the ongoing action.
R EFERENCES[1] C. Nieto Agraz, M. Pfingsthorn, P. Gliesche, M. Eichelberg, andA. Hein, “A survey of robotic systems for nursing care,” Frontiers inRobotics and AI, vol. 9, p. 832248, 2022.
[2] S. Honig and T. Oron-Gilad, “Understanding and resolving failures inhuman-robot interaction: Literature review and model development,”
Frontiers in psychology, vol. 9, p. 351644, 2018.
[3] S. Reig, E. J. Carter, T. Fong, J. Forlizzi, and A. Steinfeld, “Flailing,hailing, prevailing: Perceptions of multi-robot failure recovery strategies,” in Proceedings of the 2021 ACM/IEEE International Conferenceon Human-Robot Interaction, 2021, pp. 158–167.
[4] M. Hagenow, E. Senft, R. Radwin, M. Gleicher, B. Mutlu, and M. Zinn,“Corrective shared autonomy for addressing task variability,” IEEErobotics and automation letters, vol. 6, no. 2, pp. 3720–3727, 2021.
[5] X. Zhang, S. K. Lee, H. Maeng, and S. Hahn, “Effects of failure types
on trust repairs in human–robot interactions,” International Journal ofSocial Robotics, vol. 15, no. 9, pp. 1619–1635, 2023.
[6] S. S. White, K. W. Bisland, M. C. Collins, and Z. Li, “Design of a highlevel teleoperation interface resilient to the effects of unreliable robot
autonomy,” in 2020 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). IEEE, 2020, pp. 11 519–11 524.
[7] N. Boguslavskii, Z. Zhong, L. M. Genua, and Z. Li, “A shared autonomous nursing robot assistant with dynamic workspace for versatilemobile manipulation,” in 2023 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 2023, pp. 7040–7045.