Tele-robotic system with a robot face placed on a chair

A robot system that includes a robot face with a monitor, a camera, a speaker and a microphone. The robot face is connected to a stand that can be placed in a chair. The stand is configured so that the robot face is at a height that approximates the location of a person's head if they were sitting in the chair. The robot face is coupled to a remote station that can be operated by a user. The face includes a monitor that displays a video image of a user of the remote station. The stand may be coupled to the robot face with articulated joints that can be controlled by the remote station. By way of example, the user at the remote station can cause the face to pan and/or tilt.

TECHNICAL FIELD

The present disclosure relates generally to the field of robotics. More specifically, the present disclosure relates to a robot face that includes a camera, a monitor, a microphone and a speaker, wherein the robot face is connected to a stand that is adapted to support the robot face on a chair.

DETAILED DESCRIPTION

Disclosed is a robot system that includes a robot face with a monitor, a camera, a speaker and a microphone. The robot face is connected to a stand that can be placed in a chair. The stand is configured so that the robot face is at a height that approximates the location of a person's head if they were to be sitting in the chair. The robot face is coupled to a remote station that can be operated by a user. The face includes a monitor that displays a video image of a user of the remote station. The stand may be coupled to the robot face with articulated joints that can be controlled by the remote station. By way of example, the user at the remote station can cause the face to pan and/or tilt. The robot face and stand provide a relatively low cost telepresence system that can simulate a person being remotely present in a chair.

Referring to the drawings more particularly by reference numbers,FIGS. 1, 2 and 3show a telepresence system10. The system10includes a robot face12that is coupled to a remote control station14through a network16. By way of example, the network16may be either a packet switched network such as the Internet, or a circuit switched network such as a Public Switched Telephone Network (PSTN) or other broadband system. Alternatively, the robot face12may be coupled to the remote station14network thru a satellite.

The remote control station16may include a computer18that has a monitor20, a camera22, a microphone24and a speaker26. The computer18may also contain an input device28such as a joystick or a mouse. The control station14is typically located in a place that is remote from the robot face12. Although only one remote control station14is shown, the system10may include a plurality of remote stations14. In general any number of robot faces12may be coupled to any number of remote stations14or other robot faces12. For example, one remote station14may be coupled to a plurality of robot faces12, or one robot face12may be coupled to a plurality of remote stations14, or a plurality of robot faces12. The system may include an arbitrator (not shown) that controls access between the robot face(s)12and the remote stations14.

The robot face12is mounted to a stand30. The stand30and robot face12can be placed onto a chair32. The stand30may have a pair of leg portions34that are spaced apart to increase the stability of the face12on the chair32. The stand30preferably has a length so that the height of the face12above the chair32simulates the location of a user's head. By way of example, the robot face12could be located 20 to 40 inches above the chair. Generally speaking, the robot face position should approximate a person with a height between 5 to 6.5 feet. The face12can be moved to and from the chair by personnel at the remote site. For example, the face12can be stored at a facility. A user at the remote station14may instruct personnel at the facility to place the stand30and face12onto a chair32. The remote user can then interact with personnel at the remote station14through the robot face12. The display of the user by the robot face12simulates the user's presences at the meeting.

The stand30may be coupled to the face12with articulated joints36and38that allow the robot face12to be panned and tilted, respectively. The joints36and38may contain actuators (not shown) that can be remotely actuated through manipulation of the input device28at the remote station14.

Each robot face14includes a camera(s)40, a monitor42, a microphone(s)44and a speaker(s)46. The robot camera40is coupled to the remote monitor20. Likewise, the robot monitor42is coupled to the remote camera22so personnel at the robot site may view the user of the remote station14. The microphones24and44, and speakers26and46, allow for audible communication between the system operator and the personnel at the robot site. The face12may also include a laser pointer48that can be moved through manipulation of the input device28at the remote station.

The microphones44may include both a directional (cardioid) microphone and an omni-directional microphone. By selecting the ‘m’ key on the keyboard, a user may toggle between using one microphone or the other. For example, the user may choose to use the directional microphone when listening to a particular speaker at the other end of the table, and use the omni-directional microphone when a more rapid discussion around the table is occurring.

Each remote station computer18may operate Microsoft OS software and WINDOWS XP or other operating systems such as LINUX. The remote computer18may also operate a video driver, a camera driver, an audio driver and a joystick driver. The video images may be transmitted and received with compression software such as MPEG CODEC.

FIG. 4shows an embodiment of the robot face12. The robot face12may include a high level control system60and a low level control system62. The high level control system60may include a processor64that is connected to a bus66. The bus is coupled to the camera38by an input/output (I/O) port68, and to the monitor40by a serial output port70and a VGA driver72. The monitor40may include a touchscreen function that allows a meeting participant or a patient to enter input by touching the monitor screen.

The speaker44is coupled to the bus66by a digital to analog converter74. The microphone42is coupled to the bus66by an analog to digital converter76. The high level controller60may also contain random access memory (RAM) device78, a non-volatile RAM device80and a mass storage device82that are all coupled to the bus72. The mass storage device82may contain corporate data, or medical files of a patient that can be accessed by the user at the remote control station14. For example, the mass storage device82may contain a picture of a patient. The user, particularly in the case of a health care provider, can recall the old picture and make a side by side comparison on the monitor20with a present video image of the patient provided by the camera38. The robot antennae46may be coupled to a wireless transceiver84. By way of example, the transceiver84may transmit and receive information in accordance with IEEE 802.11. Alternatively, the transceiver84may transmit over a 3G cellular network, or may switch between WiFi and cellular-based networks or utilize them in parallel.

The remote control station may be a laptop or personal computer that has a cellular transceiver (not shown). When accessing a robot face, the control station may establish both a cellular link and a LAN link. By way of example, the cellular link may be in accordance with 3G protocol and the LAN link may operate under 802.11g. A first type of data may be sent through the cellular link and a second type of data may be transmitted with the LAN link. For example, video may be transmitted with the cellular link and audio may be sent through the LAN link. Some types of data may be sent through both links. For example, a stethoscope (not shown) may be connected to the robot and audio data of a heart beat is sent back to the control station through both the cellular and LAN links. This ensures the most rapid and robust delivery of data, which is particularly important given that delayed stethoscope audio may create false heart anomaly sounds, or mask heart anomalies. The control station can monitor one or more network parameters such as latency, packet loss and/or jitter. Unacceptable parameter values on one link may cause the station16to switch certain categories of data to the other link.

The robot platform may include a 6-inch rectangular port “arm” (not shown), which may be placed on the conference table in front of the robot face. The arm may be connected with a cable to the robot platform, or alternatively may have wireless communication to the robot face. The arm contains a variety of device ports which may be useful during the conference. For example, the arm may contain a VGA port which allows connection to a projector available on the conference table. The projector can project an image onto a screen so that viewers at the robot site can view the projected image. Consequently, a user at the remote control station can transmit information to the robot that is then projected by the projector. For example, the information may be a PowerPoint presentation that is displayed by the robot projector116and allows the remote control station user to conduct a remote meeting while “seated” at the table. The arm may also be connected to a medical instrument such as a stethoscope. This feature allows the remote station to receive instrument input from the stethoscope.

The arm may include an auxiliary video port. The auxiliary video port may include USB, VGA, Y-video/audio electrical connectors and associated electronic circuitry. A plurality of video devices can be connected to one or more of the ports. By way of example, the video devices may include a ceiling camera, a video playback machine such as a VCR or DVD player, an ultrasound device, an otoscope, an echocardiogram, and/or a dermatology camera. The video devices capture video that is transmitted to the remote station through the robot face. By way of example, the overhead camera may capture images of a top-down view of the conference that are then transmitted to the remote control station and displayed by the station monitor.

The controller64may operate with a LINUX OS operating system. The controller64may also operate MS WINDOWS along with video, camera and audio drivers for communication with the remote control station14. Video information may be transceived using MPEG CODEC compression techniques. The software may allow the user to send e-mail to someone at the robot site and vice versa, or allow someone at the robot site to access the Internet. In general the high level controller60operates to control the communication between the robot face12and the remote control station14.

The high level controller60may be linked to the low level controller62by serial port86. The low level controller62runs software routines that mechanically actuate the robot face12. For example, the low level controller62provides instructions to actuate the robot face12. The low level controller62may receive movement instructions from the high level controller60. The movement instructions may be received as movement commands from the remote control station. Although two controllers are shown, it is to be understood that the robot face12may have one controller controlling the high and low level functions.

The system may be similar to a robot system provided by InTouch Technologies, Inc. of Goleta, Calif. The system may also be similar to the system disclosed in U.S. Pat. No. 6,925,357 issued Aug. 2, 2005, which is hereby incorporated by reference.

The robot face12may be controlled by a number of different users. To accommodate for this the robot may have an arbitration system. The arbitration system may be integrated into the operating system of the robot face12. For example, the arbitration technique may be embedded into the operating system of the high-level controller60.

By way of example, the users may be divided into classes that include the robot itself, a local user, an executive, a manager, an employee or a guest. Alternatively, class may include the robot itself, a local user, a caregiver, a doctor, a family member, or a service provider. The robot face12may override input commands that conflict with robot operation. For example, if the robot face impacts a wall, the system may ignore all additional commands to continue in the direction of the wall. A local user is a person who is physically present with the robot. The robot could have an input device that allows local operation. For example, the robot may incorporate a voice recognition system that receives and interprets audible commands.

A caregiver is someone who remotely monitors the patient. A doctor is a medical professional who can remotely control the robot and also access medical files contained in the robot memory. The family and service users remotely access the robot. The service user may service the system such as by upgrading software, or setting operational parameters.

Message packets may be transmitted between a robot face12and a remote station14. The packets provide commands and feedback. Each packet may have multiple fields. By way of example, a packet may include an ID field a forward speed field, an angular speed field, a stop field, a bumper field, a sensor range field, a configuration field, a text field and a debug field.

The identification of remote users can be set in an ID field of the information that is transmitted from the remote control station14to the robot face12. For example, a user may enter a user ID into a setup table in the application software run by the remote control station14. The user ID is then sent with each message transmitted to the robot.

The robot face12may operate in one of two different modes—an exclusive mode, or a sharing mode. In the exclusive mode only one user has access control of the robot. The exclusive mode may have a priority assigned to each type of user. By way of example, the priority may be in order of local, doctor, caregiver, family and then service user. In the sharing mode, two or more users may share access with the robot. For example, a caregiver may have access to the robot; the caregiver may then enter the sharing mode to allow a doctor to also access the robot. Both the caregiver and the doctor can conduct a simultaneous tele-conference with the patient.

The arbitration scheme may have one of four mechanisms—notification, timeouts, queue and call back. The notification mechanism may inform either a present user or a requesting user that another user has, or wants, access to the robot. The timeout mechanism gives certain types of users a prescribed amount of time to finish access to the robot. The queue mechanism is an orderly waiting list for access to the robot. The call back mechanism informs a user that the robot can be accessed. By way of example, a family user may receive an e-mail message that the robot is free for usage. Tables 1 and 2 show how the mechanisms resolve access request from the various users.

The information transmitted between the station14and the robot face12may be encrypted. Additionally, the user may have to enter a password to enter the system10. A selected robot is then given an electronic key by the station14. The robot face12validates the key and returns another key to the station14. The keys are used to encrypt information transmitted in the session.

FIG. 5shows a display user interface (“DUI”)200displayed at a control station14. The DUI200may include a robot view field202that displays a video image captured by the camera of the robot. The DUI200may also include a station view field204that displays a video image provided by the camera of the remote station14. The DUI200may be part of an application program stored and operated by the computer18of the remote station14.

The DUI200may include a “Connect” button206that can be selected to connect the station to a robot. Selection of the Connect button206may cause the display of pull-down screens, etc. that allow the user to select a desired robot. System settings and options can be selected through buttons208and210, respectively.

The system includes a “shared physical whiteboard” feature. When the user points the robot's camera at a physical whiteboard, they may telestrate on the image at their remote station. The lines of their telestration (and nothing else) are projected, using the projector cabled to the robot and pointing to the same whiteboard. Thus the remote (virtual) telestration lines are overlaid onto the physical white board, allowing both local participants and remote user to “draw” on the same physical space.

At times the angle between the robot and the whiteboard may be such that the whiteboard appears in a non-orthogonal perspective angle. This would cause the resultant telestration lines, when projected, to be un-matched to the whiteboard. To compensate, the system does software-based keystoning, accelerated by the robot's GPU.

In order to calibrate the remote user's telestration on the camera image with the projected telestration on the whiteboard, the user enables a registration sequence. Upon selecting the “register telestration” button (not shown) on the remote station interface, the system projects4red registration dots on the corners of the whiteboard. The user sees the projection of these dots through the transmitted camera image from the robot. A popup message on the interface says “click the upper-left dot now”, and the user proceeds to click the point at which they see the upper-left dot. Following this is a second popup message requesting a click on the upper-right dot, and following that the lower two dots. Alternatively, a computer vision package such as OpenCV may be used to automatically detect the red dots and proceed with the registration without user intervention. At this point the system aligns those points to a software keystoning algorithm, utilizing 3D mesh functions available on the GPU.

In an alternative embodiment, the remote user can capture the image of the whiteboard with the camera and have it projected elsewhere in the room with his overlaid telestration.

The system may be accessed by a cellular phone, wherein the phone may include an accelerometer that can sense motion of the phone. The phone may transmit movement commands to the robot that are a function of the sensed movement of the cellular phone. For example, the user can move the phone about a Tilt Axis that will cause a corresponding tilt movement of the robot head. Likewise, the user may move the phone about a Spin Axis that induces a corresponding pan movement of the head. In usage, a person adjacent to the robot may request a remote user to join in a meeting thru the robot's touch-screen. A SIP-based call would be transmitted to the remote user's cellular phone. The remote user may accept the ‘call’ and find themselves remotely seated at the table. By moving the phone left or right they see who is at the table and thereby remotely participate in the meeting.

It will be apparent to those having skill in the art that changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art. In the claims, the conjunction “and” is inclusive, the conjunction “or” is exclusive and the conjunction “and/or” is either inclusive or exclusive. Embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows.