Patent Publication Number: US-2019176341-A1

Title: Movable robot capable of providing a projected interactive user interface

Description:
BACKGROUND OF THE INVENTION 
     The present application relates to robotic technologies, and in particular, to a robot having a novel mechanism for providing an interactive user interface. 
     Robotic technologies have seen a revival in recent years. Various robots are being developed for a wide range of applications such as care of senior citizens and patients, security and patrol, delivery, hotel services, and hospitality services. Typically, the robots not only need to independently move around a service area; they are also required to deliver messages to and receive commands from people. It has been a challenge to provide technologies that can effectively fulfill the above needs with simple and low-cost components and designs. With domestic technology on the rise, the quantity and complexity of social robots are becoming an important interaction design challenge to promoting the user&#39;s sense of control and engagement. Additionally, a wide range of interaction modalities for the social robots have been designed and researched, including GUI, voice control, gesture input, and augmented reality interfaces. 
     Social robots provide for an alternative mode of interaction, a way to simulate the way of communication between humans or between human and a pet using gestures, facial expression, body language and nonverbal behavior. Furthermore, by sharing physical space and objects with their users, people like to involve physical action on the part of the human for more user experience, better learning and higher engagement. 
     Other major social robot is to accompany 0-6 year old kids at home or a kindergarten. At this period of ages, kids are not good at verbal communication; instead they prefer physical body languages. There is therefore a need for a robot that can effectively interact with users and deliver content at a place and a time convenient to the users. 
     SUMMARY OF THE INVENTION 
     The presently application discloses a moveable robot having simple multi-purpose mechanisms for movement and user interface. A multi-purpose projection system can display an image with a body of the moveable robot, or project an image served as an interactive user interface on an external surface. A multi-purpose optical sensing system can detect objects in the environment as well as detecting user&#39;s movement, location, and gesture for receiving user inputs. 
     In one general aspect, the present invention relates to a moveable robot that includes a processing control system, a rotation mechanism under the control of the processing control system, a projection system that can be tilted by the rotation mechanism to a first position to project a first image on a horizontal surface outside the body of the moveable robot, wherein the projection system that can be tilted by the rotation mechanism to a second position to project a second image on a wall surface, an optical sensing system that can detect the user&#39;s movement, location, facial expression or gesture over the first image, wherein the processing control system is configured to interpret user&#39;s inputs based on the user&#39;s movement, location, facial expression or gesture over the first image projected on the horizontal surface outside the body of the moveable robot, wherein the processing control system can control one or more outputs of the moveable robot based on the interpreted user&#39;s inputs, wherein the optical sensing system is configured to detect objects surrounding the moveable robot; and a transport system that can produce, under the control of the processing control system, a movement on the horizontal surface in a moving path that avoids the objects detected by the optical sensing system. 
     Implementations of the system may include one or more of the following. The moveable robot can further include a head that houses the projection system; and a head tilt system under the control of the processing control system, wherein the head tilt system includes the rotation mechanism. The optical sensing system can detect the user&#39;s positions on the horizontal surface at a first time and at a second time, wherein the processing control system can calculate a first coordinate of the user at the first time and a second coordinate of the user at the second time, wherein the processing control system can determine if the displacement of the foot exceeds a predetermined threshold. The processing control system can calculate a direction of movement of the user&#39;s foot if the displacement of the foot exceeds a predetermined threshold. The processing control system can interpret a user input based on the direction of movement of the user&#39;s foot. The predetermined threshold can depend on the user&#39;s height. The second image can be a two-dimensional image formed on a surface. The rotation mechanism can tilt the projection system a third position to project a three-dimensional image in the air in the front the moveable robot. The optical sensing system can emit light beams to an object or a person and detect light reflected from the object or the person, wherein the processing control system can calculate locations of the objects based on the reflected lights. The optical sensing system can include a camera that detects light reflected from the surface where the first image is formed to detect the user&#39;s movement, location, facial expression or gesture over the first image. The optical sensing system can include an IR emitter and a depth camera configured to sense user&#39;s movement, location, facial expression or gesture and the objects surrounding the moveable robot. The optical sensing system can include a laser emitter and a rotating mirror configured to sense user&#39;s movement, location, facial expression or gesture and the objects surrounding the moveable robot. 
     In another general aspect, the present invention relates to a moveable robot, comprising: a processing control system; a projection system that can project a first image on a surface outside the body of the moveable robot; an optical sensing system that can detect the user&#39;s movement, location, facial expression or gesture over the first image, wherein the processing control system can interpret user&#39;s inputs based on the user&#39;s movement, location, facial expression or gesture over the first image projected on the surface outside the body of the moveable robot, wherein the optical sensing system can detect objects surrounding the moveable robot; and a transport system that can produce motion relative to the surface to plan its moving path or avoid the objects detected by the optical sensing system. 
     Implementations of the system may include one or more of the following. The moveable robot can further include a main housing body; an upper housing body comprising an optical window at a lower surface, wherein at least a portion of the project system is inside the upper housing body; a sliding platform comprising a sliding mechanism that can slide the upper housing body relative to the main housing body to expose a lower surface of the upper housing body, wherein the project system can emit light through the optical window at the lower surface of the upper housing body to project the first image on a surface outside the body of the moveable robot. The sliding mechanism can slide the upper housing body to a home position and a slide-out position, wherein the projection system is configured to display the second image on or inside a body of the moveable robot when the upper housing body is at the home position, wherein the projection system can project the first image on a surface outside the body of the moveable robot when the upper housing body is at the slide-out position. 
     The moveable robot can further include a rotation mechanism that can rotate the main housing body relative to the transport system. The projection system can include a projector configured to produce images; a mirror configured to reflect the images; and a steering mechanism configured to align the mirror to a first angle to produce the first image, and to align the mirror to a second angle to produce the second image. The projection system can display a second image on or inside a body of the moveable robot. The second image can a two-dimensional image formed on a surface. The second image can be a three-dimensional image formed inside the moveable robot. The optical sensing system can emit light beams to an object or a person and detect light reflected from the object or the person, wherein the processing control system can calculate locations of the objects based on the reflected lights. The optical sensing system can include a camera that detects light reflected from the surface where the first image is formed to detect the user&#39;s movement, location, facial expression or gesture over the first image. The optical sensing system can include an IR emitter and a depth camera configured to sense user&#39;s movement, location, facial expression or gesture and the objects surrounding the moveable robot. The optical sensing system can include a laser emitter and a rotating mirror configured to sense user&#39;s movement, location, facial expression or gesture and the objects surrounding the moveable robot. 
     These and other aspects, their implementations and other features are described in detail in the drawings, the description and the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a front view of a moveable robot in accordance with the present application. 
         FIG. 2  shows a front view of a moveable robot in a configuration for projecting an image on the ground in accordance with the present application. 
         FIG. 3  is a bottom view of the upper housing body showing a sliding platform comprising a sliding mechanism between the head and the main body of the moveable robot in  FIGS. 1 and 2 . 
         FIG. 4A  is a schematic diagram illustrating the sensing system in the moveable robot in accordance with the present application. 
         FIG. 4B  is a schematic diagram illustrating an exemplified sensing system in the moveable robot in accordance with the present application. 
         FIG. 4C  is a schematic diagram illustrating another exemplified sensing system in the moveable robot in accordance with the present application. 
         FIG. 5A  shows a configuration of the projection system in the disclosed moveable robot for projecting an image on the surface as a part of an interactive user interface. 
         FIGS. 5B and 5C  respectively show two display configurations for the projection system compatible with the disclosed moveable robot. 
         FIG. 6  is a system diagram of the moveable robot in accordance with the present application. 
         FIG. 7A  shows a front view of another moveable robot in accordance with the present application. 
         FIG. 7B  shows a side view of the moveable robot of  FIG. 7A  projecting an image on a wall. 
         FIG. 7C  shows a side view of the moveable robot of  FIG. 7A  projecting an image on the ground. 
         FIG. 8  is a system diagram of the moveable robot in  FIGS. 7A-7C . 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Referring to  FIGS. 1 and 2 , a moveable robot  100  includes a transport platform  120 , an optical sensing system  130 , a main housing body  140 , a sliding platform  150 , an upper housing body  160 , and a microphone  170  in the upper housing body  160 . The transport platform  120  includes a transport mechanism including wheels  125 , which can move the moveable robot  100  to different locations. The main housing body  140  and the transport platform  120  are connected by a rotation mechanism that allows the main housing body  140  to rotate relative to the transport platform  120  so that components in the main housing body  140  and the upper housing body  160  can detect people and objects in the environment and display information and interact with people in the right directions. 
     The optical sensing system  130  can detect objects in the environment and assist the moveable robot  100  to design the best movement path and to avoid obstacles during movement. For example, the optical sensing system  130  can emit laser beams to the environment, receive bounced back laser signals, and detect objects and their locations by analyzing the bounced back signals. As described below, an important aspect of the presently disclosed robot is that the optical sensing system  130  can also detect user&#39;s movements, locations, gestures, facial expression, body language and nonverbal behavior over a projected user interface. 
     The sliding platform  150  enables the upper housing body  160  to slide relative to the main housing body  140  to expose a lower surface of the upper housing body  160 . A project system ( 400 ,  610  in  FIGS. 4 and 7 , described below) can project a project image  180  on a surface  190  that the moveable robot  100  stands on. The surface can be a ground surface, a floor surface, a work platform, a tabletop, etc. 
     Referring to  FIGS. 1-3 , the sliding platform  150  includes a sliding mechanism  200  that includes a motor, a drive, a controller, a power supply, and mechanical components. The sliding platform  150  includes an optical window  205  for the projection optical path of the projection system ( 400 ,  FIG. 4A ). 
     In some embodiments, as shown in  FIG. 3 , the sliding mechanism  200  can further include a projector sliding plate  210 , a projector sliding reference plate  220 , a sliding block  230 , a sliding rail  240 , a pulley transmission mechanism  260 , a timing belt locking bridge  270 , a travel limit column  280 , and a photoelectric sensor  290 . The sliding mechanism  200  has two states: a home position and a slide-out position, whose positions are sensed by the photoelectric sensor  290  and locked by the travel limit column  280 . The slide rail  240 , the timing pulley drive mechanism  260 , the travel limit column  280 , and the photoelectric sensor  290  are mounted on the projector slide reference plate  220 . The slide block  230  and the timing belt lock clamp  270  are installed on the projector sliding plate  210 . The projector sliding plate  210  is further provided with a device for fixing the projector  250 . In the present implementation, at least one slide/slide-fit structure is provided; the projector sliding plate  210  is sleeved on the slide rail  240  by the slide block  230 . The timing belt locking bridge  270  is connected to the belt of the pulley transmission mechanism  260 . The driving pulley of the pulley transmission mechanism  260  is fixedly connected with the rotor shaft of a motor mounted on the projector sliding reference plate  220 . The stroke limiting post  280  engages with the notch on the projector slide-out plate  210 . The motor can be implemented using a DC motor. 
     The pulley transmission mechanism  260  can include a driving pulley, a driven pulley, a belt, and a timing belt clamping plate  270 . When the sliding mechanism  200  is in operation, the motor rotates according to the instruction sent by a processing control system ( 620  in  FIG. 6 , described below), drives the driving wheel to rotate, the driving wheel drives the belt to move left and right, the belt drives the synchronous belt clamping plate  270  to slide around, the synchronous wheel clamping plate  270 . The projector slides out of the board  210  and the projector  250  can slide corresponding to the left and right of the belt. 
     Referring to  FIGS. 4A and 5A , the moveable robot  100  includes a projection system  400  that includes a mirror  403  and a projector  408  in the upper housing body ( 160  in  FIGS. 1 and 2 ). The mirror  403  can rotate with a steering mechanism (not shown in figures) to change the angle of the mirror  403  relative to the ground  190  such that the projector  408  can project, via the mirror  403 , the image  180  on the surface  190 . One purpose of the mirror  403  can allow more flexibility for the projector  408  to be installed in very limited upper housing body  160 . If the upper housing body  160  is big enough and a slim projector  408  can be found, the mirror  403  and the steering mechanism can be optional. The image  180  can include common features of an interactive user interface such as a border, functional symbols, selection buttons, a grid, a table, text and graphics, etc. The image  180  can also include product and other commercial information. A camera  406  is installed on the upper position in the moveable robot  100 , which can detect the bounced lights from the surface  190  comprising user  450 &#39;s movement over the image  180 . 
     Referring to  FIGS. 2-3 , the components of the sliding mechanism  200  (except for the slide rail  240 ,  FIG. 3 ), the projector  408  and the mirror  403  can be mounted in the upper housing body  160 . As the upper housing body  160  is shifted relative to the main housing body  140  (shown in  FIG. 2 ), the optical window  205  for the projection optical path is exposed at the bottom of the upper housing body  160 . An image produced by the projector  408  is reflected by the mirror  403 . The steering mechanism changes the angle α of the mirror  403  relative to the surface  190 , to direct the light to casts on the surface  190  to form a projected image  180 . 
     In accordance with an important aspect of the present application, the project image  180  can provide a user interface that includes functional input areas. A user  450  can move about the project image  180  and produce gestures, facial expression, body language and other nonverbal behaviors at different functional input areas. The optical sensing system  130  has dual functions: in addition to detecting objects in the environment, it can detect user&#39;s movements, locations, gestures, facial expression, body language and other nonverbal behaviors and interpret them as inputs to the processing control system ( 620  in  FIG. 6 , described below). The laser optical sensing system  470  can emit a light beam  460 , which is reflected and bounced off from surrounding objects or by a user. The camera  406  is installed on the upper position in the moveable robot  100 , which can detect the bounced lights from the surfaces. The processing control system ( 620  in  FIG. 6 , described below) conducts computations to calculate the positions and distances of the surrounding objects, or user&#39;s movement, location, facial expression or gesture. 
     In some embodiments, LIDAR (light detection and ranging) can be used as sensors for both sensing human movement for interaction purpose and building the mapping to move in a complex environment. LIDAR is a surveying method that measures distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor. Differences in laser return times and wavelengths can then be used to make digital 2D or 3D representations of the target. Time-of-flight (TOF) cameras are sensors that can measure the depths of scene-points, by illuminating the scene with a controlled laser or LED source, and then analyzing the reflected light. The ability to remotely measure range is extremely useful and has been extensively used for mapping and surveying. 
     One type of LIDAR system uses a single laser or multiple line laser fire onto a rotating mirror spinning at high speed and hence to view objects in a single plane or multiple planes at different heights, a 360-degree field of view is generated. As shown in  FIG. 4B , a 3D image sensing system  413  comprising a spinning mirror for scanning laser beam can sense all the objects in its plane to detect objects in the environments and user movement. 
     Another type of LIDAR system includes IR emitter and depth camera. IR emitter projects a pattern of infrared light. As the light hits a surface, the pattern becomes distorted and the depth camera reads the distortion. The depth camera analyzes IR patterns to build a 3D map of the room and peoples action within it. As shown in  FIG. 4C , a 3D image sensing system  480  comprising an IR emitter and a depth camera can view all the objects in its range to detect environments and user movement. 
     The optical sensing system using LIDAR can fulfill two functions: 1. building a map for the complex environment for robot to plan it path through obstacles; 2. collecting the body movement of humans relating to the position of the projected image on the surface and analyze the motion to activate corresponding response. In this case, one component of LIDAR can replace both the camera  406  and the optical sensing system  130  because it usually integrates the light emitter and sensor into one component. 
     In some embodiments, the projection system  400  can display images on a surface of the upper housing body  160  or inside the upper housing body  160  while the upper housing body  160  is at its home position. Referring to  FIG. 5B , the upper housing body  160  includes a winder  510 . The angle β of the mirror  403  relative to the surface  190  ( FIG. 5B ) is adjusted by the steering mechanism. An image is projected and displayed at the winder  510  viewable by users. Moreover, referring to  FIG. 5C , the angle β 1  of the mirror  403  relative to the surface  190  ( FIG. 5A ) is adjusted by the steering mechanism. An image is projected and displayed on a holographic film  520  inside the upper housing body  160 . Alternatively, a three-dimensional hologram image is formed inside the upper housing body  160  by the projection system  400 . The image on the holographic film  520  or the hologram image can be viewed by a user through a transparent window in the upper housing body  160 . 
       FIG. 6  shows a system diagram of the moveable robot in accordance with the present application. The moveable robot  100  includes a projection system  610 , a processing control system  620 , an optical sensing system  630 , a transport system  640 , and a rotation mechanism  650 . The project system  610  can display two-dimensional images or three-dimensional holograms in the upper housing body  160 , viewable by users under the control of the processing control system  620 . The project system  610  can also project an image on a surface outside of in the upper housing body  160  under the control of the processing control system  620 . The projected image is not only viewable by users, but also serves as an interactive user interface for taking input from users. Under the control of the processing control system  620 , the optical sensing system  630  can not only scan and determine locations of objects in the environment (for motion path planning and object avoidance during movement); it can also detect user&#39;s movement, locations, gestures, facial expression, body language and other nonverbal behaviors over the project area, which are interpreted as user inputs by the processing control system  620 . The processing control system  620  can also controls a rotation mechanism  650  to rotate the main housing body ( 140 ,  FIGS. 1-2, 5-6B ) relative to the transport platform ( 120 ,  FIGS. 1-2, 5-6B ). 
     In some embodiments, Referring to  FIGS. 7A-7C , a moveable robot  700  includes a transport &amp; rotation system  720 , an optical sensing system  730 , a housing body  740 , a head  750 , a rotation mechanism  760 , and a projection system  770  in the head  750 . The transport &amp; rotation system  720  includes wheels  725  and can move the moveable robot  700  to different locations on a surface  790 . 
     The transport &amp; rotation system  720  can also rotate the moveable robot  700  to different directions to allow the optical sensing system  730  detect people and objects in the environment in the right directions. The optical sensing system  730  can detect objects in the environment and assist the moveable robot  700  to design the best movement path and to avoid obstacles during movement. For example, the optical sensing system  730  can emit laser beams to the environment, receive bounced back laser signals, and detect objects and their locations by analyzing the bounced back signals. As described below, an important aspect of the presently disclosed robot is that the optical sensing system  730  can also detect user&#39;s movements, locations, gestures, facial expression, body language and other nonverbal behaviors over a projected user interface. 
     The rotation of the moveable robot  700  by the transport &amp; rotation system  720  allows the projection system  770  to face right polar direction. Furthermore, the rotation mechanism  760  can tilt the head  750  up and down for projection on different surfaces. For example, the head  750  can be tilted by the rotation mechanism  760  to a position to project on a wall  780  ( FIG. 7B ). In some embodiments, the head  750  can be tilted up to another position by the rotation mechanism  760 , which allows the projection system  770  to project a 3D image in the air in front of the moveable robot  700 . In another example, the head  750  can be tilted down to yet another position by the rotation mechanism  760  to project on the ground  790  ( FIG. 7C ). 
       FIG. 8  shows a system diagram of the moveable robot  700 . The moveable robot  700  includes the projection system  770 , a processing control system  820 , the optical sensing system  730 , the transport and rotation system  720 , and a head tilt system  850  comprising the rotation mechanism  760 . The project system  770  can project an image on a surface outside of in the housing body  740  under the control of the processing control system  820 . 
     The projected image is not only viewable by users, but also serves as an interactive user interface for taking input from users. Under the control of the processing control system  620 , the optical sensing system  730  can not only scan and determine locations of objects in the environment (for motion path planning and object avoidance during movement); it can also detect the movements, locations, gestures, facial expression, body language and other nonverbal behaviors of a user  840  over the project area, which are interpreted as user inputs by the processing control system  620 . Based on the interpreted user inputs, the processing control system  820  can employ a decision-making algorithm to make decisions to further control the outputs of the moveable robot  700  to interact with or give instructions to the user  840 . The outputs of the moveable robot  700  can include one or a combination: a projected content by the projection system  770 , a sound, or a rotation or a movement by the transport and rotation system  720 . 
     Referring to  FIG. 9  and  FIG. 7C , and similar to the descriptions above relating to  FIG. 4C , the position of a user&#39;s foot is measured by the optical sensing system (step  910 ). When the foot is detected to be inside a projected area, a first coordinate of the user&#39;s foot at a first time is calculated by the processing control system (step  920 ). In some embodiments, the projected area can be divided into zones. Different users can stand and move in different zones. After a movement is detected by the user&#39;s foot, a second coordinate of the user&#39;s foot at a second time is calculated by the processing control system when the foot is inside a projected area (step  930 ). The processing control system determines if the displacement of the foot exceeds a predetermined threshold (step  940 ). The threshold can be dependent on the user&#39;s height and the specific application (e.g. the game that the user is playing, or the imaging program that the user is using). The direction of movement of the user&#39;s foot is calculated by the processing control system if the displacement of the foot is more than the predetermined threshold (step  950 ). A user input is interpreted from the direction of movement by the processing control system (step  960 ). 
     It should be noticed that the above examples are intended to illustrate the concept of the present invention. The present invention may be compatible with many other configurations and variations: for example, the shape of the moveable robot is not limited to the examples illustrated. In addition, the moveable robot can include fewer or additional housing bodies. For example, the main housing body and the transport platform can be combined and the rotation of the robot relative to the ground surface can be accomplished by the transport mechanism in the transport platform rather than accomplished by a separate rotation mechanism. Furthermore, the sliding mechanism can be realized by other mechanical and electronic components. Moreover, the optical scanning system can be implemented in other configurations to provide the capability of sensing both an object and a person&#39;s location, movements, gestures, facial expression, body language and other nonverbal behaviors. 
     While this document contains many specifics, these should not be construed as limitations on the scope of an invention that is claimed or of what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features that are described in this document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination. 
     Only a few examples and implementations are described. Other implementations, variations, modifications and enhancements to the described examples and implementations may be made without deviating from the spirit of the present invention.