Patent Description:
Robots are increasingly used not only for performing useful tasks, but also for providing a measure of companionship.

Previously proposed arrangements are disclosed in <CIT> , comprising a communication robot and a control program of a communication robot, and <CIT>, comprising a robot capable of providing a video call service, an operation method thereof, and a server connected thereto.

The invention relates to a device that includes at least one computer storage that is not a transitory signal and that in turn includes instructions executable by at least one processor to, for at least a first user, render, from at least one image of the first user by at least one imager, a full-face image representing the first user with background and body parts of the first user cropped out of the image representing the first user. The instructions may be executable to provide, to at least a first robot remote from the first user, the full-face image for presentation thereof on a display of the first robot with the full-face image filling the display. The instructions may be further executable to provide, to the first robot, information from the imager regarding motion of the first user such that a head of the first robot turns to mimic the motion of the first user while continuing to present a full-face image representing the first user on the display of the first robot regardless of whether the head of the first user turned away from the imager.

In another aspect, a method includes, for at least a first user, rendering, from at least one image captured of the first user, a full-face image representing the first user with background and body parts of the first user cropped out of the image captured of the first user. The method includes presenting, on at least one display of a first robot remote from the first user, the full-face image representing the first user with the full-face image filling the display of the first robot. The method also includes turning a head of the first robot to mimic a head turn of the first user while continuing to present a full-face image representing the first user on the display of the first robot.

Additionally, for at least a second user local to the first robot the method includes rendering, from at least one image captured of the second user, a full-face image representing the second user with background and body parts of the second user cropped out of the image representing the second user. The method includes presenting, on at least one display of a second robot local to the first user, the full-face image representing the second user with the full-face image of the second user filling the display of the second robot. Further, the method includes turning a head of the second robot to mimic a head turn of the second user while continuing to present a full-face image representing the second user on the display of the second robot.

In an embodiment, a robot includes a lower body portion on propulsion elements. An upper body portion is coupled to the lower body portion and is movable relative to the lower body portion. The upper body portion includes at least one display configured to present an image representing a person remote from the robot, with the image being a full-face image. An avatar may be presented, or an actual image of the person may be presented.

In some examples the upper body portion is movable relative to the lower body portion in accordance with motion of the person as indicated by signals received from an imager. The imager can be a webcam, smart phone cam, or other imaging device.

The full-face image can be generated from a profile image of the person, if desired using a machine learning (MI,) model executed by a processor in the robot and/or by a processor distanced from the robot.

In some examples, opposed side surfaces of the upper body portion include respective microphones.

Example implementations of the robot can include left and right cameras and at least one processor to send images from the cameras to a companion robot local to and associated with the person. A motorized vehicle may be provided with a recess configured to closely hold the lower body portion to transport the robot. At least one magnet can be disposed in the recess to magnetically couple the robot with the motorized vehicle and to charge at least one battery in the robot. If desired, at least one speaker can be provided on the robot and may be configured to play voice signals received from the person. The top surface of the robot may be implemented by at least one touch sensor to receive touch input for the processor.

The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:.

<FIG> shows a robot <NUM> that includes a lower body portion <NUM> on propulsion elements <NUM>, which may be established by four micro holonomic drives. The robot may be made of lightweight metal or plastic and may be relatively small, e.g., the robot <NUM> can be small enough to hold by hand.

An upper body or head portion <NUM> is movably coupled to the lower body portion <NUM> by one or more coupling shafts <NUM> that can be motor driven to move the head portion <NUM> relative to the lower body portion <NUM>. The lower body <NUM> and head portion <NUM> can be parallelepiped-shaped as shown and may be cubic in some examples.

The head portion <NUM> can be movable relative to the lower body portion <NUM> both rotatably and tiltably. For example, as indicated by the arrows <NUM>, the upper body or head portion <NUM> can be tiltable forward-and-back relative to the lower body portion <NUM>, while as illustrated by the arrows <NUM> the upper body or head portion <NUM> can be tiltable left-and-right. Also, as indicated by the arrows <NUM>, the upper body or head portion <NUM> can rotate about its vertical axis relative to the lower body portion <NUM>.

The front surface <NUM> of the upper body or head portion <NUM> can be established by a display <NUM> configured to present demanded images. Opposed side surfaces <NUM> of the upper body or head portion <NUM> may include respective microphones <NUM> at locations corresponding to where the ears of a human would be. The robot <NUM>, e.g., the lower body portion <NUM> thereof, can also include left and right cameras <NUM> which may be red-green-blue (RGB) cameras, depth cameras, or combinations thereof. The cameras alternately may be placed in the head portion <NUM> where the eyes of a human would be. A speaker <NUM> may be provided on the robot, e.g., on the head portion <NUM> near where the mouth of a human would be, and at least one touch sensor <NUM> can be mounted on the robot <NUM>, e.g., on the top surface of the upper body or head portion <NUM> to receive touch input for a processor within the robot <NUM> and discussed further below.

A control device <NUM> such as a smart phone may include processors, cameras, network interfaces, and the like for controlling and communicating with the robot <NUM> as discussed more fully herein.

<FIG> illustrate that the display <NUM> of the upper body or head portion <NUM> may present various different demanded images of, e.g., human faces imaged by any of the cameras herein. The images may be presented under control of any of the processors discussed herein and may be received by a network interface in the robot <NUM>. In lieu of an image of the person, the face of an avatar representing the person may be presented to preserve privacy. The avatar may be animated to have the same emotional expressions of the person by face emotion capture (including eyes, eyebrows, mouth, and nose).

Note that whether the head portion <NUM> is facing straight ahead as in <FIG> or is tilted or rotated to one side as in <FIG>, the display <NUM> presents the full-face image (of the person or the avatar), even if the original image is of a human taken from the side of the human's face. Details are discussed further below.

<FIG> illustrates a motorized vehicle <NUM> (powered by, e.g., an internal rechargeable battery) with a recess <NUM> configured to closely hold the lower body portion <NUM> of the robot <NUM> to transport the robot <NUM>. At least one magnet <NUM> can be disposed in the recess <NUM> to magnetically and electrically couple the robot <NUM> (which can include a magnet or ferromagnet) with the motorized vehicle <NUM> and to charge at least one battery in the robot. Advantageously, the propulsion elements <NUM> of the robot <NUM> need not be detached to secure the robot <NUM> in the recess <NUM>. The processor of the robot <NUM> senses the presence of a processor in the vehicle <NUM> and controls the processor in the vehicle <NUM> to move the vehicle <NUM> in lieu of moving the robot <NUM> by means of the propulsion element <NUM>.

<FIG> illustrates various components of the robot <NUM> many of which are internal to the robot <NUM>. In addition to the camera(s) <NUM>, microphone(s) <NUM>, display(s) <NUM>, speaker(s) <NUM>, and touch surface(s) <NUM>, the robot <NUM> may include one or more processors <NUM> accessing one or more computer storages <NUM> to program the processor <NUM> with instructions executable to undertake logic discussed herein. The processor <NUM> may control the components illustrated in <FIG>, including a head actuator <NUM> to move the head portion <NUM> relative to the body portion <NUM>, a propulsion motor <NUM> to activate the propulsion elements <NUM> shown in <FIG>, and a network interface <NUM> such as a wireless transceiver to communicate data to components external to the robot <NUM>.

A charge circuit <NUM> may be provided to charge one or more batteries <NUM> to provide power to the components of the robot. As discussed above, the charge circuit <NUM> may receive charge current via one or more magnetic elements <NUM> from, e.g., the vehicle <NUM> shown in <FIG>.

<FIG> illustrate example logic in example flow chart format that the processor <NUM> in <FIG> may execute. Commencing at bock <NUM> in <FIG>, input may be received from the camera(s) <NUM>. Face recognition may be executed on images from the camera, for example, to move the head <NUM> at block <NUM> to remain facing a person imaged by the camera. Also, at block <NUM> the robot may be activated to move on the propulsion elements <NUM> according to the camera signal, e.g., to turn and "hide" behind a nearby object as if "shy" in the presence of the person being imaged by the camera.

Commencing at bock <NUM> in <FIG>, input may be received from the microphone(s) <NUM>. Voice recognition may be executed on the signals, for example, to move the head <NUM> at block <NUM> to cock one of the side of the head of the robot toward the source of the signals (or toward the face of a person imaged by the cameras) as if listening attentively to the person. Also, at block <NUM> the robot may be activated to move on the propulsion elements <NUM> according to the microphone signal, e.g., to turn and approach a person being imaged by the camera.

Commencing at bock <NUM> in <FIG>, input may be received from the touch surface(s) <NUM>. At block <NUM> the processor may actuate the head <NUM> to move in response to the touch signal, e.g., to bow the head as if in respect to the person touching the head. Also, at block <NUM> the robot may be activated to move on the propulsion elements <NUM> according to the touch signal.

<FIG> illustrates a use case of the robot <NUM>. A first user <NUM> may operate a smart phone or tablet computer or other control device <NUM> to communicate with a first robot 10A. Remote from the first user <NUM>, a second user <NUM> may operate a smart phone or tablet computer or other control device <NUM> to communicate with a second robot 10B.

As indicated in <FIG>, the first robot 10A presents on its display <NUM> a full-face image <NUM> of the (frowning) second user <NUM> (equivalently, a frowning avatar face). The second robot 10B presents on its display <NUM> a full-face image <NUM> of the (smiling) first user <NUM> (equivalently, a smiling avatar face). The image <NUM> on the display of the first robot 10A may represent the second user <NUM> based on images generated by a camera in the second user control device <NUM> or a camera in the second robot 10B and communicated over, e.g., a wide area computer network or a telephony network to the first robot 10A. Likewise, the image <NUM> on the display of the second robot 10B may represent the first user <NUM> based on images generated by a camera in the first user control device <NUM> or the first robot 10A and communicated over, e.g., a wide area computer network or a telephony network to the second robot 10B. The face images of the users/avatars may be 2D or 3D and the displays <NUM> of the robots may be 2D displays or 3D displays.

Moreover, the head of the first robot 10A may be controlled by the processor in the first control device <NUM> and/or the first robot 10A to rotate and tilt in synchronization with the head of the second user <NUM> as indicated by images from the second control device <NUM> and/or second robot 10B. Likewise, the head of the second robot 10B may be controlled by the processor in the second control device <NUM> and/or the second robot 10B to rotate and tilt in synchronization with the head of the first user <NUM> as indicated by images from the first control device <NUM> and/or first robot 10A.

In both cases, however, the image of the faces on the robots remain full-face images as would be seen from a direction normal (perpendicular) to the display <NUM> from in front of the display, regardless of the orientation of the head of the respective robot. The full-face images are cropped from any background in the images of the respective user and are also cropped from body parts of the respective below the chin that may appear in the images. The full face images may be generated even as the head of the respective user turns away from the imaging camera consistent with disclosure herein, so that the front display surface of the robots present not profile images as generated by the cameras but full face images derived as described herein from camera images of a turned head no matter how the robot head is turned or tilted, just as a human face of a turned head remains a full face when viewed from directly in front of the face from a line of sight perpendicular to the face.

As below-the-head images of a user indicate movement (such as but not limited to translational movement) of the user, the corresponding (remote) robot may also move in the direction indicated by the images by activating the propulsion motor and, hence, propulsion elements <NUM> of the robot. In particular, the body portion of the robot below the display may move. Further, speech from the first user <NUM> as detected by the first control device <NUM> or first robot 10A may be sent to the second robots 10B for play on the speaker on the second robot, and vice-versa.

Thus, the first user <NUM> may interact with the first robot 10A presenting the face image of the second (remote) user <NUM> as if the second user <NUM> were located at the position of the first robot 10A, i.e., local to the first user <NUM>. Likewise, the second user <NUM> may interact with the second robot 10B presenting the face image of the first (remote) user <NUM> as if the first user <NUM> were located at the position of the second robot 10B, i.e., local to the second user <NUM>.

<FIG> further illustrates the above principles, assuming that user images are employed, it being understood that the same principles apply when avatars expressing the user emotion are used. At <NUM> the first user <NUM> of <FIG> operating the first control device <NUM> is imaged using any of the cameras discussed previously to move the second robot 10B and to send an image of the face of the first user <NUM> to the display <NUM> of the second robot 10B for presentation of a full-face image (and only a full-face image) on the second robot 10B. The image may be sent to a network address of the second robot 10B or sent to the second control device <NUM> shown in <FIG>, which relays the image to the second robot 10B via, e.g., Wi-Fi or Bluetooth. No background apart from the face image and no body portions of the first user <NUM> other than the face are presented on the display <NUM> of the second robot 10B.

As shown at <NUM>, should the first user <NUM> turn his head to the left, this motion is captured, e.g., by the camera(s) in the first control device <NUM> and/or first robot 10A and signals such as a stream of images are sent to the second robot 10B as described above to cause the processor <NUM> of the second robot 10B to activate the head actuator <NUM> to turn the head <NUM> of the second robot 10B to the left relative to the body <NUM> of the second robot 10B, as illustrated in <FIG>. However, the display <NUM> of the second robot 10B, although turned to the left relative to the front of the body <NUM>, does not show a profile view of the head of the first user <NUM> as currently being imaged by the camera(s) of the first control device <NUM> or first robot 10A. Instead, as shown in <FIG> the display <NUM> of the second robot 10B continues to show a full-face image, i.e., an image of the face of the first user <NUM> as would be seen if looking directly at the face from a line of sight perpendicular to the face.

<FIG> illustrates further principles that may be used in connection with the above description. Commencing at block <NUM>, an input set of training images is input to a machine learning (ML) model, such as one or more of a convolutional neural network (CNN), recurrent NN (RNN), and combinations thereof. The ML model is trained using the training set at block <NUM>.

The training set of images may include 3D images of human faces from various perspectives, from full frontal view through full side profile views. The training set of images may include ground truth 2D full frontal view representations of each 3D perspective view including non-full frontal 3D perspective views. The ground truth 2D images are face-only, configured to fill an entire display <NUM> of a robot <NUM>, with background and body portions other than the face cropped out from the corresponding 3D images. The full-frontal view representations show facial features as well as emotional distortions of facial muscles (smiling, frowning, etc.). In this way, the ML model learns how to generate full frontal view 2D images from a series of 3D images of a user's face as the user turns his head toward and away from a camera rendering the 3D images.

Accordingly, present principles may employ machine learning models, including deep learning models. Machine learning models use various algorithms trained in ways that include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms, which can be implemented by computer circuitry, include one or more neural networks, such as a convolutional neural network (CNN), recurrent neural network (RNN) which may be appropriate to learn information from a series of images, and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVM) and Bayesian networks also may be considered to be examples of machine learning models.

As understood herein, performing machine learning involves accessing and then training a model on training data to enable the model to process further data to make predictions. A neural network may include an input layer, an output layer, and multiple hidden layers in between that that are configured and weighted to make inferences about an appropriate output.

<FIG> illustrates logic attendant to <FIG> and <FIG> using a ML model as trained in <FIG>. It is to be understood that the logic of <FIG> may be executed by any of the processors or combinations thereof described herein, including the processor of a server on a wide area computer network communicating with the control devices <NUM>, <NUM> and/or robots 10A, 10B.

Commencing at block <NUM>, for each user (assume only two users <NUM>, <NUM> as shown in <FIG> for simplicity) images are captured at bock <NUM> of the user's face, including images showing motion of the face and body of the user. The voice of the user is captured at block <NUM> and both the voice signals and image sequence of the user as the user moves and speaks are sent at block <NUM> to the other user's local robot.

Meanwhile and proceeding to block <NUM>, the same signals - image sequences of the face and body motions and voice signals of the other user - are received at block <NUM>. The image of the face of the other user, if not already full face as would be seen looking directly at the other user along a line of sight perpendicular to the front of the face of the other user, is converted at block <NUM> to a 2D full face image using the ML model trained as described, with background and body parts of the other user other than the face being cropped. The full face 2D image is presented on the display <NUM> of the local robot preferably by entirely filling the display with the image of the face of the other user. As mentioned above, conversion of a 3D image in profile of a user's face to a full face 2D image may be effected by any one or more of the processors described herein.

In an alternative embodiment, in lieu of using ML models to convert 3D images to full face 2D images, a single 2D full face image of the other user may be obtained and presented for the duration on the local robot. As also discussed, avatars may be used for privacy instead of the image of a person, with the expression of the avatars preferably being animated according to the expression of the person.

If desired, the other user's voice may be played at block <NUM> on the local robot or the local control device. Also, at block <NUM> the head of the local robot may be turned to mimic head motion of the other user as represented by the sequence of images received at block <NUM> and as shown at <NUM> in <FIG>. Moreover, in the event that the other user moves his body by, e.g., walking, that motion is captured and received at block <NUM> and input to the processor of the local robot to actuate the propulsion elements <NUM> (or, if the robot is in a vehicle such as the vehicle <NUM> shown in <FIG>, the vehicle) to translationally move the local robot to mimic the motion of the other user.

<FIG> indicate that in lieu of the motorized vehicle <NUM> shown in <FIG>, the robot <NUM> may be mounted on other types of moving platforms such as a bicycle <NUM>, a crab-like tractor <NUM>, or an airborne drone <NUM>.

A processor may be a single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.

Network interfaces such as transceivers may be configured for communication over at least one network such as the Internet, a WAN, a LAN, etc. An interface may be, without limitation, a Wi-Fi transceiver, Bluetooth® transceiver, near filed communication transceiver, wireless telephony transceiver, etc..

Computer storage may be embodied by computer memories such as disk-based or solid-state storage that are not transitory signals.

Claim 1:
A device comprising:
at least one computer storage (<NUM>) that is not a transitory signal and that comprises instructions executable by at least one processor (<NUM>) to:
for at least a first user (<NUM>), render, from at least one image captured of the first user (<NUM>) by at least one imager (<NUM>), a full-face image representing the first user (<NUM>) with background and body parts of the first user cropped out of the image representing the first user, the full-face image (<NUM>) being a front view of the first user's face from a line of sight perpendicular to the face;
provide, to at least a first robot (10A) remote from the first user (<NUM>), the full-face image (<NUM>) for presentation thereof on a display of the first robot (10A) with the full-face image (<NUM>) filling the display (<NUM>);
provide, to the first robot (10A), information from the imager regarding motion of the first use (<NUM>) such that a head (<NUM>) of the first robot (10A) turns to mimic the motion of the first user (<NUM>) while continuing to present the full-face image (<NUM>) on the display of the first robot (10A) regardless of whether the head of the first user (<NUM>) turned away from the imager.