Mobile human-friendly assistive robot

A robotic assistant, associated software and methodology for operating the same. The described robotic assistant includes: a motorized base having at least two motor driven wheels controlled by a first control platform; a dual arm robot mounted on the motorized base, the dual arm robot having a first arm and a second arm controlled by a second control platform; a remote sip and puff mouth controller having three degrees of operational freedom; and a computer system that receives command signals from the remote sip and puff mouth controller, and includes an algorithm that translates the command signals into a first type of control signal for directing the motorized base to move, and a second type of control signal for directing the dual arm robot to move.

TECHNICAL FIELD

The subject matter of this invention relates to mobile assistive robots, and more particularly to a dual arm mobile assistive robot devices integrated into a motorized wheelchair for assisting mobility impaired individuals.

BACKGROUND

Mobility disabled individuals usually require full-time caretakers to be present with them to help with daily activities, which can be very expensive in the long run. One approach in addressing this need is to develop a robotic assistant that could help the user perform daily chores that they are unable to perform themselves.

Despite the proliferation of industrial robots in fields such as automotive and semiconductor manufacturing, food processing, and material handling, existing robots largely operate in a teach-and-repeat mode, with limited human interaction. For less structured environments, such as in assistive living, human guidance and intervention is essential.

Current robotic solutions for mobility impaired individuals mostly consist of a single robotic arm mounted on a wheelchair, which limits its usage to simple tasks like ‘pick and place’. However, these arms are usually controlled via a joystick and as a result, are not appropriate for individuals who are paralyzed from the neck down (approx. 5000 per year in US), or have limited use of their hands. Furthermore, adapting existing robots that are controllable via human guidance for mobility impaired individuals are faced with numerous economic and technological hurdles that have yet to be fully addressed by the prior art.

Research work is also being done to develop fully automated robots that can navigate and pick/place objects by themselves. However, these systems only exist in academia and are tailored for specific scenarios or workspace. As such, most such solutions could potentially be useful for mobility impaired individuals are still some ways off of being commercially available and are also very expensive. Accordingly, a need exists for a robotic assistant that can help severely handicapped individuals.

SUMMARY

The present disclosure describes a robotic assistant that is more versatile, cheaper, and which can be remotely controllable by anyone whose mobility is impaired. The disclosed robotic assistant generally comprises a motorized base and dual arm robot mounted thereon. The robotic assistant is designed to be utilized by mobility impaired individuals through a sip-and-blow interface, tongue command device, eye/blink tracker, a joystick, etc., to, e.g., command the motion of the assistant to pick up objects from floor or shelf, to inspect suspicious noise, etc.

The assistive robot can also serve as a surrogate for medical staff/caregivers at a remote location, e.g., via the Internet, to inspect and examine the individual through vision and touch (with possible haptic feedback) and deployment of different biometric sensors such as thermometer, stethoscope, oximetry, ophthalmoscope, etc.

Sensors (e.g., vision, proximity, touch) on the robotic assistant may be used to provide feedback to the user to facilitate task execution (e.g., warning signal for impending collision, etc.). The multi-arm feature of the platform may also be used in a social companion/assistant context, to encourage exercise, play games, etc. The platform may be connected to the Internet to allow remote operation by medical staff/caregivers to inspect/examine/interact with a patient from remote surroundings. Through the distributed software architecture, the platform may be integrated into sensors/actuators in the user environment (e.g., a smart home) as a mobile sensor for home lighting, HVAC, and security systems.

The described embodiments have advantages over existing solutions, including:

(1) The multi-arm capability allows a wider range of tasks to be performed such as handling large and bulky objects, manipulating flexible objects such as clothing, cables and sheets, and inherent two arm tasks such as opening jars, preparing food, etc.

(2) The interface is adapted to handle a suite of input devices, such as mouth controllers, joysticks, etc., accessible to anybody who is mobility impaired.

(3) The robotic assistant can serve as a telepresence platform. Current telepresence systems do not have manipulation capabilities (let alone dual-arm). This platform can provide a much wider range of diagnostic and examination options, including touch and feel (through haptic feedback), and deployment of diagnostic devices.
(4) With the arms, this platform is anthropomorphic, allowing it to provide a social companion function. Because the system is teleoperated, the solution is an order of magnitude cheaper compared to systems that utilize fully automated robots.

A first aspect provides a robotic assistant, comprising: a motorized base having at least two motor driven wheels controlled by a first control platform; a dual arm robot mounted on the motorized base, the dual arm robot having a first arm and a second arm controlled by a second control platform; a remote sip and puff mouth controller having three degrees of operational freedom; and a computer system that receives command signals from the remote sip and puff mouth controller, and includes a control system that translates the command signals into a first type of control signal for directing the motorized base to move, and a second type of control signal for directing the dual arm robot to move.

A second aspect provides a control platform for controlling the operation of a robotic assistant, comprising: a control system that separately communicates with a motorized based and a dual arm robot mounted on the motorized base, wherein the control system implements a first type of control signal to control the motorized base, and a second type of control signal to control the dual arm robot; an input for receiving command signals from a remotely located controller, wherein the command signals include a proposed movement in a three dimensional space of an end effector associated with a selected arm of the dual arm robot; and a software algorithm that dictates the motion of the motorized base and dual arm robot based on the proposed movement, wherein the software algorithm only instructs the motorized base to move when the selected arm is outside a defined workspace, and only instructs the selected arm to move when the motorized base is stationary.

A third aspect provides a robotic assistant, comprising: a motorized base having at least two motor driven wheels controlled by a first control platform; a dual arm robot mounted on the motorized base, the dual arm robot having a first arm and a second arm controlled by a second control platform; an input for receiving signals from a remotely located controller, the remotely located controller having three degrees of operational input for providing a proposed movement of the robotic assistant; and a control system that separately communicates with the motorized based and dual arm robot, wherein the control system implements a first type of control signal to control the motorized base, and a second type of control signal to control the dual arm robot, and wherein the control system includes a software algorithm that dictates the motion of the motorized base and dual arm robot based on the proposed movement, wherein the software algorithm only instructs the motorized base to move when a selected arm is outside a defined workspace, and only instructs the selected arm to move when the motorized base is stationary.

DETAILED DESCRIPTION

Referring now to the drawings,FIG. 1depicts a front view andFIG. 2depicts rear isometric view of a mobile assistive robot (hereinafter “robot” or “robotic assistant”)10that can be remotely controlled, e.g., by mobility disabled individuals to improve their quality of life. Recent research efforts in the field of manipulation-based assistive robotics are centered on creating autonomous robotic assistants. In contrast to this common paradigm, the present approach focuses on providing a platform for allowing a handicapped individual to manually control the robotic assistant10. Namely, a platform is provided so that the mobility disabled person is also able to control the robotic assistant10using an interface in order to increase the individual's independence and to maximize the number of tasks the robotic assistant10can complete.

Robotic assistant10generally comprises a motorized base12with at least two independently driven wheels and at least two additional small caster wheels for balance, an on-board manipulator that generally comprises a dual arm robot16sensors, a distributed control platform and/or other input/output devices. Robotic assistant10is controlled by a remote a human input device such as mouth controller, joystick, or other device.

In one embodiment, the robotic assistant10is an integration of a commercially available retrofitted power wheelchair, and an off the shelf dual arm robotic system such as the dual arm BAXTER™ robot by Rethink Robotics™, and a JAMBOXX™ sip and puff mouth controller. The control platform is generally managed by a computing system, such as a general purpose computer (e.g., a laptop26) or special purpose computing device. Although various embodiments are described in terms of commercially available systems, such as a retrofitted wheelchair, Baxter robot and Jamboxx device, it is understood that other commercially available systems could likewise be utilized, as well as custom built systems.

As noted, one example of a mouth controller is the Jamboxx, which utilizes a sip and puff controller with three degrees of freedom including: (i) a slider that can be shifted left and right with the mouth, (ii) the rotation of the same slider up and down, and (iii) a pressure sensor attached to the slider used as suck/puff interface (breath controller). The design focuses on high accuracy and low movement resistance so it can be used over longer periods of time. Using this device, an impaired user can control a computer with the precision of a computer mouse.

While dual arm robots are regularly used for various industrial applications, their use is primarily limited to preprogrammed repetitive tasks. The present invention draws on tools from both industrial robotics for object manipulation and handling, and social robotics, where robots provide human communication and assistance by performing more than simple repetitive tasks. To achieve this, a human-scale lightweight dual arm robot system16, such as the Baxter robot, is utilized. Such a robotic system can safely operate in the presence of humans and utilizes a 120V power supply, which can easily integrate with the battery power supply on the mobile base through an inverter20.

As shown inFIGS. 1 and 2, robotic assistant10generally includes a retrofitted power wheel chair base (i.e., motorized base)12that includes a motorized drive and wheels, a battery pack compartment18, a mounting adaptor plate14for mounting the dual arm robot16, and a back plate34for mounting equipment and accessories. The use of existing power wheelchair technology as the motorized base12for the robotic assistant10greatly reduces the cost, while providing a proven solution. The motorized base12may for example be retrofitted from a QUICKIE™ s646 wheelchair powered by two lead-acid batteries, which can easily be modified for computer interface and control.

Battery pack compartment18may utilize any type of system (e.g., lead or lithium batteries, fuel cells, etc.) for providing power to both the wheel chair base12and dual arm robot16. Because the Baxter robot operates on 120V AC, an onboard inverter20is utilized to convert the power for the dual arm robot16. The dual arm robot16, such as the Baxter robot, generally includes a torso28, left and right arms30, grippers32, a display screen24, plurality of cameras (e.g., proximate the grippers32and screen display24), and head sonar distance sensors22. An onboard laptop computer26is provided along with additional sensors, such as infrared or the like. In addition, a positional data collector may be implemented within the motorized base12and laptop26to track the position of the robotic assistant in a space over time.

The dual armed robot16as shown includes a total of fourteen degrees of freedom with seven degrees of freedom in each arm30and is designed to be safe to operate near humans. The robotic assistant10as a whole is controlled by receiving remote commands from the operator to the laptop26mounted on the robot10. Once a message is received, it is interpreted by the laptop26and sent to either the dual armed robot, if it is a manipulation command, or the wheelchair circuitry within the motorized base12, if it is a motion command.

As noted, the Baxter robot consists of two 7-degree-of-freedom arms fixed to a rigid base, along with a suite of sensors found along the torso and on each arm. Each arm's kinematics consists of a collocated wrist, a near-collocated shoulder with a small offset between the first and second joints, and an elbow. The kinematics for the arm are shown inFIG. 3. Each joint servo is connected to an encoder and torque sensor for closed loop control and to detect servo loading from contact. Additionally, at the “end effector” of each arm is a parallel jaw gripper32, an infrared ranger, and a camera. On the ‘head’ of the robot is a ring of sonar sensors22, a display screen24and rotatable camera.

The internal control system for the dual arm robot16may be implemented with a Robot Operating System (ROS). The robot control platform subscribes to the joint positions and publishes joint velocities at a rate of 100 Hz. There are additional services running onboard the robot to prevent self-collision, publish gravity compensation torques to the servos, and, most importantly, maintain safety by halting the servos during an unexpected impact. A custom script using the Rethink Robotics Python SDK may for example be utilized to make the Baxter robot subscriptions and publications, while wrapped in a Robot Raconteur service to host the relevant joint information on a local network. This allows for rapid development in high-level languages, distribute computational effort among a network of computers, and incorporate high-performance visualizations into our control architecture.

A Jamboxx may be used as the input device to control both arms of Baxter in task space. Typically, task space controllers use Jacobian-based methods to map the end effector velocities to joint velocities. The general form of the Jacobian matrix JTis

JT=[h1h2…hTh1x⁢p1⁢Th2x⁢p2⁢T…hTx⁢pTT],(1)
where each hiis the axis of revolution for joint i, and piTis the vector from joint i to the end effector. All vectors are represented in the Baxter base frame. Each arm on Baxter has seven joints, so the Jacobian is a 6×7 matrix. The Jacobian maps between joint velocities and task spatial velocity:

[ωTvT]=JT⁢q..(2)
Given a set of desired linear and angular velocity for the end effector, we may use the damped-least squares algorithm to command the joint motion:

The coupling between position and orientation of the robot end effector may be non-intuitive for user to command. Instead, we allow the user to command position and orientation separately. The overall control consists of the following steps:

1) The user uses Jamboxx to select from a variety of options (seeFIG. 4):

Desired direction of movement for end effector (forward, backward, left, right, up, down)Desired wrist joint movement in the positive or negative directionOpening or closing of the gripper
2) The selected option is then translated into a fixed linear end effector velocity (vT) or a fixed wrist joint velocity (qi, for i=5, 6, or 7).For linear motion, the angular velocity (wT) is set at zero and the damped-least squares controller is used to generate the desired joint velocity vector, which is then published to Baxter.For angular motion, the wrist joint velocities are directly published to Baxter.

In the described embodiment, the mobile base with batteries weighs approximately 118.8 kg while the Baxter robot weighs an additional 74.8 kg—similar to a normal adult and well within the payload range of a power wheelchair. One of the chief concerns is to ensure the safety of those around the system during use. For the mechanical design, this manifested itself into making sure that the combined system would not be in danger of tipping during normal operation. Of the possible causes for instability, the two most likely scenarios are tipping from rapid acceleration of the motorized base12, and tipping due to center of gravity displacement. The first of these is resolved by establishing speed limits on the motorized base12. A ‘Safe’ arm configuration is also chosen for rapid motion. In order to ascertain how the system's stability would be affected by the Baxter's arm movements, the center of gravity for the entire system was calculated in the most extreme possible arm configurations. A CAD model is constructed of the entire platform, complete with physical data, and used to examine the effect of arm motion on the center of gravity. It is important for the system to remain stable in all arm configurations without limiting the work envelope or payload capacity. The most effective mounting location was then chosen based on the results from these calculations. The adapter plate14(FIG. 1) was designed in order to locate the combined center of gravity as close to the center of the wheelbase as possible while still allowing full arm movement with the 2.3 kg max payload for which Baxter is rated.

With the chosen mounting location, the arms from Baxter have a working envelope that extends from the ground directly in front of the wheelchair to a maximum working height of 1.63 m off the ground. Each arm is also able to extend 1.04 m out to either side of the robot or 0.68 m in front of the motorized base12. The combined footprint of an illustrative system is only 0.914 m in length and 0.673 m in width making it only 2 cm wider than the power wheelchair itself and equally as long. This is an important feature due to door and hallway width restrictions that are likely to exist in many households. In addition, a rear mounting surface34was designed in order to accommodate the equipment needed to mobilize the system. This provides a mounting location for the inverter20, control laptop26, as well as any future hardware. By locating these objects on the opposite side of the center of mass from the robotic end effecters, the stability margin was further improved particularly for the scenario when heavy objects need to be retrieved from the ground in front of the system.

There are several devices on the robot that need to be powered including the Baxter16, laptop26and motorized base12. Both the Baxter16and the laptop26are powered through a typical wall outlet while the motorized base12requires a 24V DC line. The power architecture is structured around the power source used in the robot: two 12V/69 Ah gel-cell batteries. The batteries are connected in series to power the mobile base12and in parallel to inverter20. The inverter20converts the DC power into AC power that the Baxter16and laptop26can use.

Controlling the robot as a whole requires simultaneous control over both the motorized base12and the Baxter robot16. To accomplish this, a laptop26is mounted on the rear mounting surface34to send the appropriate control signals to a corresponding device. When the laptop26receives a message from a user, it sends a command to either the Baxter16or the motorized base12. The overall software architecture is shown inFIG. 5. The laptop26is connected to the Baxter robot16through Ethernet and has the capability to send commands directly to the computer inside Baxter16. The laptop26receives task space commands (left, right, up, down, etc.) for moving Baxter's arms from the Jamboxx40. It then uses a Jacobian based damped least-squares controller to calculate the joint velocities required to move the arms in the desired direction. These joint velocities are then transmitted via a custom Python script to read Robot Raconteur messages and publish ROS messages to the corresponding rostopics on Baxter's internal computer. This script acts as a Robot Raconteur ROS bridge. In contrast, the mobile base is controlled through intermediate circuitry. The laptop26is connected to an Arduino (via USB) which is then connected to the controller for the mobile base. When the laptop26receives a command to move the mobile base, it converts the command into a serial byte to send to the Arduino. Once the Arduino receives this byte, it sends a control signal to a digital potentiometer to mimic a physical joystick. This signal is passed to the mobile base's control system and functions as if it were receiving commands from a physical joystick.

Any type of input devices may be utilized to operate the robotic assistant10, including, e.g., a Jamboxx or a joystick. Using such devices, two possible control schemes may be implemented. The first control scheme involves placing all functionality required to control the robot10on a single input device. This results in the ability to drive the mobile base and operate the arms at the same time with ease.

A second control scheme involves a hybrid approach and utilizes both the Jamboxx and joystick. This scheme was tailored towards for use by quadriplegics as they are able to proficiently use the Jamboxx and can often control a joystick, like the ones mounted on a wheelchair, as well. In one embodiment, Baxter is controlled by the Jamboxx while the motorized base12is controlled by a joystick.

In addition to the controller layout, feedback from the system is delivered to the user in two ways. First, the Baxter robot16itself is equipped with three onboard cameras which can stream video to the operator. There is a head camera which provides an overhead view of the motion of the system, as well as two cameras located on the end effectors of the two arms. By using a combination of the viewpoints, the operator is able to see how the robotic assistant10is positioned in order to complete the desired task.

A custom MATLAB simulation package may be utilize to give the operator real-time feedback of the arm positions. This may be done by approximating the joints and linkages of the robot with a set of primitive 3D shapes. By using the known forward kinematics of the system, the current arm configuration is drawn by passing the drawing package a series of joint angles for each arm. A simulated configuration is shown inFIG. 6. Having the capability to simulate the robotic system also provides the dual functionality of allowing the designers to test different control layouts, or allowing users to get comfortable operating the robotic assistant10, without needing to activate the physical system. This will be particularly useful when individuals with different physical disabilities require custom control interfaces.

In the first control scheme noted above, both the mobile base and dual arm robot are controlled by a single input device, such as a Jamboxx. There are two typical operations: local tasks, which are within the arm workspace, and non-local tasks, which require base motion to position the task within the arm workspace. Arm and base are either controlled separately or together as an integrated unit. In the former case, the mobile base moves the arm into the area of interest and remains stationary while the arm performs the task. This requires switching between the two modes, placing added burden on the user. In the latter case, the arm and base may move together even when the base motion is unnecessary. Such unanticipated base motion compromises safety and, because of the weight of the system, reduces responsiveness and consumes more energy.

In the following embodiment, a general human-robot cooperative control approach is provided in which the system automatically determines if the mobile base needs to be moved to get an arm into a desired workspace. The arm (or arms) will only then move when the motorized base12is within the work space. The arm(s) thus stay stationary while the motorized base12moves, but the process is automated so that the user does not need to switch between the two modes.

This is achieved by providing a highly redundant mobile robotic system with the human user providing motion command through an input device with a limited number of degrees of freedom. The control problem is posed as a constrained optimization at the kinematic level that determines the robot joint velocity to minimize the deviation from the human commanded motion input. A set of equality and inequality constraints is imposed based on joint and velocity limits, singularity avoidance, and collision prevention. In addition, an equality constraint is included to ensure that the base motion follows the commanded end effector motion in a natural way by avoiding unanticipated combined arm/base motion. Since the problem is kinematic in nature, the resulting minimization becomes a convex quadratic program which guarantees a unique, and efficiently solvable, solution. This type of kinematic based algorithm is local in nature, meaning that even a globally feasible motion plan exists for the overall task, the motion may get stuck locally.

One goal is to locally follow the human command as closely as possible without violating the constraints, and rely on the human user to steer the system for the global task. A key motivation for this work is to provide assistance to severely mobility impaired individuals.

FIG. 7shows a schematics of the system at the zero configuration, where all the joint angles are at zero, and the motorized base12is centered at the origin and facing the positive x-direction of the inertia frame. The numerical values of the robot parameters are:

Only the left arm is labeled; the right arm is identical to the left, symmetrical with respect to the x-z plane of the base frame. The link parameters of the arms are a, b, c, d as shown in the diagram. The displacement of the base of the arms from the motorized base12coordinate center is given by the vector pBOLand pBOR. The joint angles of the left and right arms are denoted by the vectors qLand qR, respectively, where qLconsists of {qL1, . . . qL7} and qRconsists of {qR1, . . . qR7}. Assume no wheel slippage, the kinematics of the motorized base12is given by the standard unicycle model:

[θ.Bx.By.B]︸q.B=[100cos⁢⁢θB0sin⁢⁢θB]︸JB⁡[ωBvB]
where (xB, yB) is the motorized base12center with respect to inertia frame, θBis the orientation of the motorized base12, and (wB, vB) are the turning and linear velocities of the motorized base12. In this discussion, we only consider one arm at a time, e.g., the left arm. The differential kinematics of the left arm/base system in the mobile base frame is given by

VT=JTL⁢q.L+[ez⁢ωB-(pBOL+pOL⁢TL)×ez⁢ωB+ex⁢vB]
where VT=[wT, vT]Tis the spatial velocity (angular and linear) of the task frame at the end effector, JTLis the corresponding arm Jacobian, pOLTLis the arm base to end-effector vector, ex=[1, 0, 0]Tand ez=[0, 0, 1]Tare the unit vectors in the base frame.

Combining these two equations, we can write the overall kinematics in a compact form:

Consider the scenario where the human user provides the desired motion, in terms of either position or velocity, through an input device. The goal is to determine the mobile manipulator joint motion (including both the manipulator and the mobile base) to stay close to the human command while satisfying a set of equality and inequality constraints. The user would typically use velocity command for large motion (the input is like a gas pedal) and position command for close-up, more precise motion. The constraints could be due to environmental constraints, singularity avoidance, or specified joint pose. There is also the more qualitative requirement (for now) that the robot motion should be “natural”, avoiding unanticipated internal arm motion or the base motion.

To be more precise, denote the Euclidean frame (orientation and position) the human user is commanding by {EA, OA}. This could be the end effector frame or a frame affixed somewhere on the robot, e.g., the arm elbow joint or the base. The differential kinematics is given by VA=JA(q)u, where VAis the spatial velocity of frame A, JAthe Jacobian, q the robot configuration (manipulator joint displacement and the base position and orientation), and u the commanded velocity (manipulator joint velocities and mobile base heading velocity and angular velocity). The user may command just a subset of frame A, e.g., translation or rotation. For the purpose of this discussion, only translation is considered, as it is the most common type of operation. Rotation may be useful in dual arm operation as well, but is omitted for brevity. Denote the corresponding linear velocity, in the mobile base frame, by vH, which may be a direct velocity command from the user or the velocity control for the position command. The human-directed velocity vHis related to the vAthrough a selection matrix S:
vH=SvA=SJA(q)u:=JH(q)u.

The user can also specify some joints or frames in the robot to be in a given configuration, e.g., elbow up or down to avoid bumping into objects, or fixed wrist orientation to hold a tray level. It is also important to observe joint limits, avoid singularities which would reduce mobility, and maintain a safe distance from obstacles. All of these are holonomic constraints on the robot configuration. To be specific, the problem that we are addressing is to determine u to best achieve the commanded velocity vHwhile observing imposed equality and inequality constraints:

As q is dynamically related to u, this is an optimal control problem over a specified time period [0, T] (with a given {vH(t): tϵ[0, T]}), and the norm in the above formulations is a functional norm. We relax the problem by converting the equality constraint to a soft constraint (becoming part of the control objective) and the inequality constraint to a barrier function. For the equality constraint, consider a Jacobian-based velocity control:
vE:=JE(q)u=−kEhE(q),
where JEis the Jacobian corresponding to hE(q) and kEis the feedback gain. If the number of equality constraints is less than the number of joint velocity input, and the robot is not at a singular configuration with respect to JE, then JEis full row rank and this equation always has at least one solution.

For the inequality constraint, we require the input to steer the system towards the feasible region if it is infeasible, and remain in the feasible region if the system is already within it. This becomes a linear inequality constraint on u:

dhIidt=∂hIi∂q⁢Jq⁡(q)⁢u≥σi,i=1,…⁢,ni
where Jq is the arm Jacobian (needed for the nonholonomic base constraint), niis the number of inequality constraints, and σi, shown inFIG. 8, is given by

A regularization term ∥u∥ may need to be added to the cost function to ensure that the solution is unique. This problem is a convex quadratic program which may be efficiently solved using commercially available tools. If a solution exists, then the robot would remain within the feasible region while staying close to the human commanded motion. However, there could be the local minimum issue when the algorithm is “stuck” (u=0 before the goal is reached). For the purposes of brevity, we only consider local solution of u and rely on the human user to steer out of local minima, if encountered.

The design of the inequality and equality constraints is critical to well coordinated motion under human command. We formulate a set of design requirements below to guide the choice of these constraints.

1) When the user does not give commands (vHis zero), both the arm and the motorized base12do not move. This requirement is obviously necessary to avoid any unintended surprise motion.

2) When the arm is inside the interior of the workspace, only the arm moves but not the motorized base12. This requirement is to avoid excessive base motion when arm alone could follow the user command.

3) When the arm is at the workspace boundary, and the user commanded velocity is the direction that would move beyond the arm workspace, then the motorized base12should move to achieve the desired user motion commands with the end-effector orientation locked. This requirement allows base motion only when it is necessary to follow the user command.

4) When the arm is at the workspace boundary, the base motion should result in the end-effector motion in the same direction as the user commanded motion. This requirement will avoid the unnatural motion where the base seems to not follow the user command (with the arm also moving to compensate for the undesired base motion). This situation is illustrated inFIG. 9using the Jacobian pseudo-inverse controller. Note that during the transient (as in t=2.5 sec and t=5 sec snapshots), the base rotates in the opposite direction as the user commanded direction. Eventually the base rotates in the right direction, but such unnatural transient motion should be avoided.

5) When the obstacle is close to the mobile base, the mobile base speed is reduced, and eventually becomes zero as the minimum distance to the obstacle decreases. This requirement is necessary for safety.

We will refine the constrained optimization problem step-by-step to satisfy the above requirements. The first step is to remove the equality constraint through a reparameterization of u:
u=JE⊥ξ+JE+vE

Where the columns of JE⊥is a basis of the null space of JEand JE+is the Moore-Penrose pseudo-inverse of JE. The previous becomes

minξ⁢JH⁡(JE⊥⁢ξ+JE+⁢vE)-vH
subject to

∂hIi∂q⁢Jq⁡(q)⁢(JE⊥⁢ξ+JE+⁢vE)≥σi,i=1,…⁢,ni.
If the matrix JHJE⊥is not of full column rank or is ill conditioned, there could be large changes in u in each iteration. One solution is to add a penalty term to prevent u to deviated too far from its previous value:

minξ⁢JH⁡(JE⊥⁢ξ+JE+⁢vE)-vH+λ⁢(JE⊥⁢ξ+JE+⁢vE)-uprev
subject to the above, where uprevis the input in the previous iteration, and λ is a weighting factor for the relative emphasis between changes in u versus following the user command. The problem with this approach is that even when the user is not moving, i.e., vH=0, the robot could continue to move (violating Requirement 1). We therefore adopt an alternate approach by adding a regularization term to the cost function. Let

JHE=[JHJE]⊥
Replacing the optimization problem above by

minξ⁢JH⁡(JE⊥⁢ξ+JE+⁢vE)-vH+λ⁢(JHET⁡(JE⊥⁢ξ+JE+⁢vE)
subject to the same constraint. Since

[[JEJH]JHET]⁢JE⊥=[0JH⁢JE⊥JHET⁢JE⊥]⌊
is of full column rank. This implies

𝒩_⁡((JH+λ_⁢JHET)⁢JE⊥)-={0}
where N denotes the null space. Hence the minimization problem is strictly convex, meaning that there will be no unwanted motion, and at the same time, the user command tracking or motion imposed by the equality constraints are not be unduly penalized. The constant λ is again a weight constant between motion in the null space, and user command following. It is preferable to move the arm, instead of the base, whenever possible, as the arm is lighter weight and flexible, and therefore more responsive, consumes less energy, and less threatening to the user. To emphasize arm motion over base motion, we add a penalty term to the base motion. Let C be a matrix that select the base velocity command from u:

JHEC=[JHJEC]⊥
We then replace the above by

minξ⁢JH⁡(JE⊥⁢ξ+JE+⁢vE)-vH+λ1⁢(JHECT⁡(JE⊥⁢ξ+JE+⁢vE)+λ2⁢C⁡(JE⊥⁢ξ+JE+⁢vE)
where λ1 is the weighting for the null space motion and λ2 is the weighting for the base motion. If the user command and equality constraints may be satisfied with arm motion alone without violating the inequality constraints, this method ensures that there would be no base motion, satisfying Requirements 2 and 3. In the specific case when the user commands a fix point on the arm to translate in certain direction and the arm reaches its boundary, it is more natural to have the base follow the desired direction of motion (projected onto the x-y plane), rather than having other combined base/arm motion, as shown inFIG. 10. In this case, we add a single equality constraint
ezT(vH×(ωB×p0T+vB))=JEBu=0
where p0Tis the vector from center of the wheel axle to 0T, and JEBis the matrix that captures the constraint. The equality constraint is only enforced in the ez direction since we are only interested in the x-y components of vH. We now simply append this additional constraint to the existing equality constraint

[JEJEB]︸E⁢u=[vE0]︸β.
This ensures that we satisfy Requirement 4. Requirement 5 follows from the inequality constraint defined above. Summarizing the above, the proposed control algorithm for the kinematic control u for a given user velocity command and equality and inequality constraints is given below:

Consider a mobile manipulator at a configuration q, with the user specified velocity vH, equality constraint, and inequality constraint. Choose the robot motion input u as provided above, with x solved from the following quadratic programming problem:

subject to

∂hIi∂q⁢Jq⁡(q)⁢(E⊥⁢ξ+E+⁢β)≥σi,i=1,…⁢,ni
Where C is given by the equation above, λ1 and λ2 are chosen by weighting factors,

JD=[JHEC]⊥.
And σiis given by equation above.

Note that the minimization problem is strictly convex as the matrix

[JH⁢E⊥λ1⁢CE⊥λ2⁢JDT⁢E⊥]
is of full column rank. Therefore a unique solution always exists and may be efficiently found by using the gradient descent type of methods.

The following describes the equality and inequality constraints imposed on the system. We consider the two arms of the robot separately. Without loss of generality, consider only the left arm of the system. The kinematics as provided above given with uϵR9(7 for the arm and 2 for the motorized base), and qϵR10. We focus only on user commanding the translation of the end-effector. Hence,
JH=JTp
the translational portion of JT. The choice of the inequality and equality constraints is next described below.

1) Arm Workspace: There are two types of workspace constraints: joint limits and end-effector boundary constraint. The ranges of the joint angles are specified as follows (inequalities are interpreted component-wise):

The values are provided by the manufacturer, except for q4minwhich is to be specified based on the distance between the current position of the motorized base to the target. When the motorized base is far from the target, the arm should bend close to the body while the motorized base moves forward (to avoid bumping into nearby objects). When the motorized base is close to the target, the arm can stretch to reach to forward. Thus, we let q4min=π/2 (arm bent upward) when the distance between the motorized base12and target is larger than a threshold value (e.g., 1 m), and let q4min=0 (arm outstretched) when the distance is smaller than that threshold. The target location may either be specified by the user as the robot gets close, or detected automatically by the onboard cameras. To avoid self collision, we specify the coordinate of the end effector in x-direction,
exTp0T
to be larger than 0.3 m. This is particularly important when the motorized base is backing up. Combining the above, we have 15 inequality constraints:

hI=[qL-qLminqLmax-qLexT⁢pOT-0.3].
The corresponding Jacobian is

Jt=[I707⨯2-I707⨯2exT⁢JTLp07⨯2]
where
JTLp
is the translational portion of the left arm end effector Jacobian, JTL, I7is the 7×7 identity matrix. The parameters in the σ function are hand-tuned to be η=ε=0.15 rad for the joint limit constraints, η=0.1 m, ε=0.15 m for the self collision constraint, c is set to 0.45 in all cases. In all cases except for q4, the constraints are always satisfied, so we set eito a small number 0.01. For q4, the constraint region switches (due to q4min), so the corresponding eiis chosen larger to be 0.1.

2) End-effector Orientation Control: To maintain the endeffector orientation at a specified configuration, we use a proportional feedback control kinematic control law:
ωT=−KTqv(R0TdesTR0T),
where
R0TdesandR0T
are the desired and actual end-effector orientation matrix, KTis the feedback gain and qvis the vector quaternion. Thus, the first set of equality constraint is given with the corresponding Jacobian
JE=[JTLR03×2]
where
JTLR
is the rotational portion of the endeffector Jacobian JTL.

3) Motorized base12Movement Constraint: As described, we impose an additional equality constraint to ensure the base follows the user commanded motion when the arm reaches the workspace boundary. In the component form,
(−vH(y)p0T(y)−vH(x)p0T(x))ωB+vH(y)vB=0
with the corresponding constraint Jacobian given by:
JEB=[01×7−vH(y)p0T(y)−vH(x)p0T(x)vH(y)].

4) Obstacle Avoidance: Obstacles in the workspace may be detected by appropriate sensors, e.g., ultrasound range sensor, time-of-flight sensor, cameras, etc. In our current implementation, we use four IR sensors placed at the front side of the motorized base12to measure the distance of the obstacle ahead, di, i=1, . . . , 4. Suppose the minimum distance threshold for the sensors are dimin, i=1, . . . , 4. Then we have the inequality constraints
hI=[d1−d1mind2−d2mind3−d3mind4−d4min]T.
The corresponding Jacobian is
JI=[04×8[−1,−1,−1,−1]T]
since
di=−vB
(forward velocity decreases the distance to obstacle). In order the forward moving velocity to decrease when the motorized base12is close to the obstacle, we set a relatively small value of c and large value of ε in σ, e.g., c=0.5, ε=1.2 m, so that the speed reduction starts at 1.2 m away from the obstacle. The size of the buffer zone is set to η=0.1 m, so the motorized base12starts to decelerate as far as 1.3 m from obstacle. Since we expect the inequality constraints always to be satisfied, eiis chosen as a small number, 0.01.

Referring now toFIG. 11, an illustrative control platform is shown in which laptop26is provided with a robot manager58that includes: (1) a set of communication interfaces60for communication with various systems, and (2) a control system62for interpreting commands and controlling the robotic assistant10. In this example, robot manager58is adapted to receive remote interface commands64(e.g., from a Jamboxx) or Internet based commands66(e.g., from a health care provider). Whenever a command is received, it is interpreted by the control system62to move or control aspects of the robotic assistant10. As shown, the robot manager58is capable of sending and receiving command signals to the motorized base12, dual arm robot16, sensors68or other devices70. As noted, the motorized base12includes its own control platform having software for causing the motorized base12to move/respond according to received commands. The dual arm robot16likewise has its own unique control platform and software for causing its arms to move/respond. Because the control platforms for the two are distinct, each is controlled with unique input commands. In addition, sensors68and other devices70may also be controlled by the control system62. The control system62includes a control algorithm63, described above, for managing movements of the motorized base12and dual arm robot16, based on inputs, constraints, etc.

FIG. 12depicts a generalized flow diagram showing a process for controlling the robotic assistant10. At S1, the operator uses the input controller (e.g., Jamboxx) to select a frame of the robotic assistant to control. In one typical example, the operator would select an end effector, which is located proximate the grippers at the end of an arm. However, the operator may select other points, e.g., a wrist, shoulder, etc. Next at S2, the operator uses the input controller to command the selected frame to move within a three dimensional space. For example, the operator can use the Jamboxx to command the end effector to move towards an object on a table.

The process next determines at S3if the selected arm is within the workspace, i.e., can the end effector reach a desired point without the need to move the motorized base. If no, the motorized base is moved within the workspace at S3, without yet moving the arm. Once the arm is in the workspace, a determination is made at S5whether there are any constraints. For example, is there an object blocking the path, etc. If no, then the arm is independently moved as commanded by operator at S6. If yes, then the arm movement is limited or altered based on the constraint.

An illustrative computing system may comprise any type of computing device and, and for example includes at least one processor (e.g., CPU54ofFIG. 11), memory52, an input/output (I/O)56(e.g., one or more I/O interfaces and/or devices), and a communications pathway. In general, processor(s) execute program code which is at least partially fixed in memory52. While executing program code, processor(s) can process data, which can result in reading and/or writing transformed data from/to memory and/or I/O for further processing. The pathway provides a communications link between each of the components in computing system. I/O56can comprise one or more human I/O devices, which enable a user to interact with the computing system.

Furthermore, it is understood that the described algorithms or relevant components thereof (such as an API component) may also be automatically or semi-automatically deployed into a computer system by sending the components to a central server or a group of central servers. The components are then downloaded into a target computer that will execute the components. The components are then either detached to a directory or loaded into a directory that executes a program that detaches the components into a directory. Another alternative is to send the components directly to a directory on a client computer hard drive. When there are proxy servers, the process will, select the proxy server code, determine on which computers to place the proxy servers' code, transmit the proxy server code, then install the proxy server code on the proxy computer. The components will be transmitted to the proxy server and then it will be stored on the proxy server.