Abstract:
A shared system for robot control including integration of the human and autonomous input modalities for an improved control. 
     Autonomously planned motion trajectories are modified by a teleoperator to track unmodelled target motions, while nominal teleoperator motions are modified through compliance to accommodate geometric errors autonomously in the latter. A hierarchical shared system intelligently shares control over a remote robot between the autonomous and teleoperative portions of an overall control system. Architecture is hierarchical, and consists of two levels. The top level represents the task level, while the bottom, the execution level. 
     In space applications, the performance of pure teleoperation systems depend significantly on the communication time delays between the local and the remote sites. Selection/mixing matrices are provided with entries which relfect how each input&#39;s signals modality is weighted. The shared control minimizes the detrimental effects caused by these time delays between earth and space.

Description:
Origin of the Invention 
     The invention described herein was made in the performance of work under a NASA contract, and is subject to the provisions of Public Law 96-517 (35 USC 202) in which the Contractor has elected not to retain title. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The field of this invention is a robotic control system which shares autonomous and hand-controller commands. 
     2. Description of the Prior Art 
     Robotic manipulation research is still in its infancy, and present day manipulation systems (teleoperator or autonomous) suffer from many weaknesses. 
     Consider, for example, the teleoperator systems that are currently used in space applications. Because of the time delay in transmission of signals, terrestrial teleoperators must be predictive in sending commands to the robot. One solution is to employ teleoperation in space. This solution, however, poses many problems in cost and practicality. Moreover, if teleoperation were to be done in space (from a shuttle, for example) the time delay problem would be overcome; but then mission time would be restricted according to the study reported by Montemerlop, Merlin D., The Space Perspective: Man-Machine Redundancy in Remote Manipulator Systems, Keynote Speech, NATO Advanced Research Workshop on Robots with Redundancy: Design, Sensing &amp; Control, June 27-July 1, 1988, Salo, Lago di Garda, Italy. 
     With existing technologies, autonomous systems are incapable of accommodating large un-modelled variations. Moreover, since many problems in space applications are often detected, diagnosed and solved through human on-the-spot initiative, the presence of a human in the active loop becomes imperative. See Montemerlop, Merlin D. supra. We, therefore suggest (as many others have done) that the development of systems under shared control is of paramount importance in our attempt to automate space applications. 
     It is our observation that much of the research in shared systems, lacks a strong theoretical flavor. Effort has been directed towards building systems and implementing shared control at the servo-level. These systems consider the teleoperator as the master, record her actions in a teleoperative device, and then directly transform them into robot commands, which finally get executed under autonomous control. See, for example, Sheridan, T. B., Telerobotics&#39;, Workshop on Shared Autonomous &amp; Teleoperated Manipulator Control, 1988 IEEE International Conference on Robotics &amp; Automation, Apr. 24-29, Philadelphia, PA. 
     The Sheridan article provides a splendid historical perspective about research in telerobotics, and many shortcomings in the scenarios described above come to light in that article. Most importantly of these shortcomings, is the absence of active human intelligence that should be involved during an integration of autonomous and teleoperator inputs. By this we mean that a theory of how to intelligently share control, prior to this invention, has not yet been firmly established. The lack of an effective shared control places the responsibility of maintaining stability squarely on the shoulders of the teleoperator. This responsibility is fine for terrestrial experiments (in nuclear power plants or for prosthetics), but not for space applications. 
     There exist classes of applications for which attempting to build a robotic system that is either purely teleoperative or completely autonomous may be self defeating. Such self-defeating applications include, for example, automating small batch jobs in space involving satellite maintenance and repair. The motivation of this invention is to exploit the power of each system and integrate them into a single man-machine system. Such shared systems will have the capability of accepting commands from a high level planner and/or a teleoperator, and appropriately mixing them. 
     Other shortcomings exhibited by our present teleoperator systems are as follows. The teleoperator, often requires explicit knowledge of the relationship between her actions and that of the robot. And, in general, it is very tiresome for the operator to impart fine motions to the robot. 
     A search of the prior art was done for the purpose of evaluating this invention. The results of that search are as follows: 
     Daggett el al., U.S. Pat. No. 4,763,05 
     Takita et al., U.S. Pat. No. 4,582,026 
     Resnick, U.S. Pat. No. 4,432,063 
     Guittet el al., U.S. Pat. No. 4,510,574 
     Maruo et al., U.S. Pat. No. 4,737,697 
     Inoue, U.S. Pat. No. 4,641,251 
     Inoue discloses a divided control system in which a main control 5 is in parallel with subcontrol units, such as units 6. Such subcontrol units are used to immediately stop the robot when an unexpected obstacle is encountered. The robot is provided with a number of such subcontrol units for individually controlling each operating portion so that a high response speed is achieved if an unexpected obstacle is encountered. See Col 2, lines 40 through 56 and FIG. 2. 
     Guittet et al discloses proportioning of force control between a master actuator and a slave actuator by each transmitting to a control device of the other a position and/or speed signal with a transmission delay. The transmission delays are compared and a sum of the delay times is used in a first force control loop that is supplemented by a second control loop. The system&#39;s second control loop includes a second force signal that is related to the delay times being sensed. See FIG. 6 and Column 10, lines 39 through 58 where it is described as a desire of the patent to use the most appropriate configuration of the invention, based upon the delay time summation. 
     Maruo et al is typical of those types of systems that employ a teaching mode and a playback mode. During the teaching mode a servo is disabled and then is enabled again in the playback mode. See the abstract. 
     Takita et al discloses an anticipatory control system in which each point of operation can be changed over individually on the basis of a driving control instruction for a sub-loop controller associated with each operating point. See FIG. 2 for the master controller 10 and the subloop controllers such as 11a, 11b, etc. 
     The Daggett et al system is typical of those that operate on a cascaded control approach in which processors are individually assigned data processing and calculation tasks. These individual processors respond to extended control and basic control as shown in FIGS. 4 and 5 and as described at Column 8, lines 37 through 48. 
     Resnick is of interest for its disclosure of robot control over nonprogrammed and programmed points. In summary then, none of these references are deemed of significant relevance to this invention. 
     We conclude this background section by putting the above-described state of the art in context with some other basic background art. In an article by Sheridan, T. B., entitled &#34;Telerobotics&#34;, Workshop on Shared Autonomous &amp; Teleoperated Manipulator Control, 1988 IEEE International Conference on Robotics &amp; Automation, Apr. 24-29, Philadelphia, PA. four major areas are specified as requiring research that must be done in telerobotics. The four areas include: (i) telesensing, (ii) teleactuation, (iii) computer-aiding in control and, (iv) meta analysis of task interactions. This invention focuses on some important aspects of areas (ii) and (iv) and additionally shows how these aspects relate to each other. Such aspects are also reflected in a somewhat similar hybrid position/force task representation, used for low-level tasks, as described in an article by De Schutter J., Van Brussel H., entitled &#34;Compliant Robot Motion, I.A. Formalism for Specifying Compliant Tasks&#34;, International Journal of Robotics Research, Vol. 7, No. 4, August 1988. 
     SUMMARY OF THE INVENTION 
     An important issue that must be addressed in the development of shared systems is the actual integration of the human and autonomous input modalities as first taught, described and claimed in this our invention. Two approaches may be taken towards this; either the nominal autonomous behavior may be modified by a teleoperator, or nominal teleoperator behavior may be modified autonomously. For example, autonomously planned motion trajectories will be modified by a teleoperator to track unmodelled target motions in the former, while nominal teleoperator motions will be modified through compliance to accommodate geometric errors autonomously in the latter. 
     Ideally, one would like teleoperators to be terrestrial (based on ground) and operate robots located at remote sites with full confidence that execution will be accurate and reliable. At the execution level, these two desirable features can be obtained through a systematic design of controllers. At the task level, one way of achieving such features is to follow an integration philosophy that results in a shared system which has all the positive features of pure teleoperation and pure autonomy, and none of the negative features. We submit that our invention contributes a major step towards this &#34;ideal&#34; goal, and it lies in mixing the teleoperator and autonomous inputs in an advantageous, new and novel manner. 
     More specifically, this invention relates to a hierarchical shared system, and describes and claims a novel approach for intelligently sharing control over a remote robot between the autonomous and teleoperative control systems. In this patent application, we present a shared control architecture compatible with both approaches and discuss in detail some of the implementation issues. 
     The architecture that we present in this application is hierarchical, and consists of two levels. The top level represents the task level, while the bottom, the execution level. Inputs to our architecture are formulated in a task coordinate system and consist of the following: (1a) a specification of a task coordinate system and (1b) task trajectories, and (2) the teleoperator trajectory. A hybrid position/force task representation is used for low-level tasks. Therefore, the task coordinate system consists of orthogonal motion and force coordinates, task and teleoperator trajectories, of appropriate motion and force trajectories. Teleoperator signals are transformed from its local coordinate system (in our architecture, this is called the hand controller system) to the task coordinate system. 
     In space applications, the performance of pure teleoperation systems depend significantly on the communication time delays between the local and the remote sites. The philosophy behind any sharing of control must therefore be based on minimizing the detrimental effects caused by these time delays. 
     Our inventive approach is as follows. At the task level, should the communication delay be significant, we allow integration only along motion directions, while forces are controlled autonomously. A significant first step in the integration process, in accordance with our invention, consists therefore of deciding what signals are to be mixed, followed by the decision of when such signals may mix. Another important step is to decide how such signals may be mixed. These steps are done with the help of selection and mixing matrices. Entries in such matrices reflect how each input&#39;s signals modality is weighted. 
     At the execution level, the system is oblivious to the nature of the desired trajectory, and it may come directly from the autonomous part, teleoperator part, or the desired trajectory may be a mixture of the two. The servo controller is simply capable of tracking motion and/or force trajectories in a stable fashion. For the hybrid task representation of our invention, we choose a hybrid force/motion control architecture for servo-control. Also it should be noted that the flexibility of our invention accommodates different task representations that may result in different control architectures. 
    
    
     BRIEF DESCRIPTION OF THE DRAWING 
     FIG. 1 is a figure depicting a hierarchical system in accordance with the invention. 
     FIG. 2 is a figure depicting a shared system architecture in accordance with the invention. 
     FIG. 3 is a figure depicting a shared space system architecture in accordance with the invention. 
     FIG. 4 is a figure depicting a remote site task sharing in accordance with the invention. 
     FIG. 5 is a figure depicting a task controller in accordance with the invention. 
     FIG. 6 is a figure depicting a teleop-task transformation in accordance with the invention. 
     FIG. 7 is a figure depicting a pure hybrid control in accordance with the invention. 
     FIG. 8 is a figure depicting a traded and shared control implementation architecture in accordance with the invention. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT 
     A novel robotic control system comprises two levels of control circuitry whereby the system will accept and execute autonomous or hand-controller commands, sharing control between them such that shortcomings and limitations of one mode of control may be overcome by the other. These two levels are represented as a task level and an execution level. 
     At the task level, should the communication time delay between local and remote sites (experienced in space application) be significant, integration is used only along motion directions, while forces are controlled autonomously. The first step in integration consists of decision logic that determines when the control signals may mix and the second step is deciding how the signals may mix. 
     A novel application of a mixing matrix is employed in which input modality is weighted. 
     The invention will now be described in more detail in the following sections. 
     I. SHARED CONTROL 
     Consider a hierarchical robot system shown in FIG. 1. In a shared system, any level i must be capable of accepting commands from both autonomous and teleoperative sources, and must allow this acceptance at multiple levels. If inputs from both teleoperation 100 as well as the autonomous system 150 arrive at level i, as shown in FIG. 1, then all levels from 0 to i come under shared control of shared system 175. 
     The system approach for this invention is depicted in FIG. 2. It consists only of two levels, namely level 0 and level 1 shown on opposed sides of the dashed line 201. Level 0 is the servo control level, and accepts tool motion/force commands and runs them on the robot. Level 1 generates the motion/force commands. 
     Sharing occurs at both levels, in general. At the task level, sharing occurs in the task-level sharing circuit 215, while at the servo-level it occurs in servo-level sharing circuit 225. 
     We draw upon the ongoing research in the development of hierarchical autonomous robot systems to describe some of the details of the architecture shown in FIG. 2. At level 1 robotic task-level commands are generated, and we therefore denote that level as the task level. Level 0 executes the task and so it is denoted the execution level. 
     In this patent application, the word task is used in a local sense and denotes the generation of two pieces of information, (i) a task coordinate system, and (ii) desired trajectories of the task coordinates. To integrate teleoperator inputs with its autonomous counterpart, these inputs must be compatible, and so, if necessary, primitive arm/hand teleoperator actions must be transformed into an appropriate task coordinate system. 
     The task coordinate system is dependent on the representation of tasks. Our task representation framework is similar to that described by: De Schutter J., Van Brussel H., Compliant Robot Motion, A Formalism for Specifying Compliant Tasks, International Journal of Robotics Research, Vol. 7, No. 4, August 1988. The task coordinate system consists of orthogonal motion and force coordinates. The origin of this system is generally located at the arm endpoint (location of the contact) in the absence (presence) of contact. We assume that desired autonomous motion/force trajectories are readily available to us. 
     At the execution level, we use a hybrid force/motion control scheme disclosed by: Raibert, M. H., &amp; Craig, J. J., Hybrid Position/Force Control of Manipulators, Journal of Dynamic Systems, Measurement &amp; Control, 102 (June 1981), 126-133. 
     For space applications, the proposed hierarchy (in FIG. 2) needs additional structuring. This is due to the fact that some of the planning/control/sensing actions are done locally on the earth (or in space but physically far away from the location of the robot system), while others are done remotely by the robot system itself. In such a scenario, explicit sharing at the servo level may not be practical because of the time delays in communication. 
     FIG. 3 shows the hierarchy in FIG. 2 with some modifications. The servo-level sharing circuitry 225 of FIG. 2 has been removed completely. In addition, we show that both task and servo-level operations have now been distributed over the remote and local sites. At the local site, a planner develops a series of task-level autonomous commands. These locally-initiated commands are transmitted over line 310 to the robot system located at a remote site, where such commands are received by a task-level sharing circuit 315. 
     Meanwhile, the teleoperator 325, acquires information about (i) robot motions through TV displays, and (ii) the contact and inertial forces through force reflectance in any well known manner (not shown in FIG. 3). She compares the acquired information with her perception of the task, and generates teleoperative inputs 330 to perform the appropriate corrections. The teleoperative inputs are applied over line 331 to transformation circuitry 335. Transformation circuitry 335, in turn, sends these commands over line 333 to the remote location. 
     The actual transformation of teleoperative inputs at line 331 into the task coordinate system happens partly at the local site (circuitry 331, 335) and partly at the remote site in transformation circuitry 350. At the local site, primitive device specific actions are first converted at 335 to some local coordinate system by a hand controller, or other suitable conversion device (not shown). Then, those converted signals are transformed to a coordinate system attached to a reference location such as the base of the robot (the reference location, of course, will be known from past information) and then such coordinate information is transmitted to the remote site over line 333. 
     At the remote site, the inputs are transformed from the base of the robot to the task coordinate system by the remote transformation circuitry 350. Transformations from base to task frame is done at the remote site because the robot&#39;s joint angle information will be the most up-to-date at that location. Task-level sharing occurs completely at the remote site, as does the servo-control by servo system 365 for robot 375. 
     The architecture presented in FIG. 3 is implemented in this invention and is described in more detail with reference to FIGS. 4, 6, and 7. Such description will be given after a discussion of notation to be used in the description. 
     The notation used in this description is as follows: 
     I=Input 
     O=Output 
     The first subscript of I or O indicates the hierarchical level. The second subscript indicates the source of the signal (T for teleoperator, A for autonomous and S for shared). Therefore, 
     I ij , for i=1, . . . , n &amp; j=T,A,S 
     O ij , for i=1, . . . , n &amp; j=T,A,S 
     are the generalized expressions for the various inputs and outputs. In addition, sharing is represented by C and transformations are represented by T. With these understood, we define the following: ##EQU1## 
     Task-level sharing, (C 1 ), is discussed in section II which follows. In section III we discuss the transformation ##EQU2## of teleoperator commands to the task frame, and present the necessary control architectures for hybrid position/force control 
     II. TASK LEVEL 
     We now describe the task level of our shared control architecture in greater detail. The issue that needs to be addressed at this level is how the teleoperator and autonomous inputs (both expressed in the task coordinate system) will be combined. That is, how do we develop C 1  (the signals issuing from task-level sharing 315 in FIG. 3)? 
     The details of circuit 315 are shown in FIG. 4 where the FIG. 3 input/output connections from FIG. 3 are repeated. Note in FIG. 3 that the autonomous input 310 is from the local site location to the sharing circuit 315 and the two return signals are a modified motion/force feedback 311 and an autonomous feedback 312. Those leads are also shown in FIG. 4 for ease of comparing the two figures. 
     In FIG. 4, in the lower left-hand portion of the circuitry, the teleop input and the autonomous input terms are weighted by a series of matrices 465 through 468 and 470 through 473. The weighted values are summed in summation circuits 475 and 480 and a modified motion/force signal, as a result of such weighing and summing, is developed on output lead 410 for application to the servo control 365. The manner of modification is a significant feature of this invention and requires further explanation after digressing briefly for some background information. 
     The most important factor that affects the development of C is the transmission delay that exists between a teleoperator and the robot system itself. For terrestrial teleoperation (ground based operator) this delay time is of the order of a few seconds, and implies that modifications on the nominal trajectory will occur with a significant delay. In addition, the information obtained by the operator about the state of the robot will be obsolete, and so, her modifications are at best outdated, or may even be downright invalid. Sharing at the task level must therefore minimize these detrimental effects. 
     In general, robots may operate in one of three modes. They may either be moving freely, or just about to establish contact, or else, may have already established stable contact. In the first case, although the transmission delay causes the robot to deviate away from its desired path, and the obsolete feedback information results in errors in the specification of the desired path, the system&#39;s stability is never affected. However, in the third case, and very often in the second case (if the contact surface is very close), instabilities are generated in the system from both transmission delay as well as obsolete feedback information. 
     One way to reduce the above-mentioned undesirable features in a shared system is to allow nominal task trajectories generated by an autonomous system to be affected by the teleoperator only in those directions where contact is not established, and perform force control autonomously. At the task level, our sharing strategy can be summarized as follows. Let the motions and forces allowable by the contact type (See the notation developed by Salisbury, J. K., Kinematic &amp; force Analysis of Articulated Hands, Ph.D thesis, Department of Mechanical Engineering, Stanford University, 1982) be represented by 6×1 vectors M and F. Let M (i) represent the i th  row of M. Then: ##EQU3## 
     Similarly, F(i) is 0 or 1 depending on whether force freedom in that direction exists or not. For a hybrid task representation: 
     
         F.sup.T M=0 
    
     Also, let .sup.α M and .sup.α F represent desired motion and force trajectories expressed in the task coordinate system be represented by ##EQU4## and the corresponding Jacobian, ##EQU5## 
     A. Feedforward Path: 
     In the feedforward path, the input and output signals at the task level, FIG. 4, will consist of the following components: ##EQU6## where←indicates what information is contained in the command. 
     The mapping of I A  and I T  onto 0 S  occurs as follows. Let π M  be a 6×6 matrix 465, 466 (and π  M  (i,i) the element at its i th  row and column). For the degrees of motion freedom the π matrices in FIG. 4 are derived as follows: ##EQU7## π F  for force matrix elements 470, 471 can be constructed similarly. ##EQU8## 
     Thus the π matrices reflect the effect of task representation on sharing. Through this novel approach, we are able to specify what is being mixed along a particular direction. The question of how the actual sharing occurs is resolved through the weighing matrices Ω shown as elements 467, 468 (motion) and 472, 473 (force) in FIG. 4. The Ω matrices are determined as follows (Ω(i,i)) is the element located at the i th  row and column. 
     Now in M and F, some directions may be chosen for operation under pure teleoperation or pure autonomy. Let these be denoted by 6×1 vectors G T  and G A . Therefore, (G T  (i) (G A  (i)) is equal to 1 if and only if pure teleoperation (autonomous control) is intended in the direction i. With these, we define the elements of the Ω matrices as follows: ##EQU9## where, W denotes the weight entries in the matrices. Note that the Ω matrices will, in general, be diagonal. 
     With these stated conditions, sharing in the feedforward path occurs as given below: ##EQU10## Equations (1) and (2) completely determine C in the feedforward direction with summing taking place in summation circuits 475 and 480. 
     Note that pure autonomy and pure teleoperation can be effected within the same architecture by setting W equal to 1 and W equal to 0, respectively. 
     B. Feedback Path 
     In the feedback path of FIG. 4, actual robot motions/forces are input into the task level sharing circuitry at 410 from servo 365 in FIG. 3. This information is used to generate motion/force feedback applied by lead 312 to an autonomous task planner, as well as force reflective feedback 332 to the teleoperator. We now explain each of the feedback signals in FIG. 4 in more detail. 
     In FIG. 3, we show a signal denoted modified M/F at line 311, which signal is fed back to an autonomous planner at level 2 (See FIG. 1). The signal represents the shared desired motion/force trajectory 0 1s . The reason for this feedback is given in the following discussion. 
     The autonomous planner expects the system to track only the autonomous part I 1A . This expectation would result in an incorrect reasoning (upon completion of the task) by the planner. In short, the autonomous planner does not know that the actual signals that are controlling the robot have been a shared command. 
     What is really being tracked, of course, is not solely an autonomous input command. Note that the autonomous planner would conclude that the task execution has failed unless a mechanism for dealing with the sharing command is provided. One way to rectify the above-described incorrect conclusion is to modify the task specification of the autonomous task planner from I 1A  to 0 1s . Note that under pure teleoperation, the task planner specifications are derived from the teleoperator completely. 
     Note also that at level 2 of FIG. 1, there is an autonomous task controller 500, FIG. 5, into which the autonomous feedback 510 and modified M/F signals 520 are fed back. Based on these items of feedback information, and depending on the next task plan that it receives from a high level task planner, the task controller 500 generates appropriate autonomous inputs for the next task. These inputs are applied at line 530 by task controller 500 as is shown in FIG. 5. 
     The actual robot motions/forces may be generated at the servo level in a coordinate system different from the task coordinate system. For example, the robot motions may be expressed in terms of the robot&#39;s joint space in any well known manner. Therefore, in general, the feedback information needs to be transformed to the task coordinate system. 
     These required transformations are performed by ##EQU11## (element 415) and ##EQU12## (element 420) respectively in FIG. 4. 
     Note that the shared feedback signals appear on line 410 and are applied to the transformation circuits 415 and 420, respectively. These transformation circuits, as described above, convert the incoming joint space information to the task coordinate system&#39;s notation. The π matrices for motion, elements 425 and 427, connected to the output of circuit 415, are identical in the feedback path while the weighing matrices 435, 437 that are receiving the output terms are different. 
     In a similar manner, transformation circuit 420 applies its output to the π matrices 426 and 428 for force weighing. Those matrices, in turn, are connected to weighing matrices 434 and 436. 
     We now develope the mathematical relationship for signal weighing by the matrices 435, 437 and 434, 436. These mathematical relationships are as follows: ##EQU13## where .sub.λi reflects the scaling factor between the actual forces generated and the force that the teleoperator is intended to feel. Note that this is how we, in accordance with our invention, have force reflectance of the actual forces to the operator. This force reflectance may be openloop, where the operator just feels the contact forces, or closed-loop where the operator can effect changes in the desired contact force through teleoperation. The latter case is allowed only when the time delays are insignificant. 
     Generally speaking, the term .sup.Λ.sbsp.T M is actually very complex. We have provided a connection between the matrix 437 and the input to summing junction 475 in FIG. 4. This connection accomplishes a multifold purpose for the term .sup.Λ.sbsp.T M. 
     The purpose for the .sup.Λ.sbsp.T M term is twofold. First, it must enable the teleoperator to experience the inertia of the robot system at the remote site. Through this the operator gets a sense of how the robot is behaving. In addition, it enables the operator to adapt to the inertial characteristics of the robot system, so that the operator learns how best to teleoperate the robot 375. For example, in the presence of contacts along directions of contact forces, we simply reflect the actual forces back to the operator (with a scaling factor), as explained above. Along directions of motions, however, we obtain the difference between the desired and actual teleoperator trajectories, and use this difference information to inform the operator about the inertial characteristics of the robot. 
     We explain the actual methodology behind the above-described virtual force reflectance in the next subsection. Suffice it to say at this point, however, that at the task level, in order to obtain the existing errors in teleoperation, we must extract out of the actual motions under shared control, the contributions due to teleoperation. 
     Although the shared input signal 0 S  at lead 410 is a linear combination of autonomous and teleoperator trajectories, the non-linearities in the robot system itself (and possibly also in its controllers, if non-linear controllers are employed) makes is very difficult to perform this extraction. We therefore restrict ourselves in a mathematical development to a simple case in this patent application. The general case is valid, however, and is within the scope of the appended claims, and is covered in our invention. 
     Let us develope for the simple case; 
     
         (G.sub.A).sup.t G.sub.T =0 
    
     
         G.sub.A U G.sub.T =S.sub.M 
    
     where, U indicates a union operation and S M  represents the space of motions allowable by M. In this case, teleoperative and autonomous motions are in the orthogonal directions, and so, .sup.Λ.sbsp.T M will be: ##EQU14## 
     Note that the actual robot trajectory under shared control is then multiplied by the appropriate A matrix, and then subtracted from the desired to obtain the difference. The difference is then shipped out to the local site. Output signals from the A matrices are six element vectors. For example, if the motion output information is expressed in three digits the first three places are motion information followed by three zeroes in the last three digit places. For the force side the opposite condition is true, with the last three places containing force information while the first three digit places are filled with zeroes. A combined six digit signal is then developed at the autonomous feedback line 312. 
     To summarize, in this subsection, we have described in detail how teleoperative and autonomous inputs will be integrated at the task level. 
     III. SERVO LEVEL 
     At the servo level, three functions have to be performed in the feedforward path. The first is the actual execution of the commands issued from the task level. This function will be accomplished by any standard servo operation through equipment that is well known and located entirely at the remote site. The second function is to take the raw teleoperator input signals and convert such signals to a coordinate system attached to the device itself. This function is also accomplished by equipment that is well known and is located entirely at the local site. In connection with this second function, device-specific commands are shipped out to the robot system from the remote site. The third function is to perform the transformation of feedback information at the remote site to the appropriate task level coordinate system. 
     In the feedback path, the functions are almost identical. The only exception is that at the local site, in addition to performing the transformations between device coordinate system and the actual input device, there must also be a servo-loop in the force reflectance path. 
     We will now explain the above-mentioned features in more detail by reference to FIG. 6 which shows both the feedforward and feedback paths from and to the teleoperator. It should be recalled from FIG. 3 that an operator 325 will supply a teleoperator input to a hand controller. Such a hand controller, shown in FIG. 6 as hand controller 625, is connected in standard fashion to a servo system of any known type. 
     The servo 626 will convert the teleop inputs into a signal format that is acceptable by transformation circuitry 335. Such transformation circuitry includes a pair of transformation circuits 640 and 645 (local site), and another pair of transformation circuits 650, 655 (remote site) each of which are devoted to motion (T) and to force () respectively in accordance with our invention. 
     As one reviews the circuitry of FIG. 6, it is readily apparent that the left-hand side and the right-hand side of the figure are images of each other with the outbound or feedforward part on the left and the inbound or feedback part on the right. Description of one side is mostly self explanatory of the reverse direction that occurs in the other side. The transformation of the parameters at T and at  are described in the subsections hereinafter. 
     A. Determination of T 
     The transformation between task-level and the teleoperation-level occur in the boxes denoted transform 335 (local site) and transform 350 (remote site) as shown in FIG. 3. Once C has been determined at the task level, determination of this transformation is straightforward. 
     Transformations of the teleoperator motions to the task coordinate system are required as long as at least one degree of motion freedom exists. Therefore, the matrix ##EQU15## is required. This will in general vary with the locations of the task frame, and so requires information from higher level planning system for its construction. 
     In addition, if teleoperation is allowed along force direction, then the jacobian ##EQU16## would also be required. Similar arguments can be used for the feedback path. Thus in FIG. 6, we see that the kinematic transformations ##EQU17## in the feedforward path are done at the local and remote sites respectively. Similarly for ##EQU18## in the feedback path. The forces are transformed through the corresponding Jacobians, ##EQU19## in the feedforward path, and ##EQU20## in the feedback path. 
     In general, the teleoperator device itself has a very small motion limit. Therefore, each teleoperator input indicates the incremental change necessary in the autonomous trajectory. The rate at which a local processor within the hand controller samples the raw device signal and converts it to hand controller specific motions may, in general, be an order of magnitude faster than the rate at the task-level. 
     The above implies that the small incremental changes specified by a teleoperator will have to be accumulated and stored in order to generate the teleoperative commands at the task level. In our implementations (although not explicitly depicted in any figure) this is assumed to be done at the remote site. 
     B. Teleoperation unit 
     The transformation of motions of the input devices in the teleoperation hardware to small incremental motions in a teleoperator device specific cartesian coordinate system, occurs in the feedforward path in servo 626 in FIG. 6. The box denoted hand controller 625 represents the teleoperation hardware. The hardware not only contains encoders to read the motions of the input device, but also motors that can be actuated. 
     In the feedback path, the servo box 686 drives these motors against the operator to provide her with force reflectance in a manner well known in this art. Such force reflection can occur in two cases: (1) when there is negligible transmission time delay between the local and the remote sites. In this case actual forces from the remote site can be fed back to the operator and thus the control loop is effectively closed through the operator. And (2) when there is enough transmission time delay, only the motion subspace is directly controlled by the operator. In this latter case the motion errors in teleoperation, supplied from the task-level, are first multiplied by a stiffness matrix to generate virtual inertial forces, and then added to the actual contact forces. 
     The actuators drive the operator back (or forth) in proportion to these signals. If the operator reacts by moving the input device back or forth, she feels the effective inertia of the remote manipulator. Thus the force reflectance loop is closed only around the robot system located at the remote site. 
     C. Control Architecture 
     The output from the task level to the servo-level is made up of desired motions and/or forces as is depicted in FIG. 4. These motion and/or force commands must be actually executed on the robot. In the absence of contacts, the controller must track position trajectories. In the presence of contacts, there are certain directions along which motion trajectories are specified, while along the others, force trajectories are specified. The fundamental assumption in a hybrid task representation is that frictional effects are negligible. Under such conditions, a pure hybrid control strategy can be used for execution of the commands. 
     FIG. 7 shows a hybrid position/force control architecture, in accordance with that described by Raibert, M. H., &amp; Craig, J. J., Hybrid Position/Force Control of Manipulators, Journal of Dynamic Systems, Measurement &amp; Control, 102 (June 1981), 126-133. In FIG. 7, errors in position are compensated for by the controller 710 denoted C X , while force errors are compensated for through a force compensator 720 denoted C F . The outputs of both C x  and C F  represent actual motor current signals. These are directly added by an adder 725 and supplied to the robot 375. 
     C X  and C F  are position and force compensators respectively. These may be of any conventional design. For example, if the system is linearized, these could be directly obtained through pole placement, or designs to adapt to small geometric errors in the contact, or designed to minimize the flow of energy at the contact and so on. Such designs are well known in this art and are not believed to require any further description. 
     During implementation, we must bear in mind that each trajectory specified from the task level must be followed. One way to emulate this is to take the force/motion trajectory specified in 0 1 .sbsb.5 (denoted shared output in FIG. 4), and perform interpolations by slicing it into small incremental regions consisting of ramps or steps. Each incremental δ α specifies a small change in the state of the robot so that when accumulated over all the increments, changes specifies by α are achieved. And, each δ α will be added on to the previous alpha values to generate the present F d  and X d  signals present at 750 and 775 of FIG. 7. That is, for k=1 to n, where n is the total number of interpolations required: 
     
         X.sub.d (K+1)=X.sub.d (K)+δα.sub.S.sbsb.M 
    
     
         F.sub.d (K+1)=F.sub.d (K)+δα.sub.S.sbsb.M 
    
     And, X d  (0) and F d  (0) are known before starting the task. 
     In this subsection, we have explained in detail (i) how teleoperator inputs are generated, (ii) how task-level feedback is reflected to the teleoperator, and (iii) how each shared task command may be executed. In the next section, we will describe in somewhat more detail the shared control architecture which is representative of an implementation of the shared control of this, our invention. 
     The implementation will consist of the hardware and the software environment; and a particular way in which shared control is implemented. 
     3.0 HARDWARE 
     The hardware as shown in FIG. 8 is divided into two groups: local 800 above the dashed line 810 and remote 850 below the dashed line 810. In local site 800, there is a pair of six DOF hand controllers, shown as right and left hand controllers 801, 802 respectively. These hand controllers may be in accordance with the description given in Bejczy, A. K., Salisbury, J. K., Controlling Remote Manipulators through Kinesthetic Coupling, Computers in Mechanical Engineering, Vol. 2, No. 1, July 1983, pages 48-60. 
     These hand controllers appear at encircled numerals 1 and 2 in FIG. 8. Shown at encircled numerals 3 and 4 in FIG. 8 are a pair of associated Universal Motor Controllers (&#34;UMC&#34;) 815 and 816. The universal motor controllers 815 and 816 are connected to two VME chassis, each with two Motorola 68020/68881 based single board computers, I/O cards, and Ethernet cards (marked at encircled numerals 5 and 6 in FIG. 8. Items shown associated with the encircled numerals 1 through 6 form the hardware support for a standard teleoperation control over a robot. All operations from, and/or to, the actual tele-operator (and visa-versa) transformation boxes T and  on the local side are performed by this hardware. A Sun 3/60 work station 845 serves as the programming environment (see next subsection) and as an on line operator interface to invoke different modes of operation. 
     In general, real time signals are transmitted by using parallel I/O and non real time command invocations are through the use of ethernet based sockets. Autonomous commands are generated here. 
     The hardware of the remote site consists of a Sun 4/200 computer 885 (marked at encircled numeral 7 in FIG. 8), a VME chassis with two Motorola 68020/68881 based single board computer and serial/parallel I/O cards 856, 857 (marked 12 in FIG. 8), two UMC&#39;s 861,862 (marked at encircled numerals 8 and 9 in FIG. 8), two Puma 560&#39;s, identified as left robot 876 and right robot 886 and two Lord wrist force/torque sensors 877 and 878 (marked at encircled numerals 10 and 11 in FIG. 8). 
     All hand controllers are equipped with DC motors and encoders. The operator&#39;s hand motions are measured through the displacements it causes in the hand controller&#39;s joints using simple kinematic relations. Force feedback is possible since the hand controllers are equipped with joint motors. The hand controllers are balanced such that the operator can let go of them without the assembly dropping due to gravity. Each hand controller, in addition to providing a general six DOF motion specification capability, has three general purpose buttons and a trigger that is used for opening or closing a gripper, indexing, or any other user-defined function. 
     The UMC&#39;s may be those built at Jet Propulsion Laboratories in accordance with an article by: Bejczy, A. K., Szakaly, Z. F., A Synchronized Computational Architecture for Generalized Bilateral Control of Robot Arms, Proc. of the Conference on Advances in Intelligent Robotic Systems, SPIE &amp; International Society for Optical Engineering, Cambridge, MA., No. 1-6, 1987. and Bejczy, A. K., Szakaly, Z. F., Universal Computer Control System (UCCS) for Space Telerobots, Proc. of the 1987 IEEE International Conference on Robotics &amp; Automation, Raleigh, NC, Mar. 30-Apr. 3, 1987, pages 318-324. Such articles disclose general purpose motor controllers consisting of custom joint interface cards for reading the encoders and a multi-bus based National Semiconductor 32016 single board computer for servo control. 
     Each UMC is a stand alone robot controller with a capability to use additional microprocessors for multiprocessing. In our implementation a second NSC32016 with an onboard parallel port serves as a communication processor. This parallel port is connected to a VME based commercial parallel card. The overall system has four such connections; two in the remote site and two in the local site. The UMC&#39;s are used to send either position or voltage commands to the puma&#39;s or the hand controllers and read their encoders and potentiometer (in the case of Puma arms). When position set points are given to the UMC&#39;s, real time PID control is performed at the rate of 1000 Hz. The communication protocol which supports an array of commands and information gathering functions can be executed at the same 1000 Hz as well. 
     4.0 SOFTWARE 
     In this section we first describe the software environment that supports our implementation. Then we provide the details specific to the implementation of shared control. 
     There are three different software environments each identifiable with a particular hardware module. These are: UMC, VME/68020, and the Sun 4/200 software environments. In our implementation, the UMC&#39;s are considered black boxes with a predefined communication protocol. 
     The programming environment is IBM-PC for code development, cross compilation, and down loading. All the code is in NSC32016 assembly language. The VME/68020 uses a commercial software development package called VxWorks. This package provides all the necessary tools to write, down load, and debug code on the 68020&#39;s. The package can use one of several commercial real time kernels. 
     All the communication and 68020 software are written in the C language. The Sun 4/200 runs on a modified Sun 3.2 operating system which provides a real time kernel capability. The programming language is C. The autonomous portion is written in an enhanced (dual arm) version of Robot Control C Library (RCCL) pertaining to: Hayward, V., Paul R., Robot Manipulator Control Under Unix RCCL, International Journal of Robotics Research, Vol. 5, No. 4, pages 94-111, Winter 1987 and Lloyd, J., Parker, M., McClain, R., Extending the RCCL Programming Environment to Multiple Robots &amp; Processors, Proc. 1988 IEEE International Conference on Robotics &amp; Automation, Apr. 24-29, Philadelphia, PA, pages 465-469. 
     Now we describe the implementation of shared control. The teleoperation portion of the control, i.e., forward kinematics and transformation to the robot base coordinate frame are performed in the local site (functions on the local side in FIG. 6 are implemented on hardware items marked by encircled numerals 1 through 6 in FIG. 8). Force feedback to the operator is also implemented in the local site. The task level share control is performed on the Sun 360 in the R programming environment. 
     RCCL is general purpose robot library that provides a convenient programming environment. Since the user writes her code in the C language and makes use of RCCL library functions, the programs are not restricted to a specific syntax such as VAL or other special purpose robot languages. The user writes two pieces of code for each robot program. 
     The first part, which runs asynchronously with the robot, is the main logic of the program, such as when to start, how fast to move, Cartesian verses joint mode trajectory generation, etc. . . . The second part, which runs in real time, computes the user supplied real time functions. These real time functions can be tied to an external sensor such as a vision subsystem or force torque sensor, or an internal real time system such as joint sensors. Task level shared control is implemented using this capability of RCCL. 
     After the autonomous trajectories are determined from the task description, a ring equation of the form: 
     
         (Z . . . ) T6(. . . R)=AB . . . U 
    
     In every interval i.e., a time period during which R computes a new position set point, equation (3) is solved for T6. T6 is the transformation describing the sixth frame robot. All other matrices have to be determined before hand. Normally, the autonomous system generates plans which end up providing via point for the manipulator to pass through. In terms of the above equation, this means that matrices such as Z, . . . , U are determined by the autonomous system. One or more of these matrices can be attached to the output of the hand controllers. In this manner, the system can effectively mix the data from these two separate paths. This data can first go through appropriate filters so only the specified subspace affects the motion of the robot. 
     The output of this portion can then be given to the level 0 which actually performs the servo control. Similar but simpler mechanism is used to mix the force trajectories. It is simpler, since at the present we consider very simple (step function) trajectories for force control. 
     The servo control is performed in two 68020 CPU&#39;s in the remote site (box 12). Since the Sun 4/200 computer can compute the kinematics and dynamics equations 7 to 8 times faster than the 68020/68881 processors, most of the computational elements that do not need servo level updates reside in the Sun 4/200 computer. These include the Jacobian, inverse Jacobian, and some of the elements of the arm dynamics. The Sun 4/200 updates this data and computes the position and force set positions at the rate of 150 Hz. The servo level runs at 300 Hz. 
     In this patent application we have presented an architecture for shared control in two levels of a control hierarchy. Our philosophical approach to the invention includes a signal sharing that has retained the advantages of each mode while allowing for one&#39;s deficiencies to be overcome by the other. Particular emphasis was placed on the control of a remotely located robot in the presence of transmission time delays. The architecture provides a flexible system design such that one can start on an almost completely teleoperated mode and move towards increased autonomy in time. 
     Other applications of the principles and features of this invention will be well recognized to those of ordinary skill in this art and need no further description. The principles described and claimed hereinafter are to be construed in accordance with the applicable construction and interpretation tenets of the Patent Laws.