# Iterative Learning without Reinforcement or Reward for Multijoint Movements: A Revisit of Bernstein's DOF Problem on Dexterity

Authors: Suguru Arimoto; Masahiro Sekimoto; Kenji Tahara
Journal: Journal of Robotics
 (2010)

---

## Abstract

A robot designed to mimic a human becomes kinematically redundant and its total degrees of freedom becomes larger than the number of physical variables required for describing a given task. Kinematic redundancy may contribute to enhancement of dexterity and versatility but it incurs a problem of ill-posedness of inverse kinematics from the task
space to the joint space. This ill-posedness was originally found by Bernstein, who tried to unveil the secret of the
central nervous system and how nicely it coordinates a skeletomotor system with many DOFs interacting in complex ways. In
the history of robotics research, such ill-posedness has not yet been resolved directly but circumvented by introducing
an artificial performance index and determining uniquely an inverse kinematics solution by minimization. This paper tackles
such Bernstein's problem and proposes a new method for resolving the ill-posedness in a natural way without invoking
any artificial index. First, given a curve on a horizontal plane for a redundant robot arm whose endpoint is imposed to trace
the curve, the existence of a unique ideal joint trajectory is proved. Second, such a uniquely determined motion can be
acquired eventually as a joint control signal through iterative learning without reinforcement or reward.


---

## Body

## 1. Introduction

Almost a quarter century ago, “robotics” was defined by Professor Brady at the first International Conference of Robotics Research [1] as “the intelligent connection of perception to action.” After a great deal of researches on developments of industrial robots and their applications, a variety of research projects on “humanoid” have attracted many roboticists during the past decade and nowadays robots that can walk with a bipedal mode are not peculiar. Nevertheless, the present state of the art of humanoid still lacks dexterity in fulfillment of ordinary tasks that human encounter in their everyday life. More than a half century ago preceding the birth of “humanoid,” Bernstein [2, 3] noted that dexterity of human body movements resides in involvement of surplus degrees of freedom of limb joints but this incurs the ill-posedness of inverse kinematics. This was introduced to the robotics community through the famous textbook [4] in page 303 in such a statement as“The study of human biological motor control mechanisms led the Russian psychologist Bernstein to question how the brain could control a system with so many different degrees of freedom interacting in such a complex fashion. Many of these same complexities are also present in robotic systems and limit our ability to use multifingered hands and other robotic systems to their full advantage.”Actually, this was originally quoted from Hinton's article [5] in which he summarized what Bernstein challenged in the following way.(a) What can we infer about the code that the brain uses to communicate with the periphery, and what does that tell us about how the computation is organized?, (b) If the brain knew just what movements it wanted the body to make, could it figure out what to tell the muscles in order to make it happen? (c) How is it possible to coordinate a system with so many degrees of freedom that interact in such complex ways? (d) How does the brain make sensible choices among the myriad possibilities for movement that the body offers?This paper discusses what are the fundamentals of biomimetic control by focusing Bernstein's DOF problem and shows one way for solving such difficult problems from the standpoint of Riemannian geometry. It is shown in the case of multijoint reaching with DOF redundancy that, given a starting posture in the robot configuration space (or a Riemannian manifold as a set of all postures) and a target endpoint in the task space, there exists a unique orbit of joint motion, provided that the gravity term is compensated in a feedforward manner. Then, it is shown that such an ideal joint motion can be acquired through the iterative learning control without introducing any kind of performance index or reinforcement. In the second illustrative example, a handwriting motion with DOF redundancy is analyzed from the viewpoint of Riemannian geometry under the situation that writing with a ball pencil is imposed to trace an arbitrary smooth curve ofC∞-class on an arbitrary smooth surface in the three-dimensional Euclidean space. Even in this case there exists a unique joint motion in the base Riemannian manifold with DOF redundancy and it can be acquired through repeated exercises of handwriting motion, that is, an ILC scheme without introducing any artificial performance index. In conclusion, an ideal multijoint motion can be acquired through repeated exercises of motion even under the existence of redundancy in DOF, irrelevantly to any kind of reinforcement with the aid of some sort of reward [6, 7].


## 2. Riemannian Manifold and Euler's Equation

It is widely known among roboticists that kinematics and planning of multijoint robots are treataed in the configuration space regarded as ann-dimensional numerical space Rn [8, 9]. On the other hand, Arnold [10] pointed out the importance of Riemannian geometry in the analysis of mechanical systems and shown that the dynamics of motion of a double pendulum can be described by an orbit on a two-dimensional torus T2 that is regarded as T2=S1×S1, where S1 denotes a unit circle. In line with this notion, an n-DOF robot arm can be treated on an n-dimensional Rimeannian manifold like an n-dimensional torus Tn, and the stability problems of PD feedback with damping shaping [11] were retreated in a Riemannian-geometric manner [12, 13]. More recently, the author and his group showed that, given a robot arm, the set of all possible postures can be regarded as a Riemannian manifold with the Riemannian metric that constitutes the inertia matrix [14, 15] (see Figure 1). Thus, an orbit of motion as a geodesic solution to the Euler equation can be regarded as an inertia-induced motion without affection of damping and gravity forces [16].Figure 1
The set of all possible postures of a planar robot arm with three joints can be regarded as a Riemannian manifold with Riemannian metricgij(q) that constitutes the inertia matrix.It is well known as in a text book that motion of a robot manipulator as a serially connected rigid-body system is governed by an Euler-Lagrange equation shown in the form (see [17])(1)G(q)q̈+{12Ġ(q)+S(q,q̇)}q̇+g(q)=u,
where q=(q1,…,qn)T denotes the vector of joint angles, G(q)=(gij) does the n×n inertia matrix, u a control toque vector, g(q)=∂P(q)/∂q with a scalar function P(q) called the potential, and S(q,q̇) a skew-symmetric matrix S=(Sij) defined as(2)Sij=12{∂∂qj(∑k=1nq̇kgik)-∂∂qi(∑k=1nq̇kgjk)}.
If we consider a control torque that can exactly compensate the gravity term, that is,(3)u=g(q),
then substitution of (3) into (1) yields(4)G(q)q̈+{12Ġ(q)+S(q,q̇)}q̇=0,
which is considered to be an ideal equation of motion without affection of gravity and joint damping forces like a robot arm on an artificial satellite in space. It is pointed out in the recent papers [14, 16] that (4) is equivalently written in the form(5)gik(q)q̈i+Γikj(q)q̇jq̇i=0,k=1,…,n,
where Γikj denotes the Christoffel's symbol of the second kind and the symbol of summation with respect to i and j in (5) is omitted by obeying the Einstein's rule in differential geometry [18, 19]. Equation (5) is also expressed equivalently in the form(6)q̈k+Γijkq̇iq̇j=0,k=1,…,n,
which is called the Euler equation. In (5) and (6), Γikj and Γijk (i.e., called the Christoffel's symbol of the first kind) are defined as follows:(7)Γikj=12(∂gjk∂qi+∂gik∂qj-∂gij∂qk),Γijk=12∑l=1nglk(∂gjl∂qi+∂gil∂qj-∂gij∂ql)=12∑l=1nglkΓilj,
where (glk) denotes the inverse of G(=(gij)). The equivalence relation of (5) to (4) is shown in the previous paper [15].


## 3. Multijoint Movements with  DOF Redundancy

Let us now consider motion of a redundant planar robot arm whose endpoint is free to move in the horizontal plane as shown in Figure2. The Lagrange equation of motion of the whole arm-hand system depicted in Figure 2 is expressed as(8)G(q)q̈+{12Ġ(q)+S(q,q̇)}q̇=u,
where q=(q1,…,q4)T and each rotational axis of the four joints (shoulder, elbow, wrist, and index finger MP (metacarpophalangeal) joint) is in the direction perpendicular to the xy-plane. Given a robot arm posture q in the configuration space R4 or in the 4-dim. base manifold {M,gij}, the endpoint position x can be determined by the forward kinematics. A vector-valued function x(q)=(x(q),y(q)) of C∞-class. However, given an endpoint position xd in R4, there arises an infinite number of inverses that realize x(qd)=xd and thereby the problem for obtaining inverse kinematics from the 2-dimensional Euclidean space E2 to the 4-dimansional configuration space R4 becomes ill-posed.Figure 2
“Reaching” by means of a surplus DOF system of hand-arm dynamics.A variety of ideas for solving such ill-posedness of inverse kinematics for redundant robotic systems with excess DOFs has been proposed in the area of robotics, based upon the use of the form(9)q̇d(t)=J+(q(t))ẋd+(I-J+(q)J(q))v,
where v should be computed so as to optimize a certain performance index related to joint position variables (for example, manipulability index [20], obstacle avoidance [21], etc.). In equation (9), J(q) stands for the Jacobian matrix of x in q, that is, J(q)=∂x/∂qT, and J+(q) denotes the pseudo-inverse of J(q). The original idea of use of the pseudo-inverse J+(q) is due to Whitney [22]. Once a desired joint velocity q̇d(t) is planned, it is claimed that the computed torque method can be applied for determining the control input that must generate the whole joint motion of the robot. This is called “inverse kinematics approach”. Another idea of direct generation of a control signal called “inverse dynamics approach” is based upon a form of control input(10)u=G(q)J+(q){ẍd-J̇(q)q̇}+g(q,q̇)+(I-J+(q)J(q))v,
where v is computed so as to optimize a certain performance index related to velocity variables (e.g., kinetic energy [23], torque [24], energy dissipation [25], etc.). In (10), g(q,q̇) means compensation for the remaining nonlinear function including centrifugal and Coriolis forces and the gravity effect. In the physiological literature, main concerns are focussed on the question why human skilled multijoint reaching movements exhibit typical characteristics that (1) endpoint trajectory becomes a quasistraight line and less variable throughout repetitions, (2) velocity profiles of the endpoint velocity becomes bell-shaped, though (3) joint trajectories are rather variable trials-by-trials [26]. Then, a variety of cost functions for derivation of such properties of point-to-point reaching movements has been proposed, among which a quadratic function of endpoint jerk (rate of acceleration) was the first [27] and successively a cost function based on joint torques was introduced [28] for planning not only an endpoint trajectory but also joint trajectories. However, in the physiological literature, there is a dearth of papers that attempted to directly deal with reaching movements with redundant joints, though the importance of Bernstein's DOF problem [2] has been widely known among physiologists.Differently from the traditional approaches, a simpler control method for multijoint reaching movements was proposed very recently in our previous papers [29–31] and shown to be effective in both cases of human and robotic arms with redundant DOFs. In those papers, only planar motions confined to a horizontal plane are treated and therefore the control signal is free from gravity with a simple form (see Figure 3)(11)u=-Cq̇-JT(q)kΔx,
where C denotes a diagonal positive definite matrix as follows:(12)C=diag(c1,…,cn).
Notwithstanding this simpler form than (10), once damping factors C and single stiffness parameter k are chosen carefully, it is shown that it generates smooth reaching movements by realizing a quasistraight endpoint trajectory, bell-shaped velocity profiles, and double-peaked acceleration profiles typically seen in case of human skilled multijoint movements [26, 32]. Therefore, the method can get rid of undesirable fluctuations of the endpoint trajectory tracking as pointed out in a recent elaborate work [33]. Such fluctuations in task space tracking caused by using the computed torque under uncertainty in link parameters become more noticeable in cases of robots with redundant DOFs. In contrast, in the use of control defined in (11), there is no need of planning any desired endpoint trajectory. However, all these treatments are restricted to planar motions as well as in most of the previous papers on multijoint reaching movements. In addition, another disadvantage is that choice of damping factors recommended in [29] is not fit to the scale of coefficients of viscosity of human muscles [34, 35].Figure 3
Virtual spring-damper hypothesis.To reduce damping factors in general, another control method based upon “Virtual Spring-Damper Hypothesis” was suggested in [31], which in the case of planar motions without affection of the gravity is expressed by the form(13)u=-C0q̇-JT(q)(ζkẋ+kΔx)=-C0q̇-ζkJT(q)J(q)q̇-JT(q)kΔx,
where C0 is chosen as follows:(14)C0=ζ0diag(c1,…,cn),
together with positive constant 0<ζ0<1.0. The effectiveness of this control signal particularly in the case of middle-range reaching was demonstrated through computer simulation and its performance was compared with that of the control of (11). Theoretical verification of the effectiveness of this spring-damper hypothesis was also presented, on the basis of an energy conservation law like a Lyapunov-like relation obtained by substituting (13) into (8) and taking the inner product of this resulted closed-loop dynamics and q̇ as follows:(15)G(q)q̈+{12Ġ+S+C0}q̇+JT(q){ζkẋ+kΔx}=0,ddt{12q̇TG(q)q̇+k2∥Δx∥2}=-q̇TC0q̇-ζk∥ẋ∥2.
These results suggest that skilled multijoint movements can be generated even in the case of robot arms with redundant DOFs without construction of any inverse dynamics through “error-feedback learning” as claimed in a physiological journal [36] for modifying Equilibrium-Point hypothesis [35], End-Point hypothesis [37], and Virtual Trajectory hypothesis [37].


## 4. Existence of Desired Joint-Motion

More recently in the paper [16], an interesting result is found that the endpoint trajectory of a solution to the closed-loop dynamics for a given starting posture resembles considerably the endpoint trajectory of a geodesic solution to the Euler equation of (5) or (6) starting from the same given posture to a certain different posture with the prescribed endpoint xd. The existence of such a geodesic solution to (6) is ascertained by considering a two-dimensional Riemannian submanifold that is defined by the set of all postures q satisfying x(q)=xd, that is (see Figure 4), (16)NP'={q∣x(q)=P′(=xd)}.
Similarly, define another submanifold (17)NP={q∣x(q)=P(=x(0))}.
For a given endpoint position xd in E2, NP' constitutes a two-dimensional submanifold of the base manifold {M,gij}, that is called the equilibrium-point manifold or simply the EP-manifold in this paper. Denote by q(t) any smooth orbit of motion of the robot starting at t=a from the same posture q(t=a)=q0 and reaching the submanifold NP' at t=b so that it satisfies x(q(b))=xd. Then, consider the infimum(18)d(q0,NP′)=infq(t)∫ab∑i,jgij(q)q̇i(t)q̇j(t)dt
over the set of all such possible orbits of robot motion connecting q0 and NP'. If xd is not so distant from x(0)(=x(q0)), it is reasonable from the text books of Riemannian geometry [18, 19] that there exists a unique optimal orbit q*(t), t∈[a,b], that minimizes the right-hand side of (18). Then, the quantity d(q0,NP') is entitled to be called the Riemannian distance from q0 to the submanifold NP' in the base manifold {M,gij}. It is well known that the optimal orbit q*(t) must satisfy the Euler equation (6) for t∈[a,b] and, moreover, it must satisfy the following equation:(19){I4-J+(q*(t))J(q*(t))}q̇*(t)=0,
where J(q)=∂x(q)/∂qT, the Jacobian matrix of x(q) with respect to q. In other words, q̇*(t) must belong to the image space of JT(q*(t)) at any instant t in [a,b]. That is, q̇*(t) does not have any component in the kernel space of JT.Figure 4
A set of all postures that have the same endpoint positionxd constitutes a two-dimensional Riemannian submanifold.Consider now an endpoint trajectory tracking problem for a redundant multijoint arm of Figure2 in the case that a desired endpoint motion is given as a curve xd(t):I=[0,T]→E2. The control task is to maneuver the robot to let its endpoint trace the given trajectory xd(t) in E2 through a task space control signal(20)u=JT(q)v
provided that the robot dynamics is governed by the Lagrange equation(21)G(q)q̈+{12G(q)q̇+S(q,q̇)}q̇+C0q̇=JT(q)v,
where C0 is a 4×4 positive definite damping coefficient matrix. We assume that the initial posture of the robot at t=0 is given by q(0)=q0 and motion of the robot starts from the still state, that is, q̇(0)=0. The first problem is to find an adequate control signal v(t), t∈[0,T], so that the solution to the Lagrange equation of (21) starting from q(0)=q0 and q̇(0)=0 satisfies x(q(t))=xd(t) for t∈[0,T]. In order to find a solution to this problem, we decompose any solution trajectory of joint velocity q̇(t) in such a way that(22)q̇=(P,W)(ẋη̇),
where x=x(q), ẋ=J(q)q̇, η̇ is a 2×1 vector and P is the 4×2-matrix defined by(23)P=JT(q)(J(q)JT(q))-1=J+(q)
and W is a 4×2-matrix whose column vectors w1 and w2 are orthogonal to JT(q) (wi belongs to the kernel space of JT(q)) and satisfy ∥w1∥=∥w2∥=1 and w1Tw2=0. Then, if we define(24)Q=(P,W),Q-1=(PTWT)=QT,G1=QTGQ,C1=QTC0Q,S1=QTSQ-12Q̇TGQ+12QTGQ̇,
then, by substituting (22) into (21) and multiplying the resultant equation by the transpose of Q from the left, we have(25)G1(ẍη̈)+12{Ġ1+S1+C1}(ẋη̇)=(v0).
Note that S1 is again skew-symmetric. For convenience let us define(26)B1=12Ġ1+S1+C1
and decompose G1 and B1 in such a way that(27)G1=(G11G12G12TG22),B1=(B11B12B21B22),
where all Gij and Bij are of 2×2-matrix. Then, it follows from (25) that(28)G22η̈+B22η̇=-G12Tẍ-B21ẋ.
This equation means that if ẋ(t) is set as ẋ(t)=ẋd(t) and qd(t) is also given then η̇(t) can be determined uniquely from solving the differential equation of (28) as an initial-value problem by setting η̇(0)=0. It should be remarked at this stage that the given curve xd(t) is of C∞-class described in terms of time parameter t in E2, that is, it is an E2-valued function of t with the initial value xd(0)=x(q0), and it has the continuous time derivative ẋd(t) with the initial value ẋd(0)=0. Now, multiplying (28) by G22-1 and accompanying this with (22) by setting ẋ=ẋd and ẍ=ẍd, we have(29)q̇d=P(qd)ẋd+W(qd)η̇,(30)η̈=-G22-1(qd){B22(qd,q̇d)η̇+G12(qd)ẍd}+G22-1(qd)B21(qd,q̇d)ẋd.
This couple implies a set of six simultaneous differential equations of 1st order concerning six variables {qd,η̇} though the right hand side of (30) contains q̇d and hence it is an implicit function expression. Fortunately, it is possible to obtain an explicit expression of the six simultaneous differential equation by putting(31)B22=B22(qd,Pẋd+Wη̇),B21=B21(qd,Pẋd+Wη̇).
Hence, the right hand sides of (29) and (30) are nonlinear in qd and η̇, but they are Lipschitz continuous in qd and η̇ locally. Therefore, for given ẋd and ẍd there exists a unique solution {qd(t),η̇(t)} for an interval t∈[0,a] with some a>0 satisfying qd(0)=q0 and η̇(0)=0, where q0 signifies an initial posture satisfying x(q0)=xd(0). This fact was already discussed in our previous paper [38]. In this paper, we now prove the unique existence of the solution to the pair of 1st-order differential equations of (29) and (30) over the time interval [0,T], provided that C0 is not so small in comprison with the scale of G(q) and both the quantities of ẋd and ẍd are within an adequate physical scale.First, we analyze (30) or (28) by rewriting them in a more detailed manner as follows:(32)G22η̈+{12Ġ22+S22}η̇+C22η̇=-G12Tẍd-B¯21ẋd-C21ẋd,
where(33)G22=WTGW,C22=WTC1W,G12T=WTGP,B¯21=12Ġ21+S21,C21=WTC1P
and S22 denote the submatrix (sij) of S1 for i,j=3 or 4, and B¯21 plus C21 constitutes B21. Note that B¯21 is linear and homogeneous in η̇ but G12 are irrelevant to η̇. In this paper, we restrict our consideration to the case that overall movements of the robot is confined to a single chart of the base manifold in which at any posture in the chart the Jacobian matrix is nondegenerate and therefore there exists a positive constant σ0 such that J(q)JT(q)≥σ0I2 inside the chart. Under this condition, we analyze the following relation obtained by taking the inner product of (32) and η̈:(34)ddt(12η̇TG22η̇)=-η̇TC22η̇-η̇TB¯21ẋd-η̇T{G12Tẍd+C21ẋd}.
The second term of the right hand side is quadratic in η̇ and therefore there exists a constant β0 such that(35)∥η̇TB¯21ẋd∥≤β0∥η̇∥2
and β0 depend on the maximum magnitude of ẋd and 1/σ0. The third term of the right hand side is bounded from the above in the following way(36)∥-η̇T{G12Tẍd+C21ẋd}∥≤η̇TWTGWη̇+12ẍdPTGPẍd+12ẋdTPTC0G-1C0Pẋd.
As discussed previously in [39], the damping matrix C1 can be chosen to be of the order of G1/2 and further so as to satisfy(37)C0>2β0I4,C0>2G.
Then, (34) is reduced to the inequality relation(38)ddtηd(t)≤-γ0ηd(t)+ξd(t),
where we put with some positive constant γ0(39)ηd(t)=12η̇T(t)G22(t)η̇T(t),ξd(t)=12ẍdT(t)PTGPẍd(t)+12ẋdT(t)PTC0G-1C0Pẋd(t).
Clearly, since ηd(0)=0, (38) implies(40)ηd(t)≤∫0te-γ0(t-τ)ξd(τ)dτ
which concludes that η̇(t) is uniformly bounded in t∈[0,T].Onceq̇d(t) and η(t) are obtained for the given ẋd(t) and ẍd(t), the desired input signal vd(t) in the image space of J(qd) is obtained by setting q(t)=qd(t), ẋ(t)=ẋd(t), ẍ(t)=ẍd(t) in (25).


## 5. Iterative Learning Control in the Task Space

Given a desired trajectory of the endpoint in the task spaceE2 for a redundant robot arm, there exists a unique trajectory of robot motion in the joint space for a specified initial posture. In particular, it is shown in the previous section that there is uniquely a control signal in the task space that maneuvers the robot through the transpose of the Jacobian matrix to realize the endpoint tracking. However, such a control signal can not be obtained in any analytical form. Nevertheless, it is possible for us to acquire such a desired control signal by using a simple iterative learning control scheme, provided that the endpoint trajectory in the task space can be measured by visual sensing.At thekth trial of iterative learning, the control signal for the dynamics of (21) is designed in the form(41)vk=-JT(qk){κΔxk(t)+ζ1κΔẋk(t)}+vk-1-ΦΔẋk-1(t),
where J(qk) means ∂x/∂q at x=x(qk), and(42)Δxk(t)=x(qk(t))-xd(t),Δẋk(t)=ẋ(qk(t))-ẋd(t).
The first term of the right hand side of (41) signifies the inner task-space PD feedback, vk-1 denotes the previous control signal at the (k-1)th trial, and Φ is an adequate positive definite constant matrix. At the first trial, usually we set v0(t)=0 for t∈[0,T]. At the second trial v1(t) must contain erroneous terms. Fortunately, without knowing the desired ideal control vd(t), it is possible to expect that Δxi(t)→0 and Δẋk→0 for t∈[0,T] as k→∞. We give an illustrative example of numerical simulation conducted for the 4-DOF robot arm shown in Figure 2 with physical parameters given in Table 1. The values for length, mass, and inertia moment of the first link correspond to those of an upper arm of average human adult (male), and the values for the second link do to those of a lower arm. The third link corresponds to a human palm and the fourth an index finger. The desired task is to write a handwritten character “α” on the xy-plane. More explicitly, the endpoint trajectory is given by the equation(43)xd(t)=[0.000.30]+[0.075cosω(t)0.100cos1.5ω(t)],
where T=2.0 [s] and(44)ω(t)=2.0π{-15(tT)4+6(tT)5+10(tT)3}.
The initial posture of the arm is given in Table 2. Based on these data, the system of differential equations(45)G(qk)q̈k+{12Ġ(qk)+S(qk,q̇k)+C0}q̇k+JT(qk){κΔxk+ζ1κΔẋk}=JT(qk)vk
is numerically solved by using the Runge-Kutta method. In Figure 5 we show endpoint trajectories at 1st, 3rd, 5th, and 10th trials and their corresponding postures at t=0 and t=T. On the other side, we are able to obtain the desired control signal numerically by numerically solving a couple of 1st-order differential equations of (29) and (30). Based upon knowing physical data given in Table 1, the initial posture of q(0) given in Table 2, and the specified endpoint trajectory of (43), we obtain the desired ideal control signal shown in Figure 6. It is quite interesting to know that, through simulations of the iterative learning, calculated control signals vk(t), t∈[0,T], approach the desired one as the trial number k increases as shown in Figure 6. When k=10, the trajectory of control signal v10(t) almost coincides with the ideal one vd(t), that uniquely exists just in the image space of the Jacobian matrix  J(qd) with x(qd)=xd for all t∈[0,T].Table 1
Physical parameters of the 4-DOF robot arm.
Link numberi1234Length [m]li0.28000.28000.095000.09000Center of mass [m]lgi0.14000.14000.047500.04500Cylinder radius [m]ri0.040000.03500N/A0.009500Cuboid height [m]hiN/AN/A0.08500N/ACuboid depth [m]diN/AN/A0.03000N/AMass [kg]mi1.4071.0780.24230.02552Inertia moment [kgm2]Igiz9.758×10-37.370×10-32.004×10-41.780×10-5Table 2
Initial values and gain settings in the case of the 4-DOF planar robot arm.
(a)Terminal timeT2.0 [s]Initial postureq1(0)21.02 [deg]q2(0)75.47 [deg]q3(0)44.20 [deg]q4(0)64.87 [deg](b)Gainsk10 [N/m]ζ00.5[-]ζ12.5[-]Φ10.0[-]c10.862 [Nms]c20.569 [Nms]c30.129 [Nms]c40.0356 [Nms]Endpoint trajectories and the initial and final postures of the arm.
(a)
1st trial(b)
3rd trial(c)
5th trial(d)
10th trialTransient responses of the ILC termvd=(vd1,vd2)T.
(a)
vd1(b)
vd2Transient responses of the ILC termvk=(vk1,vk2)T.
(a)
vk1(b)
vk2It should be remarked that the desired endpoint trajectoryx(t) in the case of multijoint reaching for the robot arm of Figure 2 is obtained by solving the Euler equation of (4) or (5) as a two-point boundary-value problem when the initial posture q(0)=q0 at t=0 is given and the terminal condition at t=T is partially specified so as to move the endpoint of the arm to meet x(q)=xd at t=T and pass it away. This is the problem to find a curved orbit of the endpoint connecting two given points x(q(0))=x0 and x(q(T))=xd by selecting an adequate initial joint velocity q̇(0) that is nonzero. However, the orbit x(t) of the endpoint can be represented by a curve c(s) on E2 with the aid of length parameters. In the case of middle-range reaching, the profile of this endpoint geodesic curve c(s) quite resembles those of human-like multijoint reaching characterized typically by a quasistraight line movement of the endpoint starting from a fixed still state.


## 6. Extension to the Case of Existence of Effect of Gravity

Most of the previous results in Sections4 and 5 can be extended to the case that robot dynamics is subject to the effect of gravity. In such a robot with redundancy in DOF, robot dynamics is expressed by the Lagrange equation:(46)G(q)q̈+{12Ġ(q)+S(q,q̇)+C0}q̇+g(q)=JT(q)u,
where g(q) stands for the gravity term that can be regarded as a gradient vector of a potential function U(q) with respect to q, that is, g(q)=∂U(q)/∂q. Let us denote again the endpoint position by x(q) in the m-dimensional Euclidean space Em. Then, we split g(q) into(47)g(q)=J+(q)J(q)g(q)+(In-J+(q)J(q))g(q)=g1(q)+g2(q).
That is, g1(q) is a component of g(q) in the image space of J(q) and g2(q) is that of g(q) in the kernel space of J(q), that is orthogonally complement to the image space. In accordance with the split of the term g(q), let us choose an n×(n-m)-matrix W(q)=(w1,…,wn-m) with n-dimensional unit column vectors wi(i=1,…,n-m) that are mutually orthogonal and satisfy W(q)JT(q)=0 (i.e., W(q)J+(q)=0). Then, in a similar way to (22), we define(48)Q(q)=(J+(q),W(q))
which leads to(49)Q-1(q)=(J(q)WT(q)).
On the other hand, for a given desired endpoint trajectory xd(t) for t∈[0,T], we consider the control signal with the task space PD feedback(50)u=-{κΔx+ζ1κΔẋ}+v,
where Δx=x-xd. Substituting (50) into (46) yields the closed-loop dynamics(51)G(q)q̈+{12Ġ(q)+S(q,q̇)+C0}q̇+g1(q)+g2(q)+JT(q){κΔx+ζ1κΔẋ}=JT(q)v,
where v expresses a feedforward task space control signal. Then, consider the transform of q̇ to (ẋT,η̇T)T with an (n-m)-dimensional vector in such a form that(52)q̇=(J+(q),W(q))(ẋη̇)=Q(q)(ẋη̇).
In order that the equation of (51) with a desired control signal vd(t) has a unique solution qd(t) and q̇d(t) so that its endpoint trajectory x(t) is coincident with xd(t), and ẋ(t)=ẋd(t), the following equations must be satisfied:(53)Gd(ẍdη̈)+{12Ġd+Sd+Cd}(ẋdη̇)+((J+(qd))TgdWT(qd)gd)=(vd0),
where gd=g(qd) and(54)Gd=QdTG(qd)Qd,Cd=QdTC0Qd,Sd=QdTSQd-12Q̇dTG(qd)Qd+12QdTG(qd)Q̇d
with Qd=Q(qd). The latter (n-m)-dimensional component of (51) can be expressed by(55)G22η̈d+{12Ġ22+S22+C22}η̇+G12Tẍd+B21ẋd+WdTgd=0,
where Gd and Bd=(1/2)Ġd+Sd+Cd are decomposed into(56)Gd=(G11G12G12TG22),Bd=(B11B12B21B22),
and G22 and B22 are of (n-m)×(n-m)-matrix, and G11 and B11 are of m×m-matrix. The simultaneous differential equations (53) of 1st order together with(57)q̇d=J+(qd)ẋd+W(qd)η̇
determine a unique solution qd(t) and η̇(t) for given xd(t), ẋd(t), and ẍd(t) with the initial conditions qd(0)=q0, x(q0)=xd(0), and η̇(0)=0.


## 7. ILC for Handwriting

All the considerations in the previous sections can be extended to the case of a handwriting robot whose last link is a ball-point pen constrained on a hypersurfaceφ(x)=0, where x=(x,y,z)T in E3 and φ(x) is a scalar function of C∞-class (see Figure 8). First, we consider a pure mathematical problem of finding a geodesic curve by ignoring the effect of gravity and any joint damping. In this case, let us consider an open connected area S¯ on the hypersurface as shown in Figure 8, on which any point P(=x) satisfies the equality φ(x)=0. Then, we denote by F a local coordinate chart defined by F={q∈(M,gij)∣x(q)∈S¯} and assume that F is connected and at any q∈F the Jacobian matrix J(x(q))=∂x(q)/∂qT is nondegenerate. Then, given a point P on S¯ with the cartesian coordinates xP, the set of all q such that x(q)=xP and q∈F constitutes a single-dimensional submanifold NP (see Figure 9). Hence, for another given point P' on S¯, it is possible to consider an orbit in the submanifold F (equivalently, ϕ(F) in the configuration space) starting from q0, lying on N(S), and reaching some point lying on NP' that is another single-dimensional submanifold defined by the set of all q satisfying x(q)=xP' and q∈F. Thus, it is reasonable to suppose that there exists an optimal orbit q*(t) that gives minimization of the Riemannian distance from q0 to NP' such that(58)d(q0,NP′)=infq(t)∫01gij(q(t))q̇i(t)q̇j(t)dt=∫01gij(c(t))ċi(t)ċj(t)dt,
where the infimum is taken over all the orbits lying on N(S) (see Figure 9), starting from q0 and reaching NP', where q*(t) is rewritten by c(t). It is reasonable to conclude that the optimal orbit c(t) in (58) is a solution to the Euler equation under the constraint φ(x(c))=0:(59)G(c(t))+{12Ġ(c)+S(c,ċ)}ċ(t)=-λJcT(x(c(t)))∂φ∂xT,
where λ denotes a Lagrange multiplier. In other words, the path of c(t) from q0 to some point on NP' can be called the geodesic on the Riemannian submanifold N(S) induced naturally from the constraint φ(x(q))=0.Figure 8
A handwriting robot with four DOFs whose endpointP(=(x,y,z)) is constrained on a hypersurface φ(x)=0, where φ(x) is a scalar function of C∞-class.Figure 9
A geodesic curve starting fromq0 lying on NP and reaching some point on NP'. Here, ϕ is a homomorphism mapping F in M to ϕ(F) in R4.Next, consider the full dynamics of the handwriting robot with 4 DOFs shown in Figure8 by taking into account the effect of gravity forces and damping torques at joints (see [38]). The dynamics is described by(60)G(q)q̈+{12Ġ(q)+S(q,q̇)}q̇+C0q̇+g(q)=-λJT(q)∂φ(x)∂xT+u,
where g(q)=∂U(q)/∂q, U(q) denotes the gravity potential, C0 a positive definite damping matrix, and J(q)=∂x(q)/∂qT. It should be remarked that ∂φ/∂x stands for a vector that originates at the position x(q) in E3 and is normal to the surface φ(x)=0 which the ball pen contacts with. Now we introduce a length parameter s in E3 by the quantity(61)s(t)=∫0t∥ẋ(τ)∥dτ=∫0t∥q̇T(τ)J(q(τ))JT(q(τ))q̇(τ)∥dτ
and define the unit normal at the contact point as follows:(62)n(s)=∂φ∂x∥∂φ∂x∥-1.
Then, by rewriting f=λ∥∂φ/∂x∥, (60) can be rewritten into(63)G(q)q̈+{12Ġ(q)+S(q,q̇)}q̇+C0q̇+g(q)=-fJT(q)n(s)+u.
Note that the inner product of q̇ and the right-hand side of (63) vanish. In other words, ẋ(=J(q)q̇) is orthogonal to n(s) at the contact point in E3. Under the assumption that both the endpoint position x(t) and the velocity ẋ(t) are measured in real time by visual sensing and the Jacobian matrix J(q(t)) is calculated from the measurement data of x(t) and q(t) in real time, suppose that the control signal must be constructed through the Jacobian transpose in the form(64)u=JT(q)v.
For a given desired endpoint trajectory xd(t), t∈[0,T], together with ẋd(t) and ẍd(t) and a given desired pressing force fd(t), we are concerned with the problem to find a desired control signal vd(t) of (64) that maneuvers the robot to make the endpoint of the last link (ball pen) trace x(q(t))=xd(t) on the hypersurface with the pressing force fd in the direction normal to the surface. To show the existence of a control signal in the image space of J(q), we express it in a decomposed form such that(65)u=JT(q){vnn(s)+vbb(s)+vee(s)},
where b(s) signifies the unit vector tangent to the surface in the direction of -ẋ(t) and e(s)=n(s)×b(s). Note that substituting (65) into (63) yields(66)G(q)q̈+{12Ġ+S}q̇+C0q̇+(I4-J+J)g(q)=-JTΠ(s){(vn-fvbvn)+(gngbge)},
where(67)Π(s)=(n(s),b(s),e(s)),(gngbge)=(gT(q)J+n(s)gT(q)J+b(s)gT(q)J+e(s))=ΠT(s)(J+)Tg(q).
Note that Π(s) is an orthogonal matrix belonging to SO(3), g=(gn,gb,ge)T is a vector in E3, and a component of g(q) to(68)q̇=Q(q)(ξ̇η̇γ̇),(69)Q(g)=(J+(q)n(s),W(q),J+(q)(b(s),e(s))),
where W(q) is the 4×1 unit vector satisfying J(q)W(q)=0, ξ̇ and η̇ are a scalar and  express a 2×1 velocity vector such that (b(s),e(s))γ̇(s)=ẋ(t) at the contact point between the tip of the ball pen and the surface. Since ξ̇=0 that implies that the velocity of the tip of the pen in the direction n(s) normal to the surface is zero, substituting (68) into (66) and multiplying (66) by QT(q) yield(70)α(η̇,η̈,γ̇,γ̈)=-(vn-f+gn),(71)G11η̈+{12Ġ11+S11}η̇+WTC0Wη̇+WTg(q)+G12γ̈+B12γ̇=0,(72)G22γ̈+{12Ġ22+S22}γ̇+C22γ̇+G12Tη̈+B21η̇=-v-(gbge),
where G11 is of 1×1, G22 is of 2×2, and(73)G¯=PTGP,P=(W,J+(b,e)),G¯=(G11G12G12TG22),B¯=(B11B12B21B22),B¯=12G¯̇+S¯+PTC0P,S¯=PTSP-12ṖTGP+12PTGṖ.
In (70), α is a function of C∞-class with respect to, η̇,η̈,γ̇,γ̈, and q. Thus, for a given desired trajectory of the tip of the ball pen, γ̇(t)=ẋd(t) and γ̈(t)=ẍd(t), it is possible to prove the existence of qd and η̇(=η̇d) that satisfies   (74)q̇d=P(qd)(η̇dẋd),G11η̈d+{12Ġ11+S11}η̇d+C11η̇d+WTg(qd)+G12ẍd+B12ẋd=0,
where C11=WTC0W. The two equations (74) are the set of five simultaneous differential equations of the 1st order in five variables of η̇d and qd. Once the desired trajectory of joint motion qd(t) with its first and second derivatives is determined together with the motion η̇d(t) of excess-DOF, the desired control signal vd is given by (72) by putting γ̇=ẋd, q=qd, and η̇=η̇d. Moreover, if a desired pressing force fd is specified, the desired control signal in the direction normal to the surface can be constructed through (70) such that(75)vn=fd-gn(qd)-α(η̇d,η̈d,ẋd,ẍd).


## 8. What Should the Central Processing Unit Learn?

Given a smooth curveγ(s(t)) on a hypersurface φ(x)=0 over time interval t∈[0,T], it is shown in the previous section that there exists uniquely a control signal v(t) lying in the image space that makes the endpoint of the robot trace the curve γ(s(t)) on the hypersurface. Irrespective of the redundancy of DOF, once a starting posture of the robot is specified in the Euclidean space E3 with its endpoint position P=x(0) on the hypersurface φ(x)=0 as shown in Figure 10, the orbit qd(t) that realizes x(qd(t))=xd(t) itself is determined uniquely correspondingly to the uniquely existing control signal vd(t) (see Figure 10). Then, there arises a problem whether it is possible to acquire such a desired control signal without knowing the details of the robot dynamics. In order to discuss the problem, let us consider a learning control scheme that is expressed by the following form of the control signal (see Figure 11):(76)uk=-JT(q){κ(x-xd)+ζκ(ẋ-ẋd)}-JT(q){vk+σkn(s)},
where κ>0 stands for a constant expressing the stiffness parameter of position feedback and ζ>0 a constant lying in interval [0.5,2.0] of R1. The first term of the right-hand side expresses the task space (E3) PD feedback for a given set of position and velocity (xd,ẋd) and the second term stands for the learning control signal at the kth operation that is constructed as follows:(77)vk+1=vk-ΦΔẋk,σk+1=σk+ψΔfk,
where ψ is an appropriate constant such that 0<ψ<1 and Φ is a positive-definite 3×3 constant matrix. In the case of ordinary handwriting when the constraint surface is a horizontal plane, it is shown in the previous paper in [38] that the learning update law for the control signal based on (76) makes the motion of the endpoint trace the given curve xd(t) on the plane and at the same time the joint motions qk(t) and q̇k(t) converge to the ideal trajectories qd(t) and q̇d(t) as k→∞. The proof of convergence of the control signals vk to the desired one vd as k→∞ is also given in [39] on the basis of the “passivity” of the input v and the output ẋ of the system dynamics of (63) provided that u is given through (64) and the gravity effect g(q) is carefully compensated. Some numerical simulation results are also presented in [39].Figure 10
Handwriting on a hypersurfaceφ(x)=0.Figure 11
Iterative learning control scheme for robotic handwriting.Returning to Bernstein's problem discussed in the introductory section, we are fortunately able to quote Latash's commentary [40] to the book originally written by Bernstein in Russian cited as [2]. Latash says,Bernstein's definition of degrees of freedom relied heavily on the analysis of kinematic degrees of freedom of various systems. In particular, when analyzing the human upper limb, Bernstein considered all possible orthogonal axes of rotation in a joint as independent degrees of freedom, which were later summed over the joints. The apparent redundancy of the joints of the human arm in comparison to the three-dimensional space where we happen to live and where movements take place led Bernstein to his famous formulation that the essence of motor control is the elimination of the redundant degrees of freedom. The beauty and brevity of this formulation is stunning. A skeptic may want to decide whether control can always be reduced to the elimination of redundant degrees of freedom. In other words, I am suggesting that Bernstein's famous definition may not always be correct (blasphemy!!!)Even in the case of simple humanlike multijoint point-to-point reaching, Riemannian geometry suggests a good reason to think that the central nervous system is functioning according to a natural law of Newton's mechanics, that is, the law of inertia for multijoint mechanisms. Once a geodesic curveq*(t) connecting the given initial posture q0 and the equilibrium submanifold NP' (see (18)) is determined with some initial velocity q̇*(0), then the three-dimensional orbit x(q*(t)) for t∈[0,T] is obtained. This orbit can be rewritten into a three-dimensional curve γ(s) with the aid of length parameter s for a corresponding interval s∈[a,b]. Then this curve can be spelled out by a desired trajectory given as a movement of the tip of the arm in E3 in the form of xd(t)=γ(s(t)) for t∈[0,T] with the aid of a scale change of “time” through a monotonously increasing function s(t). Once a desired orbit xd(t) of motion of a tip of the arm is given in E3 and at the same time an initial posture of the arm is chosen, the trajectory of motion in the joint space is uniquely determined without eliminating any excess of the system's DOF. However, it is important to note that, even if there arises a small change of the initial posture, say q0+δq, it is possible to obtain the same desired orbit xd(t) in E3 but the trajectory qd(t) of joints differes slightly from that obtained when the initial posture is set as q(0)=q0 and also the desired control signal vd(t) may differ. Another noteworthy characteristic of humanlike multijoint movements is called “variability”, which was first pointed out by Bernstein [2]. Latash [32] observed that in the case of human skilled motion the grade of variability of each endpoint trajectory is quite low relative to variable profiles of joint responses at each trial of reaching. The mathematical arguments in the previous section suggest that the main part of variability in joint space is caused by small fluctuations of choice of an initial posture trial by trial and the others may be noise.We do not discuss the importance of redundancy of the muscles involved in a specific motion of multijoint reaching. The central nervous system surely uses its own means of communication with the muscles, which is not analogous to the language of joint kinematics that specifies the trajectories in individual joint. Instead of overcoming the problem of redundancy of the muscles, we implicitly assume in this paper that a desired torque of individual joint is finally generated by a total of forces of all the muscles involved in movement of the joint. This premise may not be justified by any reason, but it is important to quote Jackson's observation as a neurophysiologist [41] (see Figure 12):Figure 12
Sensory-motor coordination.“To speak figuratively, the central nervous system knows nothing of muscles, it only knows movements.”“The highest centres represent all parts of the body, literally all parts supplied by nerves.”


## 9. Conclusions

This paper discusses difficult problems of control for human-like robots with redundant DOFs from the standpoint of Riemannian geometry. Irrespective of joint redundancy, we have shown that there is no necessity to resolve the inverse kinematics problem by introducing any artificial performance index and optimizing it in both the cases of(1) multijoint reaching movement with excess joints, and (2) handwriting through iterative learning control. The most important conclusion is that Riemannian geometry does not care about the redundancy of joints through which rigid links are connected in series but it is powerful in establishing a correspondence between working-point trajectories in the external three-dimensional space (E3) and movement trajectories in a space of joint angles (a configuration space Rn or a Riemannian manifold {M,gij} as a set of whole possible postures).



---

*Source: 217867-2010-07-04.xml*