# Adaptive Image-Based Leader-Follower Approach of Mobile Robot with Omnidirectional Camera

Authors: Dejun Guo; Hesheng Wang; Weidong Chen; Xinwu Liang
Journal: Journal of Applied Mathematics
 (2015)

---

## Abstract

This paper focuses on the problem of the adaptive image-based leader-follower formation control of mobile robot with on-board omnidirectional camera. A calibrated omnidirectional camera is fixed on the follower in any position, and a feature point representing the leader can be chosen in any position. An adaptive image-based controller without depending on the velocity of the leader is proposed based on a filter technology. In other words, only by relying on the projection of the feature on the image plane, can the follower track the leader and achieve the formation control. Moreover, an observer is introduced to estimate the unknown camera extrinsic parameters and the unknown parameters of plane, where the feature point moves, relative to omnidirectional camera frame. At last, the lyapunov method is applied to prove the uniform semiglobal practical asymptotic stability (USPAS) for the closed-loop system. Simulation results are presented to validate the algorithm.


---

## Body

## 1.  Introduction

The formation control problem has been the research focus for a long time. The multirobots moving in formation have better collaboration abilities than those moving dispersedly. For example, robots in formation can achieve more complex tasks in shorter time. The leader-follower strategy is the most popular one due to its decentralized control approach, feasibility, and scalability. In the approach, the followers keep the desired relative pose to the leaders. Thus, the formation control problem can be transformed to several distributed control approaches.Many methods, focusing on leader-follower strategy, have been proposed using on-board laser sensors or perspective cameras. Choi et al. [1] proposed an adaptive position-based controller. The relative pose is measured by the on-board laser sensor, and the unknown leader’s motion was estimated by a novel observer. Dani et al. [2] and Poonawala et al. [3] measured the relative pose by pose reconstruction using a perspective camera. Dani et al. [2] estimated the relative velocity using a nonlinear estimator. Poonawala et al. [3] eliminated the need of leader’s velocity in the controller, so the design of the observer to estimate the leader’s motion could be avoided. Wang et al. [4] proposed an adaptive image-based controller based on the backstepping technology. The measurement of the relative pose could be eliminated, and the unknown height of the feature was estimated by an observer, but the measurement of object’s motion should be accurate. However, lasers are more expensive than cameras, and both lasers and perspective cameras have finite field of view and limit the robots’ motion.Compared to perspective camera, omnidirectional camera offers the panoramic view for the robot and can detect the object in the 360° surrounding scene. Due to this advantage, omnidirectional camera has been applied in many visual servoing approaches [5–8]. Many methods using the on-board omnidirectional camera have been developed to cope with leader-follower formation control. Das et al. [9, 10] designed a position-based controller depending on input-output linearization, and leader’s motion was estimated by extended Kalman filter (EKF). Mariottini et al. [11–13] achieved the leader-follower formation control only using the relative bearing measured by uncalibrated omnidirectional camera. The relative distance was estimated by EKF in [11, 12] and by immersion and invariance-based observer in [13]. Vidal et al. [14] proposed an image-based controller based on input-output linearization technology using omnidirectional camera. The optical flows method [15] was exploited to compensate for the unknown leader’s motion, but this method depended on detecting a static point all the time. However, the above methods using omnidirectional cameras all need that the camera should be mounted at the rotation center of the follower and that the image plane should be parallel with the ground plane. These assumptions limit the layout of the camera and cause the system error due to the unmatched model. What is more, most of the approaches always need to measure the relative pose, but this is not the direct way to achieve leader-follower approach. Transforming the image information to the position information is time-consuming and inaccurate.In this paper, the omnidirectional camera can be fixed on the mobile robots in any pose, and the image plane does not need to be parallel with the ground plane. The feature point on leader can be chosen in any position, and the position of the feature point can be unknown. An adaptive image-based controller is developed relying on a filter to eliminate the measurement or estimation of leader’s motion. Then, an observer is proposed to estimate the unknown pose of the omnidirectional camera relative to the follower and the unknown coefficient parameters of the plane where the feature point moves. The plane is expressed relative to the omnidirectional camera frame. At last, the Lyapunov theory is used to prove uniform semiglobal practical asymptotic stability (USPAS) of image error and formation error. The simulation results validate the performance of the proposed algorithm.


## 2. Kinematics

### 2.1. Problem Statement

The coordinate frames are defined as shown in Figures1 and 2. The world frame FW:{OW:XW,YW,ZW} is fixed on the ground. Denote follower frame by Ff:{Of:Xf,Yf,Zf} and leader frame by Fl:{Ol:Xl,Yl,Zl}. Define the omnidirectional camera frame as FC:{OC:XC,YC,ZC} and the image plane frame as FI:{OI:y1,y2}. Ol and Of coincide with robots’ rotation centers, and Zl and Zf are parallel with robots’ forward direction. The planes ZlOlXl and ZfOfXf are parallel with the ground plane. The robot’s linear velocity coincides with its Y-axis and its angular velocity is orthogonal to the ground plane. Define the original point of FC which locates in the focus point of the mirror, and the axis ZC is aligned to the symmetric axis of the mirror. The optical axis of camera coincides with ZC, and the image plane is parallel with plane XCOCYC. Axis y2 is parallel with axis YC. The problem addressed is defined as follows.Figure 1
Leader-follower system with omnidirectional camera.Figure 2
The projection model of omnidirectional camera.Problem 1.
Given a desired position on the image plane, an adaptive image-based controller is designed to make the feature point converge to an arbitrary small circle around the desired one, while the extrinsic parameters of the omnidirectional camera, the position of feature point on the leader, and the leader’s velocity are all unknown.Assumption 2.
The velocity of the leader is bounded, due to the practical reasons. The linear velocity and angular velocity of the leader areυl and ωl, respectively. υ-l and ω-l are their upper bounds; that is,(1)υl≤υ-l,ωl≤ω-l.Assumption 3.
Suppose that the parameters of the omnidirectional camera’s mirror and the intrinsic parameters of the camera are known. Both robots move on the ground plane.Notation. The notations used in this paper are listed as follows: a bold letter denotes a vector or a matrix. Otherwise, it denotes a scalar. Ik×k and 0k×l denote the k×k identify matrix and the k×l zero matrix, respectively. λmin⁡(·) and λmax⁡(·) denote the minimum and maximal eigenvalue of a positive-definite and diagonal gain matrix, respectively. · denotes the Euclidean norm of the vector. The presuper (scribe) represent the coordinate system that the variable is related to (e.g., PC denotes the position of feature point with respect to FC).


### 2.2. The Unified Model of Omnidirectional Camera

As is shown in Figure2, the omnidirectional camera includes a curved mirror and a camera. For example, a parabolic mirror is combined with an orthographic camera and a hyperbolic mirror is combined with a perspective camera. The details of the structure of the omnidirectional camera can be found in [16, 17].The 3D coordination of the feature pointP relative to frame FC is PC(t)=[XPCYPCZPC]T. As is shown in Figure 3, The intersection between the reflective ray and the plane XCOCYC is named the general-image point Q=[xy]T, and z(Q) is the proportionable coordinate corresponding to ZPC. The proportionable relationship is (2)Qz(Q)=1λPCt,where(3)λ=-e3TPC+ξPCμ=-ZPC+ξXP2C+YP2C+ZP2Cμis similar to the depth information that appeared in the case of perspective cameras and e3=[001]T. The parameters of mirror ξ, μ, and z(Q) are shown in Table 1. [y1y2]T represents the image coordinate of the image point. The mapping relation between Q and the image point is(4)y1y21=σ10y100σ2y20001xy1,where σ1, σ2, y10, y20 are the intrinsic parameters of the camera. [xy]T can be calculated through [y1y2]T when omnidirectional camera is calibrated. Therefore, without loss of generality, [xy]T, taking place of [y1y2]T, is considered as the output of the system.Table 1
The parameters of mirrors.
Parabolic
Hyperbolic
ξ
1
d
d
2
+
4
p
2
μ
2
p
2
p
d
d
2
+
4
p
2
z
(
Q
)
x
2
+
y
2
-
4
p
2
4
p
μ
+
μ
2
-
(
ξ
2
-
1
)
(
ξ
2
(
x
2
+
y
2
)
-
μ
2
)
ξ
2
-
1Figure 3
(a) The projection model of parabolic mirror. (b) The projection model of hyperbolic mirror.
(a)
(b)


### 2.3. Kinematics

The linear and angular velocities of theFC are υCC∈R3×1 and ωCC∈R3×1, respectively. The world linear velocity of the feature point P expressed in frame FC is υPC∈R3×1. Then, the relative velocity of P with respect to frame FC is(5)P˙C=υPC-υCC-ωCC×PC.The differential of λ is(6)λ˙=-e3Tμ+ξCPTμPCP˙C.The differential of Q is(7)Q˙=LP˙Cλ-PCλ˙λ2=1λL+Qe3Tμ-ξQPTCμPCυPC-υCC-ωCC×PC,where(8)L=100010.To eliminate PC, (7) can be rewritten, according to (2), as(9)1λL+Qe3Tμ-ξ2QQTz(Q)μμ+z(Q)×υPC-υCC-ωCC×λQzQ.Then, (9) can be rearranged in a more convenient matrix form(10)Q˙=J11λ,QυCCωCC+J21λ,QυPC,where the Jacobian matrixes are(11)J1=1λξ2x2μμ+z-11λξ2xyμμ+z1λξ2xzμμ+z-xμ-xyμ-z+x2μy1λξ2xyμμ+z1λξ2y2μμ+z-11λξ2yzμμ+z-yμz-y2μxyμ-x,J2=-1λ(ξ2x2μ(μ+z)-1)-1λξ2xyμ(μ+z)-1λξ2xzμμ+z-xμ-1λξ2xyμ(μ+z)-1λ(ξ2y2μ(μ+z)-1)-1λ(ξ2yzμ(μ+z)-yμ).Inspired by the depth-independent interaction matrix proposed in [18–23], a depth-like term 1/ZPC is introduced. According to (2),(12)1λ=z(Q)ZPC.Thus, (10) can be transformed to(13)Q˙=J11ZPC,QυCCωCC+J21ZPC,QυPC.The nonholonomic kinematic model of the mobile robots can be described as(14)X˙iWY˙iWϕ˙i=υicos⁡ϕiυisinϕiωi,i=f,l,where XiW and YiW denote the coordinates of Oi with respect to FW. ϕi denotes the orientation of the mobile robots and is defined as the angle from axis XW to axis Zi. υii=[00vi]T and ωii=[0-ωi0]T represent the linear and angular velocities, respectively. Moreover, let RfC(φi)∈R3×3 denote the rotation matrix of Ff relative to FC as(15)RfCφi=c1c2c3-s1s3s1c2c3+c1s3-s2c3c1s2s1s2c2c1c2s3+s1c3s1c2s3-c1c3s2s3,where si=sinφi, ci=cos⁡φi, and φi(i=1,2,3) denote Zf-Yf-Zf rotation angles defined relative to the current frames and from FC′ to FC, respectively. Frame FC′ denotes the omnidirectional frame when OC, axis YC, and axis XC coincide with Of, axis Zf, and axis Xf, respectively. Rij denotes the element of RfC in the ith row and in the jth column. The constant m=[mxmymz]T denotes the 3D coordinates of the origin of FC with respect to Ff. It is noted that υCC and ωCC are caused by the motion of the follower only. Thus, the relation between [υCCωCC]T and [υfωf]T is as follows:(16)υCCωCC=RfC(υff+ωff×m)RfCωff=Dυfωf,where the constant matrix D∈R6×2 is(17)D=R13-mzR11+mxR13R23-mzR21+mxR23R33-mzR31+mxR330-R120-R220-R32.Moreover, the time-variantRlfγ(t)∈R3×3 denotes the rotation matrix of Fl with respect to Ff. sγ=sinγ, cγ=cos⁡γ, and time-variant γ(t)=ϕl-ϕf denotes the difference of orientations between the leader and the follower. The time-invariant n=[nxnynz]T is the 3D coordinates of the feature point P with respect to Fl. υPC is caused by the motion of the leader only. Thus, the relation between υPC and [υlωl]T is as follows:(18)υPC=RfCRlfυll+ωll×n=Bυlωl,where the time-variant matrix B∈R3×2 is(19)B=-R11sγ+R13cγ-nz(R11cγ+R13sγ)+nx(-R11sγ+R13cγ)-R21sγ+R23cγ-nz(R21cγ+R23sγ)+nx(-R21sγ+R23cγ)-R31sγ+R33cγ-nz(R31cγ+R33sγ)+nx(-R31sγ+R33cγ).Substituting (16) and (18) into (13), then, (13) can be rewritten as(20)Q˙=J1Dυfωf+J2Bυlωl=J1′1ZPC,Qυfωf+J2′1ZPC,Q,γυlωl.The details of J1′ and J2′ are shown in the Appendix.According to Assumption3, the feature point and the omnidirectional camera move on the planes. Therefore, the feature point moves in a fixed plane relative to the FC. Based on a plane equation, the unknown depth 1/ZPC can be represented in terms of XPC, YPC, ZPC:(21)1ZPC=aXPCZPC+bYPCZPC+c.According to (2), (21) can be revised as(22)1ZPC=axz(x,y)+byz(x,y)+c.To eliminate 1/ZPC in (20), substituting (22) into (20), then, (20) can be revised as(23)Q˙=HQυfωf+GQ,γυlωl︸FQ,γ,υl,ωl.It is noted that γ, υl, and ωl are unknown and that they are not included in controller. Let θ∈R21×1 denote the unknown parameters including camera’s extrinsic parameters RfC, m, and the coefficient parameters of plane equation (21). The parameterized H(Q) is H(Q,θ). In addition, H(Q,θ)[υfωf]T can be parameterized in a linear form as (24)HQ,θυfωfT=MQ,υf,ωfθ.The details of θ, H(Q,θ) and the regressor matrix M can be found in the Appendix.


## 2.1. Problem Statement

The coordinate frames are defined as shown in Figures1 and 2. The world frame FW:{OW:XW,YW,ZW} is fixed on the ground. Denote follower frame by Ff:{Of:Xf,Yf,Zf} and leader frame by Fl:{Ol:Xl,Yl,Zl}. Define the omnidirectional camera frame as FC:{OC:XC,YC,ZC} and the image plane frame as FI:{OI:y1,y2}. Ol and Of coincide with robots’ rotation centers, and Zl and Zf are parallel with robots’ forward direction. The planes ZlOlXl and ZfOfXf are parallel with the ground plane. The robot’s linear velocity coincides with its Y-axis and its angular velocity is orthogonal to the ground plane. Define the original point of FC which locates in the focus point of the mirror, and the axis ZC is aligned to the symmetric axis of the mirror. The optical axis of camera coincides with ZC, and the image plane is parallel with plane XCOCYC. Axis y2 is parallel with axis YC. The problem addressed is defined as follows.Figure 1
Leader-follower system with omnidirectional camera.Figure 2
The projection model of omnidirectional camera.Problem 1.
Given a desired position on the image plane, an adaptive image-based controller is designed to make the feature point converge to an arbitrary small circle around the desired one, while the extrinsic parameters of the omnidirectional camera, the position of feature point on the leader, and the leader’s velocity are all unknown.Assumption 2.
The velocity of the leader is bounded, due to the practical reasons. The linear velocity and angular velocity of the leader areυl and ωl, respectively. υ-l and ω-l are their upper bounds; that is,(1)υl≤υ-l,ωl≤ω-l.Assumption 3.
Suppose that the parameters of the omnidirectional camera’s mirror and the intrinsic parameters of the camera are known. Both robots move on the ground plane.Notation. The notations used in this paper are listed as follows: a bold letter denotes a vector or a matrix. Otherwise, it denotes a scalar. Ik×k and 0k×l denote the k×k identify matrix and the k×l zero matrix, respectively. λmin⁡(·) and λmax⁡(·) denote the minimum and maximal eigenvalue of a positive-definite and diagonal gain matrix, respectively. · denotes the Euclidean norm of the vector. The presuper (scribe) represent the coordinate system that the variable is related to (e.g., PC denotes the position of feature point with respect to FC).


## 2.2. The Unified Model of Omnidirectional Camera

As is shown in Figure2, the omnidirectional camera includes a curved mirror and a camera. For example, a parabolic mirror is combined with an orthographic camera and a hyperbolic mirror is combined with a perspective camera. The details of the structure of the omnidirectional camera can be found in [16, 17].The 3D coordination of the feature pointP relative to frame FC is PC(t)=[XPCYPCZPC]T. As is shown in Figure 3, The intersection between the reflective ray and the plane XCOCYC is named the general-image point Q=[xy]T, and z(Q) is the proportionable coordinate corresponding to ZPC. The proportionable relationship is (2)Qz(Q)=1λPCt,where(3)λ=-e3TPC+ξPCμ=-ZPC+ξXP2C+YP2C+ZP2Cμis similar to the depth information that appeared in the case of perspective cameras and e3=[001]T. The parameters of mirror ξ, μ, and z(Q) are shown in Table 1. [y1y2]T represents the image coordinate of the image point. The mapping relation between Q and the image point is(4)y1y21=σ10y100σ2y20001xy1,where σ1, σ2, y10, y20 are the intrinsic parameters of the camera. [xy]T can be calculated through [y1y2]T when omnidirectional camera is calibrated. Therefore, without loss of generality, [xy]T, taking place of [y1y2]T, is considered as the output of the system.Table 1
The parameters of mirrors.
Parabolic
Hyperbolic
ξ
1
d
d
2
+
4
p
2
μ
2
p
2
p
d
d
2
+
4
p
2
z
(
Q
)
x
2
+
y
2
-
4
p
2
4
p
μ
+
μ
2
-
(
ξ
2
-
1
)
(
ξ
2
(
x
2
+
y
2
)
-
μ
2
)
ξ
2
-
1Figure 3
(a) The projection model of parabolic mirror. (b) The projection model of hyperbolic mirror.
(a)
(b)


## 2.3. Kinematics

The linear and angular velocities of theFC are υCC∈R3×1 and ωCC∈R3×1, respectively. The world linear velocity of the feature point P expressed in frame FC is υPC∈R3×1. Then, the relative velocity of P with respect to frame FC is(5)P˙C=υPC-υCC-ωCC×PC.The differential of λ is(6)λ˙=-e3Tμ+ξCPTμPCP˙C.The differential of Q is(7)Q˙=LP˙Cλ-PCλ˙λ2=1λL+Qe3Tμ-ξQPTCμPCυPC-υCC-ωCC×PC,where(8)L=100010.To eliminate PC, (7) can be rewritten, according to (2), as(9)1λL+Qe3Tμ-ξ2QQTz(Q)μμ+z(Q)×υPC-υCC-ωCC×λQzQ.Then, (9) can be rearranged in a more convenient matrix form(10)Q˙=J11λ,QυCCωCC+J21λ,QυPC,where the Jacobian matrixes are(11)J1=1λξ2x2μμ+z-11λξ2xyμμ+z1λξ2xzμμ+z-xμ-xyμ-z+x2μy1λξ2xyμμ+z1λξ2y2μμ+z-11λξ2yzμμ+z-yμz-y2μxyμ-x,J2=-1λ(ξ2x2μ(μ+z)-1)-1λξ2xyμ(μ+z)-1λξ2xzμμ+z-xμ-1λξ2xyμ(μ+z)-1λ(ξ2y2μ(μ+z)-1)-1λ(ξ2yzμ(μ+z)-yμ).Inspired by the depth-independent interaction matrix proposed in [18–23], a depth-like term 1/ZPC is introduced. According to (2),(12)1λ=z(Q)ZPC.Thus, (10) can be transformed to(13)Q˙=J11ZPC,QυCCωCC+J21ZPC,QυPC.The nonholonomic kinematic model of the mobile robots can be described as(14)X˙iWY˙iWϕ˙i=υicos⁡ϕiυisinϕiωi,i=f,l,where XiW and YiW denote the coordinates of Oi with respect to FW. ϕi denotes the orientation of the mobile robots and is defined as the angle from axis XW to axis Zi. υii=[00vi]T and ωii=[0-ωi0]T represent the linear and angular velocities, respectively. Moreover, let RfC(φi)∈R3×3 denote the rotation matrix of Ff relative to FC as(15)RfCφi=c1c2c3-s1s3s1c2c3+c1s3-s2c3c1s2s1s2c2c1c2s3+s1c3s1c2s3-c1c3s2s3,where si=sinφi, ci=cos⁡φi, and φi(i=1,2,3) denote Zf-Yf-Zf rotation angles defined relative to the current frames and from FC′ to FC, respectively. Frame FC′ denotes the omnidirectional frame when OC, axis YC, and axis XC coincide with Of, axis Zf, and axis Xf, respectively. Rij denotes the element of RfC in the ith row and in the jth column. The constant m=[mxmymz]T denotes the 3D coordinates of the origin of FC with respect to Ff. It is noted that υCC and ωCC are caused by the motion of the follower only. Thus, the relation between [υCCωCC]T and [υfωf]T is as follows:(16)υCCωCC=RfC(υff+ωff×m)RfCωff=Dυfωf,where the constant matrix D∈R6×2 is(17)D=R13-mzR11+mxR13R23-mzR21+mxR23R33-mzR31+mxR330-R120-R220-R32.Moreover, the time-variantRlfγ(t)∈R3×3 denotes the rotation matrix of Fl with respect to Ff. sγ=sinγ, cγ=cos⁡γ, and time-variant γ(t)=ϕl-ϕf denotes the difference of orientations between the leader and the follower. The time-invariant n=[nxnynz]T is the 3D coordinates of the feature point P with respect to Fl. υPC is caused by the motion of the leader only. Thus, the relation between υPC and [υlωl]T is as follows:(18)υPC=RfCRlfυll+ωll×n=Bυlωl,where the time-variant matrix B∈R3×2 is(19)B=-R11sγ+R13cγ-nz(R11cγ+R13sγ)+nx(-R11sγ+R13cγ)-R21sγ+R23cγ-nz(R21cγ+R23sγ)+nx(-R21sγ+R23cγ)-R31sγ+R33cγ-nz(R31cγ+R33sγ)+nx(-R31sγ+R33cγ).Substituting (16) and (18) into (13), then, (13) can be rewritten as(20)Q˙=J1Dυfωf+J2Bυlωl=J1′1ZPC,Qυfωf+J2′1ZPC,Q,γυlωl.The details of J1′ and J2′ are shown in the Appendix.According to Assumption3, the feature point and the omnidirectional camera move on the planes. Therefore, the feature point moves in a fixed plane relative to the FC. Based on a plane equation, the unknown depth 1/ZPC can be represented in terms of XPC, YPC, ZPC:(21)1ZPC=aXPCZPC+bYPCZPC+c.According to (2), (21) can be revised as(22)1ZPC=axz(x,y)+byz(x,y)+c.To eliminate 1/ZPC in (20), substituting (22) into (20), then, (20) can be revised as(23)Q˙=HQυfωf+GQ,γυlωl︸FQ,γ,υl,ωl.It is noted that γ, υl, and ωl are unknown and that they are not included in controller. Let θ∈R21×1 denote the unknown parameters including camera’s extrinsic parameters RfC, m, and the coefficient parameters of plane equation (21). The parameterized H(Q) is H(Q,θ). In addition, H(Q,θ)[υfωf]T can be parameterized in a linear form as (24)HQ,θυfωfT=MQ,υf,ωfθ.The details of θ, H(Q,θ) and the regressor matrix M can be found in the Appendix.


## 3. Adaptive Image-Based Leader-Follower Approach with Omnidirectional Camera

In image-based leader-follower approach, the position-based output, separation and bearing, can be transformed to an image point of an on-board omnidirectional camera, because there exists the injective mapping between an image point and a relative position with respect toFf. Then, general-image point is considered as the output due to the injective mapping between a general-image point and an image point. Therefore, when the general-image point converges to the desired one, the leader-follower formation is achieved. Furthermore, the desired general image point can be recorded using the omnidirectional camera when leader’s feature point locates at the desired position relative to the follower, which is known as the “teach-by-showing” approach.


### 3.1. Design of Controller and Observer

Proposition 4.
To avoid the singular point of the matrixH^(Q,θ^), the determinant p(t)=H^Q,θ^ should not be zero. θ^ Denote the estimated unknown parameters, and H^ denotes the matrix H in (23) containing θ^.According to [18], the repulsive potential field is introduced(25)U(θ^)=1eκ2p2-1+κ1,p<ε10,p≥ε1.And the potential force is introduced as(26)∂U(θ^)∂θ^=-2κ2peκ2p2eκ2p2-1+κ12∂p∂θ^,p<ε1-ε2-uε2ε1-pp∂p∂θ^,ε1-ε2≤p≤ε10,p>ε1,ε1, ε2, κ1, and κ2 are all positive constants, and(27)u=2κ2eκ2ε1-ε22eκ2ε1-ε22-1+κ12.Define the general-image error ase=Q-Qd=[x-xdy-yd]T. The observer is proposed as(28)θ^˙=Γ-1MTe+eTeN∂Uθ^∂θ^,which can be considered as the update law of the estimated parameters θ^. The regressor matrix M is calculated from (24). The second term in (28) serves as a repulsive force which can push θ^ away from the singular point of matrix H^(Q,θ^). Γ,N∈R21×21 are all positive gain symmetric matrices.Inspired by [24, 25], a general-image based filter Φ˙=-βΦ-Ke is proposed, where Φ=[Φ1Φ2]T can be regarded as a pseudoerror which is used to compensate for the general-image’s velocity caused by leader’s motion. β,K∈R2×2 are positive definite and diagonal gain matrices. The controller is proposed as(29)υfωf=H^-1-αe+KΠ-c∂U∂θ^e,where H^-1 is the inverse of the matrix H^(Q,θ^). Π=[tanh(δ1Φ1)tanh(δ2Φ2)]T, and c,α∈R2×2 are positive-definite and diagonal gain matrices. δ1 and δ2 are positive gains.


### 3.2. Stability Analysis

Theorem 5.
Utilizing the controller (29) and the observer (28), the general-image error e is uniform semiglobal practical asymptotic stability (USPAS), which means USPAS of image error. Moreover, the image-based leader-follower system with omnidirectional camera is stable under Assumptions 2 and 3 and the initial relative heading φ is bounded away from ±π, υlZPf>0.Proof.
The Lyapunov function is proposed as(30)V=V1+V2,V1=12eTe+TTΛT+12θ~TΓθ~,(31)V2=12γ2,where T=[ln⁡cosh⁡(δ1Φ1)ln⁡cosh⁡(δ2Φ2)]T, Λ=diag[1/δ11/δ2], and θ~=θ^-θ. The differential of V1 is(32)V˙1=eTe˙-βΦ+KeTΠ+θ~TΓθ^˙,where e˙=Q˙. Equation (23) can be refined as(33)e˙=-H^-Hυfωf+H^υfωf+F.Substituting (29) into (33), e˙ can be rewritten as(34)e˙=-H^-Hυfωf+-αe+KΠ-c∂U∂θ^e+F.Substituting (34) and the observer (28) into (32), (32) can be revised as(35)V˙1=-eTαe+eTF-βΦTΠ-eT(H^-H)υfωf+θ~TMTe+θ~TeTeN∂U∂θ^-c∂U∂θ^eTe.According to (24), -eT(H^-H)υfωf=-θ~TMTe. Then, (35) can be revised as(36)V˙1≤-λmin⁡(α)-2Fee2-βΦTΠ-eTc-θ~TNI2×2∂U∂θ^e,where βΦTΠ≥0 is an odd function; thus the second term in (36) is nonpositive. There is a small positive ε, which can be adjusted by the gains α, making sure that λmin⁡(α)-2F/ε=0. If e>ε, λmin⁡(α)-2F/ε>0; thus the first term in (36) is negative. Moreover, the gain matrix c should satisfy c≥λmax⁡(N)θ~I2×2; thus the third term in (36) is nonpositive. Therefore, when e>ε, V˙1<0. The scalar of ε can be arbitrarily decreased. So e is USPAS, according to [26].
The differential of (31) is(37)V˙2=γγ˙=γωl-ωf.Due to USPAS of e, [XPfZPf]T is also USAPS. The differential of [XPfZPf]T is(38)X˙PfZ˙Pf=0ZPf-1-XPfυfWωfW+-sγ-dzcγ-dxsγcγ-dzsγ+dxcγυlWωlW,where(39)ωf=υlZPfsγ+1ZPfX˙Pf+1ZPfnzcγ+nxsγωl.Substituting (39) into (37), (37) can be rewritten as(40)γ˙=ωl-ωf=-υlZPfsγ+1ZPfZPf-nzcγ-nxsγωl-1ZPfX˙Pf︸η1.The nominal system γ˙=-(υl/ZPf)sγ is exponentially stable when υlZPf>0 is satisfied. Obviously, η1 is bounded. Therefore, the perturbed system (40) is stable, and γ is bounded when γ(0)≠±π based on the stability of perturbed system [27].
In summary,e is USPAS, which is equivalent to USPAS of image error. Also, the relative heading γ is bounded. Therefore, the image-based leader-follower system with omnidirectional camera is stable.


## 3.1. Design of Controller and Observer

Proposition 4.
To avoid the singular point of the matrixH^(Q,θ^), the determinant p(t)=H^Q,θ^ should not be zero. θ^ Denote the estimated unknown parameters, and H^ denotes the matrix H in (23) containing θ^.According to [18], the repulsive potential field is introduced(25)U(θ^)=1eκ2p2-1+κ1,p<ε10,p≥ε1.And the potential force is introduced as(26)∂U(θ^)∂θ^=-2κ2peκ2p2eκ2p2-1+κ12∂p∂θ^,p<ε1-ε2-uε2ε1-pp∂p∂θ^,ε1-ε2≤p≤ε10,p>ε1,ε1, ε2, κ1, and κ2 are all positive constants, and(27)u=2κ2eκ2ε1-ε22eκ2ε1-ε22-1+κ12.Define the general-image error ase=Q-Qd=[x-xdy-yd]T. The observer is proposed as(28)θ^˙=Γ-1MTe+eTeN∂Uθ^∂θ^,which can be considered as the update law of the estimated parameters θ^. The regressor matrix M is calculated from (24). The second term in (28) serves as a repulsive force which can push θ^ away from the singular point of matrix H^(Q,θ^). Γ,N∈R21×21 are all positive gain symmetric matrices.Inspired by [24, 25], a general-image based filter Φ˙=-βΦ-Ke is proposed, where Φ=[Φ1Φ2]T can be regarded as a pseudoerror which is used to compensate for the general-image’s velocity caused by leader’s motion. β,K∈R2×2 are positive definite and diagonal gain matrices. The controller is proposed as(29)υfωf=H^-1-αe+KΠ-c∂U∂θ^e,where H^-1 is the inverse of the matrix H^(Q,θ^). Π=[tanh(δ1Φ1)tanh(δ2Φ2)]T, and c,α∈R2×2 are positive-definite and diagonal gain matrices. δ1 and δ2 are positive gains.


## 3.2. Stability Analysis

Theorem 5.
Utilizing the controller (29) and the observer (28), the general-image error e is uniform semiglobal practical asymptotic stability (USPAS), which means USPAS of image error. Moreover, the image-based leader-follower system with omnidirectional camera is stable under Assumptions 2 and 3 and the initial relative heading φ is bounded away from ±π, υlZPf>0.Proof.
The Lyapunov function is proposed as(30)V=V1+V2,V1=12eTe+TTΛT+12θ~TΓθ~,(31)V2=12γ2,where T=[ln⁡cosh⁡(δ1Φ1)ln⁡cosh⁡(δ2Φ2)]T, Λ=diag[1/δ11/δ2], and θ~=θ^-θ. The differential of V1 is(32)V˙1=eTe˙-βΦ+KeTΠ+θ~TΓθ^˙,where e˙=Q˙. Equation (23) can be refined as(33)e˙=-H^-Hυfωf+H^υfωf+F.Substituting (29) into (33), e˙ can be rewritten as(34)e˙=-H^-Hυfωf+-αe+KΠ-c∂U∂θ^e+F.Substituting (34) and the observer (28) into (32), (32) can be revised as(35)V˙1=-eTαe+eTF-βΦTΠ-eT(H^-H)υfωf+θ~TMTe+θ~TeTeN∂U∂θ^-c∂U∂θ^eTe.According to (24), -eT(H^-H)υfωf=-θ~TMTe. Then, (35) can be revised as(36)V˙1≤-λmin⁡(α)-2Fee2-βΦTΠ-eTc-θ~TNI2×2∂U∂θ^e,where βΦTΠ≥0 is an odd function; thus the second term in (36) is nonpositive. There is a small positive ε, which can be adjusted by the gains α, making sure that λmin⁡(α)-2F/ε=0. If e>ε, λmin⁡(α)-2F/ε>0; thus the first term in (36) is negative. Moreover, the gain matrix c should satisfy c≥λmax⁡(N)θ~I2×2; thus the third term in (36) is nonpositive. Therefore, when e>ε, V˙1<0. The scalar of ε can be arbitrarily decreased. So e is USPAS, according to [26].
The differential of (31) is(37)V˙2=γγ˙=γωl-ωf.Due to USPAS of e, [XPfZPf]T is also USAPS. The differential of [XPfZPf]T is(38)X˙PfZ˙Pf=0ZPf-1-XPfυfWωfW+-sγ-dzcγ-dxsγcγ-dzsγ+dxcγυlWωlW,where(39)ωf=υlZPfsγ+1ZPfX˙Pf+1ZPfnzcγ+nxsγωl.Substituting (39) into (37), (37) can be rewritten as(40)γ˙=ωl-ωf=-υlZPfsγ+1ZPfZPf-nzcγ-nxsγωl-1ZPfX˙Pf︸η1.The nominal system γ˙=-(υl/ZPf)sγ is exponentially stable when υlZPf>0 is satisfied. Obviously, η1 is bounded. Therefore, the perturbed system (40) is stable, and γ is bounded when γ(0)≠±π based on the stability of perturbed system [27].
In summary,e is USPAS, which is equivalent to USPAS of image error. Also, the relative heading γ is bounded. Therefore, the image-based leader-follower system with omnidirectional camera is stable.


## 4. Simulation Results

In this section, the simulation results are presented to validate the performance of the proposed algorithm.The nonholonomic two-wheeled mobile robot is used to achieve the simulation. An omnidirectional camera is fixed on the follower, and the follower detects a feature point fixed on the leader. The simulation is based on the kinematics of the vehicles. The camera can detect the object instantaneously and the robot can respond to the control instantly.The upper bounds of leader’s linear and angular velocity arev-l=0.2m/s and ω-l=0.1rad/s. The coordinate of the feature point with respect to Fl is n=[0m0.3m0m]T. Two followers are introduced in the simulation. The mirror types are hyperbolic, and their parameters are both p=0.01225 and d=0.08297. The camera intrinsic parameters are σ1=σ2=800, y10=320, y20=240. The transfer angles are both φ1=-5.7°,  φ2=-5.7°, and φ3=24.1°, respectively. The coordinates of OC with respect to Ff are m=[0.05m0m0.2m]T. The control gains are both α=diag[11,11]β=diag[15,15]K=diag[11,11] and δ1=δ2=8. The observer gains are =diag[10-4,10-4,10-2,10-4,  10-4,10-2,10-4,10-4, 10-2,10-4,10-4,10-2,10-4,10-4,10-2, 10-4,10-4,10-2,10-2,10-2,10-2], c=10-4I2×2, N=10-4I21×21. The initial estimated parameters are both chosen as θ(0)=[0,0,-1,0,0,-1,0,0,-0.1,0,0,0.1,0,0,-0.1,0,0, -0.01,0.1,-0.1,1]T. The initial pose of the leader is XlW=-0.8m,  YlW=0.8m,  ϕl=90°; the initial poses of two followers are Xf1W=0m,  Yf1W=0m,  ϕf1=135°,  Xf2W=1m,  Yf2W=0m,  ϕf2=180°, respectively. So the initial image points on Follower1’s and Follower2’s image plane are [371417]T and [464385]T, respectively. The desired positions of the leader relative to frame Follower1 and frame Follower2 are Xpdf1=0.7m, Zpdf1=0.8m and Xpdf2=-0.7m,  Zpdf2=0.8m, respectively. Then the desired image points on Follower1 and Follower2’s image plane are [466307]T and [223406]T, respectively.In the first simulation, the leader runs in a straight line with0.2m/s. The results can be found in Figure 4. In the second simulation, the leader runs along a circle with 0.2m/s and 0.1rad/s. The results can be found in Figure 5. In the third simulation, the leader runs along an arbitrary trajectory with the variational linear and angular velocity (Figure 6). The results can be found in Figure 6. The results all validated Theorem 5.Figure 4
Line tracking results. (a) Trajectory in world frame. (b) Image trajectory of Follower1. (c) Image trajectory of Follower2. (d) Image error of Follower1. (e) Image error of Follower2. (f) Position error in frame Follower1. (g) Position error in frame Follower2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 5
Circle tracking results. (a) Trajectory in world frame. (b) Image trajectory of Follower1. (c) Image trajectory of Follower2. (d) Image error of Follower1. (e) Image error of Follower2. (f) Position error in frame Follower1. (g) Position error in frame Follower2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)Figure 6
Arbitrary trajectory tracking results. (a) Trajectory in world frame. (b) Image trajectory of Follower1. (c) Image trajectory of Follower2. (d) Image error of Follower1. (e) Image error of Follower2. (f) Position error in frame Follower1. (g) Position error in frame Follower2.
(a)
(b)
(c)
(d)
(e)
(f)
(g)The simulation results above have validated the performance of the adaptive image-based leader-follower approach with the on-board omnidirectional camera. The image errors converge to approximate zero at last. The results have shown the convergence of the leader-follower formation error in three different situations. Furthermore, the results also validate the feasible adaptive algorithm by which the unknown extrinsic parameters as well as the unknown motion plane of the feature point can be estimated online.


## 5. Conclusions

In this paper, a new adaptive image-based and omnidirectional camera based controller independent of the leader’s velocity, as well as an observer estimating the unknown extrinsic parameters of the camera and the unknown motion plane of the feature point, have been proposed. The Lyapunov method is used to prove the USPAS of the image error and the formation error. The simulation results have validated the feasible performance of the algorithm. Future work will focus on the experiment in real environment.



---

*Source: 194251-2015-08-27.xml*