# Adaptive Visually Servoed Tracking Control for Wheeled Mobile Robot with Uncertain Model Parameters in Complex Environment

Authors: Fujie Wang; Yi Qin; Fang Guo; Bin Ren; John T. W. Yeow
Journal: Complexity
 (2020)

---

## Abstract

This paper investigates the stabilization and trajectory tracking problem of wheeled mobile robot with a ceiling-mounted camera in complex environment. First, an adaptive visual servoing controller is proposed based on the uncalibrated kinematic model due to the complex operation environment. Then, an adaptive controller is derived to provide a solution of uncertain dynamic control for a wheeled mobile robot subject to parametric uncertainties. Furthermore, the proposed controllers can be applied to a more general situation where the parallelism requirement between the image plane and operation plane is no more needed. The overparameterization of regressor matrices is avoided by exploring the structure of the camera-robot system, and thus, the computational complexity of the controller can be simplified. The Lyapunov method is employed to testify the stability of a closed-loop system. Finally, simulation results are presented to demonstrate the performance of the suggested control.


---

## Body

## 1. Introduction

In recent decades, the wheeled mobile robots (WMRs) have received increasing attention due to their promising applications in transportation, health care, security, and so on, which promotes the research of high-accuracy tracking control and stability analysis of the WMRs [1–4]. Particularly, WMR belongs to the nonholonomic mechanical system which is unable to be stabilized at one equilibrium by means of continuous and static state feedback controller [5–7], leading to the great complexity of the study about WMRs. A significant direction of the motion control of WMR is to employ various kinds of sensors in a closed-loop controller. The visual sensor, one of the typical noncontact sensors, has particular advantages such as abundant visual information and high efficiency; hence, visual servoing control of WMR has become a vigorous research field worldwide.Numerous scientific achievements have been reported on visual servoing and vision-based manipulations [8, 9]. Just like the robot manipulators, the vision system in a mobile robot can be formed by two kinds of configurations, namely, eye-in-hand configuration [10, 11] and fixed-camera configuration [12, 13], respectively. For the first category configuration, the camera is mounted on the end-effecter. In contrast, the camera is called a static-camera or fixed-camera configuration when the camera is located on the ceiling. Till now, there has been a plethora of prominent literature concerning the visual servoing of nonholonomic mobile robots. To mention a few, in [14], position-based visual servoing (PBVS) was employed for visual tracking between a WMR and a multi-DOF crane. In [15], a visual servoing scheme was presented for a nonholonomic mobile robot to combine the merits of PBVS and image-based visual servoing (IBVS). In [16], a novel strategy was proposed for visual servoing of a mobile robot and the difficult issue of the automatic extrinsic calibration was addressed. It should be noted that the above-mentioned works require the camera mounted on the end-effector to be tediously calibrated beforehand. Unfortunately, the controllers are very sensitive to camera calibration errors which may give rise to reduced accuracy. To obviate this limitation, the uncalibrated camera system has emerged as a valid tool for practical systems. In [17], two independent uncalibrated cameras were used to accomplish person tracking for a vision-based mobile robot subject to nonholonomic constraint. The authors in [18] addressed a visual servo regulation approach which can work well without the perfectly calibrated camera. To deal with the imperfect calibration of the camera, the visual servoing of nonholonomic mobile robots was proposed in [19], considering both unknown extrinsic parameters and unknown depth from the camera to the motion plane. In [20], without calibrating the camera, the eye-in-hand visual trajectory tracking control strategy was constructed to ensure that the WMR is able to track the desired trajectory.The aforesaid papers mainly discuss the visual servoing of nonholonomic mobile robots with eye-in-hand configuration. The fixed-camera configuration has the global sight and it enables the camera system to keep the observed object always in the field of view. Therefore, many researchers also devote themselves to the solutions of a WMR with the fixed uncalibrated camera. For instance, in [21], the unified tracking and regulation WMR visual servoing control was studied and the state information can be utilized to formulate the WMR kinematic model. In [22], a monocular camera with a fixed position and orientation was used to track the desired trajectory for a WMR and the controller does not require the camera to be mounted. Taking the limited velocity of a WMR into account, the control scheme for tracking a moving target by a WMR was presented in [23]. Despite the significant progress of visual servoing with the fixed uncalibrated camera, the adaptability of these controllers is unsatisfactory since the camera plane is always required to be parallel to the motion plane of the robots. It means that the controllers in [21–23] are no longer effective when the camera is fixed at a general orientation on the ceiling. To overcome this drawback, the authors in [24, 25] proposed the visual servoing of a mobile robot without the parallelism requirement. By employing an adaptive image-based visual servoing approach, the camera image plane and the motion plane of WMRs are free from position constraint. However, all these methods suffer from the overparameterization in the process of the decoupled linear transformation. In addition, the previous controllers are developed via a kinematics-based model and the nonlinear dynamics are not taken into consideration in controller design.Dynamic model-based control methods [26–29] reflect the motion of real mobile robots with significant dynamics characterized by mass and inertia as well as friction, which are otherwise not considered in kinematics-based model control. The nonlinear dynamics of the mobile robot usually contain uncertain and time-varying parameters. Consequently, the nonlinear dynamic controllers to deal with unmodeled robot dynamics diverse further research. Control methodologies such as adaptive control technique [6], sliding mode control technique [27], and neural network control technique [28] have been developed on dynamic model with uncertain parameters of mobile robots. By far, visual servoing control for mobile robots at the dynamic level can be found in [8, 30–32]. In [32], position/orientation tracking control of WMRs via an uncalibrated camera was considered and the adaptive controller was designed to compensate for the dynamic and the camera system uncertainties. It is noteworthy that the preceding studies are confined to visual servoing of mobile robots based on dynamic model, and these methods are invalid in a more general situation where the uncalibrated camera is fixed at an arbitrary position. Additionally, overparameterization limits the applicability of these controllers to a great extent.In this paper, the stabilization and trajectory tracking problems of a wheeled mobile robot in complex environments are studied. The main contributions of this paper are threefold:(1)
Two visual servoing controllers are proposed to stabilize a wheeled mobile robot with a ceiling-mounted camera and the desired trajectory tracking can be realized. First, an adaptive visual servoing controller is proposed based on the kinematic model. Then, an adaptive controller is derived to provide a solution of uncertain dynamic wheeled mobile robot subject to parametric uncertainties related to the camera system.(2)
An uncalibrated visual servoing control strategy is proposed to realize trajectory tracking of a WMR, whose major superiority lies in the avoidance of both the requirement that the camera plane must be parallel to the motion plane of the robots and the overparameterization as in [24, 25]. Such a solution allows the controllers to be applied in a more general situation with a simpler structure and higher efficiency.(3)
In comparison with the existing works for visual servoing mobile robot control in [19, 33], the camera parameters, including the intrinsic and extrinsic parameters, are unnecessary to be well calibrated, and the tracking control can be ensured in the presence of uncertain dynamics.


## 2. Preliminaries and System Descriptions

Throughout this paper, a typical setup for the visually servoed wheeled mobile robot is considered, as shown in Figure1, where the camera is mounted on the ceiling to observe the movement of feature point labeled on the mobile robot. Let ObXbYbZb be the base coordinate frame, OcXcYcZc be the camera coordinate frame, and OmXmYmZm be the mobile robot coordinate frame, respectively. Furthermore, let Om be the center of mass of wheeled mobile robot, P be the feature point, and d be the distance from Om to P along the positive direction of axis Xm. Without loss of generality, it is assumed that the robot moves in a specific plane. Note that both the image-based kinematic and dynamic control are fully considered in this paper.Figure 1
Visually servoed wheeled mobile robot system and coordinate representation.


### 2.1. Kinematics Model of Nonholonomic Mobile Robot in Task Space

Let us firstly review the kinematics model of a mobile robot. Denote the task-space position of wheeled mobile robot with respect to the base coordinate frame byxB,0T=xb,yb,0T and the orientation by θ, whose forward rotation direction is set to counterclockwise from axis Xb. Then, the kinematic model of the mobile robot can be written as [32, 34](1)x˙B=x˙by˙b=νcosθνsinθ(2)θ.=ω,where ν and ω denote the linear velocity and angular velocity of wheeled mobile robot in task space, respectively. From [32], the nonholonomic constraint of wheeled mobile robot can be formulated as follows:(3)x˙bsinθ−y˙bcosθ=0.This nonholonomic constraint indicates that the velocity along the connected direction between the left and right driving wheels is restricted to be zero; that is, the wheeled mobile robot will not slip during task execution. Combining the definition ofθ and mobile robot kinematics, the task-space position of P with respect to the base coordinate frame can be described as [24](4)x0=xpyp0=xb+dcosθyb+dsinθ0.Differentiating (4) with respect to time gives rise to(5)x˙=cosθ−dsinθsinθdcosθνω.


### 2.2. Transformation from Task Space to Image Space

Lety∈ℝ2 be the position of feature point P on the image plane. Via the perspective projection model [8, 35], the mapping relation of P from task space to image space is given by(6)Y1=1zDx01,where z is the depth information of feature point, D=ΩT∈ℝ3×4 is the so-called perspective projection matrix (see [8]), T∈ℝ4×4 denotes the homogenous transformation matrix from the base frame to camera frame, and Ω∈ℝ3×4 denotes the internal transformation matrix of camera. It should be noted that Ω and T depend on the intrinsic and extrinsic parameters, respectively. In addition, the depth information is defined as(7)z=D3Tx01,where D3T denotes the 3rd row of matrix D. Differentiating (6) and utilizing the definition of depth, we can obtain(8)y˙=1zD¯2×2−yD¯3Tx˙=1zD¯2×2−yD¯3Tcosθ−dsinθsinθdcosθ︸Ny,θνω︸τK,where τK can be interpreted as the kinematic control input and D¯=D¯2×2T,D¯3T∈ℝ3×2 is the left 3×2 submatrix of D. Note that Ny,θ∈ℝ2×1 depends on both the intrinsic and extrinsic parameters of the visual model. In addition, Ny,θ is called the depth-independent interaction matrix since the depth information z is separated. By exploiting the structure of Ny,θ, we can further obtain(9)Ny,θ=D¯2×2−yD¯3Tcosθ−dsinθsinθdcosθ=D¯2×2cosθ−dsinθsinθdcosθ︸Naθ−yD¯3Tcosθ−dsinθsinθdcosθ︸Nbθ.Similarly, the time differential of depth informationz can be written as(10)z˙=D¯3Tx˙=D¯3Tcosθ−dsinθsinθdcosθ︸NbθτK.The linearization properties, which are important to simplify the control design, are given as follows [34, 35].Property 1.
The products ofNaθζ and ΞNbθζ can be linearly decomposed and recombined as(11)Naθζ=EK,aθ,ζϕk,a,(12)ΞNbθζ=EK,bθ,ζ,ξϕk,b,where ζ∈ℝ2×1 is a constant vector, Ξ=diagξ∈ℝ2×2 is a diagonal matrix with ξ=ξ1,ξ2T, ϕk,a∈ℝe1×1 and ϕk,b∈ℝe2×1 are visual model parameter vector, and EK,aθ,ζ∈ℝ2×e1 and EKbθ,ζ,ξ∈ℝ2×e2 are the regressor matrices without depending on the parameter vectors ϕk,a and ϕk,b. Specifically, by observing (7), (9), and (10), it can be further obtained as follows:(13)z˙ξ=ΞNbθτK=EK,bθ,τK,ξϕk,b,(14)zξ=E¯K,bθ,x,ξϕk,b,where E¯Kbθ,x,ζ∈ℝ2×e2 is the depth regressor matrix. Note that the vector ϕk,b should involve all the depth parameters. By employing (9), (11), and (12), we have(15)Ny,θζ=EK,aθ,ζϕk,a+EK,bθ,ζ,−yϕk,b.Remark 1.
In this paper, parameter uncertainties of visual servoing robot system are addressed, which means that the real parameter valuesϕk,a and ϕk,b in (11)–(15) are unknown in the control design. Moreover, the image depth is not required to be consistent during robot operation as in [25, 36]; that is, the fixed-camera image plane can be not parallel to the operation plane, where a more realistic scenario is considered in both kinematic and dynamic control. In addition, the distance d between the feature point and the origin of the coordinate system OmXmYmZm is assumed to be uncalibrated, which, together with the above parameter uncertainties, imposes great complexity and challenge in visual tracking control.
Throughout this paper, the following assumptions hold.Assumption 1.
The feature pointP can always be detected throughout the entire robot workspace such that the image position is continuously available. Moreover, the orientation θ of mobile robot can be measured by the encoders or other optical sensors mounted on the actuators.


### 2.3. Dynamics Model of Nonholonomic Mobile Robot

The dynamic behavior of wheeled mobile robot can be expressed by the Euler-Lagrangian equation as follows [6, 37]:(16)Mθq¨+Vθ,θ˙q˙+G=BθτD+ATθλ,where q=xb,yb,θT∈ℝ3×1, Mθ∈ℝ3×3 is the symmetric and positive-definite inertia matrix, Vθ,θ⋅∈ℝ3×3 is the Coriolis and centrifugal matrix, G∈ℝ3×1 denotes the gravitational force, Bθ∈ℝ3×2 denotes the input transformation matrix, τD∈ℝ2×1 represents the dynamic input torque, Aθ∈ℝ1×3 is the so-called constraint vector with λ being the constraint force, and the constraint form can be further represented as(17)Aθq˙=0.It must be noted that matricesMθ, Vθ,θ., G, Bθ, and Aθ do not depend on the actual position of xb and yb (more details of the robotic dynamics model can be referred to [37]). Based on the kinematics (1) and (2), the following holds:(18)q˙=cosθ0sinθ001︸SθτK.Differentiating both sides of (18) and then substituting into the robot dynamics (16) and premultiplying both sides by STθ, we have(19)M¯θτ˙K+V¯θ,θ.τK+G¯θ=B¯θτD,where (17) is utilized in the process of formula simplification and M¯θ=STθMθSθ, V¯θ,θ.=STθMθS˙θ+Vθ,θ.Sθ, G¯θ=STθGθ, and B¯θ=STθBθ, respectively. To facilitate the control scheme, the dynamics properties of WMR are employed [37].Property 2.
The inertia matrixM¯θ is symmetric and positive definite, which also satisfies(20)μ1θ2≤θTM¯θθ≤μ2θ2,where μ1 and μ2 are positive constants and ⋅ denotes the standard Euclidean norm.Property 3.
The matrixM¯˙θ−2V¯θ,θ. is skew-symmetric such that(21)ℵTM¯˙θ−2V¯θ,θ.ℵ=0,with ℵ∈ℝ2×1 being a constant vector.Property 4.
The dynamic equation (19) can be linearly restructured as(22)M¯θρ˙+V¯θ,θ.ρ+G¯θ=EDθ,θ.,ρ,ρ˙ϕd,where ρ∈ℝ2×1 is a differentiable vector, ϕd∈ℝe3×1 denotes the constant parameter vector of dynamics and is unknown in the control design, and EDθ,θ.,ρ,ρ˙∈ℝ2×e3 is the regressor matrix of dynamics.Remark 2.
Via observing (8) and (19), it can be found that the kinematic control and the dynamic control are related by τK and τD, respectively. If the designed kinematic input τK is actually achievable in the task execution without any time delay, the visual tracking control can be conveniently realized by the kinematic loop. However, in most state-of-the-art researches on wheeled mobile robot control [19, 32, 33, 38], it is stressed that the motors assembled on the left and right wheels may not respond fast enough with the result that the actual kinematic control values τK may lag behind the design values. Thus, in this paper, the dynamics control for visual servoing WMR is also addressed, simultaneously taking the mechanical parameter uncertainties into consideration; that is, the precise parameter values (e.g., robot mass, inertia, and friction) are not required to be exactly measured.


### 2.4. Problem Statement

Based on the above system model and assumptions, the control problems from two different perspectives, namely, the kinematic and dynamic control, are addressed. Given a continuous desired trajectoryyd,y˙d,y¨d∈ℝ2×1 on the image plane, this paper aims to solve the following problems:P1L: assuming that the WMR responds fast enough, design an adaptive visual servoing kinematic controller (AVSKC) τK such that the precise trajectory tracking performance can be obtained in the absence of calibrated camera model; that is,(23)limt⟶∞y−yd⟶0.P2: when the kinematic input τK is not always achievable, design an adaptive visual servoing dynamic controller (AVSDC) τD such that (23) holds, simultaneously taking into account the uncalibrated camera-robot model.


## 2.1. Kinematics Model of Nonholonomic Mobile Robot in Task Space

Let us firstly review the kinematics model of a mobile robot. Denote the task-space position of wheeled mobile robot with respect to the base coordinate frame byxB,0T=xb,yb,0T and the orientation by θ, whose forward rotation direction is set to counterclockwise from axis Xb. Then, the kinematic model of the mobile robot can be written as [32, 34](1)x˙B=x˙by˙b=νcosθνsinθ(2)θ.=ω,where ν and ω denote the linear velocity and angular velocity of wheeled mobile robot in task space, respectively. From [32], the nonholonomic constraint of wheeled mobile robot can be formulated as follows:(3)x˙bsinθ−y˙bcosθ=0.This nonholonomic constraint indicates that the velocity along the connected direction between the left and right driving wheels is restricted to be zero; that is, the wheeled mobile robot will not slip during task execution. Combining the definition ofθ and mobile robot kinematics, the task-space position of P with respect to the base coordinate frame can be described as [24](4)x0=xpyp0=xb+dcosθyb+dsinθ0.Differentiating (4) with respect to time gives rise to(5)x˙=cosθ−dsinθsinθdcosθνω.


## 2.2. Transformation from Task Space to Image Space

Lety∈ℝ2 be the position of feature point P on the image plane. Via the perspective projection model [8, 35], the mapping relation of P from task space to image space is given by(6)Y1=1zDx01,where z is the depth information of feature point, D=ΩT∈ℝ3×4 is the so-called perspective projection matrix (see [8]), T∈ℝ4×4 denotes the homogenous transformation matrix from the base frame to camera frame, and Ω∈ℝ3×4 denotes the internal transformation matrix of camera. It should be noted that Ω and T depend on the intrinsic and extrinsic parameters, respectively. In addition, the depth information is defined as(7)z=D3Tx01,where D3T denotes the 3rd row of matrix D. Differentiating (6) and utilizing the definition of depth, we can obtain(8)y˙=1zD¯2×2−yD¯3Tx˙=1zD¯2×2−yD¯3Tcosθ−dsinθsinθdcosθ︸Ny,θνω︸τK,where τK can be interpreted as the kinematic control input and D¯=D¯2×2T,D¯3T∈ℝ3×2 is the left 3×2 submatrix of D. Note that Ny,θ∈ℝ2×1 depends on both the intrinsic and extrinsic parameters of the visual model. In addition, Ny,θ is called the depth-independent interaction matrix since the depth information z is separated. By exploiting the structure of Ny,θ, we can further obtain(9)Ny,θ=D¯2×2−yD¯3Tcosθ−dsinθsinθdcosθ=D¯2×2cosθ−dsinθsinθdcosθ︸Naθ−yD¯3Tcosθ−dsinθsinθdcosθ︸Nbθ.Similarly, the time differential of depth informationz can be written as(10)z˙=D¯3Tx˙=D¯3Tcosθ−dsinθsinθdcosθ︸NbθτK.The linearization properties, which are important to simplify the control design, are given as follows [34, 35].Property 1.
The products ofNaθζ and ΞNbθζ can be linearly decomposed and recombined as(11)Naθζ=EK,aθ,ζϕk,a,(12)ΞNbθζ=EK,bθ,ζ,ξϕk,b,where ζ∈ℝ2×1 is a constant vector, Ξ=diagξ∈ℝ2×2 is a diagonal matrix with ξ=ξ1,ξ2T, ϕk,a∈ℝe1×1 and ϕk,b∈ℝe2×1 are visual model parameter vector, and EK,aθ,ζ∈ℝ2×e1 and EKbθ,ζ,ξ∈ℝ2×e2 are the regressor matrices without depending on the parameter vectors ϕk,a and ϕk,b. Specifically, by observing (7), (9), and (10), it can be further obtained as follows:(13)z˙ξ=ΞNbθτK=EK,bθ,τK,ξϕk,b,(14)zξ=E¯K,bθ,x,ξϕk,b,where E¯Kbθ,x,ζ∈ℝ2×e2 is the depth regressor matrix. Note that the vector ϕk,b should involve all the depth parameters. By employing (9), (11), and (12), we have(15)Ny,θζ=EK,aθ,ζϕk,a+EK,bθ,ζ,−yϕk,b.Remark 1.
In this paper, parameter uncertainties of visual servoing robot system are addressed, which means that the real parameter valuesϕk,a and ϕk,b in (11)–(15) are unknown in the control design. Moreover, the image depth is not required to be consistent during robot operation as in [25, 36]; that is, the fixed-camera image plane can be not parallel to the operation plane, where a more realistic scenario is considered in both kinematic and dynamic control. In addition, the distance d between the feature point and the origin of the coordinate system OmXmYmZm is assumed to be uncalibrated, which, together with the above parameter uncertainties, imposes great complexity and challenge in visual tracking control.
Throughout this paper, the following assumptions hold.Assumption 1.
The feature pointP can always be detected throughout the entire robot workspace such that the image position is continuously available. Moreover, the orientation θ of mobile robot can be measured by the encoders or other optical sensors mounted on the actuators.


## 2.3. Dynamics Model of Nonholonomic Mobile Robot

The dynamic behavior of wheeled mobile robot can be expressed by the Euler-Lagrangian equation as follows [6, 37]:(16)Mθq¨+Vθ,θ˙q˙+G=BθτD+ATθλ,where q=xb,yb,θT∈ℝ3×1, Mθ∈ℝ3×3 is the symmetric and positive-definite inertia matrix, Vθ,θ⋅∈ℝ3×3 is the Coriolis and centrifugal matrix, G∈ℝ3×1 denotes the gravitational force, Bθ∈ℝ3×2 denotes the input transformation matrix, τD∈ℝ2×1 represents the dynamic input torque, Aθ∈ℝ1×3 is the so-called constraint vector with λ being the constraint force, and the constraint form can be further represented as(17)Aθq˙=0.It must be noted that matricesMθ, Vθ,θ., G, Bθ, and Aθ do not depend on the actual position of xb and yb (more details of the robotic dynamics model can be referred to [37]). Based on the kinematics (1) and (2), the following holds:(18)q˙=cosθ0sinθ001︸SθτK.Differentiating both sides of (18) and then substituting into the robot dynamics (16) and premultiplying both sides by STθ, we have(19)M¯θτ˙K+V¯θ,θ.τK+G¯θ=B¯θτD,where (17) is utilized in the process of formula simplification and M¯θ=STθMθSθ, V¯θ,θ.=STθMθS˙θ+Vθ,θ.Sθ, G¯θ=STθGθ, and B¯θ=STθBθ, respectively. To facilitate the control scheme, the dynamics properties of WMR are employed [37].Property 2.
The inertia matrixM¯θ is symmetric and positive definite, which also satisfies(20)μ1θ2≤θTM¯θθ≤μ2θ2,where μ1 and μ2 are positive constants and ⋅ denotes the standard Euclidean norm.Property 3.
The matrixM¯˙θ−2V¯θ,θ. is skew-symmetric such that(21)ℵTM¯˙θ−2V¯θ,θ.ℵ=0,with ℵ∈ℝ2×1 being a constant vector.Property 4.
The dynamic equation (19) can be linearly restructured as(22)M¯θρ˙+V¯θ,θ.ρ+G¯θ=EDθ,θ.,ρ,ρ˙ϕd,where ρ∈ℝ2×1 is a differentiable vector, ϕd∈ℝe3×1 denotes the constant parameter vector of dynamics and is unknown in the control design, and EDθ,θ.,ρ,ρ˙∈ℝ2×e3 is the regressor matrix of dynamics.Remark 2.
Via observing (8) and (19), it can be found that the kinematic control and the dynamic control are related by τK and τD, respectively. If the designed kinematic input τK is actually achievable in the task execution without any time delay, the visual tracking control can be conveniently realized by the kinematic loop. However, in most state-of-the-art researches on wheeled mobile robot control [19, 32, 33, 38], it is stressed that the motors assembled on the left and right wheels may not respond fast enough with the result that the actual kinematic control values τK may lag behind the design values. Thus, in this paper, the dynamics control for visual servoing WMR is also addressed, simultaneously taking the mechanical parameter uncertainties into consideration; that is, the precise parameter values (e.g., robot mass, inertia, and friction) are not required to be exactly measured.


## 2.4. Problem Statement

Based on the above system model and assumptions, the control problems from two different perspectives, namely, the kinematic and dynamic control, are addressed. Given a continuous desired trajectoryyd,y˙d,y¨d∈ℝ2×1 on the image plane, this paper aims to solve the following problems:P1L: assuming that the WMR responds fast enough, design an adaptive visual servoing kinematic controller (AVSKC) τK such that the precise trajectory tracking performance can be obtained in the absence of calibrated camera model; that is,(23)limt⟶∞y−yd⟶0.P2: when the kinematic input τK is not always achievable, design an adaptive visual servoing dynamic controller (AVSDC) τD such that (23) holds, simultaneously taking into account the uncalibrated camera-robot model.


## 3. Adaptive Visual Servoing Kinematic Control for Wheeled Mobile Robot under Uncalibrated Visual Model

In this part, we focus on the adaptive visual servoing kinematic control scheme for wheeled mobile robot with uncalibrated camera model, where the projection plane of camera does not need to be parallel to the operation plane during the execution of the mission, and the dynamic control will be exhibited in the next section. Since the parameters of the visual model are unknown, adaption laws are presented to estimate the real parameter values, and based on the estimated parameters, AVSKC is developed to realize the asymptotic image trajectory tracking.


### 3.1. Controller Design

LetN^y,θ, z^, and z˙^ be the estimated values of Ny,θ, z, and z˙ by replacing the unknown parameters ϕk,a and ϕk,b in Ny,θ, z, and z˙ with the estimations ϕ^k,a and ϕ^k,b, respectively, and the estimations are offered by the adaption laws. Define Δy=y−yd as the image error. Then, inspired by [25], the AVSKC is designed as(24)τK=N^−1y,θz^y˙d−α+12z˙^Δy,where α is a positive constant. In (24), the estimated visual model rather than the calibrated model is utilized, and the estimated depth and its differential are also introduced to compensate the model error since the image plane and the operation plane are nonparallel. Now, we can further analyze the closed-loop kinematics with depth information as follows:(25)zΔy˙+12z˙Δy=Ny,θτK−zy˙d+12z˙Δy=Ny,θτK−N^y,θτK+N^y,θτK+zy˙d−z^y˙d+z^y˙d+12z˙Δy−z˙^Δy+12z˙^Δy=EK,aθ,τKΔϕk,a+EK,bθ,τK,−yΔϕk,b+E¯K,bθ,x,τKΔϕk,b+EK,bθ,τK,12ΔyΔϕk,b+N^y,θτK+z^y˙d+12z˙^Δy,where Δϕk,a=ϕk,a−ϕ^k,a and Δϕk,b=ϕk,b−ϕ^k,b, and Property 1 is used. Substituting the AVSKC (24) into (25) gives rise to(26)zΔy˙+12z˙Δy=EK,aθ,τKΔϕk,a−αΔy+EK,bθ,τK,−y+yd2+E¯K,bθ,x,τKΔϕk,b.


### 3.2. Unknown Parameter Estimation

By observing (24), it is obvious that the estimation of Ny,θ is employed, which requires that the parameters ϕ^a and ϕ^b are updated online. The kinematic parameter updating laws are presented as(27)ϕ^˙a=−Φa−1EK,aTθ,τKΔy,(28)ϕ^˙b=−Φb−1EK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy,where Φa∈ℝe1×e1 and Φa∈ℝe2×e2 are the positive-definite diagonal matrices. Thus, by integrating (27) and (28), N^y,θ, z^, and z˙^ in (24) are then available.


### 3.3. Stability Analysis

At this point, we are going to formulate the first theorem.Theorem 1.
Consider the visual servoing wheeled mobile robot represented by (1), (2), (4), (6), and (8) satisfying the assumption that the estimated interaction matrix N^y,θ is nonsingular. In the case that the design kinematic input τK is actually achievable in the task execution, the adaptive visual servoing kinematic controller (AVSKC) given by (24) together with the visual parameter adaption laws (27) and (28) ensures the global stability of (26) and the asymptotical convergence of Δy to zero such that limt⟶∞y−yd⟶0.Proof.
Construct the kinematic-based Lyapunov function candidate as(29)Vk=12zΔyTΔy+12Δϕk,aTΦaΔϕk,a+12Δϕk,bTΦbΔϕk,b.
DifferentiatingVk with respect to time yields(30)V˙k=ΔyTzΔy˙+12z˙Δy+Δϕk,aTΦaϕ^˙k,a+Δϕk,bTΦbϕ^˙k,b.
Substituting the closed-loop kinematics (26) and the parameter updating laws (27) and (28) into (30), the derivative of Vk can be denoted as(31)V˙k=ΔyTEK,aθ,τKΔϕk,a−Δϕk,aTEK,aTθ,τKΔy−ΔyTαΔy+ΔyTEK,bθ,τK,−y+yd2+E¯K,bθ,x,τKΔϕk,b−Δϕk,bTEK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy=−ΔyTαΔy≤0.
SinceVK≥0 and V˙Kt≤0, we can obtain that VKt is bounded; that is, Δy, Δϕk,a, and Δϕk,b are bounded, which directly implies that ϕ^k,a and ϕ^k,b are both bounded since ϕk,a and ϕk,b are constants. Thus, N^y,θ, z^, and z˙^ are all bounded, giving rise to the boundness of τK from (24), which means that y˙,Δy˙∈ℒ∞ from (8) and the boundness of (26) is guaranteed. From the result of (29) and (38), we have Δy∈ℒ2∩ℒ∞. Therefore, we can obtain that limt⟶∞y−yd⟶0. Thus, the proof is completed.
From the result in [34], it has been proven that the matrix Ny,θ is always nonsingular. Thus, if the parameters of N^y,θ in ϕ^k,a and ϕ^k,a are updated properly, it can be ensured that N^y,θ is full rank by modifying the parameter adaption laws. In this paper, the so-called parameter projection [39] is introduced to avoid nonsingularity of N^y,θ. The adaption laws for visual kinematic parameters are presented as(32)℧a=−Φa−1EK,aTθ,τKΔy,(33)ϕ^˙a,i=proj℧a,i,(34)℧b=−Φb−1EK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy,(35)ϕ^˙b,i=proj℧b,i,where ϕ^˙a,i, ϕ^˙b,i, ℧a,i, and ℧b,i are the ith element of ϕ^˙a, ϕ^˙b, ℧a, and ℧b. Furthermore, the projection function is given as [39, 40](36)proj℧a,i=℧a,i,ifϕ^a,i>ϕ¯a,i℧a,i,ifϕ^a,i=ϕ¯a,i,℧a,i≥0,℧a,i,ifϕ^a,i=ϕ¯a,i,℧a,i≤0,℧a,i,ifϕ^a,i<ϕ¯a,i,0,otherwise,(37)proj℧b,i=℧b,i,ifϕ^b,i>ϕ¯b,i,℧b,i,ifϕ^b,i=ϕ¯b,i,℧b,i≥0,℧b,i,ifϕ^b,i=ϕ¯b,i,℧b,i≤0,℧b,i,ifϕ^b,i<ϕ¯b,i,0,otherwise,where ⋅¯ and ⋅¯ denote the upper and lower bounds of parameter ⋅. In this way, if the condition ⋅¯≤⋅^0≤⋅¯ is satisfied, then the parameter ⋅^ will locate in the region ⋅¯,⋅¯. Then, we are ready to state the following proposition.Proposition 1.
Consider the visual servoing wheeled mobile robot represented by (1), (2), (4), (6), and (8). In the case that τK is achievable, the adaptive visual servoing kinematic controller (AVSKC) given by (24) together with the visual parameter adaption laws (33) and (35) ensures the global stability of (26) and the asymptotical convergence of image errors such that limt⟶∞y−yd⟶0.Proof.
Choose the same Lyapunov function candidateVk (29), whose time derivative can be written as(38)V˙k=−ΔyTαΔy+Δϕk,aTΦaϕ^˙k,a−℧a+Δϕk,bTΦbϕ^˙k,b−℧b≤−ΔyTαΔy.
Thus, the proof of Proposition1 can be referred to that of Theorem 1.Remark 3.
It is noted that the considerations on visual kinematic model in this paper are similar to that in [19, 24, 25, 32, 33, 36], where the visual model parameters are uncalibrated in kinematic control design. However, the projection plane is set to be parallel to the operation plane in [32]. Also, note that eye-in-hand configuration is addressed in [19, 33] rather than the eye-to-hand setup in this paper, and in addition, only partial camera intrinsic parameters are taken into consideration in [33], and only the extrinsic camera parameters are considered in [19], respectively. Furthermore, the overparametrization problem is still unresolved in [24, 25], where a 2×14 regressor matrix needs to be determined. Extra two particular feature points are introduced in [36] for the purpose of preventing the direct use of image Jacobian matrix from control design. The major difference between the proposed AVSKC scheme and the uncalibrated visual tracking control scheme [19, 24, 25, 32, 33, 36] is that the parameters in image projection matrix, including the intrinsic and extrinsic camera parameters, are estimated by adaption laws while avoiding the overparametrization problem. This is realized by exploiting the structure of image Jacobian matrix inspired by [35] (see Property 1). Specifically, the salient features of the proposed AVSKC scheme lie in (I) the structurally simple implementation of control law τK (24); (II) the inexpensive obtainment of regressor matrices EK,a and EK,b in adaption laws (27) and (28) (or using (33) and (35)); (III) the nonsingularity property of N^y,θ and the receptivity of uncertain parameters processed by adaption laws (33) and (35).Remark 4.
If the actuators of WMR perform effectively, the visual tracking on image plane can be conveniently realized by the proposed AVSKC scheme. However, it is well recognized that the presence of dynamic uncertainties will cause a great negative impact on control performance. In the next section, we will pertinently propose the dynamic control scheme for visual servoing WMR together with the handling of dynamic uncertainties.


## 3.1. Controller Design

LetN^y,θ, z^, and z˙^ be the estimated values of Ny,θ, z, and z˙ by replacing the unknown parameters ϕk,a and ϕk,b in Ny,θ, z, and z˙ with the estimations ϕ^k,a and ϕ^k,b, respectively, and the estimations are offered by the adaption laws. Define Δy=y−yd as the image error. Then, inspired by [25], the AVSKC is designed as(24)τK=N^−1y,θz^y˙d−α+12z˙^Δy,where α is a positive constant. In (24), the estimated visual model rather than the calibrated model is utilized, and the estimated depth and its differential are also introduced to compensate the model error since the image plane and the operation plane are nonparallel. Now, we can further analyze the closed-loop kinematics with depth information as follows:(25)zΔy˙+12z˙Δy=Ny,θτK−zy˙d+12z˙Δy=Ny,θτK−N^y,θτK+N^y,θτK+zy˙d−z^y˙d+z^y˙d+12z˙Δy−z˙^Δy+12z˙^Δy=EK,aθ,τKΔϕk,a+EK,bθ,τK,−yΔϕk,b+E¯K,bθ,x,τKΔϕk,b+EK,bθ,τK,12ΔyΔϕk,b+N^y,θτK+z^y˙d+12z˙^Δy,where Δϕk,a=ϕk,a−ϕ^k,a and Δϕk,b=ϕk,b−ϕ^k,b, and Property 1 is used. Substituting the AVSKC (24) into (25) gives rise to(26)zΔy˙+12z˙Δy=EK,aθ,τKΔϕk,a−αΔy+EK,bθ,τK,−y+yd2+E¯K,bθ,x,τKΔϕk,b.


## 3.2. Unknown Parameter Estimation

By observing (24), it is obvious that the estimation of Ny,θ is employed, which requires that the parameters ϕ^a and ϕ^b are updated online. The kinematic parameter updating laws are presented as(27)ϕ^˙a=−Φa−1EK,aTθ,τKΔy,(28)ϕ^˙b=−Φb−1EK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy,where Φa∈ℝe1×e1 and Φa∈ℝe2×e2 are the positive-definite diagonal matrices. Thus, by integrating (27) and (28), N^y,θ, z^, and z˙^ in (24) are then available.


## 3.3. Stability Analysis

At this point, we are going to formulate the first theorem.Theorem 1.
Consider the visual servoing wheeled mobile robot represented by (1), (2), (4), (6), and (8) satisfying the assumption that the estimated interaction matrix N^y,θ is nonsingular. In the case that the design kinematic input τK is actually achievable in the task execution, the adaptive visual servoing kinematic controller (AVSKC) given by (24) together with the visual parameter adaption laws (27) and (28) ensures the global stability of (26) and the asymptotical convergence of Δy to zero such that limt⟶∞y−yd⟶0.Proof.
Construct the kinematic-based Lyapunov function candidate as(29)Vk=12zΔyTΔy+12Δϕk,aTΦaΔϕk,a+12Δϕk,bTΦbΔϕk,b.
DifferentiatingVk with respect to time yields(30)V˙k=ΔyTzΔy˙+12z˙Δy+Δϕk,aTΦaϕ^˙k,a+Δϕk,bTΦbϕ^˙k,b.
Substituting the closed-loop kinematics (26) and the parameter updating laws (27) and (28) into (30), the derivative of Vk can be denoted as(31)V˙k=ΔyTEK,aθ,τKΔϕk,a−Δϕk,aTEK,aTθ,τKΔy−ΔyTαΔy+ΔyTEK,bθ,τK,−y+yd2+E¯K,bθ,x,τKΔϕk,b−Δϕk,bTEK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy=−ΔyTαΔy≤0.
SinceVK≥0 and V˙Kt≤0, we can obtain that VKt is bounded; that is, Δy, Δϕk,a, and Δϕk,b are bounded, which directly implies that ϕ^k,a and ϕ^k,b are both bounded since ϕk,a and ϕk,b are constants. Thus, N^y,θ, z^, and z˙^ are all bounded, giving rise to the boundness of τK from (24), which means that y˙,Δy˙∈ℒ∞ from (8) and the boundness of (26) is guaranteed. From the result of (29) and (38), we have Δy∈ℒ2∩ℒ∞. Therefore, we can obtain that limt⟶∞y−yd⟶0. Thus, the proof is completed.
From the result in [34], it has been proven that the matrix Ny,θ is always nonsingular. Thus, if the parameters of N^y,θ in ϕ^k,a and ϕ^k,a are updated properly, it can be ensured that N^y,θ is full rank by modifying the parameter adaption laws. In this paper, the so-called parameter projection [39] is introduced to avoid nonsingularity of N^y,θ. The adaption laws for visual kinematic parameters are presented as(32)℧a=−Φa−1EK,aTθ,τKΔy,(33)ϕ^˙a,i=proj℧a,i,(34)℧b=−Φb−1EK,bTθ,τK,−y+yd2+E¯K,bTθ,x,τKΔy,(35)ϕ^˙b,i=proj℧b,i,where ϕ^˙a,i, ϕ^˙b,i, ℧a,i, and ℧b,i are the ith element of ϕ^˙a, ϕ^˙b, ℧a, and ℧b. Furthermore, the projection function is given as [39, 40](36)proj℧a,i=℧a,i,ifϕ^a,i>ϕ¯a,i℧a,i,ifϕ^a,i=ϕ¯a,i,℧a,i≥0,℧a,i,ifϕ^a,i=ϕ¯a,i,℧a,i≤0,℧a,i,ifϕ^a,i<ϕ¯a,i,0,otherwise,(37)proj℧b,i=℧b,i,ifϕ^b,i>ϕ¯b,i,℧b,i,ifϕ^b,i=ϕ¯b,i,℧b,i≥0,℧b,i,ifϕ^b,i=ϕ¯b,i,℧b,i≤0,℧b,i,ifϕ^b,i<ϕ¯b,i,0,otherwise,where ⋅¯ and ⋅¯ denote the upper and lower bounds of parameter ⋅. In this way, if the condition ⋅¯≤⋅^0≤⋅¯ is satisfied, then the parameter ⋅^ will locate in the region ⋅¯,⋅¯. Then, we are ready to state the following proposition.Proposition 1.
Consider the visual servoing wheeled mobile robot represented by (1), (2), (4), (6), and (8). In the case that τK is achievable, the adaptive visual servoing kinematic controller (AVSKC) given by (24) together with the visual parameter adaption laws (33) and (35) ensures the global stability of (26) and the asymptotical convergence of image errors such that limt⟶∞y−yd⟶0.Proof.
Choose the same Lyapunov function candidateVk (29), whose time derivative can be written as(38)V˙k=−ΔyTαΔy+Δϕk,aTΦaϕ^˙k,a−℧a+Δϕk,bTΦbϕ^˙k,b−℧b≤−ΔyTαΔy.
Thus, the proof of Proposition1 can be referred to that of Theorem 1.Remark 3.
It is noted that the considerations on visual kinematic model in this paper are similar to that in [19, 24, 25, 32, 33, 36], where the visual model parameters are uncalibrated in kinematic control design. However, the projection plane is set to be parallel to the operation plane in [32]. Also, note that eye-in-hand configuration is addressed in [19, 33] rather than the eye-to-hand setup in this paper, and in addition, only partial camera intrinsic parameters are taken into consideration in [33], and only the extrinsic camera parameters are considered in [19], respectively. Furthermore, the overparametrization problem is still unresolved in [24, 25], where a 2×14 regressor matrix needs to be determined. Extra two particular feature points are introduced in [36] for the purpose of preventing the direct use of image Jacobian matrix from control design. The major difference between the proposed AVSKC scheme and the uncalibrated visual tracking control scheme [19, 24, 25, 32, 33, 36] is that the parameters in image projection matrix, including the intrinsic and extrinsic camera parameters, are estimated by adaption laws while avoiding the overparametrization problem. This is realized by exploiting the structure of image Jacobian matrix inspired by [35] (see Property 1). Specifically, the salient features of the proposed AVSKC scheme lie in (I) the structurally simple implementation of control law τK (24); (II) the inexpensive obtainment of regressor matrices EK,a and EK,b in adaption laws (27) and (28) (or using (33) and (35)); (III) the nonsingularity property of N^y,θ and the receptivity of uncertain parameters processed by adaption laws (33) and (35).Remark 4.
If the actuators of WMR perform effectively, the visual tracking on image plane can be conveniently realized by the proposed AVSKC scheme. However, it is well recognized that the presence of dynamic uncertainties will cause a great negative impact on control performance. In the next section, we will pertinently propose the dynamic control scheme for visual servoing WMR together with the handling of dynamic uncertainties.


## 4. Adaptive Visual Servoing Dynamic Control for Wheeled Mobile Robot with Uncalibrated Visual Model and Dynamics

The focus in this section is extending the wheeled mobile robot kinematic control to dynamic control in the presence of uncalibrated visual model and uncertain dynamics; that is, both the kinematics and dynamics parameters are not required to be measured accurately. Note that, in this case, the designed controller becomes the dynamic input torqueτD, in which case a deeper control loop is considered.


### 4.1. Controller Design

Define a referenced image velocity as(39)y˙r=y˙d−γΔy,where γ is a positive constant. Then, the reference errors of image velocity can be denoted as(40)ry=y˙−y˙r=Δy˙+γΔy.Thus, the reference errorsry contain both the image errors and image velocity errors. Furthermore, define a kinematic auxiliary variable(41)τr=N^−1y,θz^y˙r.Differentiating (41) with respect to time, we have(42)τ˙r=−N^−1y,θN^˙y,θN^−1y,θ+N^−1y,θz^˙y˙r+N^−1y,θz^ry.The purpose of designing this auxiliary variable is to connect the kinematic control variableτK, which can be specified as(43)rτ=τK−τr.Note that, in (41), τK is a real response due to the evolution of robot dynamics (19) rather than a designed input. The relation between rτ and ry can be interestingly derived as(44)rτTN^Ty,θβyz^ry=N^y,θτK−N^y,θτrTβyz^ry=N^y,θτK−Ny,θτK+zy˙−z^y˙+z^y˙−z^y˙rTβyz^ry,where βy is a positive constant. Using Property 1, we can further obtain N^y,θτK−N^y,θτr=−EK,aθ,τKΔϕk,a+EK,bθ,τK,yΔϕk,b and zy˙−z^y˙=E¯K,bθ,x,y˙Δϕk,b. Thus, (44) can be rewritten as(45)rτN^Ty,θβyz^ry=−Δϕk,aTEK,aTθ,τKβyz^ry+ryTz^βyz^ry+Δϕk,bTE¯K,bTθ,x,y˙+EK,bTθ,τK,yβyz^ry.Based on the kinematic control errorsrτ and reference errors ry, the adaptive visual servoing dynamic controller (AVSDC) is proposed as(46)τD=B¯−1θEDθ,θ.,τr,τ˙rϕ^d−βτrτ−N^Ty,θβyz^ry.where βτ is a positive constant.Remark 5.
SinceB¯θ=STθBθ is related to the actuator dynamics rather than the robot dynamics, in this paper, we assume that the exact structure and parameters of B¯θ are known. In particular, B¯θ is defined as the I2×2 identity matrix in [6, 38] if the actuators are free of operation faults.Remark 6.
It seems interesting that, in the proposed AVSDC scheme, both image errors and velocity errorsry=Δy˙+γΔy are introduced in the control law, which will strengthen the tracking performance and robustness of the dynamic closed-loop system. As will be shown in the stability analysis, the asymptotical convergence of ry leads to the asymptotical convergence of both Δy˙ and Δy.
By substituting the AVSDC (46) into (19) and employing Property 4, one can obtain the closed-loop dynamics:(47)M¯θr˙τ+V¯θ,θ.rτ=EDθ,θ.,τr,τ˙rΔϕd−βτrτ−N^Ty,θβyz^ry.


### 4.2. Unknown Parameter Estimation

In the AVSDC design (46), the estimated dynamics and visual kinematics are employed, and the estimated parameters are updated online by(48)ϕ^˙d=−Φd−1EDTθ,θ.,τr,τ˙rrτ,(49)ϕ^˙a,i=proj℧¯a,i,(50)ϕ^˙b,i=proj℧¯b,i,where Φd∈ℝe3×e3 is a positive-definite diagonal matrix, the projection operation is given in (36) and (37), and(51)℧¯a=Φa−1EK,aTθ,τKβyz^ry,℧¯b=−Φb−1E¯K,bTθ,x,y˙+EK,bTθ,τK,yβyz^ry.


### 4.3. Stability Analysis

Based on the above system analysis of closed-loop dynamics, we have the following result.Theorem 2.
Consider the visually servoed WMR system consisting of the uncalibrated kinematics (1), (2), (4), (6), and (8) and uncertain dynamics (19), under the control of AVSDC (46) with parameter updating laws (48), (49), and (50). Then, the closed-loop dynamics system of WMR is globally bounded, and the image errors are convergent to zero asymptotically.Proof.
Similarly, consider the Lyapunov function candidate(52)VD=12rτTM¯θrτ+12Δϕk,aTΦaΔϕk,a+12Δϕk,bTΦbΔϕk,b+12ΔϕdTΦbΔϕd.
Taking the derivative ofVD yields(53)V˙D=rτTM¯θr˙τ+12rτTM¯˙θrτ+Δϕk,aTΦaϕ^˙k,aΔϕk,bTΦbϕ^˙k,b+ΔϕdTΦbϕ^˙d.
Premultiplying both sides of (47) by rτT and then substituting it into (53), we have(54)V˙D=rτTEDθ,θ.,τr,τ˙rΔϕd−rτTβτrτ−rτTN^Ty,θβyz^ry+Δϕk,aTΦaϕ^˙k,a+Δϕk,bTΦbϕ^˙k,b+ΔϕdTΦbϕ^˙d,where Property 3 is used. Subsequently, substituting (45) and the parameter updating laws (48), (49), and (50) into (54), we can obtain the following result:(55)V˙D=−rτTβτrτ−ryTz^βyz^ry+Δϕk,aTΦaϕ^˙k,a−℧¯a+Δϕk,bTΦbϕ^˙k,b−℧¯b≤−rτTβτrτ−ryTz^βyz^ry.
AsVD≥0 and VD≤0 are simultaneously established, we can obtain that VDt must be bounded; that is, rτ, M¯θ, Δϕk,a, Δϕk,b, and Δϕd are all bounded, giving rise to the boundedness of θ, ϕ^k,a, ϕ^˙k,a, ϕ^k,b, and ϕ^d since ϕk,a, ϕk,b, and ϕd are all constants. Moreover, N^y,θ is bounded and nonsingular, and z^, y˙r, and ry∈ℒ∞ by observing (40) and (43). From (42), we have τ˙r∈ℒ∞, leading to τD∈ℒ∞ from (46). According to the robot dynamics (19), we have τ˙K∈ℒ∞, which directly implies that r˙τ∈ℒ∞. Thus, the closed-loop dynamics of WMR in (47) is globally bounded.
Furthermore, from the result of (52) and (55), we get rτ∈ℒ∞∩ℒ2 and z^ry∈ℒ∞∩ℒ2. Differentiating (8) with respect to time leads to y¨=N˙y,θτK+Ny,θτ˙K/z−Ny,θτKz˙/z2∈ℒ∞. Additionally, from the above analysis, y¨r=y¨d−γΔy˙∈ℒ∞, which thus results in z^˙ry+z^r˙y∈ℒ∞. Therefore, we have limt⟶∞z^ry⟶0, since z^≠0 is defined by the projection function, which finally indicates that the image errors converge to zero such that limt⟶∞y−yd⟶0 and limt⟶∞y˙−y˙d⟶0.Remark 7.
Compared with the recent work on handing uncertain parameters for WMR [19, 33, 34, 36], where only kinematic uncertainties are addressed, this paper extends the uncalibrated visual servoing control to a dynamic control loop in the presence of parameter uncertainties and varying depth. It can be seen in AVSDC (46) that both the reference image errors ry and kinematic control errors rτ, which contain velocity errors Δy˙ and y˙r, are concurrently employed, giving rise to asymptotical convergence of both Δy˙ and Δy by comparing with the AVSKC (24). Furthermore, the parameter uncertainties of visual kinematics and dynamics are adaptively compensated by the parameter updating laws (48), (49), and (50). Actually, the AVSDC (46), to some extent, can be potentially regarded as containing the kinematic control by designing the kinematic auxiliary variable τr.


## 4.1. Controller Design

Define a referenced image velocity as(39)y˙r=y˙d−γΔy,where γ is a positive constant. Then, the reference errors of image velocity can be denoted as(40)ry=y˙−y˙r=Δy˙+γΔy.Thus, the reference errorsry contain both the image errors and image velocity errors. Furthermore, define a kinematic auxiliary variable(41)τr=N^−1y,θz^y˙r.Differentiating (41) with respect to time, we have(42)τ˙r=−N^−1y,θN^˙y,θN^−1y,θ+N^−1y,θz^˙y˙r+N^−1y,θz^ry.The purpose of designing this auxiliary variable is to connect the kinematic control variableτK, which can be specified as(43)rτ=τK−τr.Note that, in (41), τK is a real response due to the evolution of robot dynamics (19) rather than a designed input. The relation between rτ and ry can be interestingly derived as(44)rτTN^Ty,θβyz^ry=N^y,θτK−N^y,θτrTβyz^ry=N^y,θτK−Ny,θτK+zy˙−z^y˙+z^y˙−z^y˙rTβyz^ry,where βy is a positive constant. Using Property 1, we can further obtain N^y,θτK−N^y,θτr=−EK,aθ,τKΔϕk,a+EK,bθ,τK,yΔϕk,b and zy˙−z^y˙=E¯K,bθ,x,y˙Δϕk,b. Thus, (44) can be rewritten as(45)rτN^Ty,θβyz^ry=−Δϕk,aTEK,aTθ,τKβyz^ry+ryTz^βyz^ry+Δϕk,bTE¯K,bTθ,x,y˙+EK,bTθ,τK,yβyz^ry.Based on the kinematic control errorsrτ and reference errors ry, the adaptive visual servoing dynamic controller (AVSDC) is proposed as(46)τD=B¯−1θEDθ,θ.,τr,τ˙rϕ^d−βτrτ−N^Ty,θβyz^ry.where βτ is a positive constant.Remark 5.
SinceB¯θ=STθBθ is related to the actuator dynamics rather than the robot dynamics, in this paper, we assume that the exact structure and parameters of B¯θ are known. In particular, B¯θ is defined as the I2×2 identity matrix in [6, 38] if the actuators are free of operation faults.Remark 6.
It seems interesting that, in the proposed AVSDC scheme, both image errors and velocity errorsry=Δy˙+γΔy are introduced in the control law, which will strengthen the tracking performance and robustness of the dynamic closed-loop system. As will be shown in the stability analysis, the asymptotical convergence of ry leads to the asymptotical convergence of both Δy˙ and Δy.
By substituting the AVSDC (46) into (19) and employing Property 4, one can obtain the closed-loop dynamics:(47)M¯θr˙τ+V¯θ,θ.rτ=EDθ,θ.,τr,τ˙rΔϕd−βτrτ−N^Ty,θβyz^ry.


## 4.2. Unknown Parameter Estimation

In the AVSDC design (46), the estimated dynamics and visual kinematics are employed, and the estimated parameters are updated online by(48)ϕ^˙d=−Φd−1EDTθ,θ.,τr,τ˙rrτ,(49)ϕ^˙a,i=proj℧¯a,i,(50)ϕ^˙b,i=proj℧¯b,i,where Φd∈ℝe3×e3 is a positive-definite diagonal matrix, the projection operation is given in (36) and (37), and(51)℧¯a=Φa−1EK,aTθ,τKβyz^ry,℧¯b=−Φb−1E¯K,bTθ,x,y˙+EK,bTθ,τK,yβyz^ry.


## 4.3. Stability Analysis

Based on the above system analysis of closed-loop dynamics, we have the following result.Theorem 2.
Consider the visually servoed WMR system consisting of the uncalibrated kinematics (1), (2), (4), (6), and (8) and uncertain dynamics (19), under the control of AVSDC (46) with parameter updating laws (48), (49), and (50). Then, the closed-loop dynamics system of WMR is globally bounded, and the image errors are convergent to zero asymptotically.Proof.
Similarly, consider the Lyapunov function candidate(52)VD=12rτTM¯θrτ+12Δϕk,aTΦaΔϕk,a+12Δϕk,bTΦbΔϕk,b+12ΔϕdTΦbΔϕd.
Taking the derivative ofVD yields(53)V˙D=rτTM¯θr˙τ+12rτTM¯˙θrτ+Δϕk,aTΦaϕ^˙k,aΔϕk,bTΦbϕ^˙k,b+ΔϕdTΦbϕ^˙d.
Premultiplying both sides of (47) by rτT and then substituting it into (53), we have(54)V˙D=rτTEDθ,θ.,τr,τ˙rΔϕd−rτTβτrτ−rτTN^Ty,θβyz^ry+Δϕk,aTΦaϕ^˙k,a+Δϕk,bTΦbϕ^˙k,b+ΔϕdTΦbϕ^˙d,where Property 3 is used. Subsequently, substituting (45) and the parameter updating laws (48), (49), and (50) into (54), we can obtain the following result:(55)V˙D=−rτTβτrτ−ryTz^βyz^ry+Δϕk,aTΦaϕ^˙k,a−℧¯a+Δϕk,bTΦbϕ^˙k,b−℧¯b≤−rτTβτrτ−ryTz^βyz^ry.
AsVD≥0 and VD≤0 are simultaneously established, we can obtain that VDt must be bounded; that is, rτ, M¯θ, Δϕk,a, Δϕk,b, and Δϕd are all bounded, giving rise to the boundedness of θ, ϕ^k,a, ϕ^˙k,a, ϕ^k,b, and ϕ^d since ϕk,a, ϕk,b, and ϕd are all constants. Moreover, N^y,θ is bounded and nonsingular, and z^, y˙r, and ry∈ℒ∞ by observing (40) and (43). From (42), we have τ˙r∈ℒ∞, leading to τD∈ℒ∞ from (46). According to the robot dynamics (19), we have τ˙K∈ℒ∞, which directly implies that r˙τ∈ℒ∞. Thus, the closed-loop dynamics of WMR in (47) is globally bounded.
Furthermore, from the result of (52) and (55), we get rτ∈ℒ∞∩ℒ2 and z^ry∈ℒ∞∩ℒ2. Differentiating (8) with respect to time leads to y¨=N˙y,θτK+Ny,θτ˙K/z−Ny,θτKz˙/z2∈ℒ∞. Additionally, from the above analysis, y¨r=y¨d−γΔy˙∈ℒ∞, which thus results in z^˙ry+z^r˙y∈ℒ∞. Therefore, we have limt⟶∞z^ry⟶0, since z^≠0 is defined by the projection function, which finally indicates that the image errors converge to zero such that limt⟶∞y−yd⟶0 and limt⟶∞y˙−y˙d⟶0.Remark 7.
Compared with the recent work on handing uncertain parameters for WMR [19, 33, 34, 36], where only kinematic uncertainties are addressed, this paper extends the uncalibrated visual servoing control to a dynamic control loop in the presence of parameter uncertainties and varying depth. It can be seen in AVSDC (46) that both the reference image errors ry and kinematic control errors rτ, which contain velocity errors Δy˙ and y˙r, are concurrently employed, giving rise to asymptotical convergence of both Δy˙ and Δy by comparing with the AVSKC (24). Furthermore, the parameter uncertainties of visual kinematics and dynamics are adaptively compensated by the parameter updating laws (48), (49), and (50). Actually, the AVSDC (46), to some extent, can be potentially regarded as containing the kinematic control by designing the kinematic auxiliary variable τr.


## 5. Numerical Simulations

In order to demonstrate the tracking performance of AVSKC and AVSDC schemes, simulation studies are carried out. As in Figure1, a two-wheeled mobile robot with a camera in a fixed place is considered.


### 5.1. Trajectory Tracking for AVSKC

In the simulation task, we firstly address the AVSKC scheme under parallel and unparallel camera case. Assume that one feature point is marked on the WMR, and the distanced is set to be 0.3m. The simulated parameters of the perspective projection matrix in [8, 25] are given in Table 1, where f denotes the focal length of the camera, uk and vk denote the scalar factors of two-dimensional axes on image plane, uo and vo are the positions of principal point, ε is the included angle between the coordinate axis and is assumed to be known, and the rotation matrix in transformation matrix is set as Rot−1x,π in parallel camera case and Rot−1x,π−π/10 in unparallel camera case, respectively. Note that these parameters are only used to construct the simulated model, but are unavailable in the control design. For simplicity, the detailed expression of system model in (6) and (7) and Property 1 can be referred to in [8, 35]. The designed control gains and adaption gains are set as α=10 and Φa=Φb=10000, respectively. The upper and lower bounds in parameter projection are designed as ϕ¯a,i=0.6ϕa,i, ϕ¯a,i=1.3ϕa,i, ϕ¯b,i=0.6ϕb,i, and ϕ¯b,i=1.3ϕb,i, and the initial parameters are ϕ^a,i0=0.8ϕa,i and ϕ^b,i0=0.8ϕb,i, respectively. The initial states of WMR are θ10=θ20=θ30=0, where the superscript i denotes ith initial condition in simulation task, and the referenced trajectory is given as(56)yd=10∗sint+31010∗cost+340pixel.Table 1
Visual model parameters.
fukvkuovoεCamera parameters0.0420002000300300π/2Transformation matrix in parallel camera caseRotx,π11−301×31−1Transformation matrix in unparallel camera caseRotx,π−π/1011301×31−1


#### 5.1.1. Parallel Camera Case (PCC)

In this case, the image plane is parallel to the operation plane. The initial states of WMR are set asxb10=yb10=0.1m, xb20=yb20=0.1m, and xb30=yb30=0.1m, respectively. The graphs in Figures 2(a)–2(c) demonstrate the corresponding simulation results, from which we can observe that the real trajectory converges to the referenced trajectory in about 2.5 s. Note that, in this case, the desired velocity y˙d is time varying; however, smoothly real trajectory and bounded states are still achievable.Figure 2
Control responses of AVSKC schemes. (a) Position errors on image plane in PCC. (b) The real and desired trajectory in PCC. (c) Control input in PCC. (d) Image errors in UPCC. (e) The real and desired trajectory in UPCC. (f) Control input in UPCC.
(a)(b)(c)(d)(e)(f)


#### 5.1.2. Unparallel Camera Case (UPCC)

In this case, the camera is placed in a position unparallel to the operation plane. The initial states of WMR are the same as in PCC. The simulation results are depicted in Figures2(d)–2(f). Note that, in Figures 2(e) and 2(b), the initial points on image plane are noncoincident since the camera parameters are chosen in different values. Moreover, the image errors asymptotically converge to zero as expected, verifying the effectiveness of AVSKC scheme.


### 5.2. Trajectory Tracking for AVSDC

In this subsection, we will test the tracking performance of AVSDC scheme under tracking line case and tracking circle case. Due to the limitation of space, the parametric dynamics model of WMR is omitted, whose detailed expressions are given in [38], and the initial value for ϕ^ is set as ϕ^d0=0.8ϕd. The designed gains are set as γ=5, βτ=5, βy=10, and Φd=1000; apart from this, all the simulated model and system parameters are given in UPCC.


#### 5.2.1. Tracking Line Case (TLC)

The reference line is given as(57)yd=3t+3103t+340pixel.In this case, the desired velocity is constant. Based on the theoretical analysis in Theorem2, the real trajectory on the image plane asymptotically converges to the desired trajectory in the sense of position and velocity, confirmed by the simulation results in Figures 3(a)–3(c).Figure 3
Control responses of AVSDC schemes. (a) Position errors on image plane in TLC. (b) The real and desired trajectory in TLC. (c) Velocity errors on image plane in TLC. (d) Position errors in TCC. (e) The real and desired trajectory in TCC. (f) Velocity errors on image plane in TCC.
(a)(b)(c)(d)(e)(f)


#### 5.2.2. Tracking Circle Case (TCC)

In this case, the desired trajectory in this case is chosen as in UPCC, and the external disturbancef=3sint,3costT is applied to the robot dynamics such that(58)Mθq¨+Vθ,θ.q˙+G=BθτD+ATθλ+f.The time histories of the corresponding results are plotted in Figures3(d)–3(f). As predicted by Theorem 2, both Δy and Δy˙ asymptotically converge to zero in about 1 s even under the influence of external interference. Furthermore, faster responses are expectedly obtained as compared with the simulation results in AVSKC scheme (see Figures 2(d) and 3(d)).


## 5.1. Trajectory Tracking for AVSKC

In the simulation task, we firstly address the AVSKC scheme under parallel and unparallel camera case. Assume that one feature point is marked on the WMR, and the distanced is set to be 0.3m. The simulated parameters of the perspective projection matrix in [8, 25] are given in Table 1, where f denotes the focal length of the camera, uk and vk denote the scalar factors of two-dimensional axes on image plane, uo and vo are the positions of principal point, ε is the included angle between the coordinate axis and is assumed to be known, and the rotation matrix in transformation matrix is set as Rot−1x,π in parallel camera case and Rot−1x,π−π/10 in unparallel camera case, respectively. Note that these parameters are only used to construct the simulated model, but are unavailable in the control design. For simplicity, the detailed expression of system model in (6) and (7) and Property 1 can be referred to in [8, 35]. The designed control gains and adaption gains are set as α=10 and Φa=Φb=10000, respectively. The upper and lower bounds in parameter projection are designed as ϕ¯a,i=0.6ϕa,i, ϕ¯a,i=1.3ϕa,i, ϕ¯b,i=0.6ϕb,i, and ϕ¯b,i=1.3ϕb,i, and the initial parameters are ϕ^a,i0=0.8ϕa,i and ϕ^b,i0=0.8ϕb,i, respectively. The initial states of WMR are θ10=θ20=θ30=0, where the superscript i denotes ith initial condition in simulation task, and the referenced trajectory is given as(56)yd=10∗sint+31010∗cost+340pixel.Table 1
Visual model parameters.
fukvkuovoεCamera parameters0.0420002000300300π/2Transformation matrix in parallel camera caseRotx,π11−301×31−1Transformation matrix in unparallel camera caseRotx,π−π/1011301×31−1


### 5.1.1. Parallel Camera Case (PCC)

In this case, the image plane is parallel to the operation plane. The initial states of WMR are set asxb10=yb10=0.1m, xb20=yb20=0.1m, and xb30=yb30=0.1m, respectively. The graphs in Figures 2(a)–2(c) demonstrate the corresponding simulation results, from which we can observe that the real trajectory converges to the referenced trajectory in about 2.5 s. Note that, in this case, the desired velocity y˙d is time varying; however, smoothly real trajectory and bounded states are still achievable.Figure 2
Control responses of AVSKC schemes. (a) Position errors on image plane in PCC. (b) The real and desired trajectory in PCC. (c) Control input in PCC. (d) Image errors in UPCC. (e) The real and desired trajectory in UPCC. (f) Control input in UPCC.
(a)(b)(c)(d)(e)(f)


### 5.1.2. Unparallel Camera Case (UPCC)

In this case, the camera is placed in a position unparallel to the operation plane. The initial states of WMR are the same as in PCC. The simulation results are depicted in Figures2(d)–2(f). Note that, in Figures 2(e) and 2(b), the initial points on image plane are noncoincident since the camera parameters are chosen in different values. Moreover, the image errors asymptotically converge to zero as expected, verifying the effectiveness of AVSKC scheme.


## 5.1.1. Parallel Camera Case (PCC)

In this case, the image plane is parallel to the operation plane. The initial states of WMR are set asxb10=yb10=0.1m, xb20=yb20=0.1m, and xb30=yb30=0.1m, respectively. The graphs in Figures 2(a)–2(c) demonstrate the corresponding simulation results, from which we can observe that the real trajectory converges to the referenced trajectory in about 2.5 s. Note that, in this case, the desired velocity y˙d is time varying; however, smoothly real trajectory and bounded states are still achievable.Figure 2
Control responses of AVSKC schemes. (a) Position errors on image plane in PCC. (b) The real and desired trajectory in PCC. (c) Control input in PCC. (d) Image errors in UPCC. (e) The real and desired trajectory in UPCC. (f) Control input in UPCC.
(a)(b)(c)(d)(e)(f)


## 5.1.2. Unparallel Camera Case (UPCC)

In this case, the camera is placed in a position unparallel to the operation plane. The initial states of WMR are the same as in PCC. The simulation results are depicted in Figures2(d)–2(f). Note that, in Figures 2(e) and 2(b), the initial points on image plane are noncoincident since the camera parameters are chosen in different values. Moreover, the image errors asymptotically converge to zero as expected, verifying the effectiveness of AVSKC scheme.


## 5.2. Trajectory Tracking for AVSDC

In this subsection, we will test the tracking performance of AVSDC scheme under tracking line case and tracking circle case. Due to the limitation of space, the parametric dynamics model of WMR is omitted, whose detailed expressions are given in [38], and the initial value for ϕ^ is set as ϕ^d0=0.8ϕd. The designed gains are set as γ=5, βτ=5, βy=10, and Φd=1000; apart from this, all the simulated model and system parameters are given in UPCC.


### 5.2.1. Tracking Line Case (TLC)

The reference line is given as(57)yd=3t+3103t+340pixel.In this case, the desired velocity is constant. Based on the theoretical analysis in Theorem2, the real trajectory on the image plane asymptotically converges to the desired trajectory in the sense of position and velocity, confirmed by the simulation results in Figures 3(a)–3(c).Figure 3
Control responses of AVSDC schemes. (a) Position errors on image plane in TLC. (b) The real and desired trajectory in TLC. (c) Velocity errors on image plane in TLC. (d) Position errors in TCC. (e) The real and desired trajectory in TCC. (f) Velocity errors on image plane in TCC.
(a)(b)(c)(d)(e)(f)


### 5.2.2. Tracking Circle Case (TCC)

In this case, the desired trajectory in this case is chosen as in UPCC, and the external disturbancef=3sint,3costT is applied to the robot dynamics such that(58)Mθq¨+Vθ,θ.q˙+G=BθτD+ATθλ+f.The time histories of the corresponding results are plotted in Figures3(d)–3(f). As predicted by Theorem 2, both Δy and Δy˙ asymptotically converge to zero in about 1 s even under the influence of external interference. Furthermore, faster responses are expectedly obtained as compared with the simulation results in AVSKC scheme (see Figures 2(d) and 3(d)).


## 5.2.1. Tracking Line Case (TLC)

The reference line is given as(57)yd=3t+3103t+340pixel.In this case, the desired velocity is constant. Based on the theoretical analysis in Theorem2, the real trajectory on the image plane asymptotically converges to the desired trajectory in the sense of position and velocity, confirmed by the simulation results in Figures 3(a)–3(c).Figure 3
Control responses of AVSDC schemes. (a) Position errors on image plane in TLC. (b) The real and desired trajectory in TLC. (c) Velocity errors on image plane in TLC. (d) Position errors in TCC. (e) The real and desired trajectory in TCC. (f) Velocity errors on image plane in TCC.
(a)(b)(c)(d)(e)(f)


## 5.2.2. Tracking Circle Case (TCC)

In this case, the desired trajectory in this case is chosen as in UPCC, and the external disturbancef=3sint,3costT is applied to the robot dynamics such that(58)Mθq¨+Vθ,θ.q˙+G=BθτD+ATθλ+f.The time histories of the corresponding results are plotted in Figures3(d)–3(f). As predicted by Theorem 2, both Δy and Δy˙ asymptotically converge to zero in about 1 s even under the influence of external interference. Furthermore, faster responses are expectedly obtained as compared with the simulation results in AVSKC scheme (see Figures 2(d) and 3(d)).


## 6. Conclusions

Two uncalibrated visual servoing control schemes for the wheeled mobile robot were developed from different perspectives, namely, the kinematic control and dynamic control. By utilizing the linearization characteristics of visual kinematics and robot dynamics, image-based tracking control laws (i.e., ASVKC and ASVDC) together with the parameter adaption algorithms were proposed to realize asymptotical convergence of image errors without the knowledge of visual model robot parameters. Furthermore, the overparametrization problem is avoided by exploiting the structure of depth-independent interaction matrix, giving less dimensional regressor matrices and simple configuration of parameter adaption laws. It was proven by the Lyapunov theory that both ASVKC and ASVDC schemes are capable of achieving global stability of closed-loop system. Lastly, numerical simulations were carried out to confirm the performance of ASVKC and ASVDC.In this paper, we assume that the image trajectory is given in advance, and the external forces of robot system are not considered in complex environment. Furthermore, the applicability of visual servoing WMR control is worth further exploring. Thus, the further work encompasses the deterministic learning and accurate identification of system dynamics [41], the applicability of WMR with actuator constraint [42], and obstacle avoidance [43–45].



---

*Source: 8836468-2020-12-04.xml*