Over a predefined time horizon, the optimal solution minimizing (5.3.1) without considering any constraint corresponds to a manual propulsion strategy in which all the kinetic energy comes from the human. To avoid this trivial solution, we impose the following fatigue constraint. Knowing the initial $S_{of0}$, the final $S_{ofK}$ should reach a desired level $S_{of-ref}$:

$$S_{ofK} = S_{of-ref} \qquad (5.3.2)$$

while minimizing (5.3.1) over the considered driving profile. The wheelchair should also travel a required distance. Knowing the initial $d_0$, we impose the following distance constraint:

$$d_K = d_{ref} \qquad (5.3.3)$$

including the terminal distance $d_K$ and the desired terminal $d_{ref}$. Rather than solving explicitly a constrained problem, we represent the constraints (5.3.2)-(5.3.3) with a terminal reward, leading to the following optimal control problem:

$$\max_{U} R = -[w_{e1} w_{e2}] \begin{bmatrix} (d_K - d_{ref})^2 \\ (S_{ofK} - S_{of-ref})^2 \end{bmatrix} - \frac{1}{2} \sum_{k=0}^{K-1} U_k^2 \qquad (5.3.4)$$

with $w_{e1}, w_{e2}$ the reward weights and $K$ the finite time horizon. Note that in classical control theory the return $R$ in (5.3.4) is often replaced by a positive cost function and must be minimized. Here, we use Artificial Intelligence techniques, so we follow the maximization convention in this field.

The system considered is described in general by the deterministic state transition function:

$$x_{k+1} = \xi(x_k, u_k) \qquad (5.3.5)$$

where $x$ and $u$ are state vector and control input respectively. The general return $R$ to optimize over a finite-horizon is:

$$R(\tau) = \gamma^K T(x_K) + \sum_{k=0}^{K-1} \gamma^k r(x_k, u_k) \qquad (5.3.6)$$

where $\tau = (x_0, u_0, x_1, u_1, \dots, x_{K-1}, u_{K-1}, x_K)$ is a trajectory of the system, $T(x_K)$ is the terminal reward, and $r(x_k, u_k)$ is the stage reward. A discount factor $\gamma \in (0, 1]$ may be used; in the finite-horizon case, $\gamma$ is often taken equal to 1. The optimization problem (5.3.4) is a
specific case of the general form (5.3.6). The following algorithms are presented for the general case defined in (5.3.5)-(5.3.6).

## 5.4 Baseline solution: Approximated dynamic programming

### 5.4.1 Finite-horizon fuzzy Q-iteration

Fuzzy Q-iteration (Buşoniu et al., 2010) is originally given in the infinite-horizon case, and the horizon-K solution can be obtained simply by iterating the algorithm K times. However, the entire time-varying solution must be maintained, and special care must be taken to properly handle the terminal reward. So for clarity we restate the entire algorithm, adapting it to the finite-horizon case.

The idea is to approximate the optimal time-varying solution, which can be expressed using Q-functions of the state in the state-space X and action in the action space U. These Q-functions are generated backwards in time:

$$ 
\begin{gathered}
Q_{K-1}^*(x_{K-1}, u_{K-1}) = r(x_{K-1}, u_{K-1}) + \gamma T(\xi(x_{K-1}, u_{K-1})) \\
Q_k^*(x_k, u_k) = r(x_k, u_k) + \gamma \max_{u_{k+1}} Q_{k+1}^*(\xi(x_k, u_k), u_{k+1}), \quad (5.4.1) \\
\text{for } k = K-2, \dots, 0 \text{ and } \forall x \in X, \forall u \in U
\end{gathered}
$$

The advantage of using Q-functions is that the optimal control can then be computed relatively easily, using the following time-varying state-feedback:

$$ \pi^*(x_k, k) = \arg \max_{u_k} Q_k^*(x_k, u_k) \quad (5.4.2) $$

Since the system is nonlinear and the states and actions are continuous, in general it is impossible to compute the exact solution above. We will therefore represent $Q^*$ with an approximator that relies on an interpolation over the state space, and on a discretization of the action space. First, to handle the action, the approximate Q-value of the pair $(x, u)$ is replaced by that of the pair $(x, u_d)$, where $u_d$ has the closest Euclidean distance to $u$ in a discrete subset of actions $U_d = \{\bar{u}_j | \bar{u}_j \in U, j = 1, \dots, N_u\}$. To handle the state, a grid of discrete values $X_d = \{\bar{x}_i | \bar{x}_i \in X, i = 1, \dots, N_x\}$ in the state space is chosen for the centers of triangular membership functions $\phi(x) = [\phi_1(x), \dots, \phi_{N_x}(x)]$ (Buşoniu et al. 2010). A parameter vector
$\theta \subset \mathbb{R}^{N_x \times N_u \times K}$ is defined, and the approximate $Q$-function is linearly interpolated by overlapping the membership functions $\phi$ on the grid of the centers $X_d$ as follows:

$$ \hat{Q}_k(x, u) = \sum_{i=1}^{N_x} \phi_i(x) \theta_{i,j,k} \qquad (5.4.3) $$

with $j \in \arg\min_{j'} \|u - \bar{u}_j\|^2$. Thus, each individual parameter corresponds to a combination between a point $i$ on the state interpolation grids, a discrete action $j$, and a time stage $k$. The approximated optimal solution $\hat{\pi}$ can be obtained as follows:

$$ \hat{\pi}(x, k) = \bar{u}_j \quad \text{with} \quad j = \arg\max_{j'} \sum_{i=1}^{N_x} \phi_i(x) \theta_{i,j',k} \qquad (5.4.4) $$

Algorithm 1. Finite-horizon fuzzy Q-iteration

1 **for** $i = 1, \dots, N_x, j = 1, \dots, N_u$ **do**

2 $\theta_{i,j,K-1} = r(\bar{x}_i, \bar{u}_j) + \gamma T(\xi(\bar{x}_i, \bar{u}_j))$

3 **end for**

4 **for** $k = K - 2, \dots, 0$ **do**

5 **for** $i = 1, \dots, N_x, j = 1, \dots, N_u$ **do**

6 $\theta_{i,j,k} = r(\bar{x}_i, \bar{u}_j) + \gamma \max_{j'} \sum_{i'=1}^{N_x} \phi_{i'}(\xi(\bar{x}_{i'}, \bar{u}_{j'})) \theta_{i,j,k+1}$

7 **end for**

8 $\hat{\pi}(x, k) = \bar{u}_j, \quad j = \arg\max_{j'} \sum_{i=1}^{N_x} \phi_i(x) \theta_{i,j',k} \quad \forall x, k$

9 **end for**

Algorithm 1 gives the complete version of Fuzzy Q-iteration. To understand it, note that the main update in line 6 is equivalent to the following approximate variant of the iterative update in (5.4.1):

$$ \hat{Q}_k(\bar{x}_i, \bar{u}_j) = r(\bar{x}_i, \bar{u}_j) + \gamma \max_{\bar{u}_{j,k+1}} \hat{Q}_{k+1}(\xi(\bar{x}_i, \bar{u}_j), \bar{u}_{j,k+1}) \qquad (5.4.5) $$

This is because, firstly, due to the properties of triangular basis functions the parameter $\theta_{i,j,k}$ is equal to the approximate $Q$-value $\hat{Q}_k(\bar{x}_i, \bar{u}_j)$. Secondly, the maximization over the discretized actions is done by enumeration over $j$; and thirdly, the summation is just the approximate $Q$-value at the next step, via (5.4.3). Line 2 simply sets the parameters at step $K-1$ via the initialization in (5.4.1).
For clarity, the algorithm shows in line 8 how the near-optimal control is computed via maximization over the discrete actions. In practice, this maximization is done on-demand, only for the states encountered while controlling the system, so an explicit function $\hat{\pi}$ of the continuous state does not have to be stored. Instead, only the parameters are stored.

### 5.4.2 Optimality analysis

In contrast to the algorithm itself, the infinite-horizon analysis does not easily extend to the finite-horizon case, e.g. we need to account for the possibility that $\gamma = 1$. Thus, the upcoming study, which has been presented in (Feng et al. 2019), provides a complete analysis.

The error $\epsilon_k$ between $\hat{Q}_k$ and $Q_k^*$ for sample $k$ is defined as:

$$ \epsilon_k = \| \hat{Q}_k(x, u) - Q_k^*(x, u) \| \qquad (5.4.6) $$

The state resolution step $\delta_x$ is defined as the largest distance between any two neighbouring triangular MF cores, i.e.

$$ \delta_x = \max_{i \in \{1, \dots, N_x\}} \min_{i' \in \{1, \dots, N_x\}, i \neq i'} \| \bar{x}_i - \bar{x}_{i'} \|_2 $$

The action resolution step $\delta_u$ is defined similarly for the discrete actions. Moreover, for every $x$, only $2^{N_{state}}$ (where $N_{state}$ is the number of states) triangular membership functions are activated. Let the infinite norm $\|\theta_k\|_{\infty} = \max_{i \in \{1, \dots, N_x\}, j \in \{1, \dots, N_u\}} |\theta_{i,j,k}|$ denotes the largest parameter magnitude at sample $k$. Note that triangular membership functions are Lipschitz-continuous, so there exists a Lipschitz constant $L_{\phi} > 0$ such that $\|\phi_i(x) - \phi_i(x')\|_2 \le L_{\phi_i}(\|x - x'\|_2) \forall x, x' \in X, \forall i$. Moreover, we say that a function of the state and action, such as the deterministic state transition function $\xi$, is Lipschitz continuous with constant $L_{\xi} > 0$ if $\|\xi(x, u) - \xi(x', u')\|_2 \le L_{\xi}(\|x - x'\|_2 + \|u - u'\|_2) \forall x, x' \in X, u, u' \in U$.

**Assumption 1:** The reward function $r$, the terminal function $T$, and the deterministic state transition function $\xi$ are Lipschitz-continuous with the Lipschitz constants $L_r$, $L_T$, and $L_{\xi}$ respectively.

We present an explicit bound on the near-optimality of the $Q$-function as a function of the grid resolutions. This bound has the nice feature that it converges to zero when the grid becomes infinitely dense, which is a consistency property of the algorithm.
**Proposition 1:** Under Assumption 1, there exists an error bound $\bar{\varepsilon}_k$ so that $\hat{Q}$, i.e. the approximate $Q$-function obtained by (5.4.6) satisfies $\varepsilon_k \le \bar{\varepsilon}_k$ and $\lim_{\delta_{x},\delta_{u} \to 0} \bar{\varepsilon}_k = 0$ for $k = K - 1, \dots, 0$. Depending on the discount factor $\gamma$ and the Lipschitz constant $L_\xi$, the bound is given as follows:

$$ \gamma L_{\xi} < 1, \qquad \bar{\varepsilon}_k = (\delta_x + \delta_u) \left( (K-k)L_r + \sum_{z=1}^{K-k} \left( L_T (\gamma L_{\xi})^z + L_r \frac{(\gamma L_{\xi})^z - \gamma L_{\xi}}{\gamma L_{\xi} - 1} \right) \right) \qquad (5.4.7) $$

$$ \gamma L_{\xi} = 1, \qquad \bar{\varepsilon}_k = (\delta_x + \delta_u)(K-k) \left( L_r + L_T + \frac{(K-k-1)}{2} L_r \right) \qquad (5.4.8) $$

$$ \gamma L_{\xi} > 1, \qquad \bar{\varepsilon}_k = (\delta_x + \delta_u) \left( (K-k)L_r + 2^{N_{\text{state}}} \gamma L_{\xi} L_{\phi} \sum_{z=1}^{K-k} \|\theta_{K-z+1}\|_{\infty} \right) \qquad (5.4.9) $$

### 5.4.2.1 Lipschitz property of $Q^*$

Before giving the proof of proposition 1, we explore the Lipschitz property of $Q^*$. Hereafter, we prove that the $Q_k^*$ function is Lipschitz for $k = 0,1, \dots, K$. Knowing that $T$ function is Lipschitz and the exact optimal $Q$ function $Q_k^*$ is equal to the terminal return as $\forall x \in X, u \in U$:

$$ Q_{K}^{*}(x,u) = T(x) $$

Consequently, $Q_k^*$ is Lipschitz. Considering an arbitrary time stage $k$ ($k = 0,1, \dots, K-1$), we obtain the following inequality $\forall x, x' \in X, u, u' \in U$:

$$ \begin{aligned} & |Q_k^*(x,u) - Q_k^*(x',u')| \\ &= \left| r(x,u) + \gamma \max_{\tilde{u}} Q_{k+1}^* (\xi(x,u), \tilde{u}) - r(x',u') - \gamma \max_{\tilde{u}'} Q_{k+1}^* (\xi(x',u'), \tilde{u}') \right| \\ &\le \left| r(x,u) - r(x',u') \right| + \gamma \max_{\tilde{u}'} \left| Q_{k+1}^* (\xi(x,u), \tilde{u}') - Q_{k+1}^* (\xi(x',u'), \tilde{u}') \right| \\ &\le \left| r(x,u) - r(x',u') \right| + \gamma \max_{\tilde{u}'} \left[ Q_{k+1}^* (\xi(x,u), \tilde{u}') - Q_{k+1}^* (\xi(x',u'), \tilde{u}') \right] \end{aligned} \qquad (5.4.10) $$

Note that the stage reward function $r$ is Lipschitz. Therefore, we can bound (5.4.10) using the triangular inequality property as follows:
$$ \begin{aligned} & |Q_k^*(x, u) - Q_k^*(x', u')| \\ & \le L_r (\|x-x'\|_2 + \|u-u'\|_2) + \gamma \max_{\tilde{u}'} |Q_{k+1}^*(\xi(x, u), \tilde{u}') - Q_{k+1}^*(\xi(x', u'), \tilde{u}')| \end{aligned} \quad (5.4.11) $$

Supposing that for an arbitrary time stage $k$ ($k = 0,1, \dots, K-1$), $Q_{k+1}^*$ is Lipschitz and its Lipschitz constant is $L_{k+1}$. The inequality (5.4.11) can be expressed as:

$$ \begin{aligned} & |Q_k^*(x, u) - Q_k^*(x', u')| \\ & \le L_r (\|x-x'\|_2 + \|u-u'\|_2) + \gamma \max_{\tilde{u}'} |L_{k+1} \|\xi(x, u) - \xi(x', u')\|_2| \\ & \le L_r (\|x-x'\|_2 + \|u-u'\|_2) + L_{k+1} \gamma L_\xi (\|x-x'\|_2 + \|u-u'\|_2) \\ & \le (L_r + L_{k+1} \gamma L_\xi) (\|x-x'\|_2 + \|u-u'\|_2) \end{aligned} $$

Then, $Q_k^*$ is also Lipschitz and its Lipschitz constant is $L_k = L_r + L_{k+1}\gamma L_\xi$. As a result, the $Q_k^*$ function is Lipschitz for $k = 0,1, \dots, K-1$. Now we write the general form for the Lipschitz constant $L_k$ as follows:

When $\gamma L_\xi = 1$, $L_k = L_r + L_{k+1}$. The Lipschitz constant $L_k$ is:

$$ L_k = (K-k)L_r + L_T $$

When $\gamma L_r \neq 1$, $L_k = L_\rho + L_{k+1}\gamma L_r$. The Lipschitz constant $L_k$ is:

$$ L_k = \left( L_T + \frac{L_r}{\gamma L_\xi - 1} \right) \left( \gamma L_\xi \right)^{K-k} - \frac{L_r}{\gamma L_\xi - 1} = L_T \left( \gamma L_\xi \right)^{K-k} + L_r \frac{\left( \gamma L_\xi \right)^{K-k} - 1}{\gamma L_\xi - 1} $$

At last, before giving the proof of the proposition let us give also a property shared by the membership functions $\phi_i(x)$. They hold a convex property, i.e. for any state $x$:

$$ \sum_{i=1}^{M} \phi_i(x) = 1 \quad (5.4.12) $$

Therefore, trivially we can decompose (5.4.12) as:

$$ \sum_{i=1}^{M} \phi_i(x) = \sum_{i \in \{i | \phi_i(x) \neq 0\}} \phi_i(x) + \sum_{i \in \{i | \phi_i(x) = 0\}} \phi_i(x) $$
With $\sum_{i \in \{i | \phi_i(x) = 0\}} \phi_i(x) = 0$ and $\sum_{i \in \{i | \phi_i(x) \neq 0\}} \phi_i(x) = 1$. Moreover, for the terms of the second sum, denoting $I_k = \{i | \phi_i(x) \neq 0\}$, $i \in I_k$ defining $\delta_x$ as the state resolution step we can write:

$$ \|x - x_i\|_2 \le \delta_x $$

### 5.4.2.2 Proof of proposition 1

The exact optimal time-varying Q-function can be expressed as ($k = K - 1, K - 2, ..., 1$ and $\forall x \in X, u \in U$):

$$ Q_k^* = r_k + \gamma \max_{u_{k+1}} Q_{k+1}^* (\xi_k, u_{k+1}) $$

when $k = K$,

$$ Q_K^* = Q^*(x_K, u_K) = T(x_K) $$

Or

$$ Q^*(\xi_{K-1}, u_K) = T(\xi_{K-1}) $$

The approximate Q-function is ($k = K - 1, K - 2, ..., 1$ and $\forall x \in X, u \in U$):

$$ \hat{Q}_k = \hat{Q}(x_k, u_k) = \sum_{i=1}^{M} \phi_i(x_k) \left[ r(\bar{x}_i, \bar{u}_k) + \gamma \max_{u_{k+1}} \hat{Q}_{k+1}(\xi(\bar{x}_i, \bar{u}_k), u_{k+1}) \right] \quad (5.4.13) $$

with $\bar{u}_k = \underset{u_k^j \in U_d}{\operatorname{argmin}} \|u_k - u_k^j\|$ and $u_k^j \in U_d$. With the set $I_k = \{i | \phi_i(x) \neq 0\}$, the approximation (5.4.13) becomes:

$$ \hat{Q}_k = \sum_{i \in I_k} \phi_i(x_k) \left[ r(\bar{x}_i, \bar{u}_k) + \gamma \max_{u_{k+1}} \hat{Q}_{k+1}(\xi(\bar{x}_i, \bar{u}_k), u_{k+1}) \right] $$

When $k = K$,

$$ \hat{Q}_k = \hat{Q}_K = Q^*(x_K, u_K) = T(x_K) $$

Or

$$ \hat{Q}(\xi_{i,K-1}, u_K) = T(\xi_{i,K-1}) $$
The error between the approximate Q-function and the optimal one for arbitrary k:

$$
\begin{align}
\epsilon_k &= |\hat{Q}_k - Q_k^*| \nonumber \\
&= \left| \sum_{i \in I_k} \phi_i(x_k) \left[ r(\bar{x}_i, \bar{u}_k) + \gamma \max_{u_{k+1}} \hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) \right] - r_k - \gamma \max_{u_{k+1}} Q_{k+1}^*(\xi_k, u_{k+1}) \right| \tag{5.4.14} \\
&\le \sum_{i \in I_k} \phi_i(x_k) \left| r(\bar{x}_i, \bar{u}_k) + \gamma \max_{u_{k+1}} \hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) - r_k - \gamma \max_{u_{k+1}} Q_{k+1}^*(\xi_k, u_{k+1}) \right| \nonumber \\
&\le \sum_{i \in I_k} \phi_i(x_k) \left| L_r \left( \|\bar{x}_i - x_k\|_2 + \|\bar{u}_k - u_k\|_2 \right) + \gamma \max_{u_{k+1}} \hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) - \gamma \max_{u_{k+1}} Q_{k+1}^*(\xi_k, u_{k+1}) \right| \nonumber
\end{align}
$$

Using the triangular inequality property and introducing $Q_{k+1}^*(\xi_{i,k}, u_{k+1}) - Q_{k+1}^*(\xi_{i,k}, u_{k+1})$ the error can be bounded as:

$$
\begin{align*}
\epsilon_k &= |\hat{Q}_k - Q_k^*| \\
&\le L_r (\delta_x + \delta_u) + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} |\hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) - Q_{k+1}^*(\xi_{i,k}, u_{k+1}) + Q_{k+1}^*(\xi_{i,k}, u_{k+1}) - Q_{k+1}^*(\xi_k, u_{k+1})| \\
&\le L_r (\delta_x + \delta_u) + \epsilon_{k+1} + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} |Q_{k+1}^*(\xi_{i,k}, u_{k+1}) - Q_{k+1}^*(\xi_k, u_{k+1})|
\end{align*}
$$

Thus:

$$
\epsilon_k - \epsilon_{k+1} \le L_r (\delta_x + \delta_u) + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} \left| Q_{k+1}^*(\xi_{i,k}, u_{k+1}) - Q_{k+1}^*(\xi_k, u_{k+1}) \right| \quad (5.4.15)
$$

Since the optimal Q-function $Q_{k+1}^*$ is proved previously to be a Lipschitz function with the
corresponding Lipschitz constant $L_{k+1}$, the inequality (5.4.15) can be expressed as:

$$
\epsilon_k - \epsilon_{k+1} \le L_r (\delta_x + \delta_u) + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} L_{k+1} \|\xi_{i,k} - \xi_k\|_2
$$

With the Lipschitz property of $\xi$ and the convex sum property $\sum_{i \in I_k} \phi_i(x) = 1$, we have:

$$
\epsilon_k - \epsilon_{k+1} \le (L_r + L_{k+1} \gamma L_{\xi}) (\delta_x + \delta_u)
$$

With the same reasoning, the error between the approximate Q-function and the optimal one
for $k = K - 2, K - 1, \dots, K - m$:

$$
\epsilon_{K-1} - \epsilon_K \le (L_r + L_K \gamma L_\xi)(\delta_x + \delta_u)
$$

$$
\epsilon_{K-2} - \epsilon_{K-1} \le (L_r + L_{K-1} \gamma L_{\xi})(\delta_x + \delta_u)
$$

$$
\epsilon_{K-3} - \epsilon_{K-2} \le (L_r + L_{K-2} \gamma L_{\xi})(\delta_x + \delta_u)
$$
⋮

$$ \varepsilon_{K-m} - \varepsilon_{K-m+1} \le (L_r + L_{K-m+1} \gamma L_{\xi}) (\delta_x + \delta_u) $$

Summing up the right and the left sides of the inequalities above, we obtain the error $\varepsilon_{K-m}$ as:

$$ \varepsilon_{K-m} - \varepsilon_K \le \sum_{z=1}^{m} (L_r + L_{K-z+1} \gamma L_{\xi}) (\delta_x + \delta_u) \quad (5.4.16) $$

Since we can compute the exact Q-function of the final state, the error $\varepsilon_K = 0$. With $\varepsilon_K = 0$ and (5.4.16), we have:

$$ \varepsilon_{K-m} \le \sum_{z=1}^{m} (L_r + L_{K-z+1} \gamma L_{\xi}) (\delta_x + \delta_u) \quad (5.4.17) $$

For the special case $\gamma L_{\xi} = 1$, the bound of (5.4.17) can be expressed as:

$$ \varepsilon_{K-m} \le m(\delta_x + \delta_u) \left( L_r + L_T + \frac{m-1}{2} L_r \right) $$

And with $k = K - m$ it corresponds to (5.4.8).

Otherwise, when $\gamma L_{\xi} < 1$ and $\gamma L_{\xi} > 1$, with $k = K - m$, (5.4.17) can be bounded as:

$$ \varepsilon_k \le (K-k)L_r(\delta_x + \delta_u) + \sum_{z=1}^{K-k} \left( L_T (\gamma L_{\xi})^z + L_r \frac{(\gamma L_{\xi})^z - \gamma L_{\xi}}{\gamma L_{\xi} - 1} \right) (\delta_x + \delta_u) \quad (5.4.18) $$

Which corresponds to (5.4.7). Note that if $\gamma L_{\xi} < 1$ or $\gamma L_{\xi} = 1$, the error bound $\varepsilon_k$ converges to zero when the resolution steps $\delta_x$ and $\delta_u$ tend to zero. For $\gamma L_{\xi} > 1$, due to $(\gamma L_{\xi})^z$ the proposed error bound $\varepsilon_k$ above increases exponentially, when the horizon increases. Since the horizon is finite, the error bound converges still to zero when the resolution steps $\delta_x$ and $\delta_u$ tend to zero. In what follows, we search a new error bound which provides a better feature in terms of convergence.
When $\gamma L_f > 1$, another error bound can be considered as follows. Consider again the error $\epsilon_k$ between the approximate Q-function and the optimal one, in (5.4.14) and introducing the null quantity $\hat{Q}_{k+1}(\xi_k, u_{k+1}) - \hat{Q}_{k+1}(\xi_k, u_{k+1})$ a new bound can be obtained as:

$$ \begin{aligned} \epsilon_k &= |\hat{Q}_k - Q_k^*| \\ &\le L_r(\delta_x + \delta_u) + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} |\hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) - \hat{Q}_{k+1}(\xi_k, u_{k+1}) + \hat{Q}_{k+1}(\xi_k, u_{k+1}) - Q_{k+1}^*(\xi_k, u_{k+1})| \\ &\le L_r(\delta_x + \delta_u) + \epsilon_{k+1} + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} |\hat{Q}_{k+1}(\xi_{i,k}, u_{k+1}) - \hat{Q}_{k+1}(\xi_k, u_{k+1})| \end{aligned} $$

Define the new set of indexes: $I'_{k+1} = \{i' | \phi_{i'}(\xi_{i,k}) \neq 0 \text{ or } \phi_{i'}(\xi_k) \neq 0\}$, we have:

$$ \epsilon_k \le L_r(\delta_x + \delta_u) + \epsilon_{k+1} + \sum_{i \in I_k} \phi_i(x_k) \gamma \max_{u_{k+1}} \left| \sum_{i' \in I'_{k+1}} \left[ (\phi_{i'}(\xi_{i,k}) - \phi_{i'}(\xi_k)) \theta_{i',j,k+1} \right] \right| \quad (5.4.19) $$

With $j = \arg\min_{j'} \|u_{k+1} - u_k^{j'}\|$ and $u_k^{j'} \in U_d$. Using the Lipschitz property of $\xi$ and the convex um property of the triangular membership function $\phi$ with the Lipschitz constant $L_\xi$ and $L_\phi$ respectively,

$$ \|\phi_{i'}(\xi_{i,k}) - \phi_{i'}(\xi_k)\|_2 \le L_\xi L_\phi (\delta_x + \delta_u) \quad (5.4.20) $$

With the inequality (5.4.19), the inequality (5.4.20) can be relaxed:

$$ \epsilon_k \le L_r(\delta_x + \delta_u) + \epsilon_{k+1} + \sum_{i \in I_k} \phi_i(x_k) \gamma L_\xi L_\phi (\delta_x + \delta_u) \left| \sum_{i' \in I'_{k+1}} \theta_{i',j,k+1} \right| $$

For every $x$, only a finite number $2^{N_{\text{state}}}$ of MFs are non-zero and the cardinality of $I'_{k+1}$ $|I'_{k+1}| \le 2^{N_{\text{state}}}$. Then,

$$ \begin{aligned} \epsilon_k &\le L_r(\delta_x + \delta_u) + \epsilon_{k+1} + \sum_{i \in I_k} \phi_i(x_k) * 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_{k+1}\|_\infty \\ &\le L_r(\delta_x + \delta_u) + \epsilon_{k+1} + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_{k+1}\|_\infty \end{aligned} $$

And

$$ \epsilon_k - \epsilon_{k+1} \le L_r(\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_{k+1}\|_\infty $$
With the same reasoning, the error between the approximate Q function and the optimal one for $k = K-2, K-1, \dots, K-m$:

$$ 
\begin{aligned}
\varepsilon_{K-1} - \varepsilon_K &\le L_r(\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_K\|_\infty \\
\varepsilon_{K-2} - \varepsilon_{K-1} &\le L_r(\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_{K-1}\|_\infty \\
&\vdots \\
\varepsilon_{K-m} - \varepsilon_{K-m+1} &\le L_r(\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \|\theta_{K-m+1}\|_\infty
\end{aligned}
$$

Summing up the right and the left sides of the inequalities above, we obtain the error $\varepsilon_{K-m}$ as:

$$ \varepsilon_{K-m} \le m L_r (\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \sum_{z=1}^{m} \|\theta_{K-z+1}\|_\infty $$

And we get with $k = K-m$:

$$ \varepsilon_k \le (K-k) L_r (\delta_x + \delta_u) + 2^{N_{\text{state}}} \gamma L_\xi L_\phi (\delta_x + \delta_u) \sum_{z=1}^{K-k} \|\theta_{K-z+1}\|_\infty \quad (5.4.21) $$

That corresponds to (5.4.9), the last expression of proposition 1. At last notice that we have also for the last case (5.4.21) $\lim_{\delta_x \to 0, \delta_u \to 0} \bar{\varepsilon}_k = 0$, $k = K-1, \dots, 0$ that ends the proof. ■

## 5.5 Reinforcement Learning for Energy Optimization of PAWs

To represent the optimal control problem (5.3.4), where the objective is to minimize the electric energy consumption for a given driving task while producing a desired initial-to-final constraint of users, the terminal reward and the stage reward of (5.3.6) are defined as follows:

$$ 
\begin{aligned}
T(x_N) &= -[w_{e1} w_{e2}] \begin{bmatrix} (d_K - d_{ref})^2 \\ (S_{of K} - S_{of-ref})^2 \end{bmatrix} \quad (5.5.1) \\
r(x_k, u_k) &= -\frac{1}{2} U_k^2
\end{aligned}
$$
where the state vector is $x_k = [d_k, v_k, S_{of_k}]^T$ and the control input is the motor torque $u_k = U_k$.

Since the driving task is to travel a predefined distance, negative human torque and negative motor torque are inefficient in terms of metabolic-electrical energy consumption over the driving task. Moreover, due to the actuator limitations, the maximum torque that the motor can provide is $U_{max}$. Therefore, the control is bounded: $0 \le U \le U_{max}$. Since the distance is monotonic, it acts as a proxy for time, which can be implicitly used by the algorithm instead of an explicit time variable. Therefore, we can use a time-invariant solution $\bar{\pi}_{\lambda}(x_k)$ to approximate the optimal time-varying solution in (5.4.2). We approximate the deterministic part $\bar{\pi}$ of the motor torque by the following RBF expansion:

$$ \bar{\pi}_{\lambda}(x_k) = \lambda_{i}^{T} \varphi(x_k) \qquad (5.5.2) $$

where the RBF $\varphi_i = \exp(-\beta \|x_k - c_i\|^2)$, $c_i = c_i^{(1),...,M}$ is the center vector of the RBFs, $M$ is the total number of RBFs and $\beta$ is the radial parameter. Since the radial parameter $\beta$ is the same for each RBF, all the RBFs have the same shape.

Hereinafter, for each variable, a subscript or index P (resp. G) stands for PoWER (resp. GPOMDP).

In model-free Policy Search, exploration is indispensable to learn the unknown dynamics. Stochastic policies are needed to explore. To this end, we use a parameterized policy with the parameters $\lambda$. Then, the stochastic policy distribution is $\tilde{\pi}_{\lambda}(u_k|x_k, k)$. Under this stochastic policy, the probability distribution $p_{\lambda}(\tau)$ over trajectories $\tau$ can be expressed in the following way:

$$ p_{\lambda}(\tau) = p(x_0) \prod_{k=0}^{K-1} \tilde{\pi}_{\lambda}(u_k|x_k, k) \qquad (5.5.3) $$

where $p(x_0)$ is the initial state distribution. Under trajectories $\tau$ generated by $\tilde{\pi}_{\lambda}$, the expected return is:

$$ \bar{R}_{\lambda} = \int p_{\lambda}(\tau) R(\tau) d\tau \qquad (5.5.4) $$
### 5.5.1 GPOMDP

The GPOMDP (Gradient of a Partially Observable Markov Decision Processes) algorithm (Peters et al. 2006) updates the control parameters $\lambda$ in the steepest ascent direction so that the expected return (5.5.4) is maximized. We apply this algorithm to estimate the gradient $\nabla_{\lambda} \bar{R}_{\lambda}$, which can be obtained from the stage rewards $r_j$ and the distribution $\tilde{\pi}_{\lambda}$. The entire procedure is given in Algorithm 2, where $\Gamma$ is the total trials.

**Algorithm 2. GPOMDP**

1. Initialize $\lambda_0$

2. **for** $l = 0, 1, 2, \dots, \Gamma$

3. Generate $N_{\tau}$ trajectories $\tau$ of length $K$ using $\lambda_l$

4. $\nabla_{\lambda} \bar{R}_{\lambda} = \frac{1}{N_{\tau}} \sum_{\zeta=1}^{N_{\tau}} \left[ \sum_{k=0}^{K-1} \sum_{h=0}^{k} \left[ \nabla_{\lambda} \log \tilde{\pi}_{\lambda}(u_h^{\zeta} | x_h^{\zeta}, k) \right] r_k^{\zeta} \right]$

5. $\lambda_{l+1} = \lambda_l + \alpha \cdot \nabla_{\lambda} \bar{R}_{\lambda}$ with the learning rate $\alpha > 0$

6. **end for**

In line 3 of Algorithm 2, for each iteration $l$ we generate $N_{\tau}$ trajectories using the stochastic policy with $\lambda_l$. Applying the Likelihood Ratio Estimator, calculating the gradient $\nabla_{\lambda} \bar{R}_{\lambda}$ is transformed to calculating $\nabla_{\lambda} \log \tilde{\pi}_{\lambda}(u_k | x_k, k)$. To this end, zero mean Gaussian noise $z_G$ is added to the executed action and renders the policy (5.5.2) stochastic. In order to prevent the executed action from violating the action saturation limits, the stochastic motor torque is selected with:

$$q_{\text{sat}}\left[\lambda_l^{GT} \varphi(x_k) + z_G\right] \qquad (5.5.5)$$

where $q_{\text{sat}}$ is a smooth saturation (the Gaussian error function (5.5.5) shown at the top of Figure 5.5.1) between $[0, U_{\text{max}}]$ such that the stochastic action is differentiable with respect to $\lambda_l^G$. When the optimal action is close to the borders of the interval $[0, U_{\text{max}}]$, using the original return (5.5.1) without input saturation can lead to the divergence of the parameters. To address this problem, a penalty function $P$ is added to the stage reward (5.5.1) as follows:

$$r(x_k, u_k) = -\left[\frac{1}{2} U_k^2 + w_{e3} P(U_k)\right] \qquad (5.5.6)$$
where $w_3$ is the constraint penalty weight. The function $P$, shown in Figure 5.5.1 bottom, is
defined as follows:

$$
P = \begin{cases}
\sin\left(\frac{\pi(U - U_{\max})}{0.04U_{\max}}\right) + 1 & 0.98U_{\max} \le U \le U_{\max} \\
0 & 0.02U_{\max} \le U \le 0.98U_{\max} \\
\sin\left(\frac{\pi(-U)}{0.04U_{\max}}\right) + 1 & 0 \le U \le 0.02U_{\max}
\end{cases}
\quad (5.5.7)
$$

which penalizes the (stochastic) action when it is close to the saturation value. The objective
of $P$ is to keep the mean value of the stochastic actions inside the interval $[0, U_{\text{max}}]$.

<figure><img src="image_59.png" /><figcaption>Figure 5.5.1. Smooth saturation function $q_{\text{sat}}$ (above) and penalty function $P_s$ for $U_{\text{max}} = 50N$ (below)</figcaption></figure>

Recall that we use a time-invariant policy. Consequently, the stochastic action distribution
does not depend on the time stage *k*, but on the state *x*<sub>*k*</sub>. According to (5.5.5), the distribution
tilde pi (U_k | x_k) of the stochastic motor torque *U* is:

$$
\tilde{\pi}_{\lambda}^{G}(U_k|x_k) = \frac{1}{\sqrt{2\pi\sigma_G^2}} \exp\left(-\frac{[q_{\text{sat}}^{-1}(U_k) - \lambda_l^{GT}\varphi(x_k)]^2}{2\sigma_G^2}\right) \quad (5.5.8)
$$

The derivative of (5.5.8) with respect to $\lambda_l^G$ is used to estimate the gradient $\nabla_{\lambda} \bar{R}_{\lambda}$ in Algorithm 2 and to update the parameter vector $\lambda_l^G$. By tuning the parameters $(\beta, c, M)$ of the basis
functions (5.5.2), the standard deviation $\sigma_p$ and $\sigma_G$, the reward weights ($w_1, w_2$), the learning rate $\alpha$ and the penalty weight $w_3$, we have all the conditions to update the parameters $\lambda^G$.

The stochastic policy distribution $\pi_\lambda$ is available, so that the gradient $\nabla_\lambda \bar{R}_\lambda$ can be computed. The expected value is approximated by Monte Carlo techniques using the $N_\tau$ trajectories. The learning rate $\alpha$ has to be tuned manually in order for the control parameters $\lambda$ to converge efficiently.

### 5.5.2 Simulation validation with baseline solution

To solve the finite-horizon problem, we use first the model-based approach i.e. the algorithm 1 to derive a baseline solution. The whole set of parameters is shown in Table II. We choose the discount factor $\gamma$ as 1. For a horizon of 10s with a sampling time 0.05s, the number of the backward iteration is 200. To represent the finite-horizon return, the terminal cost is used firstly to compute the Q-function of the last time step, and then each stage is gradually added via the backward dynamic programming iterations. In total, 200 Q-functions are generated to represent a time-varying Q-function for a horizon of 10s. Moreover, we derive the policy from the obtained time-varying Q-function in the forward direction, by choosing the action which maximizes the Q-function of that step and apply it to the system.

<table><caption>Table II. PARAMETERS OF THE CONSIDERED HUMAN-WHEELCHAIR DYNAMICS</caption><thead><tr><th>Meaning</th><th>Notation [units]</th><th>Value or domain</th></tr></thead><tbody><tr><td>Sampling time</td><td>T<sub>e</sub> [s]</td><td>0.05</td></tr><tr><td colspan="3">Human parameters</td></tr><tr><td>Recovery coefficient</td><td>R[s<sup>-1</sup>]</td><td>0.0063</td></tr><tr><td>Fatigue coefficient</td><td>F [s<sup>-1</sup>]</td><td>0.153</td></tr><tr><td>MVC</td><td>M<sub>vc</sub> [N]</td><td>100</td></tr><tr><td>Fraction of V<sub>max</sub></td><td>f</td><td>0.5</td></tr><tr><td>Human control gain</td><td>K<sub>p</sub></td><td>30</td></tr><tr><td colspan="3">Wheelchair parameters</td></tr><tr><td>Wheel radius</td><td>ζ [m]</td><td>0.33</td></tr><tr><td>Maximum velocity</td><td>V<sub>max</sub> [rad/s]</td><td>7</td></tr><tr><td>System matrix</td><td>A</td><td><table><tbody><tr><td>1</td><td>0.05</td></tr><tr><td>0</td><td>0.9406</td></tr></tbody></table></td></tr><tr><td>Input matrix</td><td>B</td><td><table><tbody><tr><td>0</td></tr><tr><td>0.0059</td></tr></tbody></table></td></tr><tr><td colspan="3">Driving schedule configuration</td></tr></tbody></table>
<table><tr><td>Finite horizon</td><td>K</td><td>200</td></tr><tr><td>Initial state of fatigue</td><td>S<sub>of 0</sub></td><td>0.5</td></tr><tr><td>Desired final human fatigue</td><td>S<sub>of-ref</sub></td><td>0.5</td></tr><tr><td>Distance-to-go</td><td>d<sub>ref</sub> [rad]</td><td>20</td></tr><tr><td colspan="3">State-space and action-space region</td></tr><tr><td>Distance</td><td>d [rad]</td><td>[0,20]</td></tr><tr><td>Velocity</td><td>v [rad/s]</td><td>[0,7]</td></tr><tr><td>State of fatigue</td><td>S<sub>of</sub></td><td>[0.35,0.7]</td></tr><tr><td>Motor torque</td><td>U [Nm]</td><td>[0,50] (U<sub>max</sub> = 50)</td></tr></table>

For the PG approach, an equidistant three dimensional 5 × 5 × 8 grid is selected as the centers of the RBFs. In total, 200 RBFs ($M = 200$ and $\beta = 0.5$), together with a parameter vector $\theta \in \mathbb{R}^{200}$ are used to approximate the controller (5.5.2). The learning rate $\alpha$ is chosen as $10^{-5}$.

<figure><img src="image_60.png" /><figcaption>Figure 5.5.2. Simulation results provided by GPOMDP algorithm and ADP algorithm</figcaption></figure>

A number of 8000 trajectories of 10s are performed to learn the control parameters $\theta$. We compare the solution of PG with the solution obtained by the ADP. As shown in Figure 5.5.2,
PG approach (solid line) has a similar performance with ADP approach (dotted line). The simulation results are: the final return is -8.20e4 for PG (the energy consumption: -5.39e4 and the terminal penalty: -2.81e4). The final return is -6.98e4 for ADP (the energy consumption: -6.4726e4 and the terminal penalty: -5.1108e3). The PG approach provides 12.7% less return than ADP. However, the PG approach eliminates the need for a model by accepting this loss in return. It is important to mention this 12.7% difference includes both an electrical energy component and a difference in the final *Sof* reached by both methods.

From a practical point of view, first the “user” (of course it is an abuse of language as this is only simulation) cooperates with the motor to push the wheelchair. After reaching a suitable velocity between 1s and 3.5s, the “user” reduces his applied force to reduce his fatigue. During this time, the electrical motor provides the main input to maintain this velocity. In the reminder of the driving, the motor assistance is reduced gradually to minimize the energy consumption. Moreover, the “user” tries to attain the desired final fatigue level by reducing his force. The system uses the kinetic energy given previously by the user and the motor to end the mission. During the driving task, the provided assistive algorithm tries to provide an energy-efficient assistance to the user so that his final fatigue level reaches the desired one.

For the model-free PG approach, we have a terminal error of 0.05 between the final $S_{ofK}$ and the desired final value $S_{of-ref}$ (0.02 for the ADP approach). This error can be reduced by increasing the weighting factor $w_{e2}$. However, the energy consumption should have a significant weight in the return function (34) to fulfill the optimization objective. The weight parameters $w_{e1}$, $w_{e2}$ and $w_{e3}$ must be tuned to have a tradeoff between reaching the terminal conditions and minimizing the energy consumption. The learning rate $\alpha$ tuning also depends on the weighting factors and parameters $(\beta, c, M)$. Since no prior knowledge about the optimal policy is available, an equidistant grid on the given intervals is chosen for the centers of the RBFs. If we increase the number of RBFs, the approximate controller may tend to the optimal solution after receiving enough training. Roughly speaking, around 20-30 preliminary experiments are required to fix the 4 parameters and the RBFs used in this work.

As a considerable amount of data is needed to obtain a high performance controller, more PG learning techniques will be investigated to reduce the learning time in the next section. The ultimate objective is to develop an efficient real-time learning control of PAWs.
## 5.6 Applying PoWER to improve Data-Efficient

The energy optimization problem in the previous section requires a considerable amount of data to get a solution, which in practice would be impossible to obtain. Therefore, the main purpose of this section is to increase the data efficiency. To achieve this goal, we propose two ideas. The first one is to use a more efficient PG algorithm, namely PoWER. Secondly, as observed in (Feng et al. 2018) the operating region in the state space is concentrated on a few radial basis functions (RBFs); therefore, for the remaining RBFs the parameters remain constant or have a very small gradient. Reducing the parameters to the significant ones will accelerate the learning speed. Using Fuzzy Q-iteration as the baseline solution, we compare the performance of the two PG algorithms (PoWER and GPOMDP) with the new controller parameterizations and the one in previous section.

### 5.6.1 PoWER

To obtain a higher expected return, we may consider a new distribution $p_{\lambda'}(\tau)$ over trajectories that might provide a better expected return than the previous one i.e. $\int p_{\lambda'}(\tau)R(\tau)d\tau \ge \int p_{\lambda}(\tau)R(\tau)d\tau$. The new expected return $\int p_{\lambda'}(\tau)R(\tau)d\tau$ with parameters $\lambda'$ is lower-bounded by a quantity $L_{\lambda}(\lambda')$ that depends on $\lambda$. The analytical expression of $L_{\lambda}(\lambda')$ (Kober and Peters 2009) is expressed as follows:

$$L_{\lambda}(\lambda') = -D(p_{\lambda}(\tau)R(\tau) \| p_{\lambda'}(\tau)) \quad (5.6.1)$$

The selection of $\lambda'$ can be done by maximizing the lower bound $L_{\lambda}(\lambda')$ to implicitly maximize(5.5.4). In (Dayan and Hinton 1997), the authors show that maximizing $L_{\lambda}(\lambda')$ guarantees the improvement of the expected return. The intuition is that if $R(\tau_1) > R(\tau_2)$, the new $\lambda'$ will put more probability mass on $\tau_1$ than $\lambda$ does.

PoWER (Policy learning by Weighting Exploration with the Returns) is one of gradient-free optimization criterion which works by maximizing the lower bound $L_{\lambda}(\lambda')$. Moreover, a deterministic policy is approximated by general basis functions $\psi$ i.e. $\bar{\pi}_{\lambda}(x_k) = \lambda^T \psi(x_k, k)$. For exploration, Gaussian noise is added directly to the parameter vector $\lambda$. Using importance sampling (Neal 2001), the parameters $\lambda$ are updated with the $N_s$ trials which have the highest return among the performed trials. The formula to update the parameters is (Kober and Peters 2009):
$$ \lambda_{l+1} = \lambda_l + \frac{\sum_{s=1}^{N_s} (\lambda_s - \lambda_l) R(\tau_s)}{\sum_{s=1}^{N_s} R(\tau_s)} \quad (5.6.2) $$

The whole method is given in Algorithm 3.

**Algorithm 3. PoWER**

1. Initialize $\lambda_0$

2. **for** $l = 0, 1, 2, \dots, \Gamma$

3. Generate a new trajectory $\tau$ of length $K$ using $\lambda_l$

4. Sort the performed trials decreasingly by return

5. Select the $N_s$ trials with the highest return

6. Update parameters $\lambda_{l+1} = \lambda_l + \frac{\sum_{s=1}^{N_s} (\lambda_s - \lambda_l)R(\tau_s)}{\sum_{s=1}^{N_s} R(\tau_s)}$

7. **end for**

The exploration is carried out in the parameter space as previously explained. The zero mean Gaussian noise vector $z_p$ with the standard deviation $\sigma_p$ is added to the parameters and renders the action stochastic as follows:

$$ \text{sat}\left(0, U_{\text{max}}, (\lambda_l^P + z_p)^T \varphi(x_k)\right) \quad (5.6.3) $$

where the stochastic motor torque is saturated between 0 and $U_{\text{max}}$ and the parameter vector $\lambda_l^P$ is updated by (5.6.2). By tuning the parameters $(\beta, c, M)$ of the basis functions (5.5.2), the standard deviation $\sigma_p$ and we have all the conditions to update the parameters $\lambda^P$.

### 5.6.2 Learning time comparison between GPOMDP and PoWER

In this section, simulations are carried out to compare the proposed methods. The whole set of parameters is shown in Table II. The human model parameters are adapted from (Fayazi et al. 2013) to have a reasonable fatigue and recovery rate to avoid a trivial optimal solution. The control strategy is approximated over the state-space and action-space region given in Table III. The configurations and learning parameters of the return function, penalty function, model-based policy, and model-free policies are shown in the following Table III.
<table><caption>Table III. RETURN FUNCTION, PENALTY FUNCTION, MODEL-BASED POLICY, MODEL-FREE POLICIES CONFIGURATIONS, AND LEARNING PARAMETERS</caption><tr><td colspan="2">Return function and penalty function configuration</td></tr><tr><td>Reward weight matrix [w<sub>e1</sub> w<sub>e2</sub>]</td><td>[4000 10<sup>7</sup>]</td></tr><tr><td>Penalty weight w<sub>e3</sub></td><td>800</td></tr><tr><td colspan="2">Q-function approximation</td></tr><tr><td>Centers of triangular functions φ distributed on an equidistant grid</td><td>10 × 10 × 41 over the state-space (x = [d, v, S<sub>of</sub>]<sup>T</sup>)</td></tr><tr><td>Number of equidistant discrete actions</td><td>15</td></tr><tr><td colspan="2">Radial basis functions (5.5.2) configuration 1</td></tr><tr><td>Radial parameter β</td><td>0.5</td></tr><tr><td>Centers of RBFs distributed on an equidistant grid</td><td>5 × 5 × 8</td></tr><tr><td>Total number of RBFs M</td><td>200</td></tr><tr><td colspan="2">Radial basis functions (5.5.2) configuration 2</td></tr><tr><td>Radial parameter β</td><td>0.5</td></tr><tr><td>Centers of RBFs distributed on an equidistant grid</td><td>5 × 5 × 1</td></tr><tr><td>Total number of RBFs M</td><td>25</td></tr><tr><td colspan="2">GPOMDP parameters</td></tr><tr><td>Learning rate α</td><td>10<sup>-5</sup></td></tr><tr><td>Standard deviation σ<sub>G</sub></td><td>5</td></tr><tr><td colspan="2">POWER parameters</td></tr><tr><td>Importance sampling N<sub>s</sub></td><td>10</td></tr><tr><td>Standard deviation σ<sub>P</sub></td><td>1</td></tr></table>

The number in the legend gives the total parameters of the controller approximation (5.5.2) for each simulation. A mean value along with a 95% confidence interval calculated for 10 independent simulations is given (each simulation with 400 trials). Figure 5.6.1 shows that with the same policy parametrization, PoWER has a considerably higher data efficiency than GPOMDP. GPOMDP-25 and GPOMDP-200 give a similar final performance. Considering the mean, 90% of the baseline return is provided in around 100 trials by PoWER-25. The same performance is given in around 200 trials by PoWER-200. Overall, PoWER-25 is the best choice among the considered configurations.
<figure><img src="image_61.png" /><figcaption>Figure 5.6.1. The mean performance and 95% confidence interval on the mean value of PoWER with 25 control parameters (PoWER-25), PoWER with 200 (PoWER-200), GPOMDP with 25 control parameters (GPOMDP-25) and GPOMDP with 200 (GPOMDP-200)</figcaption></figure>

<figure><img src="image_62.png" /><figcaption>Figure 5.6.2. Controlled trajectories provided by GPOMDP and PoWER and fuzzy Q-iteration algorithm</figcaption></figure>
For the next simulations, we focus on the final near-optimal behaviours provided by PoWER-25 and GPOMDP-25. To this end, 400 trials and 8000 trials are performed to learn the parameter vectors $\lambda^P$ and $\lambda^G$, respectively. The slow learning speed of GPOMDP is mainly due to the exploration noise added directly to actions at every step. This type of exploration strategy can cause a high variance for learning algorithm (Kober et al. 2009) and leads to a poor performance in terms of data-efficiency. As shown in Figure 5.6.2, the first solution of the model-free methods PoWER (red solid line) and GPOMDP (blue solid line) are comparable to the model-based fuzzy Q-iteration (black dotted line baseline solution). PoWER-25 has the fastest convergence among other approach and other configuration. The final return is -82017, -70242, and -69837 for GPOMDP, PoWER and fuzzy Q-iteration respectively. Here again, PoWER delivers a better solution than GPOMDP in terms of final return.

The simulation was done on an Intel Core i7-6500 CPU @ 2.50GHz. The average elapsed CPU time to compute a control action is 1.0 * 10⁻⁴s, 7.6 * 10⁻⁴s, 1.5 * 10⁻⁴s, and 7.6 * 10⁻⁴s respectively for PoWER-25, PoWER-200 GPOMDP-25 and GPOMDP-200. As their elapsed CPU time is significantly less than the sampling time of 0.05s, it is possible to embed them into a real PAW.

### 5.6.3 Adaptability to different human fatigue dynamics

In this section, we turn our focus towards adaptation to human fatigue variability, which is crucial for a personalized PAW. In what follows, we investigate only the adaptability of PoWER-25 to these changes, since it provided the best results in the previous section. The objective of this investigation is to confirm the possibility of having a generic solution for different human fatigue dynamics. To represent various human fatigue dynamics, we change the parameters of the human fatigue (5.2.1) as follows:

$$F' = \frac{1}{\eta} F;$$

$$R' = \eta R;$$

$$M'_{vc} = \eta M_{vc}$$

where $F$, $R$, $M_{vc}$ are the nominal parameters used in Table II. A value $\eta > 1$ corresponds to a user physically stronger than the nominal one, because they get exhausted slower, recover faster and have more Maximum Voluntary Contraction force. On the contrary, $\eta < 1$ corresponds to a physically weaker user. Adaptation starts from the parameters found using the nominal model. As a baseline, we compare this adaptation procedure with simply
resetting the parameters to zero values when the model changes. The same variance $\sigma_p$ of Section 5.6.2 is applied for exploration. Both stronger ($\eta = 2$) and weaker ($\eta = 1/2$) users are studied. Figure 5.6.3 shows that PoWER is clearly much more efficient when initialized with the nominal model, being able to provide a good return directly and to find a new near-optimal solution for the new fatigue dynamics in less than 50 trials.

In order to confirm that the assistive control can adapt to a bigger range of parameter changes, we carry out the same comparison for $\eta = 8, 4, 3, 1/3, 1/4, 1/8$. Table III gives the baseline return for each $\eta$, the minimal return for each case and the number of trials to converge to 90% of the corresponding baseline return for both initializations. The asterisk * represents situations where the learning algorithm fails to converge to the 90% of the baseline return within 400 trials.

<figure><img src="image_63.png" /><figcaption>Figure 5.6.3. The mean performance of PoWER for both initialization (Top: $\eta = 2$ and bottom $\eta = 1/2$)</figcaption></figure>

Table IV shows that both initializations have similar convergence for $\eta = 8$. For $\eta = 3$, the initialization to zero has a faster convergence. This result may be because that the initialization to zero is closer to the optimal solution. Nevertheless, for all the other $\eta$, the initialization with the nominal model converges faster. Overall, starting learning with the nominal solution can guarantee a higher minimum return. Moreover, PoWER with prior knowledge adapts reasonably well to human fatigue dynamic changes without tuning again
the learning parameter $\sigma$. This study therefore confirms the possibility of providing an adaptive solution for different human fatigue dynamics.

<table><caption>Table IV. POWER WITH VARYING FATIGUE MODEL (ZERO: INITIALIZATION TO ZERO, NOMINAL: INITIALIZATION WITH THE NOMINAL MODEL. THE MINIMAL RETURN IS NORMALIZED BY THE CORRESPONDING BASELINE RETURN)</caption><thead><tr><th rowspan="3">η</th><th rowspan="3">Baseline return<br>(fuzzy Q-<br>iteration)</th><th colspan="4">POWER</th></tr><tr><th colspan="2">Minimal return</th><th colspan="2">Number of trials</th></tr><tr><th>Nominal</th><th>Zero</th><th>Nominal</th><th>Zero</th></tr></thead><tbody><tr><td>8</td><td>-361950</td><td>1.25</td><td>2.30</td><td>39</td><td>37</td></tr><tr><td>4</td><td>-96723</td><td>1.76</td><td>4.96</td><td>33</td><td>56</td></tr><tr><td>3</td><td>-54018</td><td>2.14</td><td>7.58</td><td>47</td><td>34</td></tr><tr><td>2</td><td>-32744</td><td>2.38</td><td>12.43</td><td>48</td><td>65</td></tr><tr><td>1/2</td><td>-150920</td><td>1.51</td><td>5.25</td><td>10</td><td>*</td></tr><tr><td>1/3</td><td>-207400</td><td>1.86</td><td>5.52</td><td>198</td><td>*</td></tr><tr><td>1/4</td><td>-299540</td><td>1.76</td><td>4.50</td><td>37</td><td>*</td></tr><tr><td>1/8</td><td>-657620</td><td>1.56</td><td>2.73</td><td>30</td><td>*</td></tr></tbody></table>

### 5.6.4 Experimental Validation

To demonstrate the effectiveness of the proposed learning algorithm, proof-of-concept experiments have been conducted on the PAW prototype. Via a joystick, the user can return a subjective evaluation of his/her $S_{of}$ to the control algorithm. When the user pushes the joystick to the negative or positive Y-direction, the joystick returns to the algorithm a discrete value $-1$ or $1$, respectively. The neutral position of the joystick returns a discrete value $0$. These three discrete values $-1, 0$ and $1$ mean respectively that the user feels too tired, is comfortable, and feels insufficiently tired (is willing to exercise more). The discrete signal is filtered so that when it changes between two levels (among $-1, 0$ and $1$), its filtered version $I$ provides a gradual transition between these levels. Furthermore, to avoid the need for too many pushes of the joystick, after such a transition the filtered signal is kept nearly constant for a certain duration.

The driving scenario consists in riding on a straight flat road with a given reference velocity $v_{ref}$ set by the user. The velocity estimated from the position encoders is available via the computer connected to the data acquisition system. The control objective is to minimize both the electrical energy and the use of the joystick, while tracking the reference velocity. Therefore, the stage reward function is:
$$r_k = -w_{e1}(v_k - v_{ref k})^2 - w_{e2}I_k^2 - w_{e3}U_k^2 \quad (5.6.4)$$

where $v_{ref k}$ is the given reference velocity at sample k. The reward weights are $w_{e1} = 10$, $w_{e2} = 0.25$ and $w_{e3} = 0.05$. Note that any joystick signal $I \neq 0$ is penalized. The controller is configured as a PI-type law:

$$U_k = \lambda_1(v_k - v_{ref k}) + \lambda_2 \sum_{i=0}^{k} (v_i - v_{ref i}) + \lambda_3 I_k + \lambda_4 \sum_{i=0}^{k} I_i - \lambda_5 F_{hk} \zeta \quad (5.6.5)$$

The first four terms of the controller (5.6.5) are used to track the reference $v_{ref}$ while keeping the filtered joystick signal $I$ to 0. The term $\lambda_5 F_{hk} \zeta$ is for compensating the human input.

One healthy male volunteer (29-year-old) performed the proof-of-concept experiments. There are 5-minute rest periods between consecutive trials. In total, 24 trials with the same driving condition have been carried out on the same day to learn the parameter vector $\lambda$ in (5.6.4). Figure 5.6.4 shows the total return of each trial. Among the 24 trials, 3 trials went unstable at the beginning of learning. For these trials, the velocity is oscillating around the set-point and the amplitude of oscillation is increasing. Therefore, the user stopped immediately the wheelchair and a very low return was given to the learning algorithm to avoid such situations in the future. The return tends to increase gradually after performing these trials. We notice that the obtained curve of return is noisy. Due to the time-consuming nature of the experiment, it is not feasible to perform many trials to obtain a smooth mean return.

<figure><img src="image_64.png" /><figcaption>Figure 5.6.4. The total return of each trial</figcaption></figure>
Figure 5.6.5 shows the trajectories of the first four stable trials and the last four trials. The motor torques are normalized between -1 and 1. The values 1 and -1 represent respectively the maximum torque in the position direction and the maximum torque in the negative direction. We remark that the user does not push the joystick anymore in the last four trials. The joystick signal *I* sums up the influence of main physiological and psychological factors to tell the learning algorithm what assistive torque is suitable to users. The fact that the user does not use the joystick at the end means that after training, the provided assistive torques are acceptable in terms of the sensation of fatigue. Another consequence of training is that the user and the controller track together the given velocity more smoothly.

Through these proof-of-concept experiments, we conclude that the proposed learning algorithm PoWER is able to improve the performance of the controller (5.6.4). For a final commercial product, there will be a certain accommodation time to obtain a satisfactory performance, during which a health professional would help the user interact with the PAW.

<figure><img src="image_65.png" /><figcaption>Figure 5.6.5. The trajectories of the first four stable trials and the last four trial. (The instant where the joystick is pushed is indicated on the *I* signal)</figcaption></figure>
## 5.7 Summary

In this chapter, a novel PAW control design has been proposed for paraplegic wheelchair users. The assistive strategy is based on energy optimization, while maintaining a suitable fatigue level for users and using minimal electrical energy over a distance-to-go. This optimal control problem was solved by the online model-free reinforcement learning methods PoWER and GPOMDP. Their near-optimality was confirmed by the model-based approach finite-horizon Q-fuzzy iteration. An important contribution is that the near-optimality of finite-horizon Q-uzzy iteration was proven. In addition, simulation results confirmed that PoWER with a simplified controller parameterization provides a considerably higher data efficiency, which renders the model-free framework better applicable in practice. Moreover, an investigation has been done to illustrate that PoWER is also able to adapt to human fatigue dynamic changes. Finally, a proof-of-concept experiment has been carried out to demonstrate the feasibility of the approach in practice.

Based on the proof-of-concept study of this chapter, next chapter gives the conclusion of the thesis and proposes an idea to integrate the model-free approach into the model-based assistive control.
# Chapter 6. Conclusion and future works

## 6.1 Conclusion

The work presented in this thesis tried to propose solutions for the assistance of Power-Assisted Wheelchair (PAW) with a minimum of sensors for the larger possible population of disabled persons. The goals were twofold: reducing the hardware cost to render the assistance kit as affordable as possible and rendering the assistance adaptable (as transparent as possible) to this highly heterogeneous population.

A “pure” classical approach of automatic control via an exhaustive modelling of both the wheelchair and the human was therefore prohibited. Effectively, not only heterogeneity would have been an important issue, but also feeding the model parameters would have been impossible case-by-case. Thus, two completely, not opposite, but different ways were explored during this work. The first one was to take profit of the best possibilities of performance and robust control techniques based on a mechanical model of the wheelchair. Results obtained were beyond our initial expectations, especially because we were able to do the assistance design without needing the main corrupted variable which is the user’s amplitude of propelling. The second way was to explore the possibilities of learning techniques applied without model for the assistive control. It was done keeping in mind, as people from automatic control community, that issues about proofs of convergence were important.

In order to highlight the current disability issues, Chapter 1 presented first the economic and social context and their corresponding challenges, such as high cost of assistive devices and heterogeneous disabled population. In such context, the chapter 2 provided a literature review of mechanical models of the wheelchair, model-based control and model-free control free techniques. This chapter were useful for the model-based PAW assistance design presented in Chapter 3 and Chapter 4 and for the model-free PAW assistance design of Chapter 5. The main contributions of this work are resumed as follows:

An innovative model-based assistive control has been proposed for a Power-Assisted Wheelchair. Using an unknown input observer (“software” or “virtual” sensor), human torques sensors are not required anymore and the cost of the assistive device is reduced
(Mohamad et al. 2015). Thanks to the reconstructed human torques, an algorithm is provided that allows defining reference trajectories for the center and yaw velocities. In addition, actuator saturation and uncertainties (user's mass and road conditions) were taken into account to design a robust observer-based tracking controller. The stability analysis of the complete closed-loop system was possible using an LMI constraints formalism under a two-steps algorithm. The effectiveness of the whole assistive control was confirmed via both simulations and experimental real-time tests.

A model-free assistance was designed for the PAW application. A case study illustrates the possibility to adapt the heterogonous disabled population (such as different human fatigue dynamics) using learning algorithms. To verify numerically the optimality of the model-free design, we used a model-based approach, such as finite-horizon fuzzy Q iteration, to derive a based-line solution. The (near-)optimality and the consistency of the finite-horizon fuzzy Q iteration were proved analytically. Moreover, a proof-of-concept experience was performed for validating the model-free design.

Based on the above design experiences, we finally proposed an idea to combine the model-based control and the model-free control to design a kind of personalized assistance of a PAW. This future direction for research seems promising as it will combine the advantages of both fields. References parameters would be adapted from learning techniques with guarantees of convergence and a high level of confidence, whereas tracking of the references would be ensured by the robust observer-based tracking controller. We give more perspectives in the next section.

## 6.2 Control-learning framework proposal and future works

In the previous three chapters, model-based control and model-free control solve separately two main problems of the PAW application. Based on a model-based design, an assistive control for PAW applications has been presented in Chapter 3 and validated with experimental results in. Adaptability to unknown human fatigue dynamic has been achieved by the model-free approach in Chapter 5. With the design experiences obtained in this thesis, we propose an innovative idea to combine control and learning for constructing an intelligent PAW. Furthermore, some theoretical perspectives are given in this chapter.
Results obtained in Chapter 5 show the proposed model-free approach is able to improve the assistive control after training. However, the wheelchair is modelled as one dimensional and goes only straight. For a practical PAW application, the rotation of the wheelchair has to be considered. For such a consideration, the state vector consists of two states e.g. center velocity and yaw velocity. The control inputs are the right and the left motor torques. Since both the number of states and the number of control inputs increase, the number of the control parameters becomes important. Thus, the time for learning a satisfying control may also increase considerably. In addition, torque sensors are needed to compute the control action.

From the results of Chapter 5, we can conclude that modelling the human-wheelchair system as a black-box may not be the best solution. Instead going from black-box to a grey-box seems a promising way. The prior knowledge of the human-wheelchair system has to be exploited. The simplified mechanical dynamic of the wheelchair is known in general. In order to remove torque sensors, a sufficiently precise model is first used to estimate human torques. To this end, an unknown input observer is designed. The simulation results in Section 3.2 and the experimental results in Section 4.1 confirm that a satisfying estimation performance is obtained.

Despite of a satisfying performance provided by the model-based assistive control, the reference generation may not be optimal respect to a particular user. To give an obvious example, we analysis the braking performances according to the two different weight users of Chapter 3. During braking, the decreasing rate of the center velocity for a heavy user should be smaller than the one for a light user. The reason is that the wheelchair with a heavier user needs more braking distance to disperse an important kinetic energy. Such a longer braking distance can be obtained by a smaller decreasing rate of the center velocity.

We show braking scenarios of both users in Figure 6.2.1 and Figure 6.2.2. These sequences are extracted from the experimental validations of Chapter 4. With these two example, we explain that a same constant decreasing rate of the center velocity for different users may not be optimal to obtain a personalized braking.
<figure><img src="image_66.png" /><figcaption>Figure 6.2.1. Center velocity and center velocity reference during braking for user A and user B (experimental results)</figcaption></figure>

<figure><img src="image_67.png" /><figcaption>Figure 6.2.2. Motor torques during braking for user A and user B (experimental results)</figcaption></figure>
Figure 6.2.1 provides the center velocity, the reference center velocity and the operating mode. Both trials have a similar initial velocity before braking. Moreover, for both trials the reference center velocity computed by the algorithm for the braking are similar, since the assistive control has the same parameters for both users. To follow such a reference, the assistive system slows down the wheelchair by reducing the assistance in assistance mode, as shown in Figure 6.2.2. Then, the wheelchair is braked by the friction in manual mode. Since user A is light and user B is heavy, more braking torque is needed for user B to stop the wheelchair than for user A. To this end, the assistive system reduces more importantly and more quickly the assistive torque for user B than for user A, see in Figure 6.2.2.

However, this quick change of assistive torque could make the user feeling uncomfortable and unsafe. From the feedback of user B, the assistive system brakes too strongly and he feels uncomfortable with it. Nonetheless, the assistive system provides a braking such that the lighter user feels comfortable and safe. Therefore, the decreasing rate $\eta$ of the center velocity should be adapted according to the user.

In addition, different users may perform different pushing frequency to achieve a same desired center velocity. For example, since user B is physically strong, his propelling is more high amplitude and low frequency, whereas for user A it is medium amplitude and frequency. Of course, some level of pathology and/or weak disabled person will end with low amplitude and high frequency. As shown in Figure 6.2.3, the center velocity under the propulsion of user B is higher than the estimated reference. Therefore, the assistive system brakes constantly the wheelchair see Figure 6.2.4. To assist better the user, the reference generation function should provide a higher reference center velocity with a same pushing frequency for user B. Thus, having an adaptive $\delta$ according to the user would be profitable.
<figure><img src="image_68.png" /><figcaption>Figure 6.2.3. Reference estimation with an inappropriate parameter $\delta$</figcaption></figure>

<figure><img src="image_69.png" /><figcaption>Figure 6.2.4. Assistive torque with an inappropriate parameter $\delta$</figcaption></figure>
Of course, this illustrative example could cover more important issues, such as, braking acceptability for some disabled person pathologies. The same kind issues can appear for acceleration, turning and so on. Therefore, a more “personalized” assistance, especially through trajectory generation has to be thought for the future. This personalized assistance could be the right place for learning.

In this context, we propose the idea shown in Figure 6.2.5 to integrate an adaptability to the proposed model-based design. In such a framework, the functionalities of the model-based control are to ensure the reference tracking and the stability of the system wheelchair + human. The functionalities of the reference generation are to produce the references based on an estimate of the human intention from the measurements. The quality of the estimated reference signals depend partly on the parameter vector [$\zeta \ \delta \ \eta]^T$. With the help of a feedback from the human (for example via a button), the learning algorithm could produce and adapt a (near-)optimal parameter vector [$\zeta * \ \delta * \ \eta^*$]ᵀ and generate a (near-)optimal reference signal for a particular user.

<figure><img src="image_70.png" /><figcaption>Figure 6.2.5. Control-Learning framework proposal for PAW designs</figcaption></figure>

If the learning function is removed in the framework of Figure 6.2.5, the assistive control can still provide the performance obtained in Chapter 3 and in Chapter 4. Based on this performance of the model-based design, this framework is expected to improve the assistive control and provide a satisfying performance at the very beginning of learning.

Furthermore, the proposed idea exploits the prior knowledge of the human-wheelchair system. Based on the model-based design and the structure of the reference generation, the learning algorithm has only the parameter vector [$\zeta \ \delta \ \eta]^T$ to learn. The objective is to find a (near-)optimal solution with few data.
To extract meaningful feature of human behaviours from the limited measurement, the learning algorithm may need long-term trials for an adaptive strategy. Therefore, in the proposed framework, the learning control is modelled as a high level control which collects enough information to update the parameter vector [$\zeta \ \delta \ \eta]^T$. The frequency of the parameter updates would be an important issue. Considering usual trips, long trips, user's state of fatigue, the frequency of update should be in the range of some hours or every day or every few days.

These three parameters are just given as an illustration of the global idea. Of course, a deeper research has to be done to determine the principal variables to adapt in order to gain a high level of drivability and fulfil the comfort requirements of final users. The way they have to return their feedback is also an important issue. The assistive control has to be as natural as possible, in order neither to increase their workload, nor to make them feel this task uncomfortable. At last, a critical issue would be to ensure that the low level model-based control can cope safely and robustly to the changes of reference. Moreover, the levels of safety and security have to be kept at a very high level. Therefore, some theoretical aspects have to be considered during the switching sequences of modification of the parameters. It is certainly challenging to combine proofs of robustness and convergence issues of the learning in a global framework.
Bibliography

Algood, S. David et al. 2004. "Impact of a Pushrim-Activated Power-Assisted Wheelchair on the Metabolic Demands, Stroke Frequency, and Range of Motion among Subjects with Tetraplegia." *Archives of physical medicine and rehabilitation* 85(11): 1865-1871.

Aula, A., S. Ahmad, and R. Akmeliawati. 2015. "PSO-Based State Feedback Regulator for Stabilizing a Two-Wheeled Wheelchair in Balancing Mode." In *2015 10th Asian Control Conference (ASCC)*, 1-5.

BA, Brian Woods et al. 2003. "A Short History of Powered Wheelchairs." *Assistive Technology* 15(2): 164-80.

Baxter, J., and P. L. Bartlett. 2000. "Direct Gradient-Based Reinforcement Learning." In *2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353)*, 271-74 vol.3.

Bellman, Richard. 1966. "Dynamic Programming." *Science* 153(3731): 34-37.

Bennani, C. et al. 2017. "A Modified Two-Step LMI Method to Design Observer-Based Controller for Linear Discrete-Time Systems with Parameter Uncertainties." In *2017 6th International Conference on Systems and Control (ICSC)*, 279-84.

Bertsekas, Dimitri P., Dimitri P. Bertsekas, Dimitri P. Bertsekas, and Dimitri P. Bertsekas. 1995. *1 Dynamic Programming and Optimal Control*. Athena scientific Belmont, MA.

Blandeau, M. et al. 2018. "Fuzzy Unknown Input Observer for Understanding Sitting Control of Persons Living with Spinal Cord Injury." *Engineering Applications of Artificial Intelligence* 67: 381-89.

Boninger, Michael L. et al. 2000. "Manual Wheelchair Pushrim Biomechanics and Axle Position." *Archives of Physical Medicine and Rehabilitation* 81(5): 608-13.

Boyan, Justin A. 2002. "Technical Update: Least-Squares Temporal Difference Learning." *Machine Learning* 49(2): 233-46.

BOYD, S. 1994. "Linear Matrix Inequalities in System and Control Theory." *SIAM*. https://ci.nii.ac.jp/naid/10000022326/ (April 30, 2019).

Bradtke, Steven J., and Andrew G. Barto. 1996. "Linear Least-Squares Algorithms for Temporal Difference Learning." *Machine learning* 22(1-3): 33-57.

Bušoniu, L., D. Ernst, B. De Schutter, and R. Babuška. 2010. "Online Least-Squares Policy Iteration for Reinforcement Learning Control." In *Proceedings of the 2010 American Control Conference*, 486-91.
Buşoniu, Lucian et al. 2018. "Reinforcement Learning for Control: Performance, Stability, and Deep Approximators." *Annual Reviews in Control* 46: 8–28.

Busoniu, Lucian, Robert Babuska, Bart De Schutter, and Damien Ernst. 2010. *Reinforcement Learning and Dynamic Programming Using Function Approximators*. 0 ed. CRC Press. https://www.taylorfrancis.com/books/9781439821091 (May 3, 2019).

Buşoniu, Lucian, Damien Ernst, Bart De Schutter, and Robert Babuška. 2010. "Approximate Dynamic Programming with a Fuzzy Parameterization." *Automatica* 46(5): 804–14.

Chadli, M., and T. M. Guerra. 2012. "LMI Solution for Robust Static Output Feedback Control of Discrete Takagi–Sugeno Fuzzy Models." *IEEE Transactions on Fuzzy Systems* 20(6): 1160–65.

Chadli, M., and H. R. Karimi. 2013. "Robust Observer Design for Unknown Inputs Takagi–Sugeno Models." *IEEE Transactions on Fuzzy Systems* 21(1): 158–64.

Chen, W., J. Yang, L. Guo, and S. Li. 2016. "Disturbance-Observer-Based Control and Related Methods—An Overview." *IEEE Transactions on Industrial Electronics* 63(2): 1083–95.

Chibani, Ali, Mohammed Chadli, and Naceur Benhadj Braiek. 2016. "A Sum of Squares Approach for Polynomial Fuzzy Observer Design for Polynomial Fuzzy Systems with Unknown Inputs." *International Journal of Control, Automation and Systems* 14(1): 323–30.

Choi, Jong-Woo, and Sang-Cheol Lee. 2009. "Antiwindup Strategy for PI-Type Speed Controller." *IEEE Transactions on Industrial Electronics* 56(6): 2039–2046.

Cooper, R. A. et al. 2002. "Performance Assessment of a Pushrim-Activated Power-Assisted Wheelchair Control System." *IEEE Transactions on Control Systems Technology* 10(1): 121–26.

Cooper, Rory A. et al. 2001. "Evaluation of a Pushrim-Activated, Power-Assisted Wheelchair." *Archives of Physical Medicine and Rehabilitation* 82(5): 702–708.

Corno, M., D. Berretta, P. Spagnol, and S. M. Savaresi. 2016. "Design, Control, and Validation of a Charge-Sustaining Parallel Hybrid Bicycle." *IEEE Transactions on Control Systems Technology* 24(3): 817–29.

Darouach, M., M. Zasadzinski, and S. J. Xu. 1994. "Full-Order Observers for Linear Systems with Unknown Inputs." *IEEE Transactions on Automatic Control* 39(3): 606–9.

Dayan, Peter, and Geoffrey E. Hinton. 1997. "Using Expectation-Maximization for Reinforcement Learning." *Neural Computation* 9(2): 271–78.

De La Cruz, Celso, Teodiano Freire Bastos, and Ricardo Carelli. 2011. "Adaptive Motion Control Law of a Robotic Wheelchair." *Control Engineering Practice* 19(2): 113–25.

Delrot, Sabrina, Thierry Marie Guerra, Michel Dambrine, and François Delmotte. 2012. "Fouling Detection in a Heat Exchanger by Observer of Takagi–Sugeno Type for
Systems with Unknown Polynomial Inputs.” *Engineering Applications of Artificial Intelligence* 25(8): 1558–1566.

Ding, B. 2010. "Homogeneous Polynomially Nonquadratic Stabilization of Discrete-Time Takagi–Sugeno Systems via Nonparallel Distributed Compensation Law." *IEEE Transactions on Fuzzy Systems* 18(5): 994–1000.

Ding, D., and R. A. Cooper. 2005. "Electric Powered Wheelchairs." *IEEE Control Systems Magazine* 25(2): 22–34.

Duan, Yan et al. 2016. “Benchmarking Deep Reinforcement Learning for Continuous Control.” In *International Conference on Machine Learning*, , 1329–1338.

Edwards, Richard, and Tara Fenwick. 2016. “Digital Analytics in Professional Work and Learning.” *Studies in Continuing Education* 38(2): 213–227.

Estrada-Manzo, V., Z. Lendek, and T. M. Guerra. 2015. “Unknown Input Estimation for Nonlinear Descriptor Systems via LMIs and Takagi-Sugeno Models.” In *2015 54th IEEE Conference on Decision and Control (CDC)*, , 6349–54.

Estrada-Manzo, Victor, Zsófia Lendek, and Thierry Marie Guerra. 2016. “Generalized LMI Observer Design for Discrete-Time Nonlinear Descriptor Models.” *Neurocomputing* 182: 210–20.

Fantuzzi, Cesare, and R. Rovatti. 1996. “On the Approximation Capabilities of the Homogeneous Takagi-Sugeno Model.” In *Proceedings of IEEE 5th International Fuzzy Systems*, IEEE, 1067–1072.

Faure, J. L. 2009. *Le Rapport de l'Observatoire National Sur La Formation, La Recherche et l'innovation Sur Le Handicap, 2008*. Paris: Observatoire national sur la formation, la recherche et l'innovation ....

Fay, Brain T., and Michael L. Boninger. 2002. “The Science behind Mobility Devices for Individuals with Multiple Sclerosis.” *Medical engineering & physics* 24(6): 375–383.

Fayazi, S. A. et al. 2013. “Optimal Pacing in a Cycling Time-Trial Considering Cyclist’s Fatigue Dynamics.” In *2013 American Control Conference*, , 6442–47.

Feng, G., L. Buşoniu, T. M. Guerra, and S. Mohammad. 2018. “Reinforcement Learning for Energy Optimization Under Human Fatigue Constraints of Power-Assisted Wheelchairs.” In *2018 Annual American Control Conference (ACC)*, , 4117–22.

Feng, G., L. Busoniu, T. Guerra, and S. Mohammad. 2019. “Data-Efficient Reinforcement Learning for Energy Optimization of Power-Assisted Wheelchairs.” *IEEE Transactions on Industrial Electronics*: 1–1.

Feng, G., T. M. Guerra, L. Busoniu, and S. Mohammad. 2017. “Unknown Input Observer in Descriptor Form via LMIs for Power-Assisted Wheelchairs.” In *2017 36th Chinese Control Conference (CCC)*, , 6299–6304.
Feng, Guoxi, Thierry Marie Guerra, Sami Mohammad, and Lucian Busoniu. 2018. "Observer-Based Assistive Control Design Under Time-Varying Sampling for Power-Assisted Wheelchairs." *IFAC-PapersOnLine* 51(10): 151-56.

Floquet, T., C. Edwards, and S. K. Spurgeon. 2007. "On Sliding Mode Observers for Systems with Unknown Inputs." *International Journal of Adaptive Control and Signal Processing* 21(8-9): 638-56.

Giesbrecht, Edward M., Jacqueline D. Ripat, Arthur O. Quanbury, and Juliette E. Cooper. 2009. "Participation in Community-Based Activities of Daily Living: Comparison of a Pushrim-Activated, Power-Assisted Wheelchair and a Power Wheelchair." *Disability and Rehabilitation: Assistive Technology* 4(3): 198-207.

Grondman, Ivo, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. 2012. "A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients." *IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)* 42(6): 1291-1307.

Guan, Y., and M. Saif. 1991. "A Novel Approach to the Design of Unknown Input Observers." *IEEE Transactions on Automatic Control* 36(5): 632-35.

Guanetti, Jacopo, Simone Formentin, Matteo Corno, and Sergio M. Savaresi. 2017. "Optimal Energy Management in Series Hybrid Electric Bicycles." *Automatica* 81: 96-106.

Guerra, T. M., H. Kerkeni, J. Lauber, and L. Vermeiren. 2012. "An Efficient Lyapunov Function for Discrete T-S Models: Observer Design." *IEEE Transactions on Fuzzy Systems* 20(1): 187-92.

Guerra, Thierry M., Antonio Sala, and Kazuo Tanaka. 2015. "Fuzzy Control Turns 50: 10 Years Later." *Fuzzy sets and systems* 281: 168-182.

Guerra, Thierry Marie, and Laurent Vermeiren. 2004. "LMI-Based Relaxed Nonquadratic Stabilization Conditions for Nonlinear Systems in the Takagi-Sugeno's Form." *Automatica* 40(5): 823-29.

Han, Hugang, Jiaying Chen, and Hamid Reza Karimi. 2017. "State and Disturbance Observers-Based Polynomial Fuzzy Controller." *Information Sciences* 382: 38-59.

Heydt, G. T. et al. 1999. "Applications of the Windowed FFT to Electric Power Quality Assessment." *IEEE Transactions on Power Delivery* 14(4): 1411-1416.

Howard, Ronald A. 1960. *Dynamic Programming and Markov Processes*. Oxford, England: John Wiley.

Ichalal, D., B. Marx, J. Ragot, and D. Maquin. 2009. "Simultaneous State and Unknown Inputs Estimation with PI and PMI Observers for Takagi Sugeno Model with Unmeasurable Premise Variables." In *2009 17th Mediterranean Conference on Control and Automation*, 353-58.

Kalsi, Karanjit, Jianming Lian, Stefen Hui, and Stanislaw H. Żak. 2010. "Sliding-Mode Observers for Systems with Unknown Inputs: A High-Gain Approach." *Automatica* 46(2): 347-53.
Kerkeni, H., J. Lauber, and T. M. Guerra. 2010. "Estimation of Individual In-Cylinder Air Mass Flow via Periodic Observer in Takagi-Sugeno Form." In *2010 IEEE Vehicle Power and Propulsion Conference*, , 1–6.

Kober, Jens, J. Andrew Bagnell, and Jan Peters. 2013. "Reinforcement Learning in Robotics: A Survey." *The International Journal of Robotics Research* 32(11): 1238–1274.

Kober, Jens, and Jan R. Peters. 2009. "Policy Search for Motor Primitives in Robotics." In *Advances in Neural Information Processing Systems 21*, eds. D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou. Curran Associates, Inc., 849–856. http://papers.nips.cc/paper/3545-policy-search-for-motor-primitives-in-robotics.pdf (April 30, 2019).

Lagoudakis, Michail G., and Ronald Parr. 2003. "Least-Squares Policy Iteration." *Journal of machine learning research* 4(Dec): 1107–1149.

Lampert, Christoph H., and Jan Peters. 2012. "Real-Time Detection of Colored Objects in Multiple Camera Streams with off-the-Shelf Hardware Components." *Journal of Real-Time Image Processing* 7(1): 31–41.

Leaman, Jesse, and Hung Manh La. 2017. "A Comprehensive Review of Smart Wheelchairs: Past, Present, and Future." *IEEE Transactions on Human-Machine Systems* 47(4): 486–499.

Lendek, Z., T. Guerra, and J. Lauber. 2015. "Controller Design for TS Models Using Delayed Nonquadratic Lyapunov Functions." *IEEE Transactions on Cybernetics* 45(3): 439–50.

LEVANT, ARIE. 1993. "Sliding Order and Sliding Accuracy in Sliding Mode Control." *International Journal of Control* 58(6): 1247–63.

Lewis, F. L., and D. Vrabie. 2009. "Reinforcement Learning and Adaptive Dynamic Programming for Feedback Control." *IEEE Circuits and Systems Magazine* 9(3): 32–50.

Löfberg, Johan. 2004. "YALMIP: A Toolbox for Modeling and Optimization in MATLAB." In *Proceedings of the CACSD Conference*, Taipei, Taiwan.

Losero, R., J. Lauber, and T. Guerra. 2015. "Discrete Angular Torque Observer Applicable to the Engine Torque and Clutch Torque Estimation via a Dual-Mass Flywheel." In *2015 IEEE 10th Conference on Industrial Electronics and Applications (ICIEA)*, , 1020–25.

Losero, R., J. Lauber, and T.-M. Guerra. 2018. "Virtual Strain Gauge Based on a Fuzzy Discrete Angular Domain Observer: Application to Engine and Clutch Torque Estimation Issues." *Fuzzy Sets and Systems* 343: 76–96.

Losero, R., J. Lauber, T. Guerra, and P. Maurel. 2016. "Dual Clutch Torque Estimation Based on an Angular Discrete Domain Takagi-Sugeno Switched Observer." In *2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)*, , 2357–63.

Luenberger, D. 1971. "An Introduction to Observers." *IEEE Transactions on Automatic Control* 16(6): 596–602.
Maeda, G., M. Ewerton, D. Koert, and J. Peters. 2016. "Acquiring and Generalizing the Embodiment Mapping From Human Observations to Robot Skills." *IEEE Robotics and Automation Letters* 1(2): 784–91.

Marx, B., D. Koenig, and J. Ragot. 2007. "Design of Observers for Takagi–Sugeno Descriptor Systems with Unknown Inputs and Application to Fault Diagnosis." *IET Control Theory & Applications* 1(5): 1487–95.

Mayne, D. Q., J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert. 2000. "Constrained Model Predictive Control: Stability and Optimality." *Automatica* 36(6): 789–814.

Mnih, Volodymyr et al. 2015. "Human-Level Control through Deep Reinforcement Learning." *Nature* 518(7540): 529.

Mulder, Eric F., Pradeep Y. Tiwari, and Mayuresh V. Kothare. 2009. "Simultaneous Linear and Anti-Windup Controller Synthesis Using Multiobjective Convex Optimization." *Automatica* 45(3): 805–11.

Nair, Ashvin et al. 2018. "Overcoming Exploration in Reinforcement Learning with Demonstrations." In *2018 IEEE International Conference on Robotics and Automation (ICRA)*, IEEE, 6292–6299.

Neal, Radford M. 2001. "Annealed Importance Sampling." *Statistics and computing* 11(2): 125–139.

Nguyen, Anh-Tu, Thierry-Marie Guerra, and Chouki Sentouh. 2018. "Simultaneous Estimation of Vehicle Lateral Dynamics and Driver Torque Using LPV Unknown Input Observer." *IFAC-PapersOnLine* 51(26): 13–18.

Oh, Sehoon, Kyoungchul Kong, and Yoichi Hori. 2014. "Operation State Observation and Condition Recognition for the Control of Power-Assisted Wheelchair." *Mechatronics* 24(8): 1101–11.

Ohishi, K., M. Nakao, K. Ohnishi, and K. Miyachi. 1987. "Microprocessor-Controlled DC Motor for Load-Insensitive Position Servo System." *IEEE Transactions on Industrial Electronics* IE-34(1): 44–49.

Oonishi, Y., S. Oh, and Y. Hori. 2010. "A New Control Method for Power-Assisted Wheelchair Based on the Surface Myoelectric Signal." *IEEE Transactions on Industrial Electronics* 57(9): 3191–96.

Organization, World Health. 2011. "World Report on Disability 2011."

Pai, M. A. 1981. *3 Power System Stability: Analysis by the Direct Method of Lyapunov.* North-Holland Amsterdam.

Peters, J., and S. Schaal. 2006. "Policy Gradient Methods for Robotics." In *2006 IEEE/RSJ International Conference on Intelligent Robots and Systems*, 2219–25.

Peters, Jan, Katharina Mulling, and Yasemin Altun. 2010. "Relative Entropy Policy Search." In *Twenty-Fourth AAAI Conference on Artificial Intelligence*.
Peters, Jan, and Stefan Schaal. 2008. "Natural Actor-Critic." *Neurocomputing* 71(7-9): 1180-1190.

Phillips, A. M., and M. Tomizuka. 1995. "Multirate Estimation and Control under Time-Varying Data Sampling with Applications to Information Storage Devices." In *Proceedings of 1995 American Control Conference - ACC'95*, 4151-55 vol.6.

Pogorzelski, Kamil, and Franz Hillenbrand. "The Incremental Encoder – Operation Principals & Fundamental Signal Evaluation Possibilities." *produktiv messen*. https://www.imc-tm.com/download-center/white-papers/the-incremental-encoder-part-1/ (June 5, 2019).

Precup, Radu-Emil, and Hans Hellendoorn. 2011. "A Survey on Industrial Applications of Fuzzy Control." *Computers in Industry* 62(3): 213-26.

Rodgers, Mary M. et al. 1994. "Biomechanics of Wheelchair Propulsion during Fatigue." *Archives of physical medicine and rehabilitation* 75(1): 85-93.

Ronchi, Enrico, Paul A. Reneke, and Richard D. Peacock. 2016. "A Conceptual Fatigue-Motivation Model to Represent Pedestrian Movement during Stair Evacuation." *Applied Mathematical Modelling* 40(7): 4380-96.

Rummery, Gavin A., and Mahesan Niranjan. 1994. 37 *On-Line Q-Learning Using Connectionist Systems*. University of Cambridge, Department of Engineering Cambridge, England.

Scherer, C W. 2001. "LPV Control and Full Block Multipliers". : 15.

Scherer, Carsten, and Siep Weiland. 2015. "Linear Matrix Inequalities in Control." : 293.

Schulman, John et al. 2015. "Trust Region Policy Optimization." In *International Conference on Machine Learning*, 1889-1897.

Seki, H., K. Ishihara, and S. Tadakuma. 2009. "Novel Regenerative Braking Control of Electric Power-Assisted Wheelchair for Safety Downhill Road Driving." *IEEE Transactions on Industrial Electronics* 56(5): 1393-1400.

Seki, H., and A. Kiso. 2011. "Disturbance Road Adaptive Driving Control of Power-Assisted Wheelchair Using Fuzzy Inference." In *2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society*, 1594-99.

Seki, H., and S. Tadakuma. 2004. "Minimum Jerk Control of Power Assisting Robot on Human Arm Behavior Characteristic." In *2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583)*, 722-27 vol.1.

———. 2006. "Straight and Circular Road Driving Control for Power Assisted Wheelchair Based on Fuzzy Algorithm." In *IECON 2006 - 32nd Annual Conference on IEEE Industrial Electronics*, 3898-3903.

Seki, Hirokazu, Takeaki Sugimoto, and Susumu Tadakuma. 2005. "Novel Straight Road Driving Control of Power Assisted Wheelchair Based on Disturbance Estimation and
Minimum Jerk Control.” In *Fourtieth IAS Annual Meeting. Conference Record of the 2005 Industry Applications Conference, 2005.*, IEEE, 1711–1717.

Shung, J. B., G. Stout, M. Tomizuka, and D. M. Auslander. 1983. "Dynamic Modeling of a Wheelchair on a Slope." *Journal of Dynamic Systems, Measurement, and Control* 105(2): 101–106.

Silver, David et al. 2016. "Mastering the Game of Go with Deep Neural Networks and Tree Search." *Nature* 529(7587): 484–89.

———. 2017. "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm." *arXiv preprint arXiv:1712.01815*.

Simpson, Richard C. 2005. "Smart Wheelchairs: A Literature Review." *Journal of rehabilitation research and development* 42(4): 423–36.

Sutton, Richard S., and Andrew G. Barto. 2018. *Reinforcement Learning: An Introduction*. MIT press.

Sutton, Richard S, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 2000. "Policy Gradient Methods for Reinforcement Learning with Function Approximation." In *Advances in Neural Information Processing Systems 12*, eds. S. A. Solla, T. K. Leen, and K. Müller. MIT Press, 1057–1063. http://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf (April 30, 2019).

Szepesvári, Csaba. 2010. "Algorithms for Reinforcement Learning." *Synthesis lectures on artificial intelligence and machine learning* 4(1): 1–103.

Takagi, TOMOHIRO, and MICHIO Sugeno. 1993. "Fuzzy Identification of Systems and Its Applications to Modeling and Control." In *Readings in Fuzzy Sets for Intelligent Systems*, eds. Didier Dubois, Henri Prade, and Ronald R. Yager. Morgan Kaufmann, 387–403. http://www.sciencedirect.com/science/article/pii/B9781483214504500456 (April 30, 2019).

Taniguchi, T., K. Tanaka, H. Ohtake, and H. O. Wang. 2001. "Model Construction, Rule Reduction, and Robust Compensation for Generalized Form of Takagi-Sugeno Fuzzy Systems." *IEEE Transactions on Fuzzy Systems* 9(4): 525–38.

Tanohata, N., H. Murakami, and H. Seki. 2010. "Battery Friendly Driving Control of Electric Power-Assisted Wheelchair Based on Fuzzy Algorithm." In *Proceedings of SICE Annual Conference 2010*, , 1595–98.

Tashiro, S., and T. Murakami. 2008. "Step Passage Control of a Power-Assisted Wheelchair for a Caregiver." *IEEE Transactions on Industrial Electronics* 55(4): 1715–21.

Tesauro, Gerald. 1995. "Temporal Difference Learning and TD-Gammon." *Communications of the ACM* 38(3): 58–68.

Thieffry, M., A. Kruszewski, C. Duriez, and T. Guerra. 2019. "Control Design for Soft Robots Based on Reduced-Order Model." *IEEE Robotics and Automation Letters* 4(1): 25–32.
Tsai, M., and P. Hsueh. 2012. "Synchronized Motion Control for 2D Joystick-Based Electric Wheelchair Driven by Two Wheel Motors." In *2012 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)*, 702-7.

Tsai, Mi-Ching, and Po-Wen Hsueh. 2013. "Force Sensorless Control of Power-Assisted Wheelchair Based on Motion Coordinate Transformation." *Mechatronics* 23(8): 1014-24.

Tsuyoshi Shibata, and Toshiyuki Murakami. 2008. "Power Assist Control by Repulsive Compliance Control of Electric Wheelchair." In *2008 10th IEEE International Workshop on Advanced Motion Control*, Trento, Italy: IEEE, 504-9. http://ieeexplore.ieee.org/document/4516118/ (April 30, 2019).

Umeno, T., T. Kaneko, and Y. Hori. 1993. "Robust Servosystem Design with Two Degrees of Freedom and Its Application to Novel Motion Control of Robot Manipulators." *IEEE Transactions on Industrial Electronics* 40(5): 473-85.

"US20170151109A1 - Method and Device Assisting with the Electric Propulsion of a Rolling System, Wheelchair Kit Comprising Such a Device and Wheelchair Equipped with Such a Device - Google Patents." https://patents.google.com/patent/US20170151109A1/en (April 30, 2019).

Vecerik, Mel et al. 2019. "A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning." In *2019 International Conference on Robotics and Automation (ICRA)*, IEEE, 754-760.

Wan, N., S. A. Fayazi, H. Saeidi, and A. Vahidi. 2014. "Optimal Power Management of an Electric Bicycle Based on Terrain Preview and Considering Human Fatigue Dynamics." In *2014 American Control Conference*, 3462-67.

Williams, Ronald J. 1992. "Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning." *Machine Learning* 8(3): 229-56.