Model-Based Control Using Neural Network: A Case Study
By: Stephen Adhisaputra
Tags: Model Predictive Control, Machine Learning, Neural Networks, Reinforcement Learning, Simulation

Suppose we have three identical point masses m = 0.5 kg, with initial positions x1, x2, and x3. The dampers have damping constant d= 0.25 N.s/m each. The springs are nonlinear, described by the Newton’s law as:

with k = 217 N/m and kp= 63.5 N/m³. The forces “u1" and “u3" are the “control inputs”, sometimes also referred as the “actuation forces” or “actions”. I will call it as actions from now on. The disturbance “dist” disrupts the motion of the system. Say, it is a random value between -100 N to 100 N.

The main goal here is to steer the masses m1 and m3 to its desired reference position values x1_ref and x3_ref. An artificial neural network will be used to predict its future positions x1 and x3 based on the forces u1, dist, and u3.

The challenge:

We only get measurements of x1 and x3 in every timestep ( Δt= 0.001s). The system starts from an arbitrary position of x1, x2, x3 with random initial velocity v1, v2, v3. The system dynamics is unknown AND we are not able to calculate the equations of motion using Newton’s law.

Sir Isaac Newton is bemused. “What do you mean we are not able to calculate the equations of motion?” [Painting by Godfrey Kneller, via Wikimedia Commons]

Approach

Sometimes in real world, its impossible to calculate the dynamics of a system without any approximations. Even if you could, who has the time to do complex calculations? You need to be an expert, study for years, maybe get a university degree, and some work experiences before being good at your domain. Mathematical derivation from first principles is absolutely not an easy task. So, we will instead try to use machine learning techniques to solve the problem!

Control of complex systems involves two parts: system identification and controller design. In system identification, we create the dynamics model.

We know that neural networks can be used to recognize patterns. Thus, we can train the network to predict future behavior based on past information. This can be done using supervised learning regression algorithms. We classify this as a time-series forecasting problem in machine learning.

Additional challenge: the mechanical system might experience a sudden dynamics change. For example, if we reduce or replace the masses with different weights. Imagine you are a taxi driver. Your passengers are changing over time. Say, you got a new load of 200 kg. This will slightly change the dynamical behavior of the vehicle. Perhaps it is not the best of examples, but you get the idea!

Knowing that dynamics can change, the designed neural network must be able to perform adaptive learning. Thus, I will design a framework to do both dynamics learning and system control online. This can be realized using reinforcement learning. Now, let’s dive deeper into the concepts!

Model

The term model here can be understood as the dynamics function, a set of “rules” which determines how the states x (position of the masses) change with respect to time t. Mathematically, it predicts the next states x based on the actions u:

where Z defines the constraints (limitations) of the states and actions.

Why use a model?

Because it increases explainability.

Explainable Artificial Intelligence [DARPA]

Many real-world physical systems also have model constraints concerning safety and system compatibility. Furthermore, model-free algorithms are not sample-efficient compared to model-based algorithms.

Comparison of general RL algorithms [article]

Neural Network Model

The neural network’s goal here is to be the model: learn the dynamics function of our mechanical system. It’s easy…

We give the neural network real-time state measurements. Then train it with some algorithm. All this without actually knowing Newton’s law!

Neural network as function approximator [1]

I won’t get into the details of how and why it works. There are enough articles and videos on the internet that explain about neural network training. In this section, I will just present my neural network design.

At first, I considered using the recurrent neural networks (RNN) such as the NARX. But, I found that the online training of RNN would take longer time in the reinforcement learning algorithm. So, I used here the simplest architecture: Multilayer Perceptron (MLP).

More layers will need more computation power, which is undesirable in real-time control. I modified the network to include past output measurements of the mechanical system, giving it the so-called time-delayed inputs.

Additionally, the neural network will predict the change in position Δx, instead of predicting the actual position of the next states x. This might sometimes be referred as residual connections. Or a data manipulation technique. This exploits the knowledge that in our dynamical system the state changes Δx are very small, since one timestep is only 0.001 seconds.

Neural Network Design

Block diagram of my neural network design

Remember: The only available real-time measurements are x1 and x3, every 0.001 seconds. Assume there are no measurements available for x2, or any state velocities to train the network.

For the experiment, I set the time delay as 2 timesteps. This results in ‘new’ information for learning: the past states (xt-1 and xt-2).

Inputs: The neural network inputs are the actions u (u1 and u3), the current states xt-0 (x1 and x3), as well as the past changes in states diff1 and diff2 (Δxt-1 and Δxt-2).

Outputs: The neural network outputs are the predicted change of states x (Δx1 and Δx3). All the neural network layers are fully-connected.

Since we have 2 measured states (x1 and x3), 2 actions (u1 and u3), and a disturbance input (dist), I will need in total (1x2) + (3x2) + 1 = 9 input neurons and 2 output neurons for my design.

I set 40 hidden neurons within the MLP architecture. The leaky rectified linear unit (LReLU) activation function was chosen with a= 0.5.

LReLU Activation function

I experimented with a lot of configurations for our case. I achieved the best training performance using Adam optimizer with learning rate 0.001 and batch size 16. The data (input and output pairs) are preprocessed prior training to have mean 0 and standard deviation 1. All the weights and biases of the neural network were initialized with Glorot uniform initializaters.

These settings are purely experimental. But just to get you the idea…

You can’t just do this. That is NOT how it works! [source]

The accuracy of the model plays a crucial role in controlling a dynamical system. So the model had to be trained well, before control is applied… Speaking of which, let’s now discuss a popular control technique: model predictive control.

Model Predictive Control (MPC)

MPC is one of the most used methods to control multivariable systems. As the name suggests, control is applied based on the predictions made by the model. The optimization is usually done using the receding horizon strategy.

This strategy provides a simple method for determining the future actions by optimizing its future trajectories... See the figure below.

MPC strategy [TU Delft]

We can express this strategy as the control “objective function”:

Now the above equation might look complicated, but it is actually very simple. We want to pick the actions u(t) which produces the least J(t).

To apply this strategy, we need to know three things:

desired reference values of y (trajectory goal, set by the user) predicted values y over some time horizon N (acquired from the model) actions u over some time horizon Nu (calculated / random guess)

I refer the J(t) here as the “cost”. Larger cost means bad control performance.

The first quadratic term is the total predicted error. You calculate this by summing the squared differences between future trajectories from desired trajectories.

The second quadratic term represents the change of actions in each timestep over the control horizon Nu. It penalizes large change in actions. Smooth (less jerky) actions are desirable because i.e. real motors can’t change its value by much over a small period.

You see the line plots? This is what I mean by “smooth”. [xkcd]

Evaluation Metrics

Before going deeper to the control algorithm, we must design the evaluation metrics. In this experiment, I define the “control performances” as two things — 1.) trajectory deviation, 2.) action smoothness and total impulse.

The trajectory deviation,

with yt being the predicted values and wt denoting the reference values in the future trajectories. Meanwhile, s is the total number of simulation timesteps.

2. Action smoothness and total impulse,

We want both of these values and the RMSE to be as small as possible. However, it is often the case of compromising between the two.

Control Strategy

Combined with the neural network-based MPC, we will apply a simple control method called the random shooting.

How does it work?

In the MPC algorithm, 300 candidate actions will be randomized for each timestep. Repeating this for N times and passing it through the model, we get 300 candidate trajectories for each state. Finally, the candidate trajectories will be used to select the best action sequence which minimizes the objective function.

Additionally, I will limit the change of possible actions u in each timestep. By doing this, we can omit the second quadratic term in the objective function. Instead, I added horizonpenalty in the equation:

where the horizonpenalty is… a term I completely made up. It can be whatever you want! In general, you should design it to achieve better control performance.

I experimented with some terms that penalizes high system velocities. How?

The trick is by substracting the last states with the first states of the predicted system trajectories. Change of distance over time = velocity. We will come back to this later.

Now you might be asking, how is this all related to reinforcement learning?

The Reinforcement Learning (RL) Framework

So in RL, the learning is done by continuous interaction of the agent with the environment. This can also be called “online learning”. In “model-free” RL algorithms, the agent learns by doing. Pretty much advanced trial-and-error.

In model-based reinforcement learning, the agent models the environment and plans the best action based on the model. If you want a detailed explanation, please read this amazing article by Jonathan Hui.

General RL framework [Mathworks]

Shortly explained: in control engineering the “agent” is a controller. The “environment” is the dynamical system + environment (where the disturbances come from). We can get the states from our “observations”.

The “reward“ can be seen as the objective function, which flows from the agent to the environment. Finally, the “action” represents the control inputs.

In general, there are two types of behavior in RL: