# Control of a Water Tank System with Value Function Approximation

Shamal Lalvani, Aggelos Katsaggelos

► To cite this version:

Shamal Lalvani, Aggelos Katsaggelos. Control of a Water Tank System with Value Function Approximation. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.36-44, 10.1007/978-3-031-34111-3_4. hal-05449792

HAL Id: hal-05449792

https://inria.hal.science/hal-05449792v1

Submitted on 9 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Control of a Water Tank System with Value Function Approximation

Shamal Lalvani(✉) and Aggelos Katsaggelos

Northwestern University, Evanston, IL 60208, USA
{shamal.lalvani,a-katsaggelos}@northwestern.edu

**Abstract.** We consider a system of two identical rectangular shaped water tanks. A source of constant water inflow is available, which may only be directed to one tank at a time. The objective is to find a control policy to maximize the final sum of the water levels at some terminal time $T$, subject to minimum water level constraints on each tank. Water exits each tank corresponding to Toricelli's law (i.e., the velocity depends on the current water level). We derive a closed form dynamic programming solution in discrete time to this problem without the water-level threshold constraints. Subsequently, we implement the value iteration algorithm on a set of support points to find a control policy with the threshold constraints, where a random forest regressor is iteratively used to update the value function. Our results show consistency between the dynamic programming solution and the value iteration solution.

**Keywords:** Reinforcement Learning · Optimal Control · Water Tank System

## 1 Introduction

Optimal control problems in which system feedback is observed between successive decisions are commonly solved using techniques that incorporate the Bellman equation (sometimes also referred to as the Hamilton-Jacobi equation) [1,2]. These techniques aim to solve a recursively defined function (sometimes called the 'cost-to-go function' or 'value function'), which defines the optimality of the different states of the system under the optimal policy. The value function may then be used to generate an optimal control policy [3]. A common approach to solve the Bellman equation is through dynamic programming, where the solution is obtained via backwards induction. However, it is generally not possible to solve all control problems through dynamic programming, and there is no guarantee that all control problems will have closed-form solutions [4].

In the machine learning literature, rather than explicitly solving for the value function, techniques that approximate the value function are commonly used [5]. One commonly used approach is the value iteration algorithm, where the value function is arbitrarily initialized at each point in the state space, and updated
through experience with the dynamics of the system. If the state space itself is large (or even infinite), however, this approach may be computationally infeasible for every point in the state space. For large state spaces, one approach is to use the value iteration algorithm on a set of support points and use a function approximator (such as a neural network, gaussian mixture model or a set of basis functions) to generalize the value function to the entire state space (see [4–7]).

In this paper, inspired by [8], we consider a system of two identical rectangular water tanks subject to minimum water level constraints, with a constant source of inflow that may be directed to only one tank at a time. Our system is identical to [8], except we assume that water exits the tanks according to Toricelli’s Law (see [9]), rather than at a constant outflow rate. We consider the objective of maximizing the sum of the water levels at a terminal time. First, we derive a solution to this problem through dynamic programming in discrete time without the water level constraints. Subsequently, we solve the problem using the value iteration algorithm with random forest regression to approximate the value function at all points of the state space. There exists a literature of control of water tank systems, such as with the use of PID controllers [10], control of water tanks in tandem [11] and control of water tanks with devices such as arduino [12]. To the best of our knowledge, however, this is the very first paper that addresses optimal control of the aforementioned model and objective.

## 2 The Model

Consider a system of two identical water-tanks, where the water levels at time $t$ of each tank are denoted $X_1(t)$ and $X_2(t)$ respectively (Fig. 1). We assume that a water source with constant inflow $w$ is available to fill either tank, and in particular may be directed at only one tank at a time. We assume the water source may be switched instantaneously between both tanks at any desired time. We let $c(t)$ denote the control variable, where

$$c(t) = \begin{cases} 1, & \text{if the hose is above tank 1 at time } t \\ 2, & \text{if the hose is above tank 2 at time } t \end{cases} \qquad (1)$$

The objective is to find a control policy that maximizes the sum of the water-levels of the two tanks at some terminal time $T > 0$, which is denoted by the objective function $m(T)$, i.e.,

$$m(T) = X_1(T) + X_2(T) \qquad (2)$$

We further impose the constraints that the water-levels of the two tanks must be above thresholds $r_1$ and $r_2$ respectively, i.e.,

$$X_i(t) \ge r_i, \text{ for all } t \in [0, T] \qquad (3)$$

Assuming satisfaction of the constraints, the dynamics of the system may be described as

$$\frac{dX_i(t)}{dt} = w\mathcal{I}(c(t) = i) - v(X_i(t)) \qquad (4)$$
where $\mathcal{I}$ denotes the indicator function, and $v(X_i(t))$ denotes the outflow rate, which is a function of the current water level of tank $i$ at time $t$. Additionally, we assume water exits the tanks according to Toricelli's law, so that,

$$v(X_i(t)) = r\sqrt{2gX_i(t)} \quad (5)$$

for some constant $r$ that depends on the dimensions of the water-tanks, where $g$ denotes the magnitude of acceleration due to gravity. We apply an approximation to model the water level of the tanks. Note that for a rectangular tank (more generally, parallelepiped tank) with inflow $w$ and an outflow rate given by Toricelli's Law, an equation exists to describe how the water levels change in a time period $dt$ ([9]),

$$t = \frac{2}{z}[\sqrt{X(T - dt)} - \sqrt{X(T)} + w[\ln(\frac{w - \sqrt{X(T - dt)}}{w - \sqrt{X(T)}})] \quad (6)$$

for some constant $z$ that depends on the dimensions of the water tank. We now derive an approximation for the equation above. Note that a first degree taylor series approximation for $ln(x)$, centered around any constant $a > 0$ yields,

$$ln(x) = ln(a) + \frac{x - a}{a} \quad (7)$$

Choosing $x = w - \sqrt{X(T - dt)}$ and $x = w - \sqrt{X(T)}$, and substitution into Eq. 7, and further substituting into Eq. 6 gives,

$$t = \frac{2}{z}[\sqrt{X(T - dt)} - \sqrt{X(T)} + w\frac{\sqrt{X(T)} - \sqrt{X(T - dt)}}{a}] \quad (8)$$

Rearranging terms, and putting $k = \frac{z}{2}$ and $\beta = 1 - \frac{w}{a}$, (where we mandate that $a$ is chosen so that $w \neq a$), it follows that,

$$\sqrt{X(T)} = \sqrt{X(T - dt)} - \frac{kdt}{\beta} \quad (9)$$

Note that if there is no inflow (i.e., $w = 0$), then $\beta = 1$, so that,

$$\sqrt{X(T)} = \sqrt{X(T - dt)} - kdt \quad (10)$$

An illustration of Eq. 9 and Eq. 10 is displayed in Fig. 2.

## 3 Dynamic Programming Solution

We provide a dynamic programming solution in discrete time to maximize the objective function $m(T)$. Let $dt > 0$ be small, and $N = \frac{T}{dt}$ denote the number of time-steps. For simplicity of notation, because $dt$ is a constant, we absorb it into the coefficient $k$ in Eq. 9 and Eq. 10. Further, we assume that $\beta < 0$ (i.e., $w > a$) if there is no source of inflow to a tank (this assumption is justified by centering $a$ around $w - \sqrt{X(T)}$ and $w - \sqrt{X(T - dt)}$ to obtain Eq. 8). The solution below involves backwards induction on time, where Sect. 3.3 corresponds to the inductive step.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Illustration of the Water Tank system. The water levels of tank 1 and tank 2 at time $t$ are denoted by $X_1(t)$ and $X_2(t)$ respectively. The system constraints impose that water levels of tank 1 and tank 2 must be kept above the thresholds $r_1$ and $r_2$ respectively. When the control variable $c(t)$ is equal to 1, the constant inflow $w$ enters tank 1, as illustrated above. When $c(t) = 2$, $w$ enters tank 2. The outflow of tank $i$ ($i = 1, 2$) at time $t$ is given by $v(X_i(t))$.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Illustration of the dynamics of the tanks with a constant inflow (blue) and with no inflow (orange) for 70 s. Both tanks initially start at a water level of 800, an inflow $w = 34$ is used, and a spacing of $dt = 0.01$ is used. (Color figure online)</figcaption></figure>
## 3.1 Optimal Decision at Time T-dt

Let $m_1(T)$ denote the sum of the final water levels if we fill tank 1 at time $t - dt$ for $dt$ seconds, and let $m_2(T)$ denote the sum of the final water levels if we will tank 2 at time $t - dt$ for $dt$ seconds. Using Eq. 9, it follows that,

$$m_1(T) - m_2(T) = 2k(1 - \frac{1}{\beta})(\sqrt{X_1(T - dt)} - \sqrt{X_2(T - dt)}) \quad (11)$$

Note that by assumption $\beta < 0$, therefore $2k(1 - \frac{1}{\beta}) > 0$. It follows that $m_1(T) - m_2(T) \ge 0$ if and only if $X_2(T - dt) \ge X_1(T - dt)$. Therefore, the optimal policy at time $T - dt$ is to fill tank 1 if it has a larger water level, and vice versa.

## 3.2 Optimal Decision at Time T-2dt

Suppose we are in period $t - 2dt$. Let $m_{i,j}$ denote the final sum of the water levels if we fill tanks i and j at times $T - 2dt$ and $T - dt$ respectively. We define $n_{i,j}$ as follows,

$$n_{i,j} = \frac{m_{i,j} - X_1(T - 2dt) - X_2(T - 2dt) - 2k^2 - 2\frac{k^2}{\beta^2}}{2k} \quad (12)$$

Note that since $2k > 0$, and because at time $T - 2dt$, we have that $-X_1(T - 2dt) - X_2(T - 2dt) - 2k^2 - 2\frac{k^2}{\beta^2}$ is a constant, maximizing $m_{i,j}$ is equivalent to maximizing $n_{i,j}$. It can be shown with substitution of Eq. 9 and Eq. 10 into Eq. 12 that,

$$n_{1,1} = -\frac{2}{\beta}\sqrt{X_1(T - 2dt)} - 2\sqrt{X_2(T - 2dt)} + k(\frac{1}{\beta^2} + 1) \quad (13)$$

$$n_{1,2} = n_{2,1} = -(1 + \frac{1}{\beta})\sqrt{X_1(T - 2dt)} - (1 + \frac{1}{\beta})\sqrt{X_2(T - 2dt)} + \frac{2k}{\beta} \quad (14)$$

$$n_{2,2} = -2\sqrt{X_1(T - 2dt)} - \frac{2}{\beta}\sqrt{X_2(T - 2dt)} + k(\frac{1}{\beta^2} + 1) \quad (15)$$

Note that $n_{1,2} = n_{2,1} \le \max\{n_{1,1}, n_{2,2}\}$. To see this, first note that the square of any real number is greater than or equal to zero so that $(1 - \frac{1}{\beta})^2 \ge 0$, which expanding gives, $1 + \frac{1}{\beta^2} \ge \frac{2}{\beta}$. Therefore, it suffices to show,

$$b(t) \le \{\max\{a(t), c(t)\}\} \quad (16)$$

where,

$$a(t) = \frac{2}{\beta}\sqrt{X_1(T - 2dt)} - 2\sqrt{X_2(T - 2dt)} \quad (17)$$

$$b(t) = -(1 + \frac{1}{\beta})\sqrt{X_1(T - 2dt)} - (1 + \frac{1}{\beta})\sqrt{X_2(T - 2dt)} \quad (18)$$

$$c(t) = -2\sqrt{X_1(T - 2dt)} - \frac{2}{\beta}\sqrt{X_2(T - 2dt)} \quad (19)$$
Assume, for a contradiction, that $b(t) > a(t)$ and that $b(t) > c(t)$. It follows that $b(t) - a(t) = (1 - \frac{1}{\beta})(\sqrt{X_1(T - 2dt)} - \sqrt{X_2(T - 2dt)}) > 0$ and that $b(t) - c(t) = -(1 - \frac{1}{\beta})(\sqrt{X_1(T - 2dt)} - \sqrt{X_2(T - 2dt)}) > 0$. Without loss of generality, we assume $\sqrt{X_1(T - 2dt)} \neq \sqrt{X_2(T - 2dt)}$ (otherwise the choice of which tank to fill doesn't matter by symmetry of the system and objective function), and therefore $(1 - \frac{1}{\beta}) > 0$ and $(1 - \frac{1}{\beta}) < 0$, a contradiction. Therefore, the monotonic transformation of the objective function is given by $\max\{n_{1,1}, n_{2,2}\}$. Note that,

$$n_{1,1}(T) - n_{2,2}(T) = 2\left(1 - \frac{1}{\beta}\right)\left(\sqrt{X_1(T - 2dt)} - \sqrt{X_2(T - 2dt)}\right) \quad (20)$$

It follows that $n_{1,1} \ge n_{2,2}$ if and only if $X_1(T - 2dt) \ge X_2(T - 2dt)$. Hence, the optimal policy at time $T - 2dt$ is to fill the tank with the larger water level. In the next section, we derive an inductive step to generalize the solution for backwards induction on time.

### 3.3 Lemma 1

Suppose that at time $t_1$, $1 \le t_1 < T$, the objective function (i.e., the final sum of the water levels), under the optimal policy is given by, $O(t_1) = \max\{m_1(t_1), m_2(t_1)\}$ where,

$$m_1(t_1) = X_1(t_1) + X_2(t_1) - \frac{k_1}{\beta}\sqrt{X_1(t_1)} - k_1\sqrt{X_2(t_1)} + c$$

$$m_2(t_1) = X_1(t_1) + X_2(t_1) - k_1\sqrt{X_1(t_1)} - \frac{k_1}{\beta}\sqrt{X_2(t_1)} + c$$

For some constants $k_1 > 0$, $\beta < 0$ and real-valued $c$. Also assume that the optimal policy is to fill tank 1 if and only if $m_1(t_1) \ge m_2(t_1)$ (i.e., $X_1(t_1) > X_2(t_1)$). It follows that at time $t = t_1 - dt$, the optimal value of the objective function and optimal policy inherit the same form.

**Proof.** Suppose at time $t$, we fill tank 1 and act optimally thereafter. Then, by assumption, the final sum of the water levels is $\max\{f(t), g(t)\}$, where,

$$f(t) = X_1(t) + X_2(t) - \left(\frac{k+k_1}{\beta}\right)\sqrt{X_1(t)} - \left(k+k_1\right)\sqrt{X_2(t)} + c + \frac{k}{\beta^2} + k^2 + \frac{kk_1}{\beta} + kk_1 \quad (21)$$

$$g(t) = X_1(t) + X_2(t) - \left(\frac{k}{\beta} + k_1\right)\sqrt{X_1(t)} - \left(k + \frac{k_1}{\beta}\right)\sqrt{X_2(t)} + c + \frac{k}{\beta^2} + k^2 + \frac{2kk_1}{\beta} \quad (22)$$

On the other hand, suppose we fill tank 2 at time $t$ and act optimally thereafter. Then, by assumption, the final sum of the water levels is given by $\max\{u(t), v(t)\}$, where,

$$u(t) = X_1(t) + X_2(t) - \left(k + \frac{k_1}{\beta}\right)\sqrt{X_1(t)} - \left(\frac{k}{\beta} + k_1\right)\sqrt{X_2(t)} + c + \frac{k}{\beta^2} + k^2 + \frac{2kk_1}{\beta} \quad (23)$$
$$v(t) = X_1(t) + X_2(t) - \left(\frac{k+k_1}{\beta}\right)\sqrt{X_1(t)} - (k+k_1)\sqrt{X_2(t)} + c + \frac{k}{\beta^2} + k^2 + \frac{kk_1}{\beta} + kk_1 \quad (24)$$

Using proof by contradiction, a similar argument to the optimal policy at time $T - 2dt$ (see Sect. 3.2) may be used to show that $g(t) \le \max\{f(t), v(t)\}$ and that $u(t) \le \max\{f(t), v(t)\}$. It follows that, under the optimal policy, the final sum of the water levels is given by $\max\{f(t), v(t)\}$, and that at time $t$, the optimal decision is to fill tank 1 if and only if $f(t) \ge v(t)$.

## 3.4 Summary of the Optimal Policy

Note that at time $t - 2dt$, the assumption of Sect. 3.3 holds by solving Eq. 12 for $m_{i,j}$, where $n\{i, j\}$ is defined in Eq. 13 to Eq. 15. Therefore, at time $t-3dt$, the Lemma may be iteratively be applied. Therefore, the optimal policy is that is always optimal to fill the tank with the higher water level.

# 4 Reinforcement Learning Solution with Function Approximation

We implement the value iteration described in Barton and Sutto (see [5]), with a set of support points (see Algorithm 1). A convergence bound $z$ is used as a tolerance – when updates to the value function are less than $z$, the algorithm is assumed to have converged or be near convergence. In the algorithm, $V(s)$ denotes the value function, which is first updated at the support points, and then generalized to the entire state space in each iteration. We note that updates to the value function (i.e., $V(s) \leftarrow \max\{r_1 + V(s_1), r_2 + V(s_2)\}$) represent the maximum value from filling tank 1 and tank 2 respectively for $dt$ seconds. In particular, note that $R_1$ and $R_2$ are the rewards for filling tanks 1 and 2 for

**Algorithm 1.** Value Iteration for Water Tank Control

**Require:** A convergence bound $z$

**Require:** A set of $N$ support points $S = \{(x_{1i}(t_i), x_{2i}(t_i), t_i)\}_{i=1}^N$, where $0 \le t_i \le T$
Initialize the value function $V(s) = 0$ for all $s$

while $\theta < z$ do

$\theta \leftarrow 0$

for $s$ in $S$ do

$v \leftarrow V(s)$

$V(s) \leftarrow \max\{R_1 + V(s_1), R_2 + V(s_2)\}$

$\theta \leftarrow \max\{\theta, |v - V_1(s)|\}$

end for

Update $V$ to the entire state space based off of support points $\{s, V(s)\}_{s \in S}$

end while

Output the function $V$
$dt$ seconds respectively, while $s_1$ and $s_2$ are the subsequent respective states. The rewards are use to optimize the objective function (i.e., the reward is zero unless the terminal time is reached, for which the reward is the sum of the water levels for both tanks). However, if the next state is less than the tank threshold (i.e., either $X_1 < r_1$ or $X_2 < r_2$), the reward is set to a negative value with large magnitude (i.e., $-10^6$) to discourage the algorithm from exploring these regions of the state space. After obtaining the value function, the optimal policy is implemented by choosing the action that achieves the larger value for the subsequent state. Note that this is computationally feasible because there are only two available actions at any time.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Illustration of the optimal policy through value iteration.</figcaption></figure>

The value function algorithm is implemented (see Fig 3) with initial values $X_1(0) = X_2(0) = 700$, thresholds $r_1 = 200$, $r_2 = 300$, time step $dt = 0.01$ and terminal time $T = 100$. The value function is updated in each iteration of Algorithm 1 using a random forest regressor with a depth of $10^6$ trees in Python using the package `Sklearn`. The tolerance is chosen to be $z = 1$.

# 5 Conclusion

In Fig. 3, we observe the similarity of the solution from Algorithm 1 to the dynamic programming solution (which, however, did not account for the water level constraints). In particular, we note that the tank with the higher level (tank 1) is always filled, except when needed to prevent violation of the constraints (i.e., to keep the water level of tank 2 above 300). This results in frequent switching between the two tanks after time $t = 20$ onwards. Note that, except for satisfaction of constraints, the policy is exactly the same as the dynamic programming
solution, which didn't take into account the constraints. In conclusion, the consistency of results from the dynamic programming approach and reinforcement learning approach illustrate that reinforcement learning with function approximation may be used to derive similar solutions to dynamic programming, without the explicit need for mathematical modeling of system dynamics. The computational cost of implementing reinforcement learning algorithms, however, may still be large with function approximation when the state space contains a variety of unique dynamics or is very large. Suggested future work involves optimizing the objective function with more than two tanks, and considering the case of when the tanks are not independent with each other (i.e., connected tanks).

## References

1. Aliyu, M.D.S.: Solving the Hamilton-jacobi equation, pp. 333-357. Nonlinear H infinity Control, Hamiltonian Systems and Hamilton-Jacobi Equations (2017)

2. Hamilton-Jacobi equations. Semiconcave Functions, Hamilton-Jacobi Equations, and Optimal Control, pp. 97-139 (2004)

3. Bertsekas, D.P.: Dynamic programming and Optimal Control. Athena Scientific, Belmont, Mass (2018)

4. Powell, W.B.: What you should know about approximate dynamic programming. Naval Res. Logist. (NRL). **56**, 239-249 (2009)

5. Sutton, R.S., Bach, F., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press Ltd, Massachusetts (2018)

6. Agostini, A., Celaya, E.: Reinforcement learning with a gaussian mixture model. In: The 2010 International Joint Conference on Neural Networks (IJCNN) (2010)

7. Ho, F., Kamel, M.: Reinforcement learning using a recurrent neural network. In: Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) (1994)

8. Lecture notes on hybrid systems - university of California, Berkeley, https://people.eecs.berkeley.edu/sastry/ee291e/lygeros.pdf

9. Atesmen, M.K.: Understanding the world around through simple mathematics: From credit card interest, basal metabolic rate to earthquake magnitude. Infinity Publishing, West Conshohocken, PA (2011)

10. Vojtesek, J., Dostal, P.: Adaptive control of water level in real model of water tank. 2015 20th International Conference on Process Control (PC). (2015)

11. Xiao, G., Liu, F.: PID- explicit predictive dual mode control for double-water tank. In: 2018 Chinese Control And Decision Conference (CCDC) (2018)

12. Godwin Premi, M.S., Malakar, J.: Automatic water tank level and Pump Control System. In: 2019 International Conference on Intelligent Computing and Control Systems (ICCS) (2019)