# Bacon, a Lightweight Deep Reinforcement Learning Benchmark Library for Flow Control
## 3 Environments  
### Shkadov  
#### 3.1.1 Physics  
The initial investigation into vertically falling fluid films was conducted by [PERSON] and [PERSON] [19], sparking extensive experimental exploration in subsequent decades. These experiments revealed that waves on the surface of a descending thin liquid film exhibit strong non-linearity, manifesting the emergence of saturated waves from small perturbations in amplitude, as well as the presence of solitary waves. For low Reynolds numbers (\(Re<300\)), it was noted that the wavelength of the non-linear waves greatly exceeded the thickness of the film, allowing for potential simplifications in their physical modeling (referred to as the long-wave regime). Various physical models were proposed, among them the Shkadov model, which was introduced in 1967 [20]. Despite being shown to be inconsistent [21], the model exhibits intriguing spatio-temporal dynamics while remaining computationally affordable. The flow rate \(q\) and the fluid height \(h\) are simultaneously evolved using the following set of equations:  
\[ \begin{split}\partial_{t}h&=-\partial_{x}q,\\ \partial_{t}q&=-\frac{6}{5}\partial_{x}\left(\frac{ q^{2}}{h}\right)+\frac{1}{5\delta}\left(h(1+\partial_{xxx}h)-\frac{q}{h^{2}} \right),\end{split}  \tag{1}\]  
where the \(\delta\) parameter encompasses all the physics of the problem,  
\[\delta=\frac{1}{15}\left(\frac{3 Re^{2}}{W}\right)^{\frac{1}{3}}, \tag{2}\] \begin{table}
\begin{tabular}{c l l} \hline \hline \(-\) & agent type & ppo-clip \\ \(\gamma\) & discount factor & 0.99 \\ \(\lambda_{x}\) & actor learning rate & \(5\times 10^{-4}\) \\ \(\lambda_{c}\) & critic learning rate & \(5\times 10^{-3}\) \\ \(-\) & optimizer & adam \\ \(-\) & weights initialization & orthogonal \\ \(-\) & activation (actor hidden layers) & tanh \\ \(-\) & activation (actor final layer, continous) & tanh, sigmoid \\ \(-\) & activation (actor final layer, discrete) & softmax \\ \(-\) & activation (critic hidden layers) & relu \\ \(-\) & activation (critic final layer) & linear \\ \(\varepsilon\) & pPO clip value & 0.2 \\ \(\beta\) & entropy bonus & 0.01 \\ \(g\) & gradient clipping value & 0.1 \\ \(-\) & actor network & \([64,[[64],[64]]]]\) \\ \(-\) & critic network & \([64,64]\) \\ \(-\) & observation normalization & yes \\ \(-\) & observation clipping & no \\ \(-\) & advantage type & GAE \\ \(\lambda_{\text{GAE}}\) & bias–variance trade-off & 0.99 \\ \(-\) & advantage normalization & yes \\ \hline \(n_{\text{rollout}}\) & no. of transitions per update & env. specific \\ \(n_{\text{batch}}\) & no. of minibatches per update & env. specific \\ \(n_{\text{epoch}}\) & no. of epochs per update & env. specific \\ \(n_{\text{max}}\) & total no. of transitions per training & env. specific \\ \(n_{\text{training}}\) & total no. of averaged trainings & 5 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Default parameters used for the ppo agent. and where \(Re\) and \(W\) are the Reynolds and the Webber numbers, respectively, defined by the flat film thickness and the flat film average velocity [22]. System (1) is solved on a 1D domain of length \(L\) with the following initial and boundary conditions:  
\[ \begin{split} q(x,0)&=1\text{ and }h(x,0)=1,\\ q(0,t)&=1\text{ and }h(0,t)=1+\mathcal{U}(- \varepsilon,\varepsilon),\\ \partial_{x}q(L,t)&=0\text{ and }\partial_{x}q(L,t)=0, \end{split}  \tag{3}\]  
with \(\varepsilon\ll 1\) being the noise level generated by the uniform distribution \(\mathcal{U}\). As depicted in Figure 1, the introduction of random uniform noise at the inlet initiates the onset of exponentially growing instabilities (blue region), which subsequently exhibit a pseudo-periodic pattern (orange region). In the subsequent area, the periodicity of the waves breaks, and the instabilities transform into pulse-like formations, characterized by a sharp leading edge followed by minor ripples [23]. Of the sharp pulses, some, known as solitary pulses, move faster than others, leading to the amalgamation of upstream pulses in coalescence events.