# Bacon, a Lightweight Deep Reinforcement Learning Benchmark Library for Flow Control
## 3 Environments
### Shkadov
#### 3.1.3 Environment  
The provided setup is a re-implementation based on the original work by [PERSON] et al. [12], albeit with several distinctions. Notably, the translational invariance aspect introduced in [12] is _not_ utilized in this context. Instead, this control scenario is viewed as an opportunity to evaluate algorithms dealing with problems featuring arbitrarily high action dimensionality. For the control of system (1), a forcing term \(\delta q_{j}\) is introduced into the equation governing the temporal evolution of the flow rate. Practically, localized jets are inserted at specific locations within the domain, as illustrated in Figure 3 below. The intensities of these jets are determined by the DRL agent. Initially, the first jet is positioned at \(x_{0}=150\), with the default jet spacing set to \(\Delta x_{\text{jets}}=10\), akin to in [12]. To conserve computational resources, the domain length is conditioned by the number of jets, \(n_{\text{jets}}\), and their spacing as follows:  
\[L=L_{0}+\left(n_{\text{jets}}+2\right)\Delta x_{\text{jets}}. \tag{6}\]  
By default, \(L_{0}\) is set to be equal to 150 (which corresponds to the beginning of the pseudo-periodic region for \(\delta=0.1\)), and \(n_{\text{jets}}\) is initialized as 1. The spatial discretization step is designated as \(\Delta x=0.5\), while the numerical time-step is \(\Delta t=0.005\) time units. The inlet noise level is configured as \(\varepsilon=5\times 10^{-4}\), mirroring the setup in [12]. The injected flow rate \(\delta q_{j}\) takes the following form:  
\[\delta q_{j}(x,t)=Au_{j}(t)\frac{4\Big{(}x-x_{j}^{t}\Big{)}\Big{(}x_{j}^{r}-x \Big{)}}{\Big{(}x_{j}^{r}-x_{j}^{t}\Big{)}^{2}}, \tag{7}\]  
where \(A=5\), an ad hoc non-dimensional amplitude factor, \(x_{j}^{t}\) and \(x_{j}^{r}\) represent the left and right limits of jet \(j\), and \(u_{j}(t)\in[-1,1]\) denotes the agent-provided jet amplitude. Equation (7) corresponds to a parabolic profile of the jet in \(x\), ensuring a zero flow rate at the boundaries. The jet width \(x_{j}^{r}-x_{j}^{l}\) is fixed at 4, in line with [12]. To transition from one action to another, a time dependence is introduced to \(u(t)\), involving the following saturated linear variation:  
\[u_{j}(t)=(1-\alpha(t))u_{j}^{n-1}+\alpha(t)u_{j}^{n}\text{, with }\alpha(t)= \min\left(\frac{t-t_{n}}{\Delta t_{\text{int}}},1\right), \tag{8}\]  
Hence, when the actor provides a new action to the environment at time \(t=t_{n}\), the effective jet amplitude is linearly interpolated from the previous action \(u_{j}^{n-1}\) to the next one \(u_{j}^{n}\) over period \(\Delta t_{\text{int}}\) (set here to 0.01 time units). Following this interpolation, the new jet amplitude remains constant for the remainder of time interval \(\Delta t_{\text{const}}\) (set here to 0.04 time units). Consequently, the total action time-step is \(\Delta t_{\text{act}}=\Delta t_{\text{int}}+\Delta t_{\text{const}}=0.05\) time units. The overall episode duration is fixed at 20 time units, corresponding to 400 actions. The observations provided to the agent consist of the mass flow rates gathered from the combination of regions \(A_{\text{obs}}^{j}\) of length \(l_{\text{obs}}=10\), located upstream of each jet. In contrast to the original work, the fluid heights in this area are not relayed to the agent, and the observations remain unclipped. The reward for each jet \(j\) is calculated over region \(A_{\text{rwd}}^{j}\) of length \(l_{\text{rwd}}=10\), positioned downstream of it, with the overall reward being the following weighted summation of individual rewards:  
\[r(t)=-\frac{1}{l_{\text{rwd}}\,n_{\text{jets}}}\sum_{j=0}^{n_{\text{jets}}-1 }\sum_{x\in A_{\text{rwd}}^{j}}(h(x,t)-1)^{2} \tag{9}\]  
Thus, a perfect flat film yields a maximum reward value of 0. Moreover, the use of a normalization factor enables comparison of scores across different numbers of jets. Lastly, an evolved initial state is initialized at the onset of each episode.