# Obtaining Human Experience for Intelligent Dredger Control: A Reinforcement Learning Approach
## 3 Neural Network Model to Construct Virtual Environment
### Representation of State Space  
As we intent to develop a learning system that can imitate the dredging manipulation of an experienced human operator, the variables involved in the state space are consistent with the data that the operator can observe. The states are also related to the production of a CSD, since the main objective of the dredging operation is to maintain an appropriate production rate. In this work, the data are collected from a real CSD, and we selected the following variables that are observable for human operators to form the state space  
\[s:=[C_{v},V_{s},I_{c},I_{p},D_{v},V_{f}]\in\mathbb{R}^{6} \tag{1}\]  
where \(C_{v}\in\mathbb{R}\) is the direct factor that indicates the solids level in the pipeline, and it is measured by a nuclear-based gamma densitometer installed at the stern of a CSD. Other five factors are indirect indicators that are related to the slurry density, but also can reflect the internal information about the dredger and external information about the environment.  
Figure 2: The learning task for the cutting process control.  
* _Swing speed_, denoted by \(V_{s}\in\mathbb{R}\), can indicate the among of the soils that are dredged in a swing movement along an arc trajectory. In such a swing, the CSD rotates around the working pole by slacking (or pulling) the cable of the starboard anchor (or the port side anchor). Those anchor cables are connected to the deck winches via the sheaves on both side of the cuter head. Thus, when the deck winches pull or slack the cables for the swing movement, it also produces a reaction force on the cutter head.
* _Motor current of the cutter head_, denoted by \(l_{c}\in\mathbb{R}\), indicates the reaction forces on the cutter head. The reaction forces are mainly affected by the soil type, as well as the rotation and swing movements of the cutter head. Even if the swing speed remains the same, the reaction forces of the clockwise rotation and the counterclockwise rotation must be different. As mentioned above, the swing movement also produces a reaction force on the cutter head via the anchor cables.
* _Motor current of the underwater pump_, denoted by \(I_{p}\in\mathbb{R}\), can indicate the extent to which the dredged soil is sucked up by the underwater pump. If the slurry density between the cutter head and the underwater pump remains at a high level, more torque is needed to drive the impeller of the pump for rotating, which results in the increase of the motor current of the underwater pump.
* _Degree of suction vacuum_, denoted by \(D_{v}\in\mathbb{R}\), can also indicate the extent to which the dredged soil is sucked up by the underwater pump, but this parameter is different from the motor of the underwater pump. Here the reason is that the slurry with high density requires more power to be sucked up, and the degree of the suction vacuum must be maintained at a high level to pump the dredged soil from the inlet of the cutter head.
* _Flow velocity_, denoted by \(V_{f}\in\mathbb{R}\), can reflect the among of sediment particles in the pipeline. Usually, the rotating speed of the pump(s) is not adjusted frequently, so the changes of the flow velocity are directly related to the slurry density in the pipeline. If the slurry density drops, it will be easier to transport the sediment particles in the pipeline, which will result in the increase of the flow velocity.