## 3 Materials and Methods
### Reinforcement Learning Problem Setup  
Here we present how the AVG navigation problem can be formulated as an DRL problem. In this work, the observation space \(O\) is defined as a concatenation of \([O_{v},O_{I},O_{ar}]\). To deal with partial observability, we stack 4 most recent RGB image \(O_{v}\) along the channel dimension exactly as how DQN processed Atari games [19]. The second observation component is LiDAR observation \(O_{I}\), since it mostly reaches fully-observability, we just retain the most recent LiDAR readings. To further increase the information content, we also keep the same length of historical actions and rewards as a part of observation similarly to [65]. Note that no additional handcrafted high-level information, e.g., the position of the robot or the dolly is given. Furthermore, neither a method for localization nor mapping is used. The complete state design is summarized in Table 1. The visual perception \(O_{v}\) and the LiDAR readings \(O_{I}\) are rescaled to \([0,1]\). All these post processed features serve as input to critic and actor network in SAC, as shown in Figure 2. \begin{table}
\begin{tabular}{l c} \hline \hline \multicolumn{2}{c}{**Observation Components**} \\ \hline
**Description** & **Dimensions** \\ \hline Sequence of the four most recent camera RGB images & \(\mathbf{R}^{4\times 3\times 80\times 80}\) \\ \hline Current LiDAR sensor input & \(\mathbf{R}^{1\times 256}\) \\ (front and back sensor concatenated) & \(\mathbf{R}^{4\times 2}\) \\ \hline History of the four previously taken & \(\mathbf{R}^{4\times 1}\) \\ \hline History of the four previously received & \(\mathbf{R}^{4\times 1}\) \\ rewards & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of the sensory observations and additional statistics that describe the state design of this thesis. The action space \(A\) is defined as differential drive command for the AVG, which is a 2-dimensional input \(\vec{a}_{t}=\left[v_{t},\omega_{t}\right]\) with \(v_{t}\in[-1\text{ m/s},1\text{ m/s}]\) and \(\omega_{t}\in[-1\text{ rad/s},1\text{ rad/s}]\). Here \(v_{t}\) and \(\omega_{t}\) are linear and angular target velocity inputs for the differential drive. The actions are carried out for 180 ms, resulting in an approximately 5.5 Hz operation frequency. The reward function is formulated as:  
\[r(t)=r_{S}(t)+\mathds{1}_{CD}r_{CD}(t)+\mathds{1}_{C}r_{C}(t)+\mathds{1}_{F}r_ {F}(t)+\mathds{1}_{G}r_{G}(t),\]  
where \(r_{S}=-0.1\) represents a negative reward for each time step, \(r_{CD}=-0.1\) denotes a small negative reward for collision with the dolly, and \(r_{C}=-10\) corresponds to a large penalty for collision with non-dolly objects, i.e., walls or other obstacles. We set a small penalty for collision with dolly so as to encourage the agent to reach the proximities to the dolly. The term \(r_{C}=10\) is a positive reward when the forklift ends up with successfully reaching underneath the dolly. The last component \(r_{F}=-0.05\) acts as a penalty of not driving forwards, i.e., when the velocity of the AVG is below 0.3 m/s. The symbol \(\mathds{1}_{CD}\), \(\mathds{1}_{C}\), \(\mathds{1}_{F}\) and \(\mathds{1}_{G}\) denotes the corresponding indicator function of whether that event happens. Note that our reward design does not require any map information, e.g., the distance between the dolly and the agent, so that it can be well applicable also to real-world training. Terminal state is reached once the Euclidean distance between the center of the dolly and the center of the robot is less than 0.3 m. Collisions with any objects will also result in an instant termination of the task. To increase the generalizability of the learned policy, we inject domain randomization for each worker environment, e.g., shift of light sources, shape of cells, pattern of the floors and walls etc. The designed arena is shown in Figure 3. Particularly, we randomize the number of obstacles, the position of target dolly and initial pose of the AVG in the cell to avoid overfitting of the sensor readings on a single environment.