## 4 Experiments and Results
### Policy Learning  
The task-relevant simulated environment was generated automatically based on our proposed method in the Mujoco physics engine [48] interfaced with OpenAI Gym [49], as is shown in Figure 6. For the manipulation task, the action dimension is 3, which moves the gripper in 3D space. For the navigation task, the action dimension is 2, which controls the TurtleBot's navigation in 2D space. Figure 5: Semantic segmentation results of robot working scenarios. The first row shows the RGB images, and the second row shows their corresponding semantic images. The left two are for the manipulation task, and the right two are for the navigation task. Best viewed in color. We compared our method against several baseline methods. 1. Transfer-RGB: direct training policy with simulated RGB images and using real-world RGB images in the inference period. 2. Transfer-Depth: training policy with simulated depth images and using real-world depth images in the inference period. 3. DR (domain randomization): In the policy training period, the mesh of the objects in the simulated environment is randomly chosen from 50 textures. The camera position and orientation remain fixed, which matches the real-world scenario. The trained policy is directly employed in the real-world scenario in the policy inference period. 4. DA (domain adaptation): First, the training policy is implemented in the simulated environment and then the trained policy is fine-tuned using the same amount of real-world training data as in the simulated environment. For the Transfer-RGB, Transfer-Depth, DR, and DA methods, we built a simulated environment similar to our real-world robot working scenario in Gazebo simulator ([[https://gazebosim.org](https://gazebosim.org)]([https://gazebosim.org](https://gazebosim.org))). The domain randomization technique refers to [9] ([[https://github.com/neka-nat/gazebo_domain_randomization](https://github.com/neka-nat/gazebo_domain_randomization)]([https://github.com/neka-nat/gazebo_domain_randomization](https://github.com/neka-nat/gazebo_domain_randomization))). All of the methods share the same neural networks. We initialized the convolutional layers of the policy networks by the weights pre-trained on ImageNet. The policy networks are trained with TensorFlow ([[https://www.tensorflow.org](https://www.tensorflow.org)]([https://www.tensorflow.org](https://www.tensorflow.org))) on NVIDIA GTX1080. Table 1 summarizes the parameters used in our experiments. In the policy training phase, to evaluate the success rates of the trained policies for the manipulation task, we randomly chose one of the toys (toy dolphin, toy hedgehog, toy squirrel, and toy lion) as the target object to be randomly put in front of the UR5 robot, within its workspace. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Learning rate for policy \(\alpha\) & \(3.0\times 10^{-4}\) \\ Learning rate for value function \(\beta\) & \(1.0\times 10^{-3}\) \\ Length of horizon \(T\) & 100 (Manipulation); 200 (Navigation) \\ Discount \(\gamma\) & 0.99 \\ Rollouts per iteration & 20 \\ Batch size & 32 \\ Optimization method & Adam [50] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameters in our experiments. Figure 6: Task-relevant simulated environments generated based on our proposed method. (**a**) Manipulation task. (**b**) Navigation task. The blue objects are the obstacles, solid red balls are the virtual agents, and green balls denote the target objects (positions). The gray areas illustrate control information of the constructed environment. Best viewed in color. The success rates were evaluated by performing the manipulation task 20 times with the robot gripper in different starting positions and the target object in different goal positions every 30 policy iteration steps.