Introduction to ML Agents 

# Introduction

I am a member of **Arcanain**, a volunteer team working on autonomous driving robots in Japan. Recently, we decided to explore **reinforcement learning** for motion planning. To do this, we chose **Unity ML-Agents** as our simulation and training environment. Eventually, we hope to achieve something similar to what is described in [this article](https://www.jstage.jst.go.jp/article/pjsai/JSAI2019/0/JSAI2019_4Rin119/_pdf), but in Unity. 

In this post, we’ll focus on how to set up the environment for Unity ML-Agents. We’ll continue to build on this foundation and share updates as we progress toward our goal of autonomous driving simulation in Unity.

https://github.com/Arcanain

#  Prior Information

**Environment**

For this tutorial, we'll use either **Ubuntu 22.04** or **macOS 15.2** with the following key components on the table:


- Table 

| name     | version     |
| -------- | ----------- |
| Unity Editor   | 2022.3.55f1|
| Python   | 3.10.8      |
| ML-Agents |  [ML-Agents Release 21](https://github.com/Unity-Technologies/ml-agents/releases/tag/release_21)           |

<details><summary>python packages</summary>

The result of the command  ``$ pip list`` is bellow:

| Packages                | version    | 
|-----------------------------|--------------|
| **mlagents**                | 0.30.0       |
| **mlagents-envs**           | 0.30.0       | 
| **gym**                     | 0.26.2       | 
| **gym-notices**             | 0.0.8        | 
| **numpy**                   | 1.21.2       | 
| **protobuf**                | 5.29.2       | 
| **torch**                   | 1.11.0       | 
| **tensorboard**             | 2.18.0       | 
| **tensorboard-data-server** | 0.7.2        | 
| **PyYAML**                  | 6.0.2        | 
| **attrs**                   | 24.3.0       | 
| **cattrs**                  | 1.5.0        | 
| **grpcio**                  | 1.69.0       | 



</details>

> **Note**  
> It’s important that the versions of **mlagents** and **mlagents-envs** match the requirements for the specific **ML-Agents Unity Toolkit** release you’re using. In this article, we’ll install **Release 21**, so we’ll use **mlagents** and **mlagents-envs** at version **0.30.0**. For more deteils see [this release note](https://github.com/Unity-Technologies/ml-agents/releases/tag/release_21).


# 1. Install Unity Hub

Reference[^1].
Add the public signing key, run the following command:
```
$ wget -qO - https://hub.unity3d.com/linux/keys/public | gpg --dearmor | sudo tee /usr/share/keyrings/Unity_Technologies_ApS.gpg > /dev/null
```
To add the Unity Hub repository, you need an entry in /etc/apt/sources.list.d. Run the following command to add the Unity Hub repository:
```
$ sudo sh -c 'echo "deb [signed-by=/usr/share/keyrings/Unity_Technologies_ApS.gpg] https://hub.unity3d.com/linux/repos/deb stable main" > /etc/apt/sources.list.d/unityhub.list'
```
Update the package cache and install the package:
> Mac user can install it on this page: [Start Your Creative Projects and Download the Unity Hub | Unity](https://unity.com/download)


```
$ sudo apt update
$ sudo apt-get install unityhub
```
Then, intsallation is complete.
Start Unity Hub by terminal command
```
$ unityhub
```
If the installation finished without any problems, you should see a window like this:
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/f4acc32a-0107-fd23-730f-9b14de9c0a8e.png)

### Create Account

If you don't have an Unity account, you have to create new one.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/18c6d2bf-03cd-1b06-5881-89dd478b7e96.png)


# 2. Install **Unity Editor**
After cretating an account or loggin, you can install the Unity Editor.
The minimum supported Unity version was updated to 2022.3. [^7], make sure to select **2022.3 or newer** when installing.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/079e4cf6-8063-aba3-5f46-3c14c072cf4a.png)

[^7]:https://github.com/Unity-Technologies/ml-agents/releases/tag/release_21

<details>
<summary>Licence (Optional)

</summary>

Recently, you don't have to add a license for personal use. A Free Personal License is added by default.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/57fb4e27-55cc-fce5-da0b-76224d76ab4b.png)


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/722103ba-ae2b-579a-3907-1e2f8f0efdac.png)



![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/05dd6ae6-0737-0278-0d4c-f8f000864eee.png)



![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/2fb0ffe7-1b64-a7cc-1b01-2b2e66ceef7f.png)


</details>



# 2. Install ML-Agents 

### 2.1 Install ML-Agents Tool Kit


First, clone the ML-Agents Toolkit from GitHub by running the following command:



```
$ git clone https://github.com/Unity-Technologies/ml-agents.git
```
After the cloning finished, checkout the branch to ``release 21``.

```
$ cd ml-agents
$ git checkout release_21
```

### 2.2 Install the mlagents for python

The `mlagents` Python package provides the core training scripts and interfaces for communicating with the Unity environment. 

Install it by running:
```
$ pip3 install mlagents
```


When the installation finishes, verify it with the following command:

```
$ mlagents-learn --help
```

If you see an output similar to:

```
usage: mlagents-learn [-h] [--env ENV_PATH] [--resume] [--deterministic]
                      [--force] [--run-id RUN_ID] [--initialize-from RUN_ID]
                      [--seed SEED] [--inference] [--base-port BASE_PORT]
                      [--num-envs NUM_ENVS] [--num-areas NUM_AREAS] [--debug]
                      [--env-args ...]
                      [--max-lifetime-restarts MAX_LIFETIME_RESTARTS]
                      [--restarts-rate-limit-n RESTARTS_RATE_LIMIT_N]
                      [--restarts-rate-limit-period-s RESTARTS_RATE_LIMIT_PERIOD_S]
                .
                .
                .
                .
                
```
then you have successfully installed the **mlagents** Python package!




# 3. Tutorial
Start unity hub with the following command:
```
$ unityhub
```

First, you have to create a new Project.
Push [Add] -> Add project from disk.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/3280d5c5-833c-6de2-6ae7-4245dc30057a.png)

In this tutorial, we use cloened repository ``ml-agents``.
Choose the directory which you have cloned before
- ``.../ml-agents/Project``
    - ``ml-agents`` is the cloned directory


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/d418f56d-3e0a-b7ad-0ef5-cb024f76969b.png)


If you can see this window it is working.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/55781a22-35eb-1071-d3ee-6ced0968bd7a.png)


## demo
You can see ``Assets`` on the bottom of the previous window.

**click  "「ML-Agents」→「Examples」→「3DBall」→「Scenes」→「3DBall」"**
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/0a455eaa-6a3a-e8bf-a389-fda83703e0a2.png)



If you can see these objects, loading has successed.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/e19dc8ef-7739-5112-8581-f1540d25579e.png)

run this command:

```
$ cd /path/to/Unity/ml-agents 
$ mlagents-learn config/ppo/3DBall.yaml --run-id=first3DBallRun
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/856f47f7-9e25-a85e-8850-9820d382d9e9.png)


then, push "start botton" of Unity Editor.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/3665619/689677fa-2cbf-0ec9-915f-152f8faccd0c.png)

Reinforcement learning starts!!
After a while the learning will finish and you can see the reslult of learned model.


[^1]:[Install the Unity Hub | Unity Hub | 3.0](https://docs.unity3d.com/hub/manual/InstallHub.html#install-hub-linux)

# Acknowledgements

This project is supported by **GxP (Growth xPartners) Inc.**. They have provided us with valuable advice and essential equipment for our robots. We extend our deepest gratitude for their support.

- [GxP Official Website](https://www.gxp.co.jp/)

https://www.gxp.co.jp/home-EN

- [GxP on Qiita](https://qiita.com/organizations/gxp)

https://qiita.com/organizations/gxp