【強化学習でAIを避難させる】#4 垂直避難AIに少し手を加えてみた -Unity ML-Agents

#0. はじめに
Yuulisです。今回は、前回作ったAIに少し手を加え、学習結果の分析まで行ってみます。前回の記事を読んでいない方は、先にこちらをお読みください。

https://qiita.com/Yuulis/items/0c521eadcd1364c1efd1

#1. どこを改良するのか
前回のAIは、3階に避難した後の挙動が不安定でした。
![Videotogif.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/bd196b55-4e1f-2d2d-4e29-0c5e574f66eb.gif)
そこで、この斜めに傾く動作をなくしたいと思います。

また、学習環境の構造上、次のように移動すれば簡単に避難できてしまいます。
![スクリーンショット 2021-06-27 125623.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/cfbc013d-0940-6e64-e60a-af1c088cc70e.png)
そこで、一部の階段をふさいで、通れなくしてみます。

#2. エージェントの動作を安定させる
いろいろ試してみた結果、前回のスクリプトのこの部分が問題であることが分かりました。

```AgentControl_3Floor_Lv1_3.cs
// When Agent moves (size = 3)
    public void MoveAgent(ActionSegment<int> act)
    {
        var dirToGo = Vector3.zero;
        var rotateDir = Vector3.zero;
        var dirToGoForwardAction = act[0];
        var dirToGoSideAction = act[1];
        var rotateDirAction = act[2];

        if (dirToGoForwardAction == 1) dirToGo = 1.0f * this.transform.forward;  // Forward
        else if (dirToGoForwardAction == 2) dirToGo = -1.0f * this.transform.forward;  // Back

        if (dirToGoSideAction == 1) dirToGo = 1.0f * this.transform.right;  // Right
        else if (dirToGoSideAction == 2) dirToGo = -1.0f * this.transform.right;  // Left

        if (rotateDirAction == 1) rotateDir = 1.0f * this.transform.up;  // Rotate right
        else if (rotateDirAction == 2) rotateDir = -1.0f * this.transform.up;  // Rotate left

        this.transform.Rotate(rotateDir, Time.deltaTime * 200f);
        this.rBody.AddForce(dirToGo * AgentSpeed, ForceMode.VelocityChange);
    }
```
左右の平行移動(``dirToGoSideAction``)がふらつく原因になっていたようです。
そこで、この部分を削除し、以下のように書き換えました。

```AgentControl_3Floor_Lv4.cs
// When Agent moves (size = 3)
    public void MoveAgent(ActionSegment<int> act)
    {
        var dirToGo = Vector3.zero;
        var rotateDir = Vector3.zero;
        var dirToGoForwardAction = act[0];
        var rotateDirAction = act[1];

        if (dirToGoForwardAction == 1) dirToGo = 1.0f * this.transform.forward;  // Forward
        else if (dirToGoForwardAction == 2) dirToGo = -1.0f * this.transform.forward;  // Back

        if (rotateDirAction == 1) rotateDir = 1.0f * this.transform.up;  // Rotate right
        else if (rotateDirAction == 2) rotateDir = -1.0f * this.transform.up;  // Rotate left

        this.transform.Rotate(rotateDir, Time.deltaTime * 200f);
        this.rBody.AddForce(dirToGo * AgentSpeed, ForceMode.VelocityChange);
    }
```

#3. 一部の階段を使えなくする
続いて、合計8個ある階段(上下合わせて数える)のうち、ランダムで3つを障害物で塞いで使えなくします。
障害物の形はこんな感じ。名前は``StairObstacle``です。これは、TraingAreaの一つ下の階層に置いています。
![スクリーンショット 2021-06-27 140916.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/6fcd299b-3b8f-cfca-4933-f07b4578b4cd.png)
上下の階段をちょうど塞ぐようにしてみました。これを3個に複製し、8か所のうち3か所にランダムで配置します。
そのためのスクリプトがこちらですが、うまい書き方が分からなかったため、とても煩雑になっています。なお、``StairObstacle``は``Transform``型で取得しています。

```AgentControl_3Floor_Lv4.cs
    public void SpawnStairObstacle() 
    {
        StairObstacle1.rotation = Quaternion.identity;
        StairObstacle2.rotation = Quaternion.identity;
        StairObstacle3.rotation = Quaternion.identity;

        int RandomPosNum1 = Random.Range(1, 8 + 1);
        if (RandomPosNum1 == 1) StairObstacle1.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum1 == 2) { StairObstacle1.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle1.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum1 == 3) { StairObstacle1.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle1.Rotate(0f, 180f, 0f); }
        if (RandomPosNum1 == 4) StairObstacle1.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum1 == 5) StairObstacle1.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum1 == 6) { StairObstacle1.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle1.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum1 == 7) { StairObstacle1.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle1.Rotate(0f, 180f, 0f); }
        if (RandomPosNum1 == 8) StairObstacle1.localPosition = new Vector3(11f, 11f, -8f);

        int RandomPosNum2 = Random.Range(1, 8 + 1);
        while (RandomPosNum2 == RandomPosNum1) RandomPosNum2 = Random.Range(1, 8 + 1);

        if (RandomPosNum2 == 1) StairObstacle2.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum2 == 2) { StairObstacle2.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle2.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum2 == 3) { StairObstacle2.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle2.Rotate(0f, 180f, 0f); }
        if (RandomPosNum2 == 4) StairObstacle2.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum2 == 5) StairObstacle2.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum2 == 6) { StairObstacle2.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle2.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum2 == 7) { StairObstacle2.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle2.Rotate(0f, 180f, 0f); }
        if (RandomPosNum2 == 8) StairObstacle2.localPosition = new Vector3(11f, 11f, -8f);

        int RandomPosNum3 = Random.Range(1, 8 + 1);
        while (RandomPosNum3 == RandomPosNum1 || RandomPosNum3 == RandomPosNum2) RandomPosNum3 = Random.Range(1, 8 + 1);

        if (RandomPosNum3 == 1) StairObstacle3.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum3 == 2) { StairObstacle3.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle3.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum3 == 3) { StairObstacle3.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle3.Rotate(0f, 180f, 0f); }
        if (RandomPosNum3 == 4) StairObstacle3.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum3 == 5) StairObstacle3.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum3 == 6) { StairObstacle3.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle3.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum3 == 7) { StairObstacle3.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle3.Rotate(0f, 180f, 0f); }
        if (RandomPosNum3 == 8) StairObstacle3.localPosition = new Vector3(11f, 11f, -8f);
    }
```

#4. 1エピソード終了を視覚的に分かりやすくする
最後に、1エピソードが終わった段階で、エージェントが避難成功した、あるいは失敗したかを視覚的に分かりやすくしてみます。
新たに、空のオブジェクト``Floor3_Settings``を作り、そこに以下のスクリプトをアタッチします。なお、このオブジェクトはTraingAreaの一つ下の階層に置いています。

```Floor3_Settings.cs
using UnityEngine;

public class Floor3_Settings : MonoBehaviour
{
    // When Agent could survive, it will use this material for a few seconds.
    public Material Success_Floor;

    // When Agent couldn't survive, it will use this material for a few seconds.
    public Material Failed_Floor;

}
```
そして、2種類のマテリアルを作成、以下のようにアタッチします。
![スクリーンショット 2021-06-27 142231.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/df0e2c0a-f77b-7733-0274-9fc7e086e550.png)
![スクリーンショット 2021-06-27 142332.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/60fc7c0b-097b-64e4-b452-3dbaf2de2a7f.png)

``AgentControl_3Floor_Lv4.cs``を以下のように書き換えます(かなり長くなってきた)。

```AgentControl_3Floor_Lv4.cs
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Unity.MLAgents;
using Unity.MLAgents.Sensors;
using Unity.MLAgents.Actuators;

public class AgentControl_3Floor_Lv4 : Agent
{
    // General
    private bool onGround;
    public GameObject Floor3;
    Material Floor3Material;
    Renderer Floor3Renderer;
    Floor3_Settings Floor3_Settings;

    // Agent
    Rigidbody rBody;
    private int nowFloor;
    public float AgentSpeed = 0.6f;

    // Water
    public Transform Water;

    // Obstacle
    public Transform StairObstacle1;
    public Transform StairObstacle2;
    public Transform StairObstacle3;

    // At Initializing
    public override void Initialize()
    {
        Floor3_Settings = FindObjectOfType<Floor3_Settings>();

        this.rBody = GetComponent<Rigidbody>();
        Floor3Renderer = Floor3.GetComponent<Renderer>();
        Floor3Material = Floor3Renderer.material;
    }

    // Agent Spawn
    public void SpawnAgent()
    {
        this.transform.localPosition = new Vector3(Random.Range(-19.5f, 19.5f), 4.75f, Random.Range(-14.5f, 14.5f));
    }

    public void SpawnStairObstacle() 
    {
        StairObstacle1.rotation = Quaternion.identity;
        StairObstacle2.rotation = Quaternion.identity;
        StairObstacle3.rotation = Quaternion.identity;

        int RandomPosNum1 = Random.Range(1, 8 + 1);
        if (RandomPosNum1 == 1) StairObstacle1.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum1 == 2) { StairObstacle1.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle1.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum1 == 3) { StairObstacle1.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle1.Rotate(0f, 180f, 0f); }
        if (RandomPosNum1 == 4) StairObstacle1.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum1 == 5) StairObstacle1.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum1 == 6) { StairObstacle1.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle1.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum1 == 7) { StairObstacle1.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle1.Rotate(0f, 180f, 0f); }
        if (RandomPosNum1 == 8) StairObstacle1.localPosition = new Vector3(11f, 11f, -8f);

        int RandomPosNum2 = Random.Range(1, 8 + 1);
        while (RandomPosNum2 == RandomPosNum1) RandomPosNum2 = Random.Range(1, 8 + 1);

        if (RandomPosNum2 == 1) StairObstacle2.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum2 == 2) { StairObstacle2.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle2.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum2 == 3) { StairObstacle2.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle2.Rotate(0f, 180f, 0f); }
        if (RandomPosNum2 == 4) StairObstacle2.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum2 == 5) StairObstacle2.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum2 == 6) { StairObstacle2.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle2.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum2 == 7) { StairObstacle2.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle2.Rotate(0f, 180f, 0f); }
        if (RandomPosNum2 == 8) StairObstacle2.localPosition = new Vector3(11f, 11f, -8f);

        int RandomPosNum3 = Random.Range(1, 8 + 1);
        while (RandomPosNum3 == RandomPosNum1 || RandomPosNum3 == RandomPosNum2) RandomPosNum3 = Random.Range(1, 8 + 1);

        if (RandomPosNum3 == 1) StairObstacle3.localPosition = new Vector3(11f, 4f, 15f);
        if (RandomPosNum3 == 2) { StairObstacle3.localPosition = new Vector3(-11f, 4f, 8f); StairObstacle3.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum3 == 3) { StairObstacle3.localPosition = new Vector3(-11f, 4f, -15f); StairObstacle3.Rotate(0f, 180f, 0f); }
        if (RandomPosNum3 == 4) StairObstacle3.localPosition = new Vector3(11f, 4f, -8f);
        if (RandomPosNum3 == 5) StairObstacle3.localPosition = new Vector3(11f, 11f, 15f);
        if (RandomPosNum3 == 6) { StairObstacle3.localPosition = new Vector3(-11f, 11f, 8f); StairObstacle3.Rotate(0f, 180f, 0f); } 
        if (RandomPosNum3 == 7) { StairObstacle3.localPosition = new Vector3(-11f, 11f, -15f); StairObstacle3.Rotate(0f, 180f, 0f); }
        if (RandomPosNum3 == 8) StairObstacle3.localPosition = new Vector3(11f, 11f, -8f);
    }

    // When Episode begins
    public override void OnEpisodeBegin()
    {
        onGround = false;

        // Reset Agent's status
        this.rBody.angularVelocity = Vector3.zero;
        this.rBody.velocity = Vector3.zero;
        this.transform.rotation = Quaternion.identity;

        nowFloor = 1;
        SpawnAgent();

        SpawnStairObstacle();

        Water.localPosition = new Vector3(-25.0f, -10.0f, -25.0f);
    }

    // Get observations (size = 4)
    public override void CollectObservations(VectorSensor sensor)
    {
        sensor.AddObservation(this.transform.localPosition);  // Agent(x, y, z)
        sensor.AddObservation(Water.localPosition.y);  // Water(y)
    }

    // When Agent moves (size = 3)
    public void MoveAgent(ActionSegment<int> act)
    {
        var dirToGo = Vector3.zero;
        var rotateDir = Vector3.zero;
        var dirToGoForwardAction = act[0];
        var rotateDirAction = act[1];

        if (dirToGoForwardAction == 1) dirToGo = 1.0f * this.transform.forward;  // Forward
        else if (dirToGoForwardAction == 2) dirToGo = -1.0f * this.transform.forward;  // Back

        if (rotateDirAction == 1) rotateDir = 1.0f * this.transform.up;  // Rotate right
        else if (rotateDirAction == 2) rotateDir = -1.0f * this.transform.up;  // Rotate left

        this.transform.Rotate(rotateDir, Time.deltaTime * 200f);
        this.rBody.AddForce(dirToGo * AgentSpeed, ForceMode.VelocityChange);
    }

    IEnumerator GoalScoredSwapGroundMaterial(Material mat, float time)
    {
        Floor3Renderer.material = mat;
        yield return new WaitForSeconds(time);
        Floor3Renderer.material = Floor3Material;
    }

    // Agent's action
    public override void OnActionReceived(ActionBuffers actions)
    {
        if (onGround) 
        {
            MoveAgent(actions.DiscreteActions);
            if (Water.localPosition.y < 14.0f) Water.Translate(0f, 0.01f, 0f);

            // When Water comes
            if (this.transform.localPosition.y < Water.localPosition.y)
            {
                AddReward(-1.0f);
                EndEpisode();
                StartCoroutine(
                    GoalScoredSwapGroundMaterial(Floor3_Settings.Failed_Floor, 0.5f));
            }
            
            float DistanceToWater = this.transform.localPosition.y - Water.localPosition.y;
            AddReward(((DistanceToWater / 10f) + (0.15f * nowFloor)) / MaxStep);
        }

    }

    // Human Controling
    public override void Heuristic(in ActionBuffers actionsOut)
    {
        var discreteActionsOut = actionsOut.DiscreteActions;

        if (Input.GetKey(KeyCode.W)) discreteActionsOut[0] = 1;  // Forward
        if (Input.GetKey(KeyCode.S)) discreteActionsOut[0] = 2;  // Back
        if (Input.GetKey(KeyCode.D)) discreteActionsOut[1] = 1;  // Rotate right
        if (Input.GetKey(KeyCode.A)) discreteActionsOut[1] = 2;  // Rotate left
    }

    // When touching something
    void OnCollisionEnter(Collision col)
    {
        // Touching floor?
        if (onGround)
        {
            // Wall or Obstacle
            if (col.gameObject.CompareTag("Wall") || col.gameObject.CompareTag("Obstacle"))
            {
                AddReward(-0.7f);
                EndEpisode();
                StartCoroutine(
                GoalScoredSwapGroundMaterial(Floor3_Settings.Failed_Floor, 0.5f));
            }

            // UpStair
            else if (col.gameObject.CompareTag("UpStair"))
            {

                this.rBody.angularVelocity = Vector3.zero;
                this.rBody.velocity = Vector3.zero;
                this.transform.rotation = Quaternion.identity;

                if (this.transform.localPosition.x > 0 && this.transform.localPosition.z > 0) { this.transform.Translate(-1.0f, 7.0f, 4.0f); this.transform.Rotate(0f, -90.0f, 0f); }
                else if (this.transform.localPosition.x < 0 && this.transform.localPosition.z > 0) { this.transform.Translate(1.0f, 7.0f, -4.0f); this.transform.Rotate(0f, 90.0f, 0f); }
                else if (this.transform.localPosition.x < 0 && this.transform.localPosition.z < 0) { this.transform.Translate(1.0f, 7.0f, -4.0f); this.transform.Rotate(0f, 90.0f, 0f); }
                else if (this.transform.localPosition.x > 0 && this.transform.localPosition.z < 0) { this.transform.Translate(-1.0f, 7.0f, 4.0f); this.transform.Rotate(0f, -90.0f, 0f); }
                
                nowFloor++;
                AddReward(0.3f);
            }

            // DownStair
            else if (col.gameObject.CompareTag("DownStair"))
            {

                this.rBody.angularVelocity = Vector3.zero;
                this.rBody.velocity = Vector3.zero;
                this.transform.rotation = Quaternion.identity;

                if (this.transform.localPosition.x > 0 && this.transform.localPosition.z > 0) { this.transform.Translate(-1.0f, -7.0f, -4.0f); this.transform.Rotate(0f, -90.0f, 0f); }
                else if (this.transform.localPosition.x < 0 && this.transform.localPosition.z > 0) { this.transform.Translate(1.0f, -7.0f, 4.0f); this.transform.Rotate(0f, 90.0f, 0f); }
                else if (this.transform.localPosition.x < 0 && this.transform.localPosition.z < 0) { this.transform.Translate(1.0f, -7.0f, 4.0f); this.transform.Rotate(0f, 90.0f, 0f); }
                else if (this.transform.localPosition.x > 0 && this.transform.localPosition.z < 0) { this.transform.Translate(-1.0f, -7.0f, -4.0f); this.transform.Rotate(0f, -90.0f, 0f); }

                nowFloor--;
                AddReward(-0.5f);
            }
        }

        else
        {
            // Floor
            if (col.gameObject.CompareTag("Floor"))
            {
                onGround = true;
            }

            else if (this.transform.localPosition.y < Water.localPosition.y)
            {
                SpawnAgent();
            }

            // else
            else SpawnAgent();
        }
    }
}
```
``IEnumerator GoalScoredSwapGroundMaterial(Material mat, float time)``という関数で、3階部分の床の色を変えています。

#5. 学習結果
この条件で、500万Step学習させました。今回は、学習結果を数値に取って分析してみました。
3階に5秒ほどとどまれたら避難成功としています。

https://youtu.be/XVN1bZe-0pQ

報酬グラフはこんな感じ。
![スクリーンショット 2021-07-03 132201.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/19c92f13-b16e-ac3f-4c2e-51d706f0c0f6.png)
また、5000回試行すると、こんな結果に。
![スクリーンショット 2021-07-03 135958.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1285757/2d409acb-70f6-f5e8-3d1e-d4040cf9d4d2.png)
約70%が避難に成功しているようです。なかなかいい感じ。

#6. 終わりに
今回で、垂直避難AIは完成とします。
今後の課題としては、このAIの振る舞いを実際の避難行動にどう応用していくかだと思っています。強化学習では、AIがなぜそのような行動に至ったのかを説明することが難しいので、個々の状況に応じた分析が難しいからです。

次回以降は、津波避難を学習させる予定です。ありがとうございました。