Reinforcement Learning a Self-driving Car AI in Unity | by Adam Streck | Towards Data Science

The era of self-driving cars is almost upon us, at least according to Elon Musk. Hence, it’s probably about time to learn how to make one for ourselves. While this article does not go into such depths, it’s enough to make simulated cars that are proficient on any track.
EDIT: This story has been updated on 1.1.2021 to the match the release (1.0) version of ML-Agents.
Note: This article is for people who already have a basic idea about working with Unity and are interested in Neural Networks and Reinforcement Learning. No prior experience with neural networks and/or PhD required! This article provides all needed to obtain the following result:
All the necessary code can be found in a GitHub repository for this article.
We first build a basic functionality for a racing game — for this you only need a functional version of Unity (tested with 2020.1). As we are not looking to make a fun or a realistic racing, this can be done in just a few lines.
To make a track, I’ve created a set of 3 tiles (straight, turn left, turn right) which we can combine in any manner to build a track.
As you see, each tile has an arrow. This arrow shows the exit direction from the tile and matches the forward vector of each tile, which we will use later.
By placing the tiles in distances of 10 meters from each other (I recommend setting Snap Settings to 10), we can build any custom track. This is the one we will use for training:
Second, we make the car. I have provided a b̶e̶a̶u̶t̶i̶f̶u̶l convenient 3D model as a base for this project.
Likewise, the car’s forward vector aims in its direction of movement. We can now get our car to move, which is done using the following script.
public class Car : MonoBehaviour{    public float speed = 10f;    public float torque = 10f;    private void Update()    {        float horizontal = Input.GetAxis("Horizontal");        float vertical = Input.GetAxis("Vertical");        float dt = Time.deltaTime;        MoveCar(horizontal, vertical, dt);    }    private void MoveCar(float horizontal, float vertical, float dt)    {            // Translated in the direction the car is facing        float moveDist = speed * vertical;        transform.Translate(dt * moveDist * Vector3.forward);        // Rotate alongside it up axis         float rotation = horizontal * torque * 90f;        transform.Rotate(0f, rotation * dt, 0f);    }}
That’s it. If you followed so far, you can take your car for a drive.
Notice that the script only moves forward/backward based on the up or down arrow keys, and rotates left or right.
We’ve done it, our racing game for one is now complete! Make a few test runs on your own and then let’s see how an AI will fare on it.
Now we will move to the ML-Agents part of the project. For this a ML-agents plugin needs to be installed if you have not yet done so. For details please refer to ML-Agents installation instructions.
For an agent to self-train we need conduct the following:
Observe the state of the agent (position, direction, etc.)Decide an action based on the state.Evaluate the outcome of that action.
Observe the state of the agent (position, direction, etc.)
Decide an action based on the state.
Evaluate the outcome of that action.
To this end we create a new class called CarAgent.cs that extends the Car.cs and gives us the ability to learn from the environment. This class needs to inherit from an Agent instead of MonoBehaviour in order to connect to the underlying ML-Agents machinery.
First we will compare the direction of the agent with the direction of the track. For this we will store the current tile of the track in a global variable _track . We obtain the current track using the following function:
private Transform GetTrackTile(){    var carCenter = transform.position + Vector3.up;    Physics.Raycast(carCenter, Vector3.down, out var hit, 2f);    return hit.transform;}
We can then compare the difference in the direction of the tile and the car, as show in the figure above. This gives us our initial observation — logically the car will try to minimize this angle in order to follow the track. We also use a signed angle (between -180 and 180 degrees) to tell the car whether to steer right or left.
public override void CollectObservations(VectorSensor vectorSensor){    _track = GetTrackTile();    float angle = Vector3         .SignedAngle(track.forward, transform.forward, Vector3.up);    vectorSensor.AddObservation(angle / 180f); // Map to (-1,1)}
Note that the method extend the CollectObservations() method of the Agent class, assuring that the observations get correctly interpreted.
Secondly we need to judge the distance of the car from the obstacles. In the real self-driving application, often used technology is LIDAR — a method of measuring distance using light bounces. Here we do something similar using Raycasting to observe the distance from solid objects around.
To cast the ray we use the following helper function which takes a position and angle relative to the those of the car and checks for collision. If there is any, we store the information as a positive value between [0,1], if not, we return -1.
// Casts a ray from a point in a direction based on the car position// z: offset of the ray origin from the car centre on the z axis// x: as above but on the x// angle: direction of the ray from its originprivate float ObserveRay(float z, float x, float angle){    var tf = transform;        // Get the start position of the ray    var raySource = tf.position + Vector3.up / 2f;     const float RAY_DIST = 5f;    var position = raySource + tf.forward * z + tf.right * x;    // Get the angle of the ray    var eulerAngle = Quaternion.Euler(0, angle, 0f);    var dir = eulerAngle * tf.forward;        // See if there is a hit in the given direction    Physics.Raycast(position, dir, out var hit, RAY_DIST);    return hit.distance >= 0 ? hit.distance / RAY_DIST : -1f;}
To cast all four rays we simply add the following invocations to the AddObservation() method.
vectorSensor.AddObservation(ObserveRay(1.5f, .5f, 25f)); // FrontRvectorSensor.AddObservation(ObserveRay(1.5f, 0f, 0f)); // FrontvectorSensor.AddObservation(ObserveRay(1.5f, -.5f, -25f)); // FrontLvectorSensor.AddObservation(ObserveRay(-1.5f, 0, 180f)); // Back
This is also a time to define the brain of the agent. We start with a BehaviourParameters, which will allow us to test our agent before training :
Now we are only missing an DescisionRequester— a script that controls how often the AI should make a decision:
We will now get the input actions to our academy. Now we only replace the Update function with AgentAction , which defines the basic event loop for the Agent.
public override void OnActionReceived(float[] vectorAction){          float horizontal = vectorAction[0];    float vertical = vectorAction[1];    float dt = Time.fixedDeltaTime;    MoveCar(horizontal, vertical, dt);}
Note that as AgentActionhappens at fixed time, we had to change the time difference between the individual move actions.
You can now replace the Car.cs for CarAgent.cs and continue driving using the keyboard.
Finally, for the agent to be able to self-learn we need to add notion of what is a good and a bad outcome. In particular we want to do the following two:
Reward progressing along the track.Penalize collisions.
Reward progressing along the track.
Penalize collisions.
The first is more complicated so we start with that. This is going to be done in our OnActionReceived method. We first obtain a movement vector by comparing a position before and after the move. This lets us know how much we moved “along” the track. This is then mapped from the range (180°, 0°) to (-1,1) — the bigger the angle, the lower the bonus, with angle >90° being a penalty. Note that this is a different map then the one in CollectObservation. To motivate towards moving at a high speed, we combine the movement direction with the vertical speed at the
var moveVec = transform.position - lastPos;float angle = Vector3.Angle(_track.forward, moveVec);float dirReward = 1f - (angle / 90f);AddReward((dirReward + vertical) * dt);
Note that we multiply by the time step (usually 1/60) so that we get at most reward of 1 per second. This matches the speed at which we travers one tile — at most 1 per second and assures that the bonus and reward values contribute in similar measure. When combined with the previous code we
public override void OnActionReceived(float[] vectorAction){    var lastPos = transform.position;float horizontal = vectorAction[0];    float vertical = vectorAction[1];    float dt = Time.fixedDeltaTime;    MoveCar(horizontal, vertical, dt);        var moveVec = transform.position - lastPos;    float angle = Vector3.Angle(_track.forward, moveVec);    float dirReward = 1f - (angle / 90f);    AddReward((dirReward + vertical) * dt);}
Secondly we want to avoid collisions with walls — this is penalized by setting reward to -1f and ending the current training episode.
private void OnCollisionEnter(Collision other){    if (other.gameObject.CompareTag("wall"))    {        SetReward(-1f);        EndEpisode();    }}
To complete our code we just need to add code for resetting. We want to reset the car to its starting position when it hits the wall — in combination with the negative reward this helps to train the wall-avoiding behavior (if the car got stuck along the wall we could end up with a lot of short episodes). This is done in the last function inherited from our agents:
public override void OnEpisodeBegin(){    if (resetOnCollision)    {        transform.localPosition = Vector3.zero;        transform.localRotation = Quaternion.identity;    }}
Note that we use the resetOnCollision parameter, implying that not always a collision should end in reset — indeed when playing the game the agents should deal with the collision instead of starting again.
We finally have all the components to run the training process.
As Unity supports multi-agent reinforcement learning it is preferable to use more agents at a time. For this purpose we will copy-paste the track. These need to simply be spaced out on the x-z plane so that they do not overlap:
At the very last we need to set parameters for the training process. We keep the current default parameters, only increasing the reporting frequency. To this end we create the following config file called trainer_config.yaml in the project folder:
behaviors:    CarDrive:        trainer_type: ppo            summary_freq: 10000
Now everything is set up and ready to go. If you have followed the ML-Agents installation instructions you should have python and mlagents package. Open the command line, navigate to where you put the trainer_config.yaml and run the following:
mlagents-learn trainer_config.yaml --train --run-id=CarDrive
You should get a response describing the ml-agents version and requesting you to press the play button. Switch to Unity, press play, and the training should start. The process takes a few minutes depending on your CPU (or GPU if you installed the GPU version of TensorFlow).