My Reinforcement Learning Adventures
By: Isamu Isozaki
Tags: Openai Gym, Openai Baselines, Reinforcement Learning, AI, OpenAI

While the value loss seems to improve, the policy loss seems to plateau. But, one thing I noticed was that while the wings/attack orientation seems to improve, the coordination between players doesn’t seem to be appearing. So, the main problems are:

Players can move randomly without coordination

The wings attack mechanism is too complicated

Like before, the observation space is too complicated

And finally, this is applicable to version 1 too but the action space is a bit too complicated to be human-understandable.

Version 3

Version 3 environment test. The images next to each other are what each model sees. So, the top left video is what the blue side sees and the one next to it is what the red guys see.

Environment

Action space

I made the action space a 4 by 4 vector field so that it is understandable which direction the model is going towards. I also made it so that the vector field represents the force acting on the players.

Observation space

I made the observations space a simplified version of the render above so that it is human understandable and possible easier to learn for the model than before. And now, running multiple games at once is possible!

QOL changes

I made it so, to encourage cooperation, that the players stick together by having a hiarchy and having springs attached to them like so

Spring pulling visualization

Also, for attacking, I just made it so that the velocity is the direction of attack like so and there’s a triangle of attack

Basic model explanation

I wrote a separate series on this on medium called Understanding OpenAI baseline source code and making it do self-play but I basically modified OpenAI baselines to make it so that you can do selfplay in them! I remember it taking like 1 month, maybe 2, but I’m happy with the results. I did this mainly because I wasn’t sure if the problem was in my model or an environment, and since I knew Open AI is one of the best AI companies out there, I thought I can’t go wrong by using their code. The code still doesn’t work for anything other than ppo2 for now.

For the model, I just used cnn-lstm with action space of box(continuous action space) which was quite logical for me because the observation space is an image and the output is just forces which is continuous.

Results

Policy loss

While the policy seems to be improving, when I check the videos, I noticed that there’s no significant change on what is being done across the 100 epochs run. One problem that lead to this was that the model fought against itself. So, at any moment in time, it didn’t make much sense for one side to consistently win against another. Basically, the main problem with this model was that I had no idea if the model was actually learning.

Version 4

Version 4 environment test run. As you can see, it isn’t going well.

Action space

For the action space, I changed it so that I can specify what size I want. For example, the above animation is a 2x2 vector field. I also made it so that the vector field is not an acceleration/force vector field but rather a velocity one and I just apply force that moves the model in the direction of the force. I just thought that’ll be easier to comprehend.

Observation space

No change.

QOL changes

Since I didn’t know if the model was learning, I decided to go with the easiest strategy: fighting against a stationary opponent. So, I’ll know that the model is improving if it is able to defeat the enemy in a faster way.

Another thing I did was I removed the spring mechanic because, if possible, I wanted the model to learn to stick together by itself.

Results

Reward plot with different models

Policy loss plot with different models

As you might have seen from the video, it didn’t go to great. The first plot above is basically the reward so, in the beginning, with a random policy, it worked for a while but then it just stopped doing anything productive and just went to a corner.

For the longest time I was confused about why this was happening but what happened was quite simple. For my action space, I output a continuous actions space with the shape (2, 2, 2) where action space[0][0] represents the top left vector. The problem with this was I used actor critics which outputs a mean and a log standard deviation. So let’s say for example, the model learns that shifting the mean to say -5 is quite a good idea for all actions with the standard deviation of 1. What happens there is that then all that part of the action space ever does is point towards left bottom and it’ll just remain there because all the numbers sampled will always be negative. Just to note, this problem occurred on a 1x1 action space as well.

The reason the policy seems fine is because the value loss is initially really high so the value loss improves and eventually just predicts 0 which leads to the policy loss to appear to be improving.

Version 5(Current)

Version 5 environment test run

Action space

Instead of the continuous action space I used before, I decided to just use a multidiscrete action space. Then, the model just does a softmax and chooses an action without going into all the mean changing problems. For this model, it’s just a 1x1 action space. Currently, I started training on a 2x2 action space

Observation space

No change.

Results

Reward plot with different models

Policy loss plot with different models

While the model doesn’t seem to improve with all the models, I see that for models like impala_cnn and cnn_lstm, the models rewards are increasing. Also, it’s a bit noisy but the problem of the finding an invalid policy seems to have stopped happening which I’m happy about! Also, the policy loss seems normal. So I’m finally getting something after 2 years!

Next Steps

Since I have a self play environment set up, I plan to just have a model fight an older version of itself. For example, making one model learn for 1 epoch and then make the other model fight against it and see how it improves.