You remember how Boston Dynamics published a video of Atlas where it did? And then... And then we were all like, LOOOOO! And then, a wise man who did not own Twitter at the time said... This is nothing. In a few years, that bot will move so fast you'll need a strobe light to see it, sweet dreams. Well, I'm still waiting for my atomic car and my hoverboard. However, the progress in robotics locomotion last few years has been nothing short of amazing. Today, I would like to talk to you about how robots can imitate the motion of humans or animals. And I know I can get a little bit technical at times, but believe me, by the end of this 8th net video, you will have a really good idea of how this works. In one of my previous videos, I used reinforcement learning to teach robotic dog beetle how to move forward and left and right with the help of reinforcement learning. And there I used handcrafted reward function. Basically, reward function incentivizes the behavior you want the robot to do in this case, walking and it penalizes whatever else robot might be doing, such as flying, walking on its knees, standing on its head, jumping, creating, accurate, stable reward function is really challenging. The robot may and will exhibit some strange behaviors what I told you about, right? Flying, walking on the knees, standing on the head, and maybe just moving its feet really, really fast, tapping on the floor and kind of floating about the surface. So the primary component of our reward function is the difference between the random commands, random velocity commands that we give to the robot and the actual velocity of the robot. For example, the random command that we give to particular robot is move forward with the speed of one meter per second and then we measure the actual velocity of the robot. And the difference between them is the reward or punishment. But having only primary component is not enough, it's too crude and it will lead to all sorts of unwanted behavior. So we add more things to this reward function, feed ground time, the joints allocated at their maximum positions, the termination reward or the punishment if the robot does not last until the end of the round in the simulation if it dies before that. And so on and so forth, this fine tuning of the reward function can be really time consuming. Even with Isaac Jim, which does speed up things considerably, but you'll need, I know, 10-20 minutes to go until you see the results and a reward function is correct and you will repeat many, many times. So that does not seem like a great approach because walking is not a rocket science. It's put your left feet forward, put your right feet forward and here it goes. So we don't really want the robot to figure out walking from scratch. We can tell the robot approximately what walking is like and then just make sure it can apply it to a particular structure, particular skeleton of the robot. This concept is not super complicated. Some smart people wrote papers on that topic. They have an idea that you can teach locomotion by providing hint to the policy to the robot in a form of motion capture files. This is where so-called adversarial motion priors come into play. In the paper, which is quite interesting in a few visual demos, there are two reward components. There is a discriminatory reward and there is a task reward. So the task reward is actually pretty simple. This is, for example, moving in certain direction or moving to a certain location or this is a simple velocity command. That's the one I used. And the second component is the discriminatory reward. When you combine them together, then the goal of the policy is to complete the task while imitating the style of the certain movement from the motion capture files. The motion files, they were taken from Carnegie Mellon University motion capture library and they include movements such as walking, running, jumping, yoga poses, whatsoever. So there is no need for direct correlation between the tasks and the motion capture files because the discriminator evaluates how closely the policy movement resembles the one from the motion capture files. So essentially, it's trying to tell if the observations come from a robot or the policy or they come from the motion capture files. That means that with the time the policy hopefully will become similar in style to the motion capture files movement and also will be able to complete the task given to it. And you might recognize this approach. This is actually a generative adversarial networks or GANs. Before stable diffusion was introduced, they were all the rage for generating the images. Turns out this approach works really well for robotic movements too. To emphasize what they actually say in the paper. The powerful thing about the adversarial motion priors is that in motion capture file you only have the observations. You do not have the actions. You do not know at which certain angle position was the joint to complete the action. But with the adversarial motion priors you don't need that information because the discriminator here, it only judges the observations and has nothing to do with the actions. After my video about Vittel, I sought about how it would be beneficial for the policy to have access to this kind of hint in the form of animal movements because it was a quadruped romance. And then time passed by, I forgot about it, I was doing some other projects recently. I was looking at Isaac Jim environments repository and I found that they implemented the adversarial motion priors example from the paper. They only implemented the style component. They did not implement the task component but actually that part was quite easy. And when I saw this I thought to myself, yeah, after some tweaking that would work great. After browsing a little bit on YouTube I also discovered that researchers from ETH Zurich they implemented adversarial motion priors for a wild version of animal robot and it worked fantastic for them. I decided to give it a shot and try implementing it with Boston Dynamics Appless. Yeah, I know it's not very likely that I'll be able to try out what I have on a real robot or maybe I will. Who knows? Robert, Colby maybe. What I did was I took the motion files before human because it's a humanoid robot and then went to the journey of exploration. Honestly, it was not easy. The code that in video hand for this example is a good research code. It's very reproducible. If you just want to reproduce that particular experiment with the humanoid characters, it works great. But if you want to try it with your own robots then there are a lot of rakes to step on. I want to give you the overview of the steps. This is not a production-ready code but I hope that if you follow my guide then you would only step on two or three rakes instead of 200 like I did. All right, so to see the code instructions follow onto the next video or if you're more interested in large language models and how you can run a custom GPT approximation on your computer, then here's another playlist about large language models.