How I built a bot to beat T-Rex run! | by Rajat Keshri | Towards Data Science

This article will explain the step by step procedure of building your own bot from scratch, which would play T-Rex run game by Google indefinitely.
T-Rex run game is a game which I am sure every human with a phone or a laptop with Google Chrome and no internet connection has played. For those of you who do not know, just turn off your internet connection on your phone or laptop and open any website using Google Chrome and you’ll be able to play T-Rex run! The basic objective of the game is to jump or duck and dodge the obstacles and keep your Dino alive. However, being humans we might get bored or might get distracted and probably loose the game. In order to beat this game to the fullest, we require someone with extreme concentration and someone who cannot get distracted with anything in this world, someone like a BOT!
Here, I will teach you how to build simple bot which would play the T-Rex run game for you while you sit and watch it play peacefully (Although this defeats the whole purpose of any game, but it is just for fun).
I will be building this bot in python using image processing libraries. To start with, you will be required to install Python 3.6 or above on your PC or laptop. Here are the list of libraries you’ll be required to install —
pyautogui — This is a library which has functions to interact with the T-Rex run GUIPIL — PIL Library is used for taking screenshots of the screen and applying image processing on itTime — Used for giving different delays between jumps
pyautogui — This is a library which has functions to interact with the T-Rex run GUI
PIL — PIL Library is used for taking screenshots of the screen and applying image processing on it
Time — Used for giving different delays between jumps
These libraries can be installed using the command “pip install <libary name>”. Loads of documentation is available on w3schools on using pip and other commands and basics of working with python. Here is a link to it — https://www.w3schools.com/python/python_pip.asp
We then load our prerequisites into our code by the following snippet —
import pyautoguifrom time import sleepfrom PIL import ImageGrab,ImageOps
Now, coming to the main part of the article. So the basic logic what we will be following here in building the bot is to mimic how a human plays the game. A human sees the obstacle and tries to time the jump of the Dino to avoid the obstacle. For our bot, we will be doing the same. Basically, here we take a screenshot of the T-Rex game screen and check the position of a tree or a bird obstacle in that image and if the obstacle is close enough to the Dino (defined using a threshold defined in the code), we ask the Dino to jump
I am using this website which I found randomly which is the same T-Rex game, but on the internet — http://www.trex-game.skipser.com/.
The first step is to take the screen shot of the web page with the game and locate the the exact pixel values of the Dino. We find the top and bottom corner pixel values of the Dino. This can be done by importing the image into paint.net and finding the pixel values. Also, if you notice, the position of the Dino is constant and only the obstacles are moving towards it. Hence we take this approach of creating an action on the Dino based on its static position with respect to the obstacles moving forward towards it. The below figure shows how we can get the pixel positions of the Dino figure, using the rulers and the grid view. Usually the sum of all pixel values in an image adds upto a number, and if the image does not change, the sum always remains constant. We keep this in mind and ill explain why this is important in the next few paragraphs.
Similarly, we take a screenshot of the game when a tree is present, a bird is present and replay button is present and note down its pixel locations. This is not much required as of now, but in the future can be helpful.
There are 5 different objects in this game — Dino, tree 1, tree 2, bird and replay button if you crash. You can use the pixel location values which I have used for each of these characters. I have used a class definition hold onto these values for making it simple. This is how to looks —
class coordinates():    replay=(353,388)    tree1=(195,416)     tree2=(245,433)     bird=(155,379)    dino=(170,391)
The next step is to actually write a logic for this code. We create different functions for different processes or objectives.
First we create a function which would capture every frame or take screen shot of every frame on the game in order to process it. Think it like how humans see the game while playing it, the bot also needs to see the game using screenshots. This is shown in the following snippet.
def imagegrab():    box=     (coordinates.dino[0]+20,coordinates.dino[1],coordinates.dino[0]+100,  coordinates.dino[1]+30)    image = ImageGrab.grab(box)    grayimage = ImageOps.grayscale(image)    a = array(grayimage.getcolors())    return a.sum()
In the above function imagegrab(), we get the coordinates of the Dino and create a box around it. Basically the box is a rectangle with 4 pixel location values of each corner, with the Dino inside the box. The box has some open space on right side of the Dino. We then take a screen shot of the box and the box acts like our logic for creating a jump or duck action. The basic logic here is that if a tree or a bird comes inside the box, the Dino is asked to jump or duck respectively.
But bots actually cannot see whats happening in the screenshot or precisely inside the box. Bots can only see and read pixel. Hence, here as hinted in the previous section, we calculate the sum of all pixels within the box. As the image of the Dino and the box is constant and Dino is not moving, the sum of pixels in the box always returns a constant. But if a tree or bird enters the box, the sum value actually increases or decreases based on the pixel values, and that is when the bot gets a hint to either jump or duck. The below image displace the screenshot or how the box would look.
Once we have the image of the Dino and the box, we convert the box into gray scale to increase up our processing as color is not an important factor here as we are just bothered with calculating the sum of the pixel values in the box.
The next function we write is the jump function, which is basically asks the bot to jump when a tree appears. Puautogui library is used to simulate keyboard controls which enables the bot to interact with the game.
Similarly a duck function is written. You can read more about pyautogui here — https://pyautogui.readthedocs.io/en/latest/keyboard.html
def jump():    pyautogui.keyDown('space')    sleep(0.05)    pyautogui.keyUp('space')    sleep(0.05)def duck():    pyautogui.keyDown('down')    sleep(0.05)    pyautogui.keyUp('down')    sleep(0.05)
The next function we write is to restart the game. If by chance the bot fails (very low chance), it should automatically restart without human aid. We use the replay button coordinates and simulate a click on that location. Here is the code snippet for it.
def restart():    pyautogui.click(coordinates.replay)
Once done with all our functions, we are ready to write our main function and run it. The below snippet shows the main function.
def main():          restart()     while True:         imagegrab()         if(imagegrab()!=2646):             jump()              sleep(0.001)If __name__=="__main__":     main()
This is how the bot runs indefinitely and thus this is how we beat the T-Rex run game. This logic can be utilized in many different games too, that is the logic of using the summation of pixel values. We should always take into consideration that the bots only see pixels and nothing else. For future scope we can actually calculate and find the values of the locations of the tree and birds dynamically using OpenCV and other image processing techniques (which i am planning to cover in my next Blog).
Thank you for your time, hope it was informative.