Creating a Game Bot to Play ‘Link Up’ with Graph Theory, Tensorflow and Keras
By: Lester So
Tags: Computer Vision, Gaming, Keras, Graph Theory, TensorFlow

Creating a Game Bot to Play ‘Link Up’ with Graph Theory, Tensorflow and Keras

用圖論, Tensorflow和Keras製作自動玩「連連看」的程式

Final Result

1.1 Gif

1.2 YouTube Link

As shown above, the final tool can effectively identify the tiles in the game panel and automatically click them according to some rules. The reaction to the dynamic change of the screen is quick and accurate.

1.3 Github Link

https://github.com/lester1027/game_bot_linkup

2. Motivation

During my leisure time, I challenged my girlfriend to play Link Up. Link Up is a game in which the players try to link two tiles of the same type together at a time. The linked tiles will disappear. The path of linking them up must consist of equal to or less than two turnings.

No Turning Point

One Turning Point

Two Turning points

After playing for many rounds, I lost all of them. It turned out that my girlfriend was a master in this game. She used to spend a lot of time on it after school. Facing her incredible reflex arc, I stood no chance as a newbie. Therefore, I thought of borrowing power from our lovely machine, computer.

3. Goal

Then I started to plan a program which could automate the whole gaming process for me. In short, a certain type of game hack, despite no real hacking being involved here. The functions of the program are listed below.

Locate the patterns on the tiles in the game panel

Identify to which category the tiles belong

Store the location and category information of the tiles

Set up the rules of the game

Determine if the target tiles abide by the rules

Move the mouse cursor to the target tiles and click them if yes

Select another pair of tiles or wait for the next timestep of the game panel screen if no

4. Methods

4.1 Categorize tiles

When I looked for insights on the web for this side project, I found a good game bot tutorial whose author made a program to automate a sushi-making game. The image recognition techniques used there is simply reading the RGB values of some target positions in the game panel. For games with simple graphical effects, this method is useful enough. However, for the game I am trying to solve, this method is not competent because of the following reasons.

I. The tiles in this game cannot be cut with the exact same height and width dimensions each time. Sometimes there are errors of one to two pixels.

II. The RGB values of the pixels from the same class of patterns can change when the tiles appear in different positions due to the lighting effects.

III. The RGB values of the pixels can also change when there are graphical effects triggered. Some of the target pixels may even be covered.

Due to the more complex situation in this game, a smarter method is required. Computer vision is a suitable approach worth trying. It is widely used in image classification tasks nowadays.

I need to train a computer vision model to do the task. Some of the codes from the book Deep Learning for Computer Vision with Python are used. I used the libraries Tensorflow and Keras for machine learning. Tensorflow is responsible for the low-level tensor operation. Keras is a high-level API for wrapping the Tensorflow operations. It makes the Tensorflow ecosystem more user-friendly. In this project, a simple CNN is trained to capture the features in the images and use them to predict the class. It involves the classification of 17 classes since there are 17 different patterns in the tiles. The CNN consists of standard convolution, activation, batch normalization, pooling and dropout layers. Then the matrix is flattened and a softmax function is applied as the final activation function. Cross-entropy is used to determine the predicted class.

model.add(Conv2D(128, (3, 3), padding=’same’)) model.add(Activation(‘relu’)) # model.add(BatchNormalization(axis=channel_dim)) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) # First (and only) set of FC => RELU layers model.add(Flatten()) model.add(Dense(512)) model.add(Activation(‘relu’)) model.add(BatchNormalization()) model.add(Dropout(0.5)) # Softmax classifier model.add(Dense(classes)) model.add(Activation(‘softmax’))

First, a screenshot of the game panel is captured. Then the tiles in the screenshot are extracted according to the predefined pos_dict. The keys of the pos_dict are the row and column indices of the tiles. The values of it are the x and y coordinates of the four vertices of each tile. Grayscaling is applied on the split images so that only one channel of the images is left to reduce the processing time during classification. All the pixels values in the processed images are rescaled to a (0,1) range. After preprocessing, the pixels values are stored in val_dict.

def categorize(model): ‘’’ #input:None #output: a dictionary containing the cropped and grayscaled images ‘’’ val_dict =grabBox() type_dict={} #for each position of the game panel for row in range(n_row): for col in range(n_col): val=val_dict[(row,col)] #resize sp = SimplePreprocessor(pixels,pixels) val=sp.preprocess(val) #image to array operation iap = ImageToArrayPreprocessor() val=iap.preprocess(val) val=val.reshape(-1,pixels,pixels,1) type_dict[(row,col)]=type_list[model.predict(val,batch_size=1).argmax()] return type_dict

4.2 Graph Analysis

Graph theory is used to analyze the information collected from the game panel. A graph is a set of vertices V and a set of edges E, comprising an ordered pair G=(V, E). When we only consider one class of target tile, the patterns stored can be labelled as three types: target, obstacle and empty tiles.

The Raw Screenshot

The Target Tiles: Black Duck (Blue), Obstacle Tiles (Red) and Empty Tiles (Grey) (two more columns and rows are added here because the path can be constructed outward)

The Graph formed from the Concept above

After constructing the graph, breadth-first search (BFS) algorithm is used to determine the shortest path from one tile to another. The inputs are the graph and the initial vertex when traversing the graph. The output of the function is a list with the order of traversed vertices and a dictionary containing the vertices as the keys and their parent vertices as the value.

However, the BFS algorithm cannot generate the path with the smallest number of turnings, which is necessary for this project. Therefore, a function called count_turn() is written to count the minimum number of turns in a walking path.

Shortest Path, but not with the Least Turning Points

Shortest Path, and with the Least Turning Points

If the count is smaller or equal to two, the corresponding path is a valid one.

After the analyzing progress, the program can move the mouse cursor to the centres of the target tiles and click them one after another.

When the linked tiles disappear due to successful link, the program takes another screenshot of the game panel and repeat the processes mentioned above. If there is no valid path develop in this loop, the program will continue to look for potential paths generated by other tiles.

5. Afterword

I successfully beat my girlfriend with this program. She has never played this game with me since then.