Hello, my name is Carlos and I'm going to explain you here my entry for the module Machine Learning for Creative Practice at Goldsmiths University of London. So I made a very simple game where you need to punch boxes using a lead motion Follow in the beat of the music. So let me show you how the game looks like First we will be able to see our hands next lead motion here and we see that boxes are corresponding from the bottom of the screen down to the top very similar to many written things. So what we need to do is just punch a box So every time we successfully punch a box we will see explosion. Every time we punch a box we get around 12 points And the leg is very simple to punch it. Let me show you what happens when I play the folder As you can see the gameplay gets harder and also the background reacts dynamically So for doing all of this I'm using the Rapidly Plugging which is a plugin being developed here at the Goldsmiths As you can see if I don't move like a punch I don't make this kind of movement the boxes will not react like that This is because I'm using several features for the punch but also if I don't punch the way I intended to it will misclassify the gestures of our example here I make that movement so there is low punch the network is not trained for this so it will not understand So I think you get an idea how the game works let me show you how the all the tools looks like and how the editor I will just minimize the screen and then you can see here the game editor which is the main menu scene where the game manager is And I have all my classes for handling logic audio levels audio you can also be with the audio in-game But let's jump to the machine learning bit So inside level 1 you can find here under the scenes folder bit puncher level 1 is where it's having all the components regarding the machine learning processing So inside the object I am a components you can find both classifiers for the ref hand and the right hand And I will quickly introduce the tool I'm using so this is rapidly as I told before this is a tool that I help in develop It allows to perform three types of learning classification regression and dynamic time warping I'm using classification to classify the punches And you can configure what you want to input I'm using right now dummy object which is located in the center of the pole of the hand So actually right now it doesn't matter if you have your fingers open or closed because what matters is the features that I start from this dummy object in the pole of the hand Here I have also my training examples which I can open I can explore I can modify if I want to to reach my network And here is the features section where I can choose how many features I want Right now I have three features rotation acceleration and velocity from the this dummy object But I could say that I want less for some reason I only want two so we rotation and acceleration or I want more So I can go with rotation acceleration velocity and position And I can also modify what features I want inside the code and say like well I want something else Maybe I want a relative distance between both hands so I can create a new feature extractor inside the rapidly class But what I found it worked best for me was having three these three features rotation acceleration and velocity per hand Nonetheless developing my own tools instead of using the wakenator proved to be a great challenge and really time consuming Because wakenator does a lot of things for you that work really well like serializing the data Well unity serialization system doesn't work really well so all these training examples I just couldn't serialize into one file using Jason the Jason built in unity serializer So instead I will show you what I did under the in the product folder inside assets I created the subfolder called rapid deep data and here you can find all the jasons So for the model it was really straightforward no word no problem with that because the rapidly API was already taking care of that But for the training dataset for both hands and to create a subfolder and serialize each of the feature vectors independently And this was a real pain to work with and actually really slowed me down while developing the game And not only that is really prone to errors probably because the way I am I am handling all the data So unfortunately I think this has proven to be a great problem for me But of course there are several solutions right now for this I could instead of serializing using Jason I could use binaritalization or I could try and find a more standard C sharp Jason serializer to serialize everything into one big file that I believe should be doable but not in the time I had to develop this Also there are more things that you could do with the hands probably could classify more kinds of punches punch up punch down punch left But as I told you and you can see computing all these data right now it's really a really slow process So I didn't have time for basically doing more But well this is all I hope you enjoyed and please any questions let me know Thank you