and by using this alignment matrix that you computed by predicting the lengths and therefore


End-to-End Adversarial Text-to-Speech (Paper Explained)

predicting the centers of the tokens you can then sort of shift so first you assume that


h1 h2 h3 if you were to do nothing these would just all take up like a third of the time


and now by multiplying with this matrix you have the opportunity because you predicted


a longer length for the first token you have the opportunity to shift that a bit to the


right and maybe shorten the second token a bit and then the third token goes until the


end okay that's what this aligner thing is this is not a model by itself all that this


takes in is the computation right here of the token lengths this estimates these token


lengths for each of the tokens and the rest is deterministic it's simply saying okay how


much is the offset cool that's how we know where in the sound wave we are and then where


is each of the centers and we simply do that by summing up the predicted token lengths


and then we use a Gaussian kernel with like a set hyper parameter to be a little bit fuzzy


with respect to these lengths right here so to be differentiable basically and that will


that will ultimately train this loss this model right here that computes the token lengths


right so we sum up in a weighted fashion these embeddings right here and that's what goes


into the generator so now we have embeddings and we have the alignments for the embeddings


which are these pieces of where in the sound wave these are and from that the generator


can now produce the sound wave itself okay and that's basically that's just an up sampling


here I think that's just an up convolution up sampling from 200 Hertz signal to a 24


kilohertz signal cool so that's that now they discover this doesn't work and why doesn't


it work it's because at the beginning of training these token length predictions here are pretty


crappy and so that means that I guess especially this part even where you say well where where


in the sound wave of my 20 seconds do I even need to cut out to compare with the discriminator


right if if you give if you sample this piece here and that's what you give to the discriminator


but your length predictions are so far off that the generator is trying to produce this


particular piece because it thinks it thinks oh instead of producing this tokens here which


is what the discriminator looks at it produces these tokens here of course you have no no


chance no matter how good your adversarial loss is remember the this is these length


predictions are used to see basically to see which of these tokens the generator needs


to produce the sound for and how they're aligned so they have an additional loss right here


what they do is they produce from the again they go via the spectrograms within this spectrogram


prediction loss so they say we discovered that adversarial feedback is insufficient


to learn alignment at the start of training the aligner does not produce an accurate alignment


so the information in the input tokens is incorrectly temporally distributed this encourages


the decoder to ignore the aligner output the unconditional discriminators provide no useful


signal to correct this oh yeah I should have mentioned this the discriminators here since


you don't know you don't know which tokens you should produce the discriminators are


unconditional they don't know which text is produced you don't give them the tokens you


simply give them the sound waves that's something I find particularly interesting here now you


of course this wouldn't work in a like a traditional GAN because you simply have a data sample


here and a data sample right here but in this case you of course have the corresponding


sound samples but still they are you know they are cut down to a subsequence so you


don't know which text you're producing so you have to make the discriminators unconditional


and therefore they are going to discriminate as we said between potentially between two


completely non-overlapping pieces of the sound wave which of course doesn't help you and


then the aligner can also not learn anything because there's no learning signal because


everything just says this is not the same okay and that's what they say here we face


a different problem we do not have aligned ground truth conditional discriminators which


they don't have need an aligner module which cannot function correctly at the start of


training effectively turning them into unconditional discriminators so even if they were to input


the text it would still be the wrong text because their aligner is wrong at the beginning


although it should be possible in theory to train the discriminators aligner module adversarially


we find that this does not work in practice and training gets stuck so what do they do


they say instead we propose to guide learning by using an explicit prediction loss in the


spectrogram domain we minimize the l1 loss between the log-scaled male spectrograms of


the generator output and the corresponding ground truth training window this helps learning


to take off and renders conditional discriminators unnecessary simplifying the model so they


take the they take the spectrogram of the generator output and the corresponding ground


truth training window and they simply calculate the l1 difference of these spectrograms now


this as I understand it this is different from this is different from because we said


they also have a discriminator on the spectrograms now this is different from that this is even


in addition to that so here somewhere we had yeah this this was the discriminator on the


spectrograms and I think this is even different so what they're doing is they also the discriminator


simply decides do the spectrograms look real or fake does the spectrogram look real or


fake now they also take these spectrograms and compare them with the l1 loss so this


is exactly what they said they wouldn't do right here now it's still the case right it's


still the case that they don't use spectrograms as intermediate representations but they now


do have a supervised loss on the spectrograms and one of the motivations to do this end


to end is saying you know maybe these auxiliary losses and supervised losses they they sort


of distract they're good to guide the training but they sort of distract and now they see


okay maybe we have to introduce this one right here in order to make the training start because


this is a this is a real signal but again you run into a problem namely if you produce


something with the generator and so first of all this is not a discriminator anymore


this is a true l1 loss so we potentially run into this problem right of the of the generator


simply copying the input because you always tell it what the correct input is this is


now a supervised loss that we guide the training with and what was I gonna say yeah so you


take the generator output you transform it into a spectrogram you take the real output


transform it into a spectrum compare the l1 loss now you sort of run into the same problem


in that if these are completely not aligned then this is not going to work but since you


have a supervised loss this it can it gives you a much stronger learning signal of what


the generator should produce so you're kind of counting at the beginning of training you're


counting on sort of a reverse reverse learning process in that the real the real sound will


go into a spectrogram and the generator will go here and then that learning signal will


sort of travel to make the generator produce more of whatever the real sound is and that


almost like if you think that the aligner is so bad that we have even non-overlapping


fragments basically you teach the generator to ignore the input that it gets from down


here that it gets from its entire backbone you teach it to sort of ignore all of that


if if that makes any sense it simply produces the sound according to this supervised loss


now of course it doesn't ignore it it still takes the features but it ignores the this


whole alignment thing and now once the generator gets a better signal of what it should produce


that signal can travel back to the aligner module to this length estimation module and


guide that one to make better predictions about the lengths okay so that's how you at


the beginning of training you sort of rely on this path of learning to make to initialize


this module of the aligner and then once these length predictors are better then the the


loss can travel in its intended path where you forward produce these aligned sound waves


and then these discriminators take over i don't exactly know if they trade this off


during training or they simply set it to a number such that it helps them at the beginning


but it's a it's a good idea and it's a it's a good trick to introduce here a a supervised


portion to make the beginning easier but of course you'd run into the same problem as


i said and that the fact that if you have two spectrograms they not don't necessarily


align again and here they use this dynamic time warping loss now this looks very very


similar to the aligner but it is something different because now you have to the difference


here is you have two things that you know should match right you have this thing and


you have this thing and they both have the same amount of entries so they both have a


b c d e this has an a a b a c a d and an e slot and this also has an a a b a c a d and


an e slot and you know that um you assume so here is something you assume you assume


that the beginning and the ends match this is not true of course because we could have


completely unaligned but they say in practice this works so you assume that sort of at least


a little bit these are aligned all right so they have by the way there's so much to this


paper by the way they have an auxiliary loss where the produced lengths all the lengths


that the this length prediction module produces they i don't remember where that is but they


have an auxiliary loss where all the lengths must add up right here all the lengths that


these length predictors produce must add up to the total length of the sound which in


our case i guess is the the two seconds okay so that's how they if so really quickly these


length predictions will sort of at least the least thing they can do is they can all predict


like um l over n and that will give you a sort of a rough alignment such that it it


kind of makes sense to to do this um dynamic time warping to assume that the beginnings


and the endings align all right so we have two things we they have the same amount of


of slots we know the beginnings and ends align or we assume that how do we make it how do


we find out which slots align to which and this is a dynamic programming they formulate


this as a dynamic programming problem that you might you know from you might know from


from like these are often taught in algorithms and data structure courses and so on where


you you can figure out which of these align so if you go a step here that means that you


go one step in each in each of the sequences and then if you go a step here that means


only this one advances and this one still corresponds to this one right here and um


okay i formulated it wrong at the beginning you don't have abcde i guess you would actually


have all of these slots and you would figure out which ones correspond to which and yeah


but i hope you recognize these sort of problems where and here you align them again so these


are classic dynamic programming alignment problems and they align it like this and they


simply say the more that this path deviates from the straight path the larger penalty


we give so they give a penalty with respect to how much this path deviates so here you


can see how much the spectrogram of the generated uh the generated sound aligns with the spectrogram


of the ground truth and here is a penalty for each time that the two spectrograms don't


align correctly they do this in a soft way so they do every single possible path right


here and you can again do this using dynamic programming and the entire catch here is that


the alignment must be monotonic because no matter how long you know or short these sequences


are they always follow one after another in both of these spectrograms and both of these


sounds so that's why you can optimize it in a way so over all the possible paths that


you can align them you weigh these paths by their score that you give them here and then


you calculate the loss across all these different paths and that will give you that is sort


of a fuzzy loss so you don't compare the spectrograms directly but you compare them and you sort


of forgive them for not aligning too well but the more they don't align you give a penalty


and that's how you sort of force the generator again you force the generator to produce things


that are aligned you produce produce these length predictions that make these spectrograms


closer to each other so that's how you calculate the spectrogram loss this is entirely deterministic


there's no learned weights right here okay cool last thing they say is that they use


these phony miser that's the very beginning but they also ablate that so in the results


they do a lot lot of ablation studies which I don't want to go into right now I've already


shown you some they do a even I think they do a human evaluation do they do a human evaluation


I know this might have been in another paper but as you have heard from the examples this


sounds extremely realistic I'll link the website to these samples in the in in the video description


for sure so I think we've gone over everything the generator starts off with text puts that


into normalized text calculates hidden features right here these hidden features on one hand


are used to predict the lengths of each of the tokens in the sound and are also used


to as an input to the generator here now they can only be used as an input to the generator


if the generator knows how to align them in time and how to align them in time is predicted


from these predicted lengths right here via this aligner algorithm this is an asset the


lengths are the only thing that is predicted everything then is deterministic the aligner


is simply a Gaussian kernel over the predicted locations on the on the time axis it is so


the Gaussian kernel is to make it to make this alignment a bit fuzzy to make this prediction


fuzzy you perform a weighted sum with these features and then the generator knows where


to put the feet where to put the tokens finally the generator can up sample the talk the now


aligned tokens into sound this goes into the discriminator the discriminator is actually


five different discriminators which try each try to discriminate the original from the


real sorry the generated from the real at different time scales in addition to that


you have a discriminator on the spectrograms and you also have an L one loss on the spectrograms


which helps especially at the beginning of training for the L one loss of the spectrograms


you have to again compute an alignment but you do this in a deterministic way by this


thing down here this dynamic time warping where you simply assume that they are aligned


and forgive them for not being aligned with a with a a soft penalty and not a hard hard


zero score all right this was the paper again if you like this leave a like a comment share


AI made this music video | What happens when OpenAI's CLIP meets BigGAN?

I wrote a song with lyrics made from ImageNet class labels and then I used OpenAI's clip


model together with a big GAN and a backpropagation procedure to generate a music video that fits


the lyrics of the song. The song is performed on a live looper and the lyrics mean absolutely


nothing. I hope you think this is as cool as I do. Enjoy!


Soon I'll be on a larger screen with my head in a guillotine. My hair smells like an old dish


rack, my face looks like a used door mat, my spine is like a horizontal bar. These are just


some things you'll find on ImageNet. A thousand cups of joy, but mostly things to pet. Be my weasel, be my pig, be my badger, on an offshore rig. Find a beaver, catch a slug, bring them all to my whiskey jug. Watch out for the king snake, the blind snake, the green snake, and don't forget


the night snake, the sea snake, and the pug. Find a beagle, catch a slug, bring them all to my whiskey jug. And here I sit in my rocking chair, looking for my purple hair. What's inside that wooden chest?


Maybe it is my bulletproof vest. I hear a porticolly cry a bird, he's non-dog, goes flying all the while, two hummingbirds stay near. Those are just some things you'll find on ImageNet. A


A thousand cuts of joy, but mostly things to pet


Be my weasel, be my pig, be my badger


On an offshore rig, find a beagle, catch a slug


Bring them whole to my whiskey jug


Watch out for the king snake, the vine snake, the green snake


And don't forget the night snake, the sea snake and the pug


Find a beagle, catch a slug


GANs are amazingly good at producing high quality images.


The cool thing about a GAN is that what you need to do is you need to sample a point in what's called the latent space


and then you'll get out a picture in picture space.


Now, if you have two points in latent space, you can also go from one to the other in a stepwise fashion.


We call that interpolation or traversal.


If you sequence those pictures one after another, it gives you a video of morphing one picture into the other.


We came up with a picture for each line of lyric and then we simply traverse the latent space in sync with the music in order to produce this video.


But how did we even get the initial pictures and how did we make them fit the text?


That's where OpenAI's CLIP model comes in.


So CLIP is a model that takes a piece of text and a picture and it will give you a number telling you how well the two fit together or not.


Now that in itself will not be useful, but the useful part comes when you realize that the picture part of the pipeline is fully differentiable.


That means we can back propagate the error signal all the way to the image space.


So what we do in practice is we take CLIP and we put a piece of text in our case, one line of lyrics for the picture, we don't just put a picture, we actually put the output of a GAN.


In our case, we use BigGAN that has been trained on a variety of images and can produce amazing images by itself.


We take the output of BigGAN and feed it into the input of CLIP.


And now that we have all of this, we back propagate the error that CLIP tells us through the image part of CLIP, through the GAN into the latent space of the GAN.


So in essence, we start off with a random picture that might not fit the text at all.


But then through back propagation over many hundreds of steps, we find a point in the input space of the GAN that more and more and more makes the CLIP model happy.


Now this doesn't always give you very realistic images.


However, it usually gives you pretty cool images like this one is the spine being a horizontal bar.


It's not exactly horizontal, but still very, very cool.


And this here is the face being a used doormat.


I think this is amazing.


So we feed each line of lyrics through the system, get out a point in the latent space that gives us a picture that is fitting to that line of lyrics.


And then with all these points in the latent space, all we need to do is traverse them in order synchronized up with the music, and we have ourselves a music video.


For the song itself, I took ImageNet lyrics and made them into a song text.


This isn't because I'm superbly musically talented or anything, but usually YouTube and music copyright aren't best friend.


I just wanted to avoid all of that stuff.


And so I came up with my own song.


So the lyrics mean absolutely nothing.


There's no hidden meaning.


I struggled already enough to actually find some rhymes and yeah, that's what came out.


The song is played in a loop fashion.


So all the songs are produced by me in some form or another.


My gear is I use a Boss VE2 as a voice processor for harmonies.


Though I only use it at the very end in this song.


I use a Boss RC-500 for looping.


It's pretty new to me and I still have my troubles with it.


And the Boss Octave OC5 pedal.


In order to simulate a bass with my guitar.


My guitar is a Little Martin electro acoustic guitar.


It sounds pretty good, honestly.


The flaw in this setup is probably the microphone I used to record this with as it is an iPad


microphone and I didn't have anything else.


I guess I could have used this one.


Yeah it was pretty stupid for not thinking of that.


And yes, I did buy this combo after I saw Ed Sheeran perform live.


Absolutely amazing.


So usually I'm pretty comfortable playing in front of people.


I have terrible stage fright but I do overcome it pretty quickly.


Cameras is a different thing.


As soon as the camera is rolling, like my brain just turns off.


So this was certainly my 20th attempt or so at recording this song and not even now I


have it down.


So forgive a little bit of cracks in voices and my whistling was a bit tired at this point.


I hope you still enjoy it.


I'm going to let the play the song one more time with a different generation of the music


video.


One, two, three, four, five, six, seven, eight, nine, ten.


In my shower cap, soon I'll be on a larger screen.


With my head in a guillotine.


My hair smells like an old dish rack.


My face looks like a used door mat.


My spine is like a horizontal bar.


These are just some things you'll find on an internet.


A thousand cups of joy, but mostly things to pet.


Be my weasel, be my pig, be my badger on an offshore rig.


Find a beagle, catch a slug, bring them all to my whiskey jug.


Watch out for the king snake, the vine snake, the green snake, and don't forget the night


snake, the sea snake and the bird.


And here I sit in my rocking chair, looking for my purple hair.


What's inside that wooden chest?


Maybe it is my bulletproof vest.


Here a porticolly cry birdies.


Mountain dog goes by and all the while two hummingbirds stay near.


Those are just some things you'll find on an internet.


Thank you so much for watching.


Of course this is not all my work.


It's built upon the work of many great people and I'll link to as much as I can in the description


of the video.


Please check this out.


A lot of people have worked very hard and I'm simply building on top of them.


And the same people are actually pushing the state of the art of what's possible with the


clip model to an entirely new level that you wouldn't believe how cool this is.


So check it out.


I've also linked my code that I've used to produce the music video.


You can produce your own if you want to, or play around with it.


Special thanks to JR for helping me with the code, to Lance for editing and to you for


watching.


I'll see you next time, at Json windows.


[Classic] Deep Residual Learning for Image Recognition (Paper Explained)

Hi there, today we'll look at deep residual learning for image recognition by Kaiming


He, Xiangyu Cheng, Shaoqing Ren and Jian Sun.


So this, you know it, this is an old paper.


It is from 2015.


But I thought we'd still look at it because this not only is it one of the most influential


papers in modern deep learning, it is also a very well written paper.


And I remember it like it was yesterday when this came out.


This was like a bomb.


So around that time, this meme was going around.


I was winning ImageNet, but then someone made a deeper net.


This was a, this was a time when after AlexNet, people were trying to build bigger and bigger


networks.


And every time someone managed to build a bigger network, the accuracy on ImageNet data


set would increase pretty much in lockstep with how much bigger the network was.


But people got to the limit of building big networks.


And then this paper drops and changed everything.


And now residual connections are everywhere, not only in image recognition, they are in


transformers, they are in whatever, wherever you go, you'll probably find some residual


connections somewhere in there.


So yeah, let's let's look at this paper.


And let's revisit what kind of problems people had then and how they solved it.


So here they go directly into into this problem of deep neural networks.


And the problem that people had was they knew that if you can increase the if you can increase


the depth of a neural network, you can make it perform better, you can make it generalized


better, you can reach lower training loss, but optimizing it was hard.


Specifically, this was a phenomenon that people observed.


So if you have a 20 layer neural network, you could train it and you know, there's this


learning rate drop, people have had already figured out that you need to drop drop the


learning rate, and it would reach a certain level.


And here, this would be the test error over here.


However, if after a certain point, if they increase the depth even more, the training


error would actually go up again.


And so with the test error, and this is not a problem of overfitting, because overfitting


would be when the training error is lower or as low, and then the test error went up.


So this is the first thing, this is not a phenomenon of overfitting of too many parameters.


So why can't we train bigger layers networks until that time have very much followed kind


of the original network design that was envisioned by sort of people like Yann LeCun, and also


Alex net, and the most popular ones were these VGG nets.


And they were very much of the philosophy that you'd have like some, you have the image


here, and you input that into into convolutional layers, which first would kind of keep a big


resolution, but would increase the channel size by you know, some amount, and then you


would sort of downscale the image as you increase the number of filters.


So you would stack more and more filters, when I draw more filters, you would stack


more and more filters while downscaling the resolution of the image.


The reasoning was that if you do image classification, right, then, you know, where on this where


on this, maybe you want to classify this into a Lego tower or whatever that is.


It's not that important where it is.


So on the lower levels, you would want to parse out like very low layer features like


edges and so on.


And these are still important where they are right, the fact that here's an edge, here's


an edge, here's an edge.


But then as you go higher up and go to more and more abstract features, and we already


knew that these neural networks, they tend to learn more and more abstract features as


you go up the layers, the hypothesis was that the exact localization of these abstract features


would be less and less important.


So if there is if you recognize that there is a rectangle, it's not that important where


it is, just that it's somewhere there, and maybe where it is in relation to the other.


So if you have, if you recognize, want to recognize a car, the lower layers would recognize


the fact that there are edges.


And then the intermediate layers would recognize the geometric shapes of maybe here, the wheels


and these bodies.


But it's not that important where exactly they are.


And then the higher layers would learn to combine the individual parts to each other.


And again, it becomes less and less important where these things are.


And more and more important that you build more expressive features.


So people would downscale the resolution upscale the number of filters.


Now that's a good heuristic.


But this is based this was basically the architecture of these networks.


And we would question why would if we increase the number of layers, so if we instead of


one here, we have two of these layers, right, we simply have two of these layers.


And here we have two of these layers.


Why does it get worse?


Especially this paper here makes an interesting observation.


So it is not caused by overfitting.


And adding more layers leads to a higher training error.


The degradation indicates that is not all systems are similarly easy to optimize.


Let us consider a shallower architecture and its deeper counterparts that adds more layers


onto it.


There exists a solution by construction to the deeper model, the added layers are identity


mapping and the other layers are copied from the learned shallower model.


So pretty easy if you have a shallow model, like five layers that learns a particular


function, I can pretty easily prove that there is a deep model that learns the same function


by simply copying over these five layers and having these here learn the identity function.


Okay, so if we're able to learn this, we should be able to train this network to at least


the same accuracy.


That's what this paper argues, because it can, you know, these layers can simply learn


the identity function.


So it must have something to do with the easiness of optimizing these deep architectures, not


with overfitting.


And this is, I think, if you read the entire text here, it's very, very clear.


If you read it, they lead you through this reasoning saying that, look, all these layers


have to do is learn the identity function, and then we could at least get the same accuracy.


So so what?


Why don't they learn the identity function?


Well, because we initialize most weights, you know, towards zero, we initialize them


randomly, but mostly we initialize them around zero, our initialization procedure, usually


sample from some Gaussian with some kind of a standard deviation, but around the mean


of zero.


And also, if we use things like weight decay, l two regularization, all of these things,


they do they bias the weights towards zero.


So if there is any natural thing that these networks are good at is they learn the zero


function really well, learning the identity function is as difficult as learning any other


function, the identity function, the convolutional filter is actually pretty difficult to learn.


Because you know, if I have a my, if I have my three by three filter, where is my no,


no, this is my three by three filter, the identity function is it like a one here and


zeros everywhere else.


That's the that that would be one of the things it's not that easy, you need to learn nine


weights in the correct way.


So this paper says, Can we do something to make the default function of the network not


be the zero function, or whatever the randomly initialized function?


Can we make the default function, the one function, can we make the default function,


the identity function?


And that brings you to residual connection.


So instead of learning to transform x, via a neural network into x, which is the identity


function, why don't we have x, stay x, and then learn whatever we need to change?


Okay, so if let's call that tilde, if the assumption is that it's a good default to


not change much, so this is almost the same as this, we might make this built this directly


into the architecture, the fact that these two are equal, plus plus some deviation that


is learned right here.


And the hypothesis is that, especially the deeper you go, if you go very deep, each function


here will actually learn not that much, it will learn to basically change the signal


a little bit, but mostly, it will learn the identity function if it behaves well.


And therefore, it might be, you know, reasonable to build this into the architecture.


And of course, this has turned out to be very accurate, it has actually been reasonable


to build this into the architecture.


So that's what they propose right here.


So instead of just having weight layers one after another, what they propose is to have


these skip connections in here.


So these skip connections, they will, instead of learning the function, they call this entire


function h of x, which might be very complicated, they learn the function, whatever f and f


is whatever you need to change about x, you see, at the end, you add x to it.


So these weight layers here, they simply learn whatever makes this next, this output different


from this input, and learning differences.


Now you have the desired property, because what do we know about weight layers from before?


Well, they tend towards the zero function, right?


If we use weight decay, or generally how we initialize them, they tend towards the zero


function.


Well, if f tends towards the zero function, then h becomes the identity function.


So the default function of this network is the identity function.


And whenever we learn something, we learn how to deviate from the identity function.


And that is, that is a much better default function.


Now it's not entirely true that the default function is the identity function, you see


that here, for example, there's after the skip connection, there is actually a relu.


So there's still it's still a nonlinear function in total the network in total.


But the default for the individual blocks here is the identity.


Okay, now if you chain these blocks, you get a residual network.


And that's what they propose right here.


So on the left, you see this original VGG architecture, like we described it.


So you can see you have an image which has four channels, and you first up it to 64 channels,


you keep the resolution, and then you max pool, which halves the resolution, but you


go up with the filters to 128, you max pool again, go up with the filters, and so on.


Now this has, even though it doesn't look like it, this has a lot of parameters, and


it needs a lot of computation.


So it has 19.6 billion floating point operation for a forward pass.


In contrast, the networks we're going to build here, the residual networks have 3.6 billion


flops.


So they are much, much less in terms of complexity than the old VGG networks, while still being


much deeper.


Okay, the hypothesis is the deeper, the better.


And as a trade off per layer, you don't actually need to have that many parameters because


you don't learn that much per layer.


But the succession of layers gains you much more than simply having single massive layers.


You can see at the same size of resolution here, you the the resonance can get away with


much less amounts of filters.


And that's why they are less, they are of less size.


So this is the comparison, the VGG 19.


Now they do build this 34 layer network, which they call plane.


And you can see, it is simply a 34 layer network with no pooling right here.


And here instead of pooling, they do a stride to convolution, which has also become, this


has become kind of more standard than doing max or average pooling to downscale to do


simply stride to convolution.


So this paper has actually set the standards for a lot of things in modern deep learning.


So our goal is to going to be to compare, first of all, the VGG 19 to the 34 layer plane


to show that you lose performance when you simply up the number of layers.


But then when you introduce the residual connections, as you can see right here, so there is always


this jumping connection right here.


So along these jumping connections, the signal can travel as the identity function.


What we're going to see is that if we go from plane to residual, introducing no extra parameters,


just the skip connections will change everything.


We'll make this network all of a sudden trainable and make the deeper networks the better networks.


Okay, the only little caveat here is, of course, in order to build a residual connection, the


output has to be of the same size as the input because you need to add the input to the output.


And this here, for example, is not given.


So here, you can see this signal after this layer is going to be half as big because it's


a stride to convolution.


So the output right here is only half the size, but it is twice the number of filters.


You can see right here, this has 64 filters.


And here we go to 128 filters.


That's why this connection right here has parameters in order to simply expand the number


of filters.


These are these one by one convolutions that simply project the 64 filters to 128 filters.


However, this doesn't introduce too many parameters because it's only one by one.


In fact, here, the 34 parameters residual network.


No, I'm wrong.


You have different options.


So the world has ended up at the option of doing one by one convolutions.


But in this paper, they still explore three different options.


And I guess here in this particular experiment, option A is simply to zero pad.


So to leave the first 64 channels, but to simply append 128 zero padded filters there


or channels, option B is the one by one convolution.


And option C is actually that all of these connections right here also have the one by


one convolutions, which introduces extra parameters.


And they realize that option C isn't improving over option B substantially, and in fact is


only improving marginally.


And they say, okay, that's probably just because we have more parameters.


So ultimately, they went with option B. And I think that's what the world does right now.


I also, when I read this first, I particularly enjoyed this paragraph right here.


Let's read it together.


Our implementation for ImageNet follows the practice in da da da da.


The image is resized with the shorter randomly sampled in this.


For scale augmentation, this crop is randomly sampled from the image or its horizontal flip


with the per pixel as being subtracted.


The standard color augmentation is used, we adopt the batch normalization right after


each convolution before activation.


This an age old discussion was born when to use batch normalization before, before the


activation or after the activation.


I still I think people are still fighting over this today.


We initialize the weights as in 13 and train all plain residual nets from scratch use SGD


data data data, the learning rate starts from this is divided by then.


So here in this paragraph, in this paragraph, they detail basically all the training procedure


and all the tricks that they use that I remember specifically that, you know, I've read all


of this, which was the idea and I could follow like, oh, this is super well explained, this


is so cool, and so on.


And then I expect basically an implementation of that.


And then there's one single paragraph with like 20 lines saying, Oh, and by the way,


we use these 50 tricks from these other papers.


And yeah, that's when it I guess, it was already happening, you needed to do all the modern


tricks in order to really reach the top accuracies.


But you know, in hindsight, we know it wasn't the tricks that helped them, it was actually


their idea.


I just I just thought it was rather funny.


So you can see right here the results of this.


If you look at the left, these are the plane networks.


And we've already sort of seen this.


Now this is on ImageNet right here, you can see the 18 layer network simply has lower


train and validation accuracy.


So the solid line here is the validation.


On ImageNet, bold curves denote validation error of the center crops.


So I guess they do, yeah, they do center crops.


So the training error is going to be higher because they do these different augmentations.


But you can see the training and the validation error are higher in the deeper network if


you don't reuse residual connections.


Again, this is not due to overfitting.


And this is because we can't train these deep networks, because we should be able to the


solution space of the 18 layer network is a subspace of the solution space of the 34


layer network.


Everything tells us we should be able to learn the 34 layers to at least the accuracy of


the 18 layers, but we can't.


However, introduce residual connections, bum, bum, bum, bum.


And you can see that the trend is exactly reversed.


Now the 34 layer with residual connections has a much, much lower training and validation


error than the 18 layer.


In fact, look at this table right here.


If you introduce the residual connections to the 18 layers, it's marginally better.


However, if you introduce the residual connections to the 34 layers, it is a lot better.


And this is another testament to the fact that these residual connections, they really


help more and more the deeper you go, you can see the effect in.


So this 18 layers, this is sort of a VGG 19 depth network.


While it and there, we already know we can train these without residual connections,


right, because we were able to train VGG 19.


However, if we go higher to more layers, we can these residual connections all of a sudden,


make it a lot a lot better.


You can see that it's not it's not that we can't train the 34 layers, but the residual


connections just help a lot more.


And it most of a sudden, most of most importantly, they don't degrade the performance from the


shallower network.


So they, they explore the different options right here and compare it to others.


Different options, as I said, being a, b, and c, where a is the zero padding for the


projection, b is having projections simply between where the channels don't fit, and


c being having projections in every single residual connection.


And you can see right here that the option b gives you quite a bit of a boost, while


option c doesn't give you that much of a boost introduces many more parameters.


And you know, overall, is, I guess, they decided against it, which, since then the world has


also decided against it.


They also do deeper networks.


So they built deeper networks, like 50 layer ResNet, 101 layer ResNet and 152 layer ResNet


and the 152 layer ResNet ended up being the best one, as you can see here.


And you can see a pretty gain like it almost almost lockstep gain depth.


More depth means better network and this at the time, this these numbers, they were unheard


of, like even 50 layer deep neural network was bombastic, but 152 layers, it was, it


was crazy.


And the fact that still it has less parameters than the VGG 19 and performs better.


That was mind mind blowing, absolutely mind blowing.


And then at the end, they built an ensemble of these models and ended up taking the 2015


ImageNet competition winner.


That was still like very important back then it was still very important who wins who wins


ImageNet that year, where I think I haven't even followed up on the last few years.


It's some kind of wide fix ResNet whatnot with pre trained and 50 billion extra data.


Yeah.


So for the deeper networks, they decide that they are computationally rather become rather


expensive.


So they introduce these bottleneck blocks here on the right, where as you can see, so


here, if you have a 64 dimensional input, you do 64 feature channels in your convolution,


have a 64 dimensional output, you can save computation if you first project the higher.


So here you have a 256 dimensional input, and they say we can save computational power


by pretty much projecting down to 64 first, because then our complexity of this layer,


which is the expensive layer, will be the same as the complexity of one of these layers.


And then we can project up again, the one by one convolution, they are significantly


lower computational intensive than the three by three convolutions, like it's nine times


less operations, if you think about it.


So that's what they use to build the deeper residual networks.


And these residual networks, the ResNet 50, 101, 152, they are still staples today, you


can have these are you can have pre trained versions of those and people still use it


like ResNet 50 is used in every segmentation whatnot application.


So yeah, this has turned out these decisions here have, you know, made it long way.


Here you can see the number of parameters in these residual networks.


And this was the absolute craziest thing right here.


1202 layers, okay.


So you can see still until here, ResNet 110.


Now this is on CIFAR 10, right here, not on ImageNet anymore.


But you can see that even 110 layers still had less parameters or actually the select


the same order of parameters than these previous networks that were only 19 layers deep.


This was unheard of and much more unheard of 1202 layer network to train on CIFAR 10.


It's a bit of an overkill, but they say their goal was explicitly to study depth.


And you can see here that with the deeper and deeper networks, they outperformed all


of the previous networks, so all of the baselines and themselves as they went deeper and deeper


and deeper.


However, once you go to 1002 layers, you go up again.


So here's the question, was this all just kind of a trick, a hack?


And do we run into the same problem again?


And that's the question they asked themselves.


And the answer is no.


So if you look right here, so here you see again, the plane networks.


In the plane networks, you can pretty easily see that the more layers you have, the higher


your error goes, whereas in the residual network, it's exactly the opposite way.


The more layers you have, the lower your error.


And if you compare this 110 layer network with the 1200 layer network, you see your


validation error going up again.


However, your training error and I can't zoom in more, but it's the same.


It's the same and it's at zero.


So here they conclude and the the here they conclude, now we are overfitting.


They don't use like the biggest data augmentation like we use today.


So overfitting was still a thing back then.


So now they conclude, okay, now we have actually built a large enough network that is overfitting


and then and the fact that we go up again in the training error is due to the fact that


we are probably overfitting.


So not only have they enabled us to build deeper networks, they have effectively shown


that this can get you to the to the point where you don't need deeper networks anymore,


at least on C410 because you are overfitting and it can effectively get you there.


This is a lot of evidence for the fact that this biasing the networks towards the identity


function is a very valid thing to do and is the solution to the we can't train deep networks


problems.


Lastly, they investigate the size of the responses.


So their hypothesis is that if if it is really beneficial to bias the network towards the


identity function, and if it is really true that each of these layers only learns a little


bit, right, because the identity function is already very good.


Each of these layers only needs to learn kind of a small function, they look at the responses


of these things.


So the response magnitude of these layers, right here of the signal through the layers,


and they compare those with the response magnitude of the other neural networks where you don't


have the skip connection.


The hypothesis is, if we look at these, then the responses of these layers should be much


larger because they have to learn much more.


And the responses here will be much smaller because the identity function is already doing


most of the work.


And that's exactly what you find.


So here, the layers are ordered by response and you can see the plane networks in the


dashed lines are significantly above the residual network, even and that's not a function of


the depth because if the depth was actually equal here, you would expect that the dashed


lines would would stretch like this, right, they would kind of stretch out.


However, exactly the opposite is happening.


You can see that the residual networks even at the beginning, their responses are very


much smaller.


And this is kind of what I like about this paper.


It's, it's one narrative.


It is a hypothesis.


And then every single, like the hypothesis is taken, and they make predictions from the


hypothesis, they say, Okay, if we are right with our hypothesis, not only should our idea


get us better accuracy, that's what most people most papers do today.


But also, you know, but also, it should be that we can, for example, push our network


to the brink of where we actually are overfitting, like here.


And it should also be that the responses of our signal through our layers is smaller.


And yeah, that's research like this is just pretty, pretty cool.


And it's, I think, a lesson for us that sadly, the world has taken the ResNets, but the world


hasn't all taken the research methodology of this paper.


So yeah, if you again, if you want a good read, it's very well written, you I'm very


sure you can follow it even if you have read very few papers.


And with that, yeah, I hope you enjoyed this.


Please tell me what you think of going through kind of old papers, looking at whether or


not they have stood the test of time.


And yeah, any other comments, leave them in the comments.


I do read them.


And I'll see you next time.


[ML News] Facebook AI adapting robots | Baidu autonomous excavators | Happy Birthday EleutherAI

Facebook AI builds crazy walking robots by do builds automatic excavators and Luther


AI turns one. Welcome to ML news. Hello and welcome to ML news, your moderately regular


update of what's going on in the machine learning world. Let's dive in. Facebook AI blog writes


AI now enables robots to adapt rapidly to changing real world conditions. These are


robots that you might be used to from things like Boston Dynamics. However, Facebook trained


those robots purely in simulation and also end to end. While most people who make robots


like this, they rely on sort of predefined policies and then some controller that classifies


what policy must be active at any given point. These robots are trained end to end, meaning


that the input signal is directly converted into the force values on the actuators that


should be applied. So the cool thing here is that this robot can adapt really rapidly


to changing conditions in its environment, which means that it can handle a number of


different terrains. So here you can see the robot going off path into grass. And here


you can see that it quickly adapts to its leg being blocked by a rock. Now the interesting


thing is that this robot was never trained in the real world. This is a pure simulation


trained robot to achieve this and to quickly adapt to different environments. Facebook


AI trained two policies. One is a reinforcement learned policy, essentially the base layer


of just moving around in different types of worlds with different parameters and so on


in simulation. By now we have a pretty good idea of what it takes of how we need to set


up the simulations such that things work moderately well in the real world. However, to bridge


the gap to actually go into the world and deal with problems, there is a second policy


that sort of adapts to changes in the environments. So the robot constantly predicts from what


it has done so far what it expects the next sensor readings to be. And if those sensor


readings turn out to be different from what it expects, it knows that the environment


has changed or is somehow different than what it's used to. And it can rapidly adapt to


that. And that's how the robot can deal with such different environments. So safe to say


these robots are getting to the sort of level of where they can actually do really good


things and the potential applications of them are nearly endless. There's a paper going


along with this called rapid motor adaptation for a legged robot. robot. That details this


to strategy approach to making the robot really adaptive and it's by researchers of UC Berkeley,


Carnegie Mellon and as I said, Facebook AI research. Check out the paper and the blog


post if you're interested.


Baidu research comes up with an autonomous excavator system for material loading tasks.


So in this article, they detail the development and research on an automatic excavator system.


Now this is a pretty cool thing. Apparently excavator operators are in short supply around


the world and also the job can be dangerous sometimes. Machines give us an advantage here


in that they can operate 24 seven and we can send them into maybe dangerous maybe toxic


environments. So with all of this being pretty cool, there is a video to go along with this


and something's very strange in that video. Listen up.


Baidu research robotics and auto driving lab and UMD have developed an autonomous excavator


system AES. The result was published in science robotics.


This is an AI generated voice. No, like, how meta is this that the video on the fully autonomous


excavator system is AI generated? Like listen up like I might be super but this is AI generated.


system AES. The result was published in science robotics. The construction industry has been


booming, fueled by demand for new infrastructure and digital transformation.


This is a robot voice. Nice, nice. If this is supposed to be like an Easter egg by Baidu


researchers, well done.


Alright, next news in Luther AI turns one year old. In this blog post written by Connor


Lee, one of the co founders of Luther AI, he details sort of the coming about of the


whole organization. Of course, starting with the effort to replicate GPT three in the open,


the blog post details how they went about it, how they organized when the various members


joined, how the initial successes looked like. It's a pretty funny article. And it details


more than just GPT three replication, such things as the pile data set, which is now


publicly available, and the various successors to GPT Neo be that GPT Neo x or GPT J. And


also the recent pushes into biology research and ML art, mostly using models such as clip.


Apparently this is also the origin of the Unreal Engine trick by J Buster, which I reported


on previously, but good to see where it actually came from. The article finishes with a bunch


of reflections by the individual members, and also an outlook on the near and maybe


far future. And of course, a bunch of memes, I totally encourage you to check out the article.


It's a pretty fun and entertaining read.


Okay, next news, the verge writes Elon Musk just now realizing that self driving cars


are a hard problem. This after Elon Musk tweeted out that the full self driving beta is shipping


soon and that generalized self driving is a hard problem as it requires solving a large


part of real world AI didn't expect it to be so hard but the difficulty is obvious in


retrospect. Nothing has more degrees of freedom than reality. Of course, Elon Musk is known


to sort of over promise things and then under deliver or deliver too late, but he's also


known to actually deliver on stuff and I've done an analysis on Andre Karpati's talk on


the fully self driving system that Tesla is building up. And honestly, it looks pretty


cool. So for some reason, right now it's fashionable to dunk on Elon Musk, which is exactly what


this article does and what the whole article is about. And of course, there's all kinds


of reasons to dunk on Elon Musk. But for some reason, it seems to be the hip thing to do


much more than to dunk on various other personalities. And this is not lost in the comments. People


notice that the coverage here is a bit less favorable than coverages of similar things,


for example, by Uber. But beside all of this, I've noticed something interesting in that


the slug the URL of the article, which you do usually for search engine optimization,


you kind of want to condense the title of the article into the URL such that the search


engines pick up on it. It is Tesla Elon Musk, full self driving admission autopilot crash.


There's no crash in the title. There's no crash in the subtitle. In fact, the word crash


appears only about after half of the article talking about various crashes Tesla had. But


you know, I just found this to be funny that it was in the URL make of that whatever you


want. Next news MIT technology review writes we tested AI interview tools. Here's what


we found. And the subtitle is one gave our candidate a high score for English proficiency


when she spoke only in German. So the experiment is pretty funny in that the candidate is supposed


to undergo some sort of an English competency test. And when she did it regularly, she received


an 8.5 out of nine. And then she did it a second time and just read the German Wikipedia


entry for psychometrics and the system awarded her a six out of nine for English competency.


Now of course, the funny thing is that the machine gives a relatively high score for


not even speaking the correct language. Safe to say the message one should get from this


experiment is we have a long way to go when it comes to deploying these systems. Really,


there should be checks to see whether the candidate actually speaks English about the


topic they're asked to and so on and so on. What this is not really is an effective criticism


of the model itself. The article even says she completed the interview again and received


the same score. So at least the system is moderately reliable, giving the same output


when you give the same input, we all can see that these systems aren't perfect yet. And


there are other studies that show that the background you have during an interview, whether


you wear glasses or not, and so on, can all skew these automatic systems to one direction


or another. And there are also big questions with respect to where the data is sampled


from that goes into the systems. And of course, you wouldn't dare to use the horrible, horrible,


horrible biased L2 L1 whatever loss, all the losses are problematic, apparently. So the


article tested multiple systems and all the systems gave essentially a response whenever


the interviewee was doing German instead of English trick. Now again, is this a problem


with the model itself? Probably not because the model was mostly trained to distinguish


better English or more standard English, whatever you want to do out of that from less standard


or less desired English, whatever that means model was not designed to distinguish not


English at all. And I think the thing to take away from this is that if you deploy these


systems in the real world, especially if they work on human inputs, if they deal with humans,


if they have some decision power over some input into decision power, it is important


to think of the outliers, the edge cases, the out of distributions, things that could


come into the model that you didn't necessarily intended, and to build in some safety measures


to have some sanity checks here and there. And in the future, I hope we're able to find


a way to take the best of what these AI systems have to offer and infuse just a little bit


of the human process back into them. Alright, next news, RunwayML releases SQL, which is


a video editor, which is one in the browser, which is already pretty cool, but two has


a lot of built in AI tools. So right now, the main feature is the automated green screen,


but they also advertise automatic depth maps, automatic optical flow and other things. So


it's not entirely there yet on the level of a sophisticated video editing software, but


do give it a try. If you're interested, you can try it out for free and get an impression


of what's possible right now. I did it and the auto green screening is pretty nice. Next


news, the mine RL basalt challenge is now a official NeurIPS 2021 competition. The interesting


thing in this challenge is there is no reward function, but your system is judged by humans.


So the way it works is that you get a textual description of what you need to do, for example,


make a waterfall or build a village house, and you just let your agent run. And then


at the end, a human gets two runs from two different agents that have tried to perform


this task, the human has to rate which one did it better, there is no other reward function


inherent, you may design one yourself as a developer in training the system. But ultimately,


you're only evaluated on those human judgments. Since human judgments are expensive, there


is sort of a bit of a marketplace system in place with respect to evaluating those things.


So in order for your agent to be evaluated on the platform, you first have to go and


evaluate a bunch of other agents. How exactly this is going to turn out is not clear yet.


I can't imagine the research community being good spirits and actually evaluating the agents


rather than just really fast click on a random scoring. But we'll see I hope the best for


the challenge. And if you're interested, participate. So there's an article by Francesco Orabona,


who recently got tenure and having gotten tenure apparently now feels okay to speak


out about some of the problems that plague the review system. This one is the myth of


the expert reviewer. It is a pretty entertaining article that makes the point that if we go


more and more into the direction of expert evaluation, this is not necessarily a good


thing. His main point is that the more expert you are, the narrower your domain of expertise


and therefore anything falling outside of that domain you either don't care about you


think it's bad because it's not in your domain, you think it's bad because it's not done by


you or you just don't know anything about it because it's outside of your area of expertise.


This delivers a little bit of pushback that expert reviewers are a good way to solve the


reviewing problem in machine learning. The reviewing problem being that because of the


explosion of the field, we have not enough reviewers and therefore more and more non


expert more and more inexperienced at the beginning of their careers, researchers come


and review for the big conferences and generally that signal is very noisy. The author here


identifies that with expert reviewers, you get a whole different set of problems which


aren't necessarily an improvement to the old system. The article outlines one particular


story where the author fought really hard to get a paper past other reviewers simply


because the other reviewers dismissed it and that was in a system featuring expert reviewers.


He says in reality in my 15 years of experience, I rarely saw the reviewing system working


as it should. Most of the time in order to get a meaningful decision on a paper, you


have to work hard so hard that people might end up deciding that it is not worth it. I


myself have less and less strength and patience to fight many of these battles. I did not


gain anything in any of them, probably only more enemies. So I fully agree with this article


and with the problems it outlines. So I invite you to read this article if you want a more


in depth and an actual example of how something like this played out. A bit of a silver lining


that I see is that the community seems to be moving away from this system of expert


reviewers. It would be really sad if we decided that in addition to the broken review system,


we would need to introduce some new on top review system featuring expert reviewers from


domains like ethics or something like this. I mean, imagine that. So NVIDIA writes NVIDIA


launches the UK's most powerful supercomputer for research in AI and healthcare. Now the


comma here makes me fairly confident that this is in fact the most powerful supercomputer


in the UK and it's applied to research in AI and healthcare. And it's not just the UK's


most powerful supercomputer for research in AI and healthcare, whichever way you want


to interpret this, this is a big, big machine. So apparently NVIDIA invested about 100 million


US dollars and the computer is for AI research as it seems mainly in industry research such


as medical research and other things. The system is called Cambridge one and features 80 DGX


a 100 systems, each of which contains eight a 100 GPUs. Of course, this is all connected


with super fast, infinity, whatever. And I'm excited to see what people will make of this


beast. It's always cool to see the photo galleries of these things. I have to say it looks pretty


slick but I can't I can't help to notice that there is a little hole in the back there.


So this is where your box would go, I guess. Charlie Snell writes an article called alien


dreams and emerging art scene documenting the rise of artists that make use of open


AI's clip model of which they released at least a small version, I guess into the public.


So of course, clip is one of the parts of Dali. Dali is the system that can take text


and turn it into images. Now open AI has not released Dali, but just a version of clip.


However, people have figured out that while it's not so easy as with Dali, you can in


fact use clip which is just sort of a classifier a judgement a similarity metrics for images


and text, you can use it to generate images. In fact, the images it generates look a lot


more trippy than classic images you get out of Dali. And there is an emerging scene that


this article documents of what people get out of these models. And it also details a


little bit of the history of how this came about first using things like big gun, which


is also something that I used in my music video if you haven't seen that yet. Check


it out. But then going beyond that, and especially the incorporation of things like VQGAN have


made big differences in this model. And lastly, also tricks like the Unreal Engine trick.


So if you look at these things now, they are really stunning pieces of art sometimes. And


they're now not only images, so little videos are made out of them, or they're being combined


with 3d photo in painting such that you get a 3d experience of the world that these models


create. I highly invite you to check out this article and try the link notebooks for yourself.


MIT news writes, infrared cameras and artificial intelligence provide insight into boiling.


So this article is actually about a very serious problem if you want to cool something using


cooling liquid because the cooling liquid needs to touch the surface that it is actually


cooling in order to transport the heat away from it. And there is a thing called a boiling


crisis where the liquid starts to boil in between. And if that happens to a certain


degree, then the liquid is essentially lifted off of the surface, which means that the cooling


effect isn't as strong anymore. So too much heat in these systems can actually lead into


a feedback loop of even more heat. And that's what they refer to as boiling in this case.


However, if you just read this as if it were about like boiling an egg or boiling your


spaghetti, it's a much funnier article. infrared cameras and artificial intelligence provide


insight into boiling. Yes, yeah, I always wondered how boiling works. I always thought


it's just making stuff warm. But we definitely need AI to investigate. It says things like


in previous research, his team spent almost five years developing a technique in which


machine learning could streamline relevant image processing. Good job. And other gems


such as machine learning is not biased by our preconceived hypotheses about boiling.


I'm not so sure about that. Have you ever thought that boiling might be a social construct?


What is the data you use for the boiling? Who made the data? What color were the eggs


that boiled? It also says to collect data, they boiled water to collect data, they boiled


water. That's what I would do too. And also, this is a big deal. I agree. Boiling has such


complicated physics. It's been almost impossible despite at least 50 years of extensive research


on this topic, boiling to develop a predictive model. Yeah, it's not as easy as if you make


stuff warm, it boils. And as an outlook, they say the idea is really to push the button


and come back to the lab once the experiment has finished. Okay, I think I've milked that


joke about as far as it can go. Next news. So Forbes writes language lessons from an


artificial intelligence. So apparently, there are companies now that make use of image generation


in order to assist language learners, which means that instead of just having some voice


talk to you in the language you want to learn, you do get an avatar with it, an AI generated


avatar that can be of any sort that you want to speak any dialect that you want. Look any


way you want, I guess they say rendering text into talk is easy. Our one's trick is to pair


that text reading capability with a friendly human face. Now while I'm totally convinced


that a what feels like a personal interaction might benefit you in learning a language rather


than just some voice processor. Yeah, that's kind of creepy. Well, if you if you like things


like this, if this is for you, you know, good for you, you've just gotten an upgrade to


your language learning skills. But you can definitely see the future where there's still


noticeable artifacts in the generation of these faces are just not enough such that


you notice and where the whole appearance and mannerisms are just a bit more human.


Honestly I think what most of these artificial avatar AI assistant systems get wrong is that


they always try to model sort of a perfect human, a absolutely polite and forever assistive


thing, which we all know doesn't exist. So it might be a bit harder to get the exact


calibration right. But all of this might feel a lot more real if the humans were just kind


of stinky sometimes and have their own opinion and aren't always and 100% friendly and polite,


maybe a startup idea, who knows. And with that, that was it from this week's ML news.


[Code] How to use Facebook's DETR object detection algorithm in Python (Full Tutorial)

Howdy ho, how's it going? So today we are going to try out the DETR, the End-to-End Object Detection


with Transformers from Facebook AI Research. And they have a GitHub repo, and they pretty much give


you everything like the model, the pre trained weights, and so on. So today, we're going to check


out how easy it is to get started with that. So in order to do that, they have like a collab,


but we won't look at it too much. I've glanced at it, and we'll basically see how far can we go


without looking at it too much and how easy is that. So what I've done is I've spun up a collab


that I will share at the end, and I've imported Torch and just loaded the model so you don't have


to wait for that to happen. So I've loaded that up, and now we have it in the cache. So now we can


basically go ahead and load an image into the model and try to detect objects in the image.


So first of all, this is super easy, right? You simply load this from Torch Hub. It's kind of like


the TensorFlow Hub. You simply give the name of the model. You say, I want it pre trained, please.


Shalga boom, you now have a model. So if we look at that model, this is going to be this entire DETR


model right here with all the transformer and ResNet and whatnot. Okay, this is almost a bit


too much right here. So what we want is an image. So let's go find an image. Where better they find


an image than Google. So let's find an image of dogs, because dogs is one of the classes in this


Cocoa data set. This one's nice, right? Okay, so we want the image address, we want to load it in


here somehow. So let the URL is, let's make this into some sort of like an input thing where we


can paste the URL right here. Okay, there we go. So we have this right here. And that's the URL.


All right, no, that's not the URL at all. Is it? But a beam, but a boom, but a bomb. Cool. Better.


Now we need to load this for that we gonna use the requests library. Always a pleasure. Requests,


requests. So the way to load a binary file is you can put the URL here and you can say streamed here.


I glance this from the other thing. And the raw entry will get you the eventual the bytes. No. Oh,


sorry. Get URL streamed stream. Yeah, so this will get you the sort of the bytes of the image and


then use just say image.open. And of course, we need the image from a the pill library,


the Python image library. So import image, we got that. And we can open that image up. And with a


bit of luck. Yeah, yeah. So this model expects I think cocoa data set is 640 by 480 images. But


they if you can see right here, and we're going to take a quick glance at their transforming,


they resize it to 800. So we're gonna we're gonna steal that part right here. People last


were some some found it really funny that I called copy pasting to go Siraj. So we'll from


now on, we'll call it just Sirajing. What we also need are the class labels, because that's in


defined in the cocoa data set, right? So these are the class labels. Let's take those. And okay,


so this T here, these are torch vision transforms, we're going to need that. So from say,


so if you don't know torch vision, it's kind of an addition to PyTorch. That just helps you with


with images and has a lot of data sets. And these transforms, they're really helpful. Because so


let's call this image. Because you can, you know, resize, but they have much more like random


cropping and rotating images, and so on pretty much everything you need for pre training. And


this here is just the standard image net, I believe the image net normalization. So these are


the means and these are the standard deviations from the image net data set. And let's already


resize our image actually to this 800. And I believe I believe if you rescale the 640 to 800,


you get 600 here, right? Fairly sure. Okay, and then let's display it just because we can. Okay,


what it's a bit squished, but we don't care. And let's put that up here. So we only need to execute


it once. Nice. Okay, so from now on, it should be a breeze. So what these transforms do is they


resize the image, okay, we don't need that anymore. They make it into a tensor. And then they normalize


by that. So if we run our image through this, because our image right now is this is pill


image, right? So our image is this pill image. But if we run it through the transforms, then we'll get


a tensor. So that's pretty cool. So the model as it is a deep learning model that expects batches,


so we'll unsqueeze that in the first dimension, and then we get batches. So shape. Let's see,


we don't have unskies. No, of course we don't. So this is a one image of three channels of 600


by 800. So this is the y and x coordinates, I guess are shifted. Yes, in pytorch. Cool. So we'll


call this our image tensor. Now we just need to put it into the model. So model, we put that in


there. And since we don't, let's actually up here, put the model in eval mode. I don't know if that's


already done. But you know, you can never be sure enough that the batch norms aren't. But I think it


probably doesn't have batch norms. Okay, you're not utilizing the GPU will do that will do that.


Thanks. So how do we use the GPU? We put our model on the GPU model equals model dot CUDA. Yes. Yes.


Yes. I think so. This is gonna work. Okay. When I come back to this later. So we forward our image,


of course, we also need that on the GPU. And this worked. Did this work? This worked. Nice. Okay. And


since this is just for evaluation, we should probably go with no grad right here, because we


don't need this whole gradient stuff. If we do that. Okay. I'm dumb. There you go. And nothing


happens, of course, because we need to capture the output somehow. Let's look at that output. Wow,


wow. Just wow. So the output is a dictionary, right? Because we get back class labels and


bounding boxes. So let's look at the pred boxes. Let's look at that tensor. That's a tensor. Very


nice. Let's look at its shape. Let's not print giant tensors anymore. Cool. So since this was


a batch of one, we should probably go with the zero. And you can see right here, there is a


hundred bounding boxes, and each one has four numbers. And if you go with the other thing


that's in there, the logits, then you'll see that there are also should be a hundred logits. And


hello, there should be a hundred logits. And each one is of size 92, because there are 92 different


classes. 92? We'll see about that. Well, one is going to be the nothing class, right? By the way,


how many classes do we have? We have 91 classes. Okay. Cool. We can deal with that. All right. So


what are we going to do next? What we want to do is for each of the logit predictions,


we want to find which class it corresponds to. So what we're going to do is we're going to take


the arg max of the last dimension. So you can see here, almost all of these things correspond to


class 91, and class 91 is not in our classes because our class is only length 91. So that


must be the nothing class. So what we can technically do is for logits and boxes in,


let's just zip them together. And like this. Okay. Class is the logits arg max. If that's 92,


or let's say, if that's larger than the length of our classes, we'll just skip it for now. Okay. So


that should work somehow. And if not, then our label should be the class index right here. So


let's just see what the detector detects right here. It detects nothing. Why does it detect


nothing? That doesn't seem good. What are we doing wrong? We zip together the logits. Oh,


yeah, of course, we still need the zero of entry. We are dumb, cool. So, so, so, so we can delete


this. And now finally, beautiful dogs, two dogs detected. Excellent. So now for each of these


dogs, we want the bounding box. Okay. So now, we somehow need to think of how are we going to draw


this on an image? And well, let's, let's actually make a copy of that image. Because I don't really


trust myself. And then at the end of this, we're just going to display that image, right? Now,


actually, the reason I make a copy is because in these, in this pillow library, you can actually


draw on these images. And we're going to use that to draw these bounding boxes. So for that,


we need an image draw, if I remember correctly. And I think later, we also want some text. So we


need an image font. Yes. All right. So let's draw a bounding box right here, where, so first of all,


let's look at that bounding box. Let's call this box, box, print box dot shape. And break right


here. What's happening? Let's not do this right now. So this is a boxes of size four. Now, this


could be two things, it could be x zero, y zero, x one, y one. So the two corner points are the kind


of the boundaries, or it could be x, y width height. Now from the paper, I know that they


predict the center and the width and the height. So I'm going to go with that. And I'm just going


to guess that it's like x, y, w, h, and not some other way around. If this is a bad guess, then


yeah, we'll see. We can just print out one of these boxes. And I honestly have no clue that


looks reason. Oh, by the way, we should scale that up. Yeah. So these are normalized coordinates


probably between zero and one. So we should scale that up. So we should probably the x


coordinates, which is scaled by 800, and the y by 600. So let's do it. So first of all, we scale our


box by 800 in the x, and here's a y and the width is the x direction. And this is the y direction.


Boom. Okay, we should probably get that on CPU. We'll just hack together a bunch of things right


here. Okay, so now this isn't the correct so we so our x and y and w and h are going to be this box.


So now we need to actually draw on the image. I'm gonna do that. So let's first go x zero,


x one is x minus w half x plus w half y zero y one is the same for y with h plus h half.


Coolio. Now we need an image draw object. So I think draw on this image. So whatever you draw


on the draw object will end up on the image. So we can use that to draw a bounding box. And let's


just quickly look it up. So pill Python, draw rectangle, maybe. There we go. Okay, so there's


this rectangle. Yeah, there's the rectangle function. And you can see you put in a shape x y


x y here, and with height like this. Wait for real, we wouldn't even have to need to transform it.


I'm pretty sure you can go x. Ah, I thought I remember you could do the different thing as well,


but it's called rectangle. Okay, so let's do that. So draw rectangle. And we'll go


we'll go x zero, or we'll go x y with height. Let's display that down here.


Yeah, that looks that looks nothing like we want. But it's, you know, it's a start.


Maybe actually we need the other thing here. We need x zero, y zero, x one, y one.


Yes, yes, doggy. Okay, we still have the break in here. Now we get both dogs.


Nice. Nice. Okay, let's do I think Phil. Yes. Red. And let's go for width five or so. Five seems like


a good width. Oh god, five is a terrible width. Oh, it's not Phil. I think it's it's outline.


Yeah, okay. Okay. Let's go still go with five. Cool. We got our dogs. Now we need to put like


some some snappy text labels. I think there is actually a pill image draw text. I think that


exists because I've this font thing. Yeah, exactly. So I think that's a good thing.


Get a font in there. And then yeah, exactly. You could put a text like this. Okay, so you probably


need the x and y coordinates of the text. So let's do that. W dot text and let's just go with x and


y right here, put it right in the middle. And the text is going to be our label, of course. And we


want the fill that's now going to be the color of the text. Let's go with white and the font.


We're going to load some font right here.


Font dot. How are we doing this? True type, true type. Okay. Ah, no, no cheating. Let's just go


with regular fonts. It won't look as fancy. But we'll be fine. So let's go with white and the


font. It won't look as fancy, but we'll be fine. So where is our text? You see it? I don't see it.


Red. Let's make it red. Yes, there we go. Okay, so it wasn't red enough. This should work on


its own. I just not see it. Cool. So we have two dogs. How easy was that? Actually, we wasted the


most time with like bounding boxes and stuff. Absolutely cool. Right. Okay, so now we can have


some fun with it. I'm going to scale this down for a bit because you don't need to see the actual


code anymore so much so you can see the image more. So we'll go to the images. And the first


thing I want to do is the dress. What does this think of the dress? Okay, so we'll copy that.


And we'll go into our colab and just paste this right here. But a boom, but a beam. Sounds nice.


And what is wrong? The size of a tensor must match the size of a tensor. We do something wrong.


Transform image. Our image is this. Maybe this is like an RGBA image. I think if this is RGBA,


we should just convert it to like an RGB. Pretty sure you can do something like this right here.


This should work. If it has an alpha channel, then that will remove it.


Yes, now it works. Okay, let's see what the model thinks of this.


There. Okay, apparently there's a car. And there's a surfboard. And there's a person. And there's a


person. Nice. See? Well, we didn't figure out whether the dress was blue or white or gold.


It was just a person. Now, you could actually like threshold by how sure you are of a given class.


But where's the fun in that? So, let's go further. Let's do some Rorschach ink blots.


Because those are always lots and lots of fun. So, which one should we go for? This one looks like fun.


Okay. So, we'll put this into here. And it's astonishing, right? This Cocoa data set, it only


has these 90 classes. It doesn't have anything else. So, it's a cake. It's a cake. And this here,


what is it? Okay, we'll have to go maybe to the left. And then we'll go to the right.


And then this here, what is it? Okay, we'll have to go maybe with blue. What is it? Stop sign.


Okay. But so, you might think, what if we want more? Like, what if we want more predictions?


So, there is a hack, right? Right now, the model can always assign math to this not a class thing.


Like right here, this class 91. In order for it to say, I don't think there's anything there.


But generally, we have 100 predictions, right? So, you see where this is going. So, yes.


Let's change it up a bit. And let's go here. Let's first extract these tensors.


And boxes. Okay. So, we have the boxes and this and logits and boxes. Okay. So, we got that. What


we want to do is basically we want to filter the we want to basically just remove the last class


before we do the arg max. And thereby, we want to force the model to make a prediction. It won't be


a very good prediction, because of course, this is only the second highest class. And it's arguable


how much that counts. But still, it'll do something. So, this must be done in the logits,


right? So, we'll look at the logits. And the logits are of shape 100. So, we have 100 predictions,


of 92 classes. Now, the first thing we want to do is just remove the last class. So,


let's go everything here until the last class. All right, so now we have 91. Actually,


let's make it more generic. Whatever is in however many classes are okay. So, we don't have this


class anymore. So, now if we do the softmax over the last thing, we can technically we get 91. But


now they're normalized. So, they add up to one. So, it's kind of a probability distribution.


Next, we want to find the max over this. And that will give us a max output.


So, we don't want to plot all the 100 predictions, because that would just be like


squares all over the place. And we'd have no clue what's happening. So, this max output right here,


what we're trying to find is we're trying to find a, let's say the five best predictions or so. The


five ones where the model is most confident. It's not really good metric, but you know.


So, these are the probability values of all of the 100 predictions. So, what we want is like the top


K. Okay, so let's go with five. And again, we'll get like a top K output. Let's call that top K.


And I think it also has like values and indices. Yes. So, now we simply need to filter from


the logits and the boxes where these these top ones are. So, we'll filter the logits.


We'll filter the logits by that top K indices. And we'll also filter the


I am not very gifted today boxes. By the way, I'm using a colab just because it's nice to kind of


play around with a model because if I were to use a file, I'd have to restart and reload the model


over and over again. So, I'm going to do that. And then I'm going to do the same thing with the


model because if I were to use a file, I'd have to restart and reload the model over and over again.


Just not as nice. So, now we have the logits and the boxes. And if we do that right now,


we get always the top five predictions. How nice is that? And you can see the top five predictions


are probably still KKKKKKK. Just to verify that and we can put its shape.


See, this is what I don't like about this stuff. Yes. Okay. So, we just have five predictions of


92 things. And we don't want the 92 we've already said. So, we just want the 91. Let's actually


could put that here. Okay. So, now we have five by 91. And now it should give us the top five.


Ah, there we go. So, many cakes and many stop signs. That's fine. That's cool. So, the ultimate


test right here is going to be, yes, the human adversarial example. Let's check it out. So,


we'll put in a Jackson Pollock image. And we'll see what the model says. Now we're actually forcing


it to make predictions, right? So, it can't escape. It will need to do something. Okay,


I made another mistake. I would need to copy the image address right here. Like this. That's what


happens when you're not an idiot. You get the actual image. So, what does the model think of


our pretty image? Okay. I can't even read that. So, let's make this into white.


Bird. Bird. Bird. Okay. Lots of birds in this image. Clearly, clearly lots of birds in this image.


Let's try another one. Let's go with this one. Yes, yes. Absolutely. Love it. Love it. Okay.


So, we copy image address. And beam. Boom. More birds. Wow. There's a lot of birds in these


Pollock images. Just so many birds. Okay. Let's try one last. How about this one?


This one is a bit more human friendly, right? Put it in here. Bang.


And, and. Okay. We get some detections. There's a clock right here. There is a, what's that? House?


Horses? Let's print. Let's print the labels. So, just so we know what they are.


Cake, horse, car, horse, and clock. Okay. So, I see the clock. Like this here is clearly a clock.


Then this rectangle on the right side must be something.


Let's put this to red as well.


Now, that's terrible. White. Back to white. How about back to white? Okay. Clock. We got horse


right here and house probably. And the entire image is in the same color.


So, it's a pretty, pretty good system. But of course, it is only these 90 classes.


But it's, for now, it's pretty cool and it works pretty well. And just the easiness with which you


can do this is really cool. And I think it's a really cool system.


It is come out of the app in butterflies per hour. They come to the fightpoint,


and the real중 So, that makes sense. To That makes sense. The movement that they make


when they attack their lab in the most difficult ways possible.


What we see all the things are real and how they bump into


like a very few lines of code you can put something together that detects these


bounding boxes. Lots of elephants and remember we only have the top five


elephants right here so what happens if we go for more? Where is our top K? So


here we can maybe say the top 15 predictions and as always if we want to


make the model to make its own decision we can simply revert back and add back


the no class label. Alright with that I hope you like this video if you did then


maybe tell YouTube that you liked it share it out and I will share this


notebook in the description for you to find and play around with. Alright thanks


RepNet: Counting Out Time - Class Agnostic Video Repetition Counting in the Wild (Paper Explained)

Hi there, check out these videos on the top. Each one kind of contains a repeating action.


So on the left you see someone doing a jumping jacks in a fairly regular pattern. In the


middle it gets a bit more difficult because what you see is a tennis ball bouncing and


it bounces faster and faster and faster as time goes on. On the right you see that there


is a short intro sequence before the repeating action the person shoveling the cement is


displayed. So the goal here is to build an AI that can detect that a repeating action


is happening. And if it detects so that it can count how often this repeating action


is happening. You can already see the difficulties here is not only the recognition itself, but


the fact that the repeating actions can be different and can be of different length and


cannot always look the same and so on. So this paper uses these self these temporal


self similarity matrices that you see at the bottom here to achieve this and we're going


to explore how that's happening. So the paper we'll look at is called counting out time


class agnostic video repetition counting in the wild by Debida Debida Dwebidi, Yusuf Aytar,


Jonathan Thompson, Pierre Sermonet and Andrew Zizerman of Google Research and DeepMind.


So as I already said, this paper detects repeating actions and is able to count them and on a


high level what they do is they encode the video using convolutional networks. Then they


build these temporal self similarity matrices between the frames in order to detect the


repetitions and they decode that into the predictions using another neural network.


This is all trained end to end and they also make a new data set for this task. So that's


the high level if you want to find out how exactly that's done, I invite you to stick


around because we'll go into the paper. If you like content like this, don't forget to


share it out to leave a like and tell me what you think of the paper in the comments. I


do read the comments. So yeah, I'm very happy to read what you all think about it. Okay,


so as we already said, they say we present an approach for estimating the period with


which an action is repeated in a video. And that's actually understating what the problem


is here. The problem is many, many fold. As you can see on the right here, even if you


don't get what this self similarity matrix is yet, the outputs that you want are at the


bottom. So what you want is first of all, a per frame periodicity prediction. So that


means that for each frame, you want to know is there even a repeating action happening


or not in that particular frame, you can see here at the beginning at the end of the video,


there's no repeating action. And then in the middle, there is a repeating action. So that's


the first thing you want to know. The second thing is this per frame period length prediction.


So for each frame, that is part of a repeating action, you want to know what is the period


length of that repeating action. And that can change throughout the video. So you need


it per frame. And once you have it per frame, you can actually count the number of repetitions.


So those are two problems already. The third problem that this paper solves is that there


is no adequate data set to train a model like this, it seems, for example, this model right


here, this q u v a data set, I believe has 100 data points. And these are mostly these


are meant for testing. So you would build your system somewhere, and then you would


test it on those 100 data points. But they claim not even those are no large and diverse


enough for the systems to be evaluated. So they build, we also collect a new challenging


data set called countix, which is 90 times larger than existing data sets, which captures


the challenges of repetition counting in real world videos. And that actually consists of


a train and test split. So you can also train a system using it. Let's dive into the architecture.


I think this paper is a very, very, very cool example of even though we're in this deep


learning paradigm, where you just throw neural networks at a problem. And it's a very cool


example that you can still achieve a lot by smartly constructing this, because we used


to achieve a lot by smartly constructing features and so on. In this case, the the goal is achieved


by smartly constructing the architecture of the neural network itself to give you back


a good performance on the particular task right here. Okay, so if that tablet lets me


actually scroll around, let's go to the architecture. So figure two shows the architecture in more


detail. So we'll go through it from the beginning to the end. Actually, let's go, let's go to


the end. So you know what is supposed to happen. So for each frame in this video, what we'll


need is a period length and a periodicity. These are two. So the bottom is a binary variable.


And the top is a, it's actually a number, but it is predicted in kind of binned as a


classification task doesn't really matter, we need two outputs that we can compare with


the labels, right? In this case, the videos are of length 64. So there's 64 frames per


video. And for each of those frames, we want a period length. And for each of those frames,


we want a periodicity binary prediction. And that, as I said, we compare it with our labels,


and then we can calculate a loss. So this is the loss, the losses at the end, comparing


these labels, and then everything else is trained using back propagation on that loss.


Okay. So now with that in mind, let's go to the beginning. So the video is taken and fed


through an encoder in order to produce these per frame embeddings. So we want an embedding


for each frame, that means for each of the 64 frames, we want to obtain one vector of


length 512. That describes that particular frame in terms that the model can understand.


And we do that using an encoder. Now the encoder, it has a bunch of parts to it, it's not just


a blob, as you see right here. So the encoder consists of three things. First of all, there's


this convolutional feature extractor, which is a resonant 50 architecture is simply a


convolution. And you let the convolutional neural network run on each frame independently.


This is simply a feature extractor from images, right? Like, you know it from any other image


processing task. But of course, here we have a video. So it would be nice if the it would


be nice if the frames knew something about the other frames, right? Especially if you


think of something like a jumping jack. If you are in this position right here, it, it


doesn't tell you everything about that video frame, if you consider the frame before it,


and maybe the frame before it is this, and maybe the frame after it is that you can clearly


see that the hands or the arms are in an upward motion. So the next step of the encoder tries


to integrate that temporal information into the embeddings. And that is achieved via a


3d convolution. So once once we process each frame individually, then we feed it into one


layer through a layer of 3d convolution to add local temporal information to the per


frame features. So if you don't know what a 3d convolution is, oh, this already drew.


So in a 2d convolution, what you want to do is you want to have this filter right here,


which is a 2d filter for each of the channels. And you slide it across the image like this,


okay. And you can have multiple channels of the input image right here. And you can actually


do this multiple times, which corresponds to multiple output channels of the filters.


But the actual convolution is happening in two dimensions. So the sliding is across the


width and the height of the image. In contrast to that, if you have a 3d convolution, and


you have the same input stack of images, and now this, we have to you have to pay attention


here, this right, this stack right here is the individual channels of one image. So of


one video frame, this stack right here that I'm drawing, these are the video frames stacked.


Now each of these stacked video frames can have multiple channels resulting from its


2d convolution, or even just RGB. So I can't really draw in four dimensions. But now we


stack the video frames. And now our kernel, our filter will be also in the direction of


the video frame. So I don't know if this is really recognizable if I draw it like this.