but yeah, it kind of makes sense because it's cleaner and you can essentially scale and distribute.


Transformer Memory as a Differentiable Search Index (Machine Learning Research Paper Explained)

There is no ordering effect.


There's no catastrophic forgetting or anything like this.


And yeah, so that makes sense.


So that's what they do.


All right, we'll get into the experiments now.


The data set is natural questions.


This is a question answering data set and it can be used for retrieval


because the data set essentially always contains a question,


a passage, which is usually called the context, and an answer.


This is one data point.


Now, the idea is that you look at the context and the question and you find the answer inside of it.


However, you can make a retrieval data set out of this by forgetting about the answer


and by severing the connection between the context and the query and considering the entire data set.


And essentially the task is now, if I have a given query, a given question,


which context is the correct one to go with that question?


So you can make a retrieval data set, which is usually quite hard


because the data set is made with the fact in mind that you will get the context, right?


So it is not necessarily the same as a user typing something into Google


where they need to look for a document.


The question is a question about the document if you already have the document.


So it is a little bit, it is an okay data set for retrieval,


but it's just not a direct retrieval data set.


Also, note that it's kind of like 300, there's 300k data points.


They make subset of that.


So they make a 10k, a 100k, 10k data set, a 100k data set, and a 300k data set.


So a small, medium, and large, although even the large one, right, is not very large.


You can, because in a search task, 300,000 documents, it seems a lot,


but if you build search applications, that is not a lot of documents, right?


A lot of document collections have millions of documents and more that you need to retrieve from.


But it is good to observe scaling properties right here.


But just keep in mind that their largest data set is still not super duper large.


The other thing you can see they have train pairs and validation pairs.


And that kind of, yeah, so all of these things, they have a special notion right here,


which I'm not exactly sure I have to be honest how this is exactly done.


So the training pairs, I have the queries and the context both, right?


And for the validation pairs, I also have queries and context.


Now, usually I train a question answering system.


I train on these things, right, with the answers,


and then I input these things over here at inference time.


However, if I train a search index, I certainly need to index at least the contexts of the validation pairs,


and I simply prohibit myself from ever seeing the queries.


So what I think they do, what I think they do is that I think they take these together.


They, these are all the contexts, all the documents, and they take the queries from the training set.


And that makes sort of the quote unquote training set, right?


This, this here would be indexing, and this here would be fine tuning.


And then they evaluate, this here would be eval.


But this is a hypothesis of mine.


I'm not exactly sure that that's what they do, because certainly they can't just not index the data that they're going to retrieve from, right?


But I hope they don't actually fine tune on the queries that are in the validation set.


But again, maybe they also first do this, and then as a last step, they then index the validation set.


I'm not sure, just honestly, and I couldn't read from the paper.


Maybe I've overlooked something, but it would be a good question to the authors how this exactly is done.


Training regimen seems pretty decent.


So this, it's Google research, so they have the big chips.


Yeah, T5 isn't exactly a small model, right?


Especially the larger ones.


So here are the results, and they are all over the place, which makes me a little bit skeptical.


First, you can see in general, the larger models for the differentiable search index generally outperform the smaller models by a lot, right?


You can see here, for example, these are large models.


These are small models on the same task.


These are hits at one and hits at ten, which means if the correct answer is in the top one or the top ten, respectively.


For all of the DSI models, that's the case.


By the way, when it says T5 here, that is a dual encoder baseline, and above here you can see the BM25 baseline.


Now, also, I would like to draw your attention to the fact that BM25 on the small data set, it gets like a performance of 12.4.


On the large data set, it gets like 11.6, which, you know, is reasonable.


It kind of goes down a bit if the data set is larger because it can confuse the documents a bit more.


But in general, it's constant.


But then there's like a big jump in this 100k data set.


Like, what's up? What's up with that?


This seems to be weird.


So you can't really see that in the dual encoder setup.


There is a jump here, but that remains.


Then if you look at the small models here, it goes up and it goes down again.


Yeah, that's the same trend.


But then here, if you can see, it kind of goes down in performance and then it goes up.


No, it goes, it kind of remains down.


All I'm saying is this is not OK.


This might be to be expected.


This might be expected because going down in performance is what I would expect if it goes, if the data set becomes larger.


OK, but there are some inconsistencies among here.


Yeah, all the weirder that here actually goes up.


And as you can see, the highlighted bits right here, for example, this thing, the methods that work, they seem to be all over the place.


Sometimes this naive string doc ID is the best.


Sometimes this semantic string doc ID is the best.


The clear trend is that pretty much everywhere the larger models are better,


which I think is reasonable to say because they're going to have more capacity of adopting the data into their weights.


And in other trends, the larger the data set gets, the worse the models become.


Like, look at this. It goes down to be expected.


It goes up again. What's up?


So this data set is just is cursed, so we won't look at it.


So let's just compare the very left and the very right things.


You can also see that there isn't a big improvement over BM25, which is surprising, right?


That even the dual encoders improve over BM25.


But this differentiable search index, especially if it gets large, improves by quite a bit.


Now, I suspect again that that is kind of the nature of the data set right here.


But it might as well be that the all the embedding techniques are very good.


But yeah. Lastly, what I want to point out.


Oh, yeah, the improvement over the dual encoders of the differentiable search index.


So over this baseline right here, this gets smaller and smaller as the data set grows, right?


Which we discussed at the beginning and which I think is a little bit of a bad sign for these types of techniques in that obviously,


as I have more data, I cannot really save it into my weights as easily.


And the dual encoders, they are not like the embedding space.


High dimensional embedding space is kind of infinite, right?


So I can save a lot of stuff there no matter how much data I have.


It'd be interesting, though, because there are techniques in which you can like if I have a matrix and I want to store stuff in that matrix,


as long as that stuff, as long as I build like low rank matrices that I add to it or in vector terms,


if I build like vectors that are largely orthogonal to one another, I can, you know,


save a lot of stuff in a single matrix by just adding to it or to a vector space or to a set of vectors.


And maybe, maybe, you know, with a bit of trickery in how the weights are updated exactly for the different documents,


one could improve this quite a bit.


This here is zero, the zero shot setting, which means this models, they never see any queries.


They never learn to map queries to document IDs.


They simply learn to map documents to doc IDs, which is an additional difficulty.


Again, you can see that the weirdness of BM25, right, that's exactly the same.


BM25 is going to perform the same because BM25 is always zero shot.


It never sees labeled queries.


You can, you just can, I guess you can, you can also run it through indexing, but yeah.


Interestingly, the dual encoder in a zero shot fashion just sucks, it really sucks.


The sentence T5, which is explicitly made for like sentence similarity, it is apparently OK.


It apparently outperforms BM25.


Also, I have trouble believing that, but you know, if they say so.


But then these DSI, they really shine in this, especially here, this atomic doc ID method.


For some reason, it really is, is really good.


As you can see, it outperforms the semantic string doc ID, which was kind of the best one before or one of the best one.


Also, this naive string doc ID was really good before it outperforms that in a zero shot setting.


So the results are kind of all over the place.


And that is what worries me a little bit in that it seems to be quite noisy.


They themselves admit or report that training with these atomic doc IDs seems to perform well in the zero shot setting, but it's also quite unstable.


So, yeah, it's a it's a cool method, cool paper, and it shows some really interesting results.


But it also seems that there's quite a bit of noise and probably we haven't exactly figured out many of those things yet,


which is a good thing if you're in research. Yeah.


So they find a bunch of things like in general, they say structured semantic identifiers are helpful and improve over unstructured ones.


However, we also note that unstructured atomic identifiers perform the best by a wide margin on the zero shot retrieval setup.


Who knows why? We can I guess we can hypothesize.


The other methods I've already discussed a little bit, especially model size.


It seems to be really important, as you can see for dual encoders, that doesn't pay that much of a that doesn't make super duper difference.


It makes much more difference for the differentiable search index.


Whereas if you talk about data set size, a higher data set size seems to be much more detrimental to the differentiable search index than it is to a dual encoder.


Interestingly, also, the length of the tokens you index per document seems to be better if it's kind of shorter, which is interesting.


So if you index the same documents for longer, for more tokens, that seems to hurt performance and really, if you go much, much longer.


And lastly, here they investigate how much indexing versus retrieval they have to feed in during the multitask training.


If they train index and labeled query pairs at the same time, turns out that's also fairly noisy, but you can't go too high.


One seems to be fine, right? So you can get an improvement if you have more indexing, but one seems to be fine, which is already relieving.


I think you could just mix them together and you'd be fine. Yeah, I wanted to say one one more thing.


Yes, so in their conclusion, they talk about document identifiers and they say it would be interesting to explore alternative strategies for representing documents and doc IDs,


including end-to-end strategies for learning semantic identifiers.


That's what they say because they're kind of unsatisfied with the way they represent the document IDs


because the height of their method is this hierarchical clustering, which is also it uses a separate encoder and so on.


However, I'm thinking myself, if you want this to be learned like end-to-end and so on,


isn't that like isn't that exactly like regressing to cross encoder setup and dense retrieval setup?


Isn't that essentially what you're doing if you're learning these things end-to-end?


I don't know exactly how then that's going to be different in principle.


And this is my a little bit of my worry about this paper as well, that they didn't compare at all to any cross encoder setup,


to any kind of re-ranking setup that are very prevalent in neural search these days, any dense retriever setup.


Maybe dense retriever is by encoder.


I'm not even sure, but I feel these are some some baselines that are missing right here, along with the smaller size of the data set.


But all in all, pretty cool.


Again, I don't think this is necessarily going to be such a use in search in itself, like search through document collections,


but much more could be very useful as a part in, for example, a reinforcement learning agent


who has to store stuff during the episode and then retrieve it later in a very differentiable manner, in an addressable manner.


It would also be interesting to see whether outputting document IDs is better than outputting the information that I want directly.


Because you could also think of that. You could also say, you know, here is a query,


just output the document itself or the part of the document that matches instead of outputting the document ID.


You know, how does that perform? It would be equally interesting to see that.


So lots of things to research.


I really like this paper because it does something different.


It does something weird and it puts in the engineering effort to figure out what makes it work and what doesn't.


And yeah, that's it. Let me know what you think in the comments.


Functions with Parameters with Tkinter GUI buttons - Tkinter tutorial Python 3.4 p. 3

Hey everybody, what is going on?


Welcome to another TKinter with Python tutorial for making windows and GUIs.


In this video what we're going to be talking about is how we can add pages.


So like I was saying before, this whole part, I mean this was a lot of stuff that got thrown


at you, but basically we've got a baseline here of how we're going to display pages and


now we're going to be talking about how we can add a bunch of pages here and make it


really easy.


So as promised, this will be significantly easier than what we did before.


We will have to change a slight bit of code right in here basically to make the backend


work with multiple pages, but you'll see what I mean and then adding new pages is really


really simple.


So let's go ahead and we're going to start here in start page.


So in start page, since we're going to have multiple pages, generally you want to allow


for some sort of navigation right from page to page, that only makes sense.


So we're going to add a button that is going to allow the user to go to another page.


So we're going to say here, button1 and button1 is going to equal tk.button, not three t's,


just two, and don't forget your capital B there.


And so button1 is going to be the object that's returned from this TKinter button here.


And as usual, we pass self through it.


And then we're going to pass text and text is going to be visit page one.


Okay so this is the text that's going to go on the button.


Then what we're going to do is comma and then the other thing that a button has is a command.


So you can pass through a command.


And so for example, we would have command and command, we can say command equals and


then you can equal a function.


So just to give you a really quick example here, I'm going to make a quick function.


Feel free to follow along as well, but we'll say define qf for quick function, empty parms,


print and then you did it.


Okay, so we can use that as an example command here.


So we can say tk.button self text equals visit page one and then command, we can say command


equals qf.


Then as usual, we have to pack this button.


Or you could use grid, but we need to add it.


So we're going to go ahead and use pack because we just have a few really simple stuff here.


So I'm just going to say button one dot pack, easy enough.


So now when we go to run this, you're going to see that we have a slight issue.


Well not this one, but you'll see eventually we'll have a slight issue.


So here's our little window.


It's too little.


Oh goodness, let me bring it back.


Here we go.


I noticed that in my videos, if I enlarge these windows as I'm recording, it makes it


go crazy.


If you're prone to seizures, close your eyes for a minute.


But when I make it big like this, for whatever reason, that makes my video recorder really


mad.


Anyway, I've brought it back down for you seizure folk.


So now we see start page, and when they click visit page one, it prints out you did it.


So obviously we ran that function.


But here's the kicker.


What if, what if, we want to do QF and then we want to say QF and then print, and then


you know string to print is our parameter.


And then we really just want to print out whatever the string to print happens to be.


So you might think to yourself, well here's what I'll do.


I'll do this and then I'll do a string to print as a yo.


Well what happens here is we'll go to run it and you see it just prints out yo yo yo


immediately.


And of course, you know, we can hit visit page one and it does absolutely nothing.


And so that's no bueno.


So what happened here?


The way that this command works is it's going to run this basically because you're calling


the function.


It's going to run this function on load and then it's not going to work after.


So there is a slight way that we can get around that using lambda.


So what lambda is going to allow us to do is kind of like hack our way around this.


And what lambda is going to allow us to do is create like a quick throwaway function,


but it's only here when we call it.


So it's not going to run immediately.


So the only major downfall here is if that function returns something and then saving


that sometimes is problematic, but also just the inherent problem with a throwaway function.


But you'll see that in a minute.


Well not in a minute, quite a few videos probably.


But anyway, you'll see it eventually.


So what we can do instead is we can use lambda though.


And I'm going to pop this down to a new line.


So after this comma here, I'm just going to hit enter.


And then we're going to say command equals and command is going to equal lambda and then


you do colon space.


And then what we could do for example, did I get rid of, I guess I deleted that function.


Let me just show you really quick.


Let's define QF param and then we'll say print param.


So QF print param and then here button tk button lambda and then you can do something


like this.


Like you could go QF and then param.


And that should allow you to get away with that.


The other thing that's going to be a problem is when you go to pass variables through.


So because of what we're doing here, if you wanted to pass a variable through or even


do like some other complex stuff, it's just not going to work.


But you'll see what I mean in a minute.


Oh and we need to change param into, see this worked, like that.


So let's go ahead and save and run this really quick.


We get start page.


It didn't print anything out and we can click this and see this worked.


So then we can build this function here to actually do whatever we need it to do.


And this is just a way that we can go about doing that.


So there's another use for lambda.


So anyway we'll delete that function and I think I'll probably just cut the video off


here and then we'll be talking about adding a new page and all of that.


But obviously this is a really integral thing that you need to know because bringing another


page we have to run things through here and all that.


So that's how we can run things.


So anyways if you guys have any questions or comments feel free to post them below.


Otherwise as always thanks for watching, thanks for all the support and subscriptions and


Convolutional Neural Networks - Deep Learning basics with Python, TensorFlow and Keras p.3

What's going on everybody and welcome to part three of our deep learning with Python, TensorFlow, and Keras tutorial series.


In this video what we're going to be covering is convolutional neural networks and applying the data set that we worked on building in the last video


to a convolutional neural network in hopes that we can get this neural network to classify dogs versus cats.


Now first what we should do is cover quickly how convolutional neural networks work just to get some of an intuition of why we use ConvNets with imagery types of data.


So let me just pull this over from the text based version. I didn't really want to have to redraw everything again.


So in general you're going to have the following steps are going to be taken.


You're going to do some convolution and then pooling and then convolution again and pooling.


So let's talk about that. So first of all the idea is convolution is to find some sort of useful thing and then we pool these useful things together


for the combination of convolution and pooling for your convolutional neural network.


So the idea of this is like let's say you take an image and then we convert that image to pixel data. So this is just my bad representation of pixels.


But anyways you convert to grid or after it's been converted to pixels like this in theory could be a convolution on top of already,


there's already been a few convolution layers. But anyways the idea is then you have a window like a convolutional window.


In this case it's a 3 by 3 window and in that window the convolution is going to attempt to simplify what it finds down to some sort of value.


Now when it does that the window is going to shift over and do the exact same thing just a bunch of times.


So in general you've got the window, you've got how big of a stride over do you want to take and so on.


Now with Keras you can really just specify the window size and that's it.


And like everything else like someone was asking in the previous, well the first video about bias.


If you want to learn more about the deeper details of deep learning go to the practical machine learning tutorial series


where everything is much more deeply explained because it has to be deeply explained because the way we were writing that was with raw TensorFlow code.


So definitely go check that out if you want to see more of the inner workings and how things are done.


Like for example as you keep sliding what happens when you slide over the edge?


You know most cases we're probably going to pad it but then do you want to pad it like or same or all this stuff.


So anyways getting back to what we're doing here.


So it'll produce features so in general and then you'll do it many many times.


So it's a way to kind of slowly extract features from an image.


So convolution, convolution and let's say the convolution produces a 2, a 1, a 4 and a 3.


Now these aren't, it's not going to look just like that but just for simplicity sake let's say it's a 2, 1, 4, 3.


Then let's say you've got a bunch of convolutions right this is just a bunch.


The next thing that's going to likely happen you don't have to but this is pretty common is to do a pooling layer.


So in general the most common form of pooling is max pooling.


So again let's say you do a max pooling of a 3 by 3 window.


Well all max pooling does is find the max value.


So in this window the maximum value it's a 4 then it shifts it over and again and again.


So you can see how it worked basically down here.


So that's convolution and max pooling.


Now the higher level idea of how this actually works is basically it's slowly extracting value from the image.


So lower the more initial layers in a convolutional neural network are going to be finding things like edges.


And lines and then as you go you're going to find like maybe squiggles and then maybe you'll find circles.


And then maybe you'll find squares and then soon you're going to find I don't know like very small like features.


So for example if you're curious I don't know if this is going to play some sound it is.


Let me just mute it. This was a deep dream and what you can see actually going through this is how what a convolutional neural network is kind of basically a deep dream is going to accentuate what it finds.


So these are the initial layers where it just finds a bunch of straight lines and then as you go a little little deeper you start finding slight curvatures and lines.


And then as you continue on you'll find hopefully something different.


So you get kind of more square shapes right and then as you continue along we start getting like faces and eyes stuff like that.


As we continue even further you start getting like you can almost recognize a dog there.


And then as you get further and further in the network you start seeing things like scales and stuff like this was trained on like what was it not pets.


I'm trying to think of the term but this was off of I'm trying to think it was the image I think was the ImageNet data set.


Anyways this was inception off of the ImageNet data set.


But anyway you can check out a deep dream of a neural network if you want.


Also you can check out the that series the unconventional neural network series for more on more interesting projects like that.


So okay so the first thing that we want to do is bring in tensor flow and all the things that we need to make a convolutional neural network.


So we're going to import tensor flow as tf we're going to say from tensor flow dot keras dot data sets.


Actually we don't need to import any data sets just kidding.


What we're going to do first is bring in from tensor flow dot keras dot models.


We're going to import sequential because we're going to use that sequential model then from tensor flow dot keras dot layers.


We're going to import dense so your typical dense layer we're going to import dropout we're going to import activation.


And so first of all we're going to import the dense layer because a lot of times you're going to end either a recurrent net or a connet with a final dense.


Just fully connected layer and then also flatten and we're going to use flatten before we feed it through that final dense layer.


Then we also want to bring in conv 2d capital D and then max pooling 2d.


Okay so then we're going to import pickle and we're going to do that pickle or actually we can just do this we can say x equals pickle dot load.


And we'll load in open x dot pickle pickle rb and then we'll do the same thing for y.


So x and y.


So then the first thing we want to do before we feed data through a neural network is consider normalizing that data.


The easiest way to normalize data is going to be to scale that data.


In our case we're using imagery data so actually the easiest way to scale the imagery data is we know what the maximum value and the min value is.


The min is zero the max is 255 for pixel data so we can actually just do that divided by 255.


I need more tea. So but there is I think it's keras dot utils dot normalize I think it is.


Anyways in many cases you're going to need something more like that but for now we can actually just do x equals x divided by 255.


Then once we've done that we can come down and begin building our model so we'll start with not a capital M model equals sequential.


Because it's just going to be a simple sequential model and then we're going to start using that model dot add syntax that we've been using.


So we're going to say conv 2d so we're going to start with that convolutional layer and this can be we could probably we're probably get away with almost anything we pass here.


I'm going to pass 64 though and then we're going to pass the window that we want to use three by three.


And then finally the input underscore shape is going to be equal to the x dot shape and then here you can just say one colon.


So we're going to skip that first like negative one so each because negative one again is was how many of those feature sets did we have right.


So we don't need that that's not actually input shape it's going to be the shape of the data we're actually throwing in so it's not negative one but it is you know 50 by 50 by one.


That's true so that is how we can get that input shape really dynamically.


So once we've done that the next thing that we're going to do is I feel like we shouldn't be.


OK whatever the next thing we're going to do is model dot add.


And we're going to add so in this case you.


So the the the max pooling again to have so in general convolutional and max pooling are going to go together.


So in this case what we could say is after the convolutional layer we could pass either an activation or a pooling.


I'm going to go with we're going to just do an activation layer so activation and the activation will go with here is just the same one that we've been using which is rectified linear.


Then we're going to do the pooling so model dot add.


Well I'm just going to copy this actually copy paste max pooling 2D and then again here you're going to specify the pool size and let's just do a two by two.


OK so that's our first sort of layer now we can do this again.


And this time we don't need an input shape.


So now we've got a two by sixty four layer convolutional neural network and then like I said before we're going to pass a final.


You don't have to do this again we probably don't even need it for this data set but just for good measure model dot add.


We'll add a final dense layer and that's going to be a 64 node.


Now before you do that you have to flatten the data because convolutional is 2D whereas that dense layer again wants a 1D data set.


So we're going to say model dot add and then we just throw in a flatten.


Don't forget whenever you want to do stuff like this you do have to import that layer.


But flattened 64 cool.


So then now all we need is that output layer so model dot add and we'll add a dense and we'll go with one and then we need that activation layer or the activation for it.


Add I feel so silly calling that a layer but it is coming from the layers.


I'm kind of surprised that we would call activation for example a layer but OK that's what we're doing.


So anyways model dot add and then activation I mean I guess it's one more thing but in the old ways of doing things like you would because you could also say dense.


So like here you could say dense one and then you can actually pass activation equals something.


So there's a kind of two ways to do it.


So I'm not really sure why they have that in layers but anyways activation.


Someone could correct me if I'm wrong but I don't really think like I think if you if you walked around calling the activation function a layer in a neural network like if you consider this neural network to not know when I look at this I see one layer two layer three layer and then like the output layer.


So really I would say this is three layers but if you started calling this a one two three four five six seven eight nine ten layer content I think most people would disagree but I don't know they're throwing that in layers.


So I don't know if that kind of opinion has changed in the ML community or what.


Anyways model dot compile and then we'll do loss loss equals and we're going to use category cool actually will use so you could use cat is dogs and cats so in this case we can use categorical or I'm sorry binary cross entropy.


Wow.


How many times my trip over that.


But you could use categorical and then optimizer in this case we're going to go with Adam because that's what we always go with and then metrics equals accuracy.


Now what we want to do is run the fitment so we're just going to say model not fit x y.


We can specify the batch size this is how many at a time do we want to pass again we got 25,000 samples, you probably don't want to pass one at a time, but you also don't really want to pass all at the same time it's generally a good idea to do something between like.


Somewhere in the 20s to maybe 200 usually I find negative results start coming in if I start trying to do like a batch size of like thousands, but it depends on the size of your data, so it's really going to kind of scale with how big your data is if you got like a data set of a billion samples, you could probably get away with a batch size of a few thousand.


But anyways yeah so model.fit batch size we'll talk more on that in a little in the next tutorial so anyways validation split this is the kind of out of sample data.


We're going to say we're going to save 30% and in fact that's pretty high let's not do 30% let's just do 10% so model.fit 10% and great so let's let's train the model make sure we don't hit an error.


We did X looks like a lowercase X we need a uppercase the X.


I knew I felt weird there let's see one requirement colonel size do.


Okay now I know what I've done I knew I felt like I was like wait something's wrong here so I closed off with this 64 and we need to put it here so so I always get confused with why are you doing it I always get confused when it comes time to do parentheses while I'm doing tutorials so let me just space everything out real nicely.


So this is the conv layer right it's 64 units followed by the next parameter which is the the window size followed by the next parameter which is the input shape.


Okay.


Oh my gosh how many.


Oh we did it because we copy and pasted got it chickity check.


Please no more errors to be really nice.


All right, so I think we're training finally yes all right so we're already at 6465 we're doing pretty good.


I don't know if we said, oh we didn't specify epochs so let's go ahead and specify so even after one.


We got a 73% validation accuracy which is pretty good so let's go ahead and pass in one more epochs equals three let's do it a few more epochs of this data.


Even better than the last one.


Okay, so that's looking pretty good we were up to 74 78 looks like we're gonna maybe breach 80 we'll see not yet anyway, but if we kept training this probably for like 10 epochs or something like that.


In fact I'll plug in 10 for the rest of my my talking here.


Chances are will probably breach 80 which is pretty good because that means we're classifying cats and dogs pretty darn accurately.


So in the next tutorial what we're going to talk about is okay how do we read these numbers like what is you know how do we actually read loss how do we measure you know how we like read the accuracy and especially like what about accuracy versus validation accuracy like in this case is kind of wonky.


We got a better validation accuracy over accuracy it's kind of weird and we did it again.


This is the first time we didn't so I'll talk about like little stuff like that that should be kind of red flags at least initially it's not that big of a deal but if your model consistently does that that's a problem.


So we'll talk about some of the patterns that you should start seeing and that you should watch out for as you begin to train models so anyways that's all for now.


If you've got questions comments concerns whatever you can feel free to leave them below if you like this content you can support the content by clicking on that beautiful join button right down there.


If you join it helps support me in the channel also you get various perks like a little badge next to your name you also get early access to videos just like this one and also shout out from time to time when people join.


So my most recent member is Tejas thank you very much for your support good sir you help support my mug addiction.


Okay so that's it for now in the next tutorial we're going to be talking all about TensorBoard which is absolutely essential for training models and just kind of analyzing them as they go it's super super useful.


DETR: End-to-End Object Detection with Transformers (Paper Explained)

Hi there, today we're going to look at end-to-end object detection with transformers by Nicolas


Carion, Francisco Massa and others at Facebook AI Research.


So on a high level, this paper does object detection in images using first a CNN and


then a transformer to detect objects and it does so via a bipartite matching training


objective and this leaves you basically with an architecture that is super super simple


compared to the previous architectures that had all kinds of engineering hurdles and thresholds


and hyper parameters.


So really excited for this.


As always if you like content like this consider leaving a like, a comment or subscribing.


Let's get into it.


So let's say you have a picture like this here and you're supposed to detect all the


objects in it and also where they are and what they are.


This task is called object detection.


So a good classifier here would say there's a bird right here and so this is a bird and


then this here is also a bird.


Right, they can be overlapping these bounding boxes.


So this is you see the first problem that bird, why is that green?


Never mind.


Okay and those are the only two objects.


So there's a number of very difficult things here.


First of all, you need to sort of detect the objects.


You need to know how many there are.


It's not always the same in each image.


There can be multiple objects of the same class.


There can be multiple objects of different classes.


They can be anywhere of any size that can be overlapping in the background small or


across the entire image.


They can occlude each other partially.


So the problem is a very, very difficult problem.


And previous work has done a lot of engineering on this like building detectors and then kind


of you want to classify every single pixel here and then you get like two detections


right here that are very close for the same class.


Ah, that must maybe be the same instance, right?


So there's only one thing here and not two things, and so on.


So there, there used to be very complicated architectures that solve these problems.


And this paper here comes up with a super simple architecture, and will kind of go from


the high level to the low to the implementation of each of the parts.


So what does this paper propose?


How do we solve a task like this?


First of all, we put the image and the image here without the labels, of course, we put


it through a convolutional neural network encoder.


Since this is an image task, it's, you know, kind of understandable that we do this.


Mostly because CNN is just work so well for images.


So this gives us this set of image features.


And I think this this vector here is not really representative of what's happening.


So let's actually take this picture right here and throw it in kind of an angled way.


And what we'll do with CNN is we'll simply sort of scale it down, but have it multiple.


So here, it's three channels, right?


It's red, green and blue, like this three channels, but we'll scale it down, but we


make it more channels.


So


yeah, so more channels.


Okay, but it's still sort of an image right here, it still has the image form.


Okay, so the CNN basically gives us this thing, which is sort of a higher level representation


of the image with many more feature channels, but still kind of information where in the


image those features are.


And this is going to be important in a second.


Because now this thing, which is this set of image features goes into a transformer


encoder decoder.


And this is sort of the magic thing here as as a component, we'll look into that in this


in a second.


But what they get out right here is this set of box predictions.


So outcomes, each of these boxes here is going to be consisting of a tuple and the tuple


is going to be the class and the bounding box.


Okay, so an example for this could be bird, bird at x equals two y equals five.


Okay, that's an example.


Another example of this could also be there is nothing at x equals seven y equals nine.


Okay, so nothing, the nothing class is a valid class right here.


And that's also important.


But safe to say there is this set of box predictions.


And then that is basically your output, right?


These things are your output.


If you have those things, you can draw these bounding boxes, you can assign the labels,


the question is, how do you train it?


Now what you're given is a database of images.


And these images, as you see here on the right, these images already have by human annotators


drawn these bounding boxes in and also labels.


So this here would be annotated with bird.


And this here would be annotated with bird.


But it doesn't have any of these like it doesn't annotate the nothing classes or and so on.


So the question is, how do you compare the two?


Can you simply say, okay, if the first one here is the bird, and then and the second


one is this bird, then it's good.


But then, you know, the ordering shouldn't matter, you simply care whether you have the


correct bounding boxes, you don't care whether you output them in the correct order.


And also, what if your classifier does something like this, it outputs those two boxes we see


here, but it also outputs this year and says bird, or like one that is slightly off and


says bird, and so on.


So how do you deal with all of these cases?


So the way that this paper deals with all of these cases is with their bipartite matching


loss, this thing right here.


So how does it work?


Let's say you're, where can we go?


Let's say your classifier.


So here is an image.


I have to wait for this to catch up.


Here is an image and we put it through this entire pipeline, right?


And we get a set of predictions, right?


And they're going to be class bounding box, class bounding box, class bounding box.


Now, the first thing you need to know is that there are always the same amount of predictions,


right?


There are always this size here is fixed, that's large n, okay?


That is sort of, that's kind of a maximum of predictions, since you can always predict


either a class or the nothing class.


In this case, you could predict anywhere from zero to five objects in the scene, right?


Okay.


And then the second thing is from your from your database, you get out an image with its


bounding box annotations, right that are made by human labellers.


Let's say these two.


And you also do class bounding box, class bounding box.


But now you see we only have two, two instances.


So here we just pad with the nothing class.


So I don't know what the bounding box should be for the nothing class.


It doesn't really matter.


Nothing, no bounding box, nothing, no bounding box, no bounding box.


So your ground truth labels, if you will, are also of size n.


So you always compare n things here on the left that your classifier output with n things


on the right.


Now, as we already said, the question is, how do you deal with you can't simply compare


one by one because the ordering should not be important.


But also, you don't want to encourage your classifier to always kind of if there is if


if the one bird is very prominent, right, you don't want to encourage your classifier


to say to say that here's a bird, here's a bird, there's a bird right here.


Hey, hey, there's a bird, there's a bird, there's a bird.


And basically, just because the signal for that bird is stronger and basically ignore


the other bird.


What you want to do is you want to encourage some sort of your classifier to detect if


it has already detected an object, it shouldn't detect it again in a slightly different place.


So what the way you do this is with this bipartite matching loss.


So at the time when you compute a loss, you go here, and you compute what's called a maximum


matching.


Now, what you have to provide is a loss function.


So we can, there's a loss function, l, and l will take two of these things, l will take


the red, the predicted thing of your model.


And l will take the true under one of the true underlying things.


And l will compute a number.


And we'll say how well do these two agree.


So you can say, for example, if either of them is the nothing class, then I have no


loss, like I don't care about them, that gives you no loss.


But if the two if the two classes agree, and the two bounding boxes agree, then it's very


good, right?


And we maybe even gives like some negative loss or give loss zero.


But if if the bounding boxes agree, but the classes don't agree, then you say that's bad.


Or the other way around, if the classes agree in the bounding, or even if if everything


disagrees, it's the worst.


What you're basically saying is, if if these two would correspond to each other, right,


if the thing on the left were the prediction for the thing on the right, which we don't


know, right, it could be that the thing on the right refers to the bird on the right.


And the thing on the left refers to the bird on the left.


So it would be natural that the bounding boxes aren't the same.


But you say if these were corresponding to each other, what, what would the loss be?


How well would they do?


And now, if you compute this bipartite matching, what you want, I guess it's a it's a minimum


matching in this case, what you want is you want to find an assignment of things on the


left to things on the right, a one to one assignment.


This is an example of a one to one assignment, everything on the left is assigned exactly


one thing on the right, such that the total loss is minimized.


So you're going to say, I'm going to align the things on the left with the things on


the right, such that it's maximally favorable, right, I give you the maximum benefit of the


doubt by aligning these things.


And what so in the best possible case, what's the loss?


Okay.


I hope this is something this is somehow clear.


So this, you're trying to find the assignment from the left to the right, that makes that


basically is the best case for this output right here, where you really say, Oh, okay.


Here you output it output a bird, very close to the bird here in the ground truth label.


That's this here.


So I'm going to connect, I'm going to connect these two, because that's sort of it's it's


it gives the model the most benefit of the doubt.


And the loss that you have at the end of that matching, so this loss here would only then


count wherever these connections are.


That loss is going to be your training loss.


So this solves the problems we had before it is not dependent on the order.


Because if you reorder the things your minimum matching will simply find will simply swap


with it.


It is it is if you output the same bird multiple times, only one of these is going to be assigned.


So if if this here is that bird, only one of them, only this one maybe is going to be


assigned to that one.


And the other ones can't be assigned to that one are forced to be assigned to a different


one, let's say this one here, and are going to incur a loss.


So you encourage your model to output, let's say diverse bounding boxes, different bounding


boxes for things.


Okay, so this this solves these problems.


And it's very clever.


And there are algorithms to compute these these minimum matchings.


They use the Hungarian algorithm, which will give you exactly such a matching.


Again, this is possible because you have n things on each side.


And the n is in effect here is the maximum of objects that you can detect at once, I


guess.


And if there is less, you can simply pad right here.


And then the model, of course, is encouraged to come up with the equal number of no class


predictions, because if it outputs a prediction, when it shouldn't, right, if it already predicts


two things, and these are assigned to these two things, and then it outputs one more thing,


it is going to be penalized because it should output three things with no class, but it


has output one too many with a with a class is going to be penalized.


Okay, so the this is a pretty, pretty cool thing.


It again, it relies on the fact that you have n on both sides, but you can make n so large


that basically, it covers all of the cases.


So you can make n like 50.


So you can detect up to 50 things in a scene.


Alright, that's the algorithm in a high level.


They do show their loss here, you see the loss ultimately is going to be so is going


to be over this matching right here.


That's the minimum bipartite assignment that basically minimizes this total loss over your


prediction and label matchings.


And the loss they come up with here, I said you have to give the algorithm a loss is this


one.


And they kind of go into how they do it, I don't think it's super important.


So the class algorithm, sorry, the loss on the class labels, I think it's going to be


a softmax, or a sorry, a cross entropy loss, like an usual classification, and the loss


on the to say whether two bounding boxes agree is a mixture of the L1 loss that compares


two bounding boxes, and this IOU loss, which is not dependent on the scale of the bounding


boxes, it kind of computes how much fraction of the two bounding boxes overlap.


But in any case, the loss basically consists of saying how much do the labels agree, and


how much do the bounding boxes agree?


Okay, again, this is only possible because after that you compute this matching, otherwise


you would have no clue which boxes to come, which predictions to compare to which other


predictions.


So let's look at this architecture a bit more in detail.


As we said, you have this what they call the backbone, which is a convolutional neural


network.


And with that, you put in some positional encodings.


And I already said the, you should look at the these features right here as just smaller


feature versions of the image, but they still have some image nature, then they are flattened.


So once they are put in the transformer encoder, because the transformer is naturally a sequence


processing unit, okay, so it takes in just a sequence of vectors right here.


And since an image is not a sequence, what you'll do is, if you have your image features,


and we said we have a bunch of channels, let's say we have four channels, and their height


and width and C, you're going to unroll and flatten that into one sequence.


So this is height times width, you basically unroll across these axes right here into this


axis, and its channel size.


So basically, you have a sequence here of of of C dimensional feature vectors that you


then put into your encoder.


Okay, so your encoder will now transform this sequence into an equally long sequence yet


again of features.


The good thing about a transformer, because why do you use a transformer?


The good thing about the transformer is that in such a sequence, and I've done videos on


transformers, it you can basically mainly look at the video attention is all you need.


If you want to understand this more fully, this thing can basically have attention, so


it has attention layers, it can attend from each position to each position in a one shot


manner.


So as it transforms this representation up the transformer layers, at each step, it can


basically aggregate information from everywhere in the sequence to anywhere else.


And therefore, it's very, it's very powerful.


If you have a sequence and you need sort of global connections across the sequence, this


is very good for language processing, because in a sentence, let's look at this sentence,


the input images are batched together, right?


Applying blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, and then there's they,


right, the word they, and you need, you need to know that they refers to the input images,


okay, and but you see, this is very, very far away in the sentence.


So you need a model that makes use of long range dependencies.


And they make the case that in such a task right here, you also need the long range dependencies,


because these bounding boxes, as you see right here, they can be quite large.


So if you have an image, you need that this part here communicates with these and this


and this and this part, basically anywhere in the bounding box, and these bounding boxes


can be quite large.


So the transformer architecture actually makes sense here.


Now, I want to go a bit later into why I think it actually makes even more sense for bounding


box detection.


But right now, I just want to keep going through this, through this architecture right here.


So if my computer here decides to come back, yes, we can go on.


So what we'll get out is yet another.


So in here, we put this thing we put down here, we put into the transformer encoder,


and we get an equally sized, equally shaped sequence out of the transformer encoder.


And you see that this thing here goes as a side input into this transformer decoder.


So the transformer encoder here is just a bit more of a feature mapping.


Like just for the architecture, you could think of just putting this into here.


But of course, it's going to go better with the transformer encoder, the transformer decoder.


Now does something similar, but you see it has the encoder as a side input.


This is very much like this is not like BERT BERT is like a only encoder transformer.


Whereas this is much like the original attention is all you need transformer that has an encoder.


And then the decoder as a side input, basically as conditioning information has the encoder


output.


What does the decoder do?


Again, since it's a transformer, it's going to take a sequence and output a sequence.


The sequence it takes is right here is what they call object queries.


And this also is different from the attention is all you need papers and they don't do it


auto aggressively, they just do it one shot.


What does it mean?


It means that you start with a sequence here of four things.


And these are these are the this is this big N and you out you output the sequence of a


sequence of four things.


And it's important to see what they're going to end up.


So these things are then directly going through a classifier that now outputs the so these


things here are these class label bounding box outputs.


So, each of these things is going to after transformation end up being one of these bounding


boxes either defining an object or saying that there isn't an object somewhere.


Okay, you see here, this bounding box refers to this bird, this bounding box refers to


this bird.


So each of these things is going to be one bounding box.


And the what they call object queries is the question, of course, is what do you input


here, right?


I actually I want to transform this image information that comes from the left here,


I want to transform that into the bounding boxes, what do I input here?


And the answer is, you just input at the start, you just input n random vectors, because what's


that going to give you is basically n outputs, you want an outputs because you want n of


these bounding box classifications.


So you need n things.


And if I input n things into a transformer, it's going to give me n things as an output.


And then in each step, I can simply condition on the information that comes in the images.


And it'll give me right, then I can incorporate that information.


It's a very deep learning way of thinking about it actually, that you just need the


information somewhere in there.


And I need n things.


Now, they go more into detail into this transformer architecture.


Help help in a helpful fashion in the appendix.


And we'll go there quickly.


So this, I think here makes more sense.


So the image features come in here, right?


And you see, this is just a transformer stack, an encoder stack of multi head self attention


and feed forward in instance wise or like token wise feed forward network.


And then that information is taken and is given as conditioning information over here.


Now in here, as I said, you input these object queries, which at the beginning are just n


random vectors.


And what you're going to do, you are also going to feature and code them and then you


combine it with this image information.


So ultimately, if you think of this one of these things, one of these things is going


to be a vector, right?


And then that vector is going to be transformed.


And then it will have as it is transformed, it will have the opportunity to basically


look at features that come from here, the arrow is in the wrong direction.


So you have already taken the image and you've transformed it into a feature representation,


which is also a vector, right?


You have the features of the image right here.


Now as you transform this vector, this object query queue, you have the opportunity to look


at the image features, right?


And that's how you get the image information in there.


So the image features will come in here, transform that through attention.


So this is an attention mechanism on the image.


And then what you will output is a bounding box and a class label.


It's really hard to explain.


I would guess you need to understand really what attention mechanisms are.


And of course, the crucial part of course, is what, what's this?


What do you input at the beginning?


And these object queries aren't actually random, as I said, they are learned.


So what you're going to do is you're going to learn independent of the input image, you're


going to learn n different object queries.


And these object queries, now, it's very, it's very interesting, because these object


queries are sort of going to be different.


It's like you have different people that can ask the input image different questions, right?


And this they have.


So their n is 100.


But they show 20 of these object queries that they learn.


And so they have visualization of all bounding box predictions on all images.


So it's, it's sort of like you have n different people at your disposal, and you train these


n different people to kind of ask different questions of the input image, okay?


You say this person up here will always ask irrespective of what the input image is, will


always ask sort of, hey, input image, what's what's on your bottom left?


Right?


That's I'm really interested what's on your bottom left.


And sometimes I'm a bit interested in what's here, but I'm mainly interested in what's


on the bottom left of the image.


Whereas this person right here, sorry, this person right here, is more interested in what's


in the center, and the different colors here is referred to different sizes of bounding


boxes.


So this person is also interested.


So the person on the top left is interested mainly in, I think, small bounding boxes that


are on the bottom left.


And the person here is mostly interested in what's I'm really interested, what's in the


center, what's large in the center, I want give me large things that are in the center,


right?


And then this person right here is really interested on stuff that's on the right side


of the image.


So you see, in order to get different, sort of a difference in bounding box prediction,


you train n different people to ask different questions of the input image.


And this asking of questions is exactly what an attention mechanism is.


So this person right here, let's take this person, and I'm saying person, these are vectors,


these are learned object queries.


But this person, first, they will simply ask the question, what's on what's on the right


side, and then the the image features, right?


I'm getting poor drawing, the image features, it will have an attention mechanism to this


part of the image features.


And then it will get back some signal, right?


And then it will transform that with its own signal up, and then it will ask maybe again,


okay, now that I know more, because you see that person is interested in multiple things,


it's interested in those things and those things.


So at first, it will focus on these things.


But then it says, Oh, now I'm now I know more, right?


There is there, I know I see there is actually something on the right side.


So in the higher layers, it can then go back and ask the image more questions by sending


these Q vectors of the attention mechanism.


And it will get back the V vectors from the image features that correspond to these Q


things.


So up and up the layers, this person can ask more refined questions about what that particular


person is interested in, okay.


And since you have the different people here, that ask different questions, you basically


learn the people in a way such that across the data set, they all together, they cover


every possible image pretty well.


Again, these people, what they're interested in initially, is not dependent on the picture,


you simply learn this in a global manner.


All right, this is the best way I have of describing it.


You basically learn n people that are each one is interested in different things, different


classes and different regions in the image.


And each one of these people is going to output their best guess of what is where based on


what they're interested in.


So that person might say, I'm, you know, I'm the person that's interested kind of in the


left side of things.


So I am going to output that there is a bird right here.


Now these people if this is a transformer, right, and everything can attend to everything,


they can actually communicate with each other as they incorporate information from the image.


So in each layer, they can do both, they can incorporate information from the image, and


they can communicate with each other.


And then in the next layer, they can do it again, and again, and again, and thereby,


they can sort of they can sort of say, Well, you already got the left side, I will take


the right side, you already got the bird class, I will take the elephant class, and so on.


So you see here, how the the architecture of the transformer actually is also very conducive


to doing this bounding box prediction in that these different things can sort of attend


to each other and therefore communicate with each other.


Alright, I hope that sort of makes sense.


Now, before we get into the experiments, I want to list a third reason of why the transformer,


especially the encoders might actually also make a giant amount of sense here, since you


unroll the image into height and width.


And you have to imagine what does the transformer do?


The transformer, as we said here, has this notion of attention where from any point in


the sequence, it can gather information from any other point in the sequence.


And this, that's usually one of the downsides of the transformers is done via a quadratic


attention mechanism.


So if I just list one feature channel, go over here, if I just list one feature channel,


right here, this is height times width of the image, right?


This is, this is the entire image unrolled in one vector height times width.


And here, I unroll it again, height times width, then this, this matrix that I can build


right here, which is called the attention matrix, right here, it will tell me which


parts of the sequence attend to which other parts, okay.


So if you have an image that has the, let's say the number three, and you really want


to figure out whether or not this is a three, then the bow up here must communicate with


the bow down here, right?


They need to share information to say, oh, there's a bow here, there's a bow here.


And there is a spiky thing here, that must be a three.


So you want something, this is rather at the beginning of the sequence, you want this to


attend, first of all, it will attend itself.


So you get fairly high values along the diagonal, maybe 10, 10, 10, 11, 11, 12.


I saw this oligee skit.


100 million 99.


But it also like this, this part here, at the beginning of the sequence, let's say it's


here because this is unrolled, right needs to attend to the end.


So this needs to attend to the end, which we will put an 11 here.


And the other way around doesn't always need to be symmetrical, by the way.


Okay.


So in any case, this is going to be a h times w squared matrix, because everything can attend


to everything.


And that's the attention mechanism.


Why do I think this is so good for bounding boxes?


Because let's let's imagine you actually have a matrix that is like this, okay, height times


width times height times width, every single point in here actually defines a bounding


box, because this point, this point right here in this dimension corresponds to one


location in the image.


And on this axis, it corresponds to another location.


Now in the attention matrix simply means these two points need to communicate.


But if you have two pixels, you actually have defined a bounding box right here.


Right?


You you're actually you're defining a bounding box.


And the the fact that this is happening in the exact same matrices could mean that the


transformers are uniquely well, the transformers across sequences of these height times width


unrolled images are uniquely well conducive to these bounding box prediction tasks.


I'm actually a bit astounded, because when I first just read the title, this immediately


popped to my mind.


I'm like, Oh, yes, of course.


And they're going to predict the bounding boxes by simply training.


So what you would do what I thought this was going to be is out you output an actual matrix


like this.


And then you simply each point you can you can simply classify, right?


So you can classify here, whether, whether or not like at in this direction, there is


a bird, right?


And then if you have two points like this, for example, you and you also classify whether


in this direction, there is a bird, right?


And this naturally defines a bounding box, or you could like take this matrix and actually


just classify individual points in this matrix to be the bounding boxes, because they already


define bounding boxes.


So I just I think these these quadratic things are are uniquely I mean, someone must have


thought of this, or if not cite the YouTube channel.


It would be funny first paper ever to actually have to cite the YouTube channel.


But again, yeah, so transformers seem to be a good idea for these kinds of things.


So how do they do?


Of course, they do well.


They are on par with these other much, much, much more complex architectures, these faster


rCNN models, they are apparently much more complex, but they are on par with this.


They do however, train forever.


I think they train for like six days on a GPU is not that much if you compare to like


language models on hundreds of TPUs, but still, okay, I don't want to go into the numbers


of experiments.


But what is pretty cool is that they can now visualize this sort of attention.


And you can see right here that if they look at a particular point in the image and visualize


the attention, it will actually attend to the instance itself.


So it will, like, these are usually the problems for these detection algorithms when things


overlap and are partially occluded.


But you can see right here that the attention is on the part of the image that makes the


instance in the back.


And the attention here is in the part of this, and it doesn't sort of overlap into the others.


So that is one thing that's pretty impressive about these architectures.


The other thing they show is, for example, can generalize to many, many instances.


So here, it has never seen 24 giraffes in one image, but yet it can absolutely do that.


And giraffe, giraffe, giraffe, giraffe, giraffe.


And the one of the coolest images I find are these here, where you can see right here,


the attention visualization.


And you see that even within the bounding box of the front elephant here, you see that


the attention on this foot of the back elephant is assigned to this blue bounding box.


So this is the blue, basically the blue bounding box person that is attending to that back


foot.


That means they, they, these things really sort of understand, or they learn these things


like occlusion.


And I just had a hard time describing it.


But you can see it visually here, right, like how it clearly learns that these are two instances


that are sort of occluding each other.


But this, this, this instance can actually appear within the bounding box of the other


instance.


And the same goes for the zebra here that are partially occluding each other.


And you can see that the attention is correctly like even here, that this back foot of this


zebra is correctly labeled.


So all in all, that is pretty cool.


And they take it a step further.


And they say, well, with this architecture, we can actually pretty easily do pixel wise


classification.


So this is just this cocoa stuff and things data set, where I don't know which one is


the stuff and which one is the things.


I think things is the objects and stuff is like sky and mountains and so on.


And so this is a classification task where you actually have to label every single pixel.


So what they do is they simply input this through their detector.


And they detect the instances.


They take the attention maps of the instances.


And then they scale it up.


This right here is just a CNN sort of in reverse that scales up the image because they have


scaled it down.


As we said, they scale it up again.


And then they can simply classify each pixel where each of these you remember we had these


different people here that that cared about different things in the image.


Each of these people will classify their respective pixels, the pixels they feel responsible for.


And then you simply merge all of these people's predictions together into this prediction.


And again, this gives pretty, pretty impressive results.


I am I mean, this is this is fun.


This looks like it sort of works.


I haven't.


They do quantitative analysis, of course, but I'm just impressed by the examples right


here.


All right, that was sort of it.


I really enjoyed reading this papers.


The simplicity is pretty cool.


They do have not only do they have code in the paper to show how ridiculously easy it


is to get this to run.


This is all you need in pytorch, but they do actually have code.


And as I understand, they also have pre trained models.


So they have this model zoo right here, where they give you the pre trained models.


So you can play with it and you can even load it from torch hub yourself and you can train


it yourself.


They have a collab always there.


All right.


Again, if you enjoyed this video, consider leaving a like subscribing and I'll see you


next time.


[ML News] AI-generated patent approved | Germany gets an analog to OpenAI | ML cheats video games

And AI is now officially listed as the inventor in a patent, Aleph Alpha raises $27 million to


build Europe's open AI and an open source replication of Dali is released. Welcome to ML News.


All right, before we get into all this stuff, this video is sponsored by


Weights and Biases. Weights and Biases is a one stop shop for machine learning researchers to track


their experiments, save their models, recreate their old experiments, share work with others


and generally analyze their results. Weights and Biases allows you with one single line of code


to track your experiments, which means that Weights and Biases will track the execution


run of your experiment will track the results, it will track saved models and checkpoints,


upload it all to a convenient central place in your profile. And that allows you to analyze,


visualize all of your experiments and data. Think of it like effortless tensor board in the cloud,


Weights and Biases has integrations across all of the deep learning frameworks, pytorch,


TensorFlow, hugging phase, you name it, they probably have an integration available. Today,


I want to tell you about a new feature that they have, which is called tables. Now the name is


deceptively simple. Table is simply a grid of stuff. But in Weights and Biases, tables allow you


to view things like data sets, but also outputs of your runs, any kind of artifacts you have,


you can analyze in tables, tables allow you to sort group filter and do anything with the data


you're looking at. And you can take advantage of all the visualization capabilities that you're


used to from Weights and Biases dashboards. For example, here, we automatically visualize the


results of pixel level annotations. I mean, look at that left hand side, that model sucks.


Look at the bottom, why is the sky labeled as trees? Clearly, you have to do something here.


So as you can see, you can analyze the output of your runs, you can see where the model still makes


mistakes by filtering for the samples that are classified incorrectly. If for some reason,


Weights and Biases doesn't have a visualization for your type of data, which is unlikely if they


don't have it, they allow you to actually integrate with their framework in order to produce one,


the capabilities here are really endless. Here you can see we visualize anything from sound files,


to training plots to spectrograms, whatever you can think of. So as a special bonus, viewers of


this channel only get 80% off today off the basic plan, which you don't need actually, because it's


free. Yes, it's completely free. There's really nothing stopping you from going there and making


accounts, personal accounts, free unlimited experiments. If you're a bit more involved,


if you want a team, and if that team is large, and does a lot of tracking, you'll have to give them


some money, but their main income comes from big enterprises that want to use this internally. If


you are such a big enterprise, don't hesitate to give them a call and give them a lot of money.


In that way, you'll be supporting all the free accounts for all us plebs. There are special


options for academic research teams, which do get free team accounts. And you can also self host if


you need to be compliant with some sort of regulations. So again, go over to weights and


biases and check it out. There's a lot of features that I haven't even talked about yet, such as


hyper parameter optimization that's done automatically check it out. And now let's get


into the news. I'm back. Yay. What did I miss? What has been going on? How do I do? How do I do


news? I forgot. Alright, the global legal post right South Africa issues world's first patent


listing AI as inventor. So this person right here is Professor Ryan Abbott, he and his legal team


have been fighting around the world applying for patents that list the AI named Davos as the


inventor of two particular inventions. So now they finally succeeded in South Africa. And also as ABC


News writes, an Australian court has equally ruled that AI can be listed as an inventor on a patent


application. Now the situation is a little bit complex, and I'm not a lawyer, so don't take my


word for it. But the ownership of the patent rests with the creator of Davos of the AI, while Davos


is listed as the inventor. So here's one of the things that Davos apparently invented, it's kind


of a fractal thing. So they're saying this is kind of a food container or something and the


fractality somehow makes it good and you can connect containers together. But there's also this


light emitting thing that has kind of a fractal ish pulse or something that makes it really


noticeable. And this here is Steven taller, who is the inventor of Davos and therefore the owner of


the patent. Now I was immensely interested into this and I have spent way too much time researching


this here is kind of a few takeaways. First, I thought this is a PR stunt. Come on, you know,


why can't you just list yourself as an inventor? Because ultimately AI is like a tool, right? And


how does an AI even come up with new ideas? Like what counts as new ideas? And like, how does an AI


come up with this or this? Like, what was the part that the AI did? What was the starting point? What


was it doing? Like, I'm so confused. Okay, so this is the website of the team of the legal professionals


that got the patents through to through the courts. And they answer some of these questions.


And their claim here is that in the various legal systems, the granting of a patent requires the


inventor to perform like the invention step, like there's a specific step in the conception of an


idea that is like the innovative step. And it is actually criminal offense to list the wrong


individual as an inventor. So the inventor does the creative step, and you have to list that person


as the inventor. Otherwise, it's criminal offense. Now, the question is, if legally the AI did that


inventive step, whatever that means, technically, you should list the AI there because you can't


list any of your employees, you can't list yourself because you've only controlled and


built the AI. But the AI did the actual step that the law requires to be listed under the


inventor. And apparently, they claim at places patent applications have been rejected because


of this. So from this perspective, it kind of makes sense that you should be able to list the AI


as the inventor. Now, counter to that, some legal systems also reject this notion, saying only a


natural person can be an inventor. And therefore, on some of these inventions, simply no patent can


be granted, which would be discouraging from researching stuff. Remember, AI is used to make


inventions in such field as drug discovery, where the AI simply comes up with new compounds, and


then you test them. So in a way, the inventive step is performed by the AI. If you could not apply for


a patent in that that would discourage research in these directions. Alright, so this seemed to


me like to be a reasonable explanation. But that's only the surface right here. I was much more


interested in the question of how, how does this system that I have never heard of come up with new


invention? And here on this hideous website of this legal team, this question appears to be answered


and cut. So this has gotten so long through the edits, that it just completely blows the format


of ML news. So what we're going to do is we're going to cut the rest of this into its own video


because this is really weird. This data system is weird. This whole case is weird. The too long


didn't read is there might be a valid legal reason why AI needs to be listed as an inventor on a


patent. Also at the same time, this is probably a giant PR stunt. And the inventions themselves


aren't they're nothing. So, you know, look forward to the next video, make up your own mind. Let's


go on with the news. All right, German startup olive alpha raises 27 million US dollar series


a round to build Europe's open AI from TechCrunch. This is Jonas under rule is the founder of olive


alpha with headquarters in Heidelberg in Germany, which is not too far from here. And the goal is to


build the equivalent of open AI, but in a European fashion. So it says the German AI startup olive


alpha has now raised 23 million euro, which is 27 million in real money in a series a founding


co led by early bird VC, Lake star and UBC partners. The team says it will have a strong


commitment to open source communities such as Luther AI academic partnerships and will be


pushing European values and ethical standards it says supporting fair access to modern AI research


aimed at counteracting the ongoing de democratization, monopolization and loss of


control or transparency. So while these are a laudable goals, and I really hope they achieve


and stick to these goals. Remember that open AI has set the same at the beginning and now open AI


is mostly interested in closing down access to their stuff and charging for it. But luckily,


venture capitalists, which are the main founders of this venture right here, are not known to ever


wanting their money back or anything like this. So this should just be a breeze for olive alpha.


So I wish Jonas and co founder Samuel and anyone part of olive alpha all the best and big success


in their endeavors. It's going to be fun having sort of a counterforce to the US here in Europe.


Robotics 24 seven says a and pay robotics marks milestone in data pick rates for automated


recycling. So speaking of companies and raising money, this company is now a raising series B


for about 55 million US dollars. And they're in the space of garbage, sorting and disposal and


recycling. So they've developed these analysis and gripper technologies. And this is incredibly cool


to watch. I mean, we're always talking about AI taking away our jobs. I don't think people will


be too sad that AI is going to take away their jobs in this particular field. So here the AI


automatically analyzes the streams of garbage and sorts them by the materials in them. And so


these blocks of cans just look really cool. Also, there is such a thing as waste Expo didn't know


excellent must be a blast. Next news DeepMind releases a paper called open ended learning


leads to generally capable agents. So what they do is they build an environment called excellent,


this is kind of a 3d environment. And the agents in here you can see on the top left and top right,


this is what they see, apparently, and they have to fulfill various goals in these environments,


you can build any kind of environment you want in excellent, then you can tell the agents to


achieve that. Apparently, the paper is about when you instruct the agents who learn multiple goals,


many goals at the same time, or after one another, they become generally capable, as opposed to just


having a single objective and then ending up with a very narrow skilled agent. Now excellent can be


used to not only have many different environment spatially, but also have many different tasks or


games in this environment. So they've captured the flag king of the hill, and so on. In the paper,


they actually detail how they use population based methods in order to train these agents,


how good they are at zero shot learning, and so on. And this is all pretty cool. However,


these things and results aren't that new, we already knew that population based training


is probably good. If you want to achieve some generally skilled agents, we already knew that


multi objective or objective conditioned learning is probably a good thing. Ultimately, the agents


here are simply an observation encoder into an LSTM. And then they take in the goal conditioning,


and then it's a standard actor critic reinforcement learning. I guess what I want to say is that the


research isn't necessarily super new or exciting, but you can get a lot, lot, lot of publicity if


you build something that's 3d and looks really cool. So if you want, you can build your own


stuff in X land if you work at DeepMind because I don't think it's open source. So haha.


The New York Times writes something bothering you tell it to woe bot. And it is about the


system that delivers cognitive behavioral therapy through an app. So cognitive behavioral therapy


is one of the more successful approaches to treat things like depression or anxieties. It is rather


formulaic as this article describes, and therefore it lends itself at least a little bit to be


incorporated into some kind of algorithm. So the article is a discussion of is this good? Is this


bad? The pros are that usually a human therapist is very expensive, and there aren't enough of them,


especially in times of a global health crisis. On the other hand, critics argue that these


algorithms aren't yet good enough to replace a human because they cannot intrinsically understand


the things that the humans say. And you get the idea. The New York Times accompanies this person


right here, Eli, who has tried out the app for a given period of time. Eli details how the app


sometimes fails. Responding to my boss doesn't appreciate the work I do. And I can't seem to get


her approval. The bot answers with that sounds difficult. Does this happen more in the morning or


at night? It is a little bit of an improvement, I guess over something like Eliza. However, it still


seems to be rather formulaic. So my own personal opinion is this, if I have some problems, there


are books that I can read self help books that guide me through the process of somehow solving


my own problems. These books are necessarily impersonal. They are written by a person,


but they're not personalized to me in any way. It's the same text for every single person that


buys the book. So if a book like this can help me, then certainly a little bit of an algorithmized


version of a book like this might help me too. You know, there are ways to make it worse, but I


don't think much. So if you think that there are good books that have helped you in the past to


overcome personal issues or problems or any kind of improvement, then it's entirely possible that


an app like this does the same thing. I don't think we have to necessarily seek to replace


therapists, but there are a lot of people who cannot afford therapists or don't have one close


by. And in this case, such an app can probably help. Now, of course, it's also easy to see that


people will feel as though that actually replaces a competent therapist and not seek the attention


of an actual therapist when it's needed. So at the end, Eli breaks up with Woebot saying he was


unimpressed by the bots advice for beating back loneliness and despair, but he is not entirely


sorry that he tried it out. The mere act of typing out his problems was helpful. And through the


process, he pinpointed what he actually needed to feel better. Yes. So it worked. Now Eli is seeing


a human therapist in Philadelphia for $110 a session. Next news synced, right Google's


wordcraft text editor advances human AI collaborative story writing. So the text


editor isn't out yet just a paper and a demo video where a human writes something and then


clicks on a button and then the machine sort of continues the story. This seems to be sort of a


GPT three ish thing with an interface that just helps you select from different continuations and


does the prompt engineering in a smart way for you. You can even customize the prompt, you can ask the


model to elaborate on particular parts of the story and then choose from various continuation.


I think that's pretty cool if it ever will appear online, which I'm not sure, given that it's Google,


but if it ever will appear something like this might lead humans to just come up with new ideas


through this thing. So pretty cool. Next news, PC mag writes machine learning is now being used to


cheat in multiplayer games. So there's apparently this video here that demonstrates that a bot


is used for cheating in games. Now aim bots have been a thing for a while. But apparently this


thing works in a little bit of a different way. And it also works on consoles, which for now has


been a kind of a difficult thing for aim bots. So what you do is you hook up your console to a video


capture card, feed that into your PC and the PC would actually send commands to your controller.


So you'd hold the controller, but your controls would sort of be overwritten at times by the input


of the cheat engine. And that makes detecting these cheats rather hard to use. Now it just says


that machine learning is used in order to control this right here. You could also imagine this being


just kind of a classic aim bot that just recognizes some pixels and then shoots at it. But apparently


it's machine learning based. So you know, it's an ML news. Thanks. Next news, Google releases the


open buildings data set, which is a data set that across satellite images of Africa has annotations


of over 516 million buildings. This goes along with a paper where they detailed the challenges


that they had to overcome to do this. So you can devise various failure modes right here. So all of


these pictures, for examples, are not buildings. The top left are water pools, top right are rocks.


Then here, there are some buildings, but the thing in the red square is not a building. It's just a


bunch of walls. The left are containers. This is very difficult. Google has annotated over I think


a million images 1.75 million images or sorry, Google has annotated 1.75 million buildings in


100,000 images by hand and then trained a system on it. The paper details how difficult that was,


how much you have to use augmentation and regularization in order to do that. But in the


end, they've come up with this giant data set that you can now use, you can actually explore the data


set in this interactive Explorer right here. So you can switch between this view, which is I'm not


sure how helpful that is, or this view I have discovered. So if you zoom in right here, I have


discovered however, that sometimes I feel at least like this piece here, is this an actual building?


It says it's a very high confidence building. I'm not sure, honestly, also this thing here,


this might be one, but it seems like it works pretty well. Just overall, the challenges are


also recognizing buildings in both rural areas, where they kind of blend into the environment and


recognizing buildings in commercial or dense populated areas where mainly have to separate


buildings from each other. So pretty cool. Give the open buildings data set a try. If you're


interested. Next, MIT Technology Review writes, hundreds of AI tools have been built to catch


COVID. None of them helped yet another article about the shortcomings of machine learning


research. And the take of this article is somehow you know, more effort is needed and criticizing


ML research. In the meantime, I have a bit of a more cynical approach right here. Like we've known


long enough about the publication pressure in ML research and to use a buzzword topic like COVID,


in order to get a paper published by simply applying whatever your thing is in research,


whatever your topic is, and using it on some kind of COVID data set in order to get a publication


out of it. Because people think like, Oh, this is, you know, relevant, we need to publish fast. Now,


I don't think the main motivation of 99% of this research was actually to develop something that


actually works. Old methods are slapped onto new topics in order to get publications. And we will


continue to see that in the future as well. Don't expect any of these things to work in the first


place. Next news, Dali mini is an open source replication effort of open AI's Dali. So these


people have built a version of Dali that is much smaller, but has first signs of actually working.


Remember, Dali goes from text to images, and you can actually try it out yourself on an online


interactive demo on hugging face. Here's my query for creepy clown and the model does not disappoint.


It seems like there's still a gap, probably a gap in size model size and data set size until this


project reaches the level of Dali if ever, but still, it's pretty cool. And I love the avocado


chair just as much as the Dali one. Okay, we come to the helpful library section of ML news,


helpful libraries. First helpful library is kind of big news. Open AI releases Triton, which is a


language that allows you to build custom CUDA kernels. And these CUDA kernels are super duper,


duper fast, and you don't have to know low level C plus plus CUDA in order to produce them. So


there's a blog post and code to go along with it detailing in very detail what's now possible with


Triton. And apparently, open AI has made this in such a way that people who have no previous


experience with CUDA programming are able to produce kernels that are as fast or faster than


the kernels that were previously programmed by experienced CUDA programmers. So if you have


something that doesn't have a efficient CUDA kernel yet, maybe give Triton a try.


Next helpful library flammable fast and lightweight auto ML is a library for cost effective hyper


parameter optimization. So apparently, you enter your problem to optimize and your cost and the


library will optimize your hyper parameter towards your cost taking into account how much each hyper


parameter setting costs to explore. So for example, if you have something like model size as


a hyper parameter, it will preferably try the smaller sizes first because they cost less. And


you can search more before it then scales up that hyper parameter. Pretty cool. Give it a try. Next


helpful library Italian clip. Remember clip scores images and text together and Italian clip is now


available particularly can classify such things as a and oh, I'm kidding. It's a it's a cool project,


check it out if you are Italian speaking or building Italian speaking products. Next helpful


library DeepMind releases melting pot and evaluation suite for multi agent reinforcement


learning. Now other than excellent, this one is actually open. It's an environment in DeepMind


2d lab and has various scenarios for multi agent reinforcement learning. And this actually looks


like you can do some research with it and multi agent reinforcement learning, especially something


like cooperative multi agent reinforcement learning is one of these areas that is still


largely unexplored and we don't have super good algorithms for it yet. So if you're looking for


some research to do this might be a cool topic. There's an old helpful library with some news


Mujoko the 3d simulator that has been used for a long time for doing things like continuous


reinforcement learning control problems and so on is now free. The product requires a license,


but they do give out a free license to anyone at least until the 31st of October 2021. So if the


availability of the license has blocked you so far, give it a try now. Also in RL news open AI


gym has a new maintainer that is going to address the pull requests that are there project has been


kind of dead for a long time. And the new maintainer makes it clear that there aren't


going to be new environments, major breaking changes, environment wrappers, anything like this,


I think they simply want to make the gym usable and up to date as it is pretty cool. If you're


a gym user, this should give you some stability and compatibility with current libraries. The new


maintainer is JK Terry. Thanks for your work. Last news for today, the free software foundation


calls for white papers on the philosophical and legal questions around copilot. Apparently they're


contacted understandably a lot with regards to copilot and the kind of legal ramifications of


copyright and patents in what copilot does. If you don't know what copilot is, watch ml news from a


while ago. In essence, they give you 500 bucks if you publish a paper through them that somehow


elaborates on parts of these topics. So areas of interest are is copilot training on public


repositories infringing copyright? Is it fair use? How likely is the output of copilots generate


actionable claims of violations on GPL licensed works and so on. So there are some submission


guidelines. And I wonder if there's a way I can submit my ml news segment to this. Where's my 500


bucks, Richard? Come on. So the criticism of the free software foundation is that copilot is what


they call service as a software substitute, which is a term they came up with to replace as a software


as a service to make it more clear. Of course, Richard Stallman here writes, the basic point is


you can have control over a program someone else wrote if it's free, but you can never have control


over service someone else runs. So never use a service where in principle running a program


would do never Richard says never Okay, new.org. Let's look at that a certificate. What kind of


certificate is there? Hmm, details. It's by let's encrypt. G is let's encrypt the program or a


service. I wonder what's up, Richard, you're perfectly capable of generating SSL certificates


using open SSL a free program that you can run yet you elect to use a service like let's encrypt.


Well, isn't that a jolly? Alright, this was already way too long. This was it for this week's ml news.


[News] Facebook's Real-Time TTS system runs on CPUs only!

Hi there, check this out.


Modern text-to-speech systems have come a long way in using neural networks to mimic the nuances of human voice.


To generate human-like audio, one second of speech can require a TTS system to output as many as 24,000 samples, sometimes even more.


The size and complexity of state-of-the-art models require massive computation, which often needs to run on GPUs or other specialized hardware.


So this is generated by a system that Facebook AI has built, and we're going to look at it.


It's called a highly efficient real-time text-to-speech system deployed on CPUs.


There's a lot to unpack here, but this is not a paper, this is basically a technical blog post, because I think they built this into their products,


and it's mainly explaining on a high level what they did.


So they have a real-time text-to-speech system, which means that you have a text like this sentence here, share on Facebook,


you give it to the system and the system comes up with a sound wave, right, a sound wave that says this sentence.


So if you listen to it, you'll hear share on Facebook.


And it has to do that in a, let's say, credible way, such that it is a human-like voice, because people like hearing that,


not these kind of robot, old-school telephone robot voices where they just chunk together words, it has to flow naturally.


So what you want to do for this is you want to have some sort of recurrent neural network,


or any sort of autoregressive network that outputs basically these samples here one at a time.


So these points you're going to output one at a time, and this, for one second, they say for one second of audio,


it can require you to output two or 24,000 of these, of these data points, right, so 24,000 forward propagations of your autoregressive model.


That's massive. And if you want to do it in real time, and that's why real time is so impressive, is you have to do this in less than one second.


So you have to do this 24,000 forward passes in less than a second.


Okay, and even more so, this was already possible, I think, but it required, basically required a big data center with many, many, many, many GPUs in it.


So you basically would send your text to this and it would stream back the audio.


And they can do this just on CPUs. And in fact, they can do this on a quad-core CPU in real time.


In, they can generate this many samples in half a second. Pretty impressive.