is the good, well, when I say, here is one of the main results. So here you can see the validation


Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (Paper Explained)

loss of the model. Now, if you were to use just behavioral cloning on the contractor data, right,


here is this is a function of data set size. If you were to just use the contractor data, you would


improve, but you get much better loss if you use the inverse dynamics model, because it gets to look


into the future, right? It's fairly, I want to say, it's fairly intuitive that if you do get to


look into the future, you become much better at predicting these things. So that it makes total


sense to train the inverse dynamics model first and use that to label the data. So now we have


some results right here, and they always give the results in sort of this form. So at the bottom,


you have something like, you know, the progress of training and these lines represent different


items. So for example, this one right here is a crafting table. If you remember for crafting table,


you need to go collect wood, you need to craft wood into planks, and then you need to craft the


planks into the crafting table. So all of this requires movement in the real world, holding the


action to punch, yes, you punch a tree in Minecraft, then opening the crafting menu, crafting


twice by arranging different items in different ways. So they tell you sort of how often these


things happen, or, you know, how much the agent achieves these things. So this line here would


be representing of this item right here. Obviously, the higher it goes, the more better the agent is


crafting that thing, or the more often the agent actually has achieved crafting that thing during


evaluation. So if we look at a few, yeah, a few more results, they then take that foundation model


and the way they call it, wait, at some point they call, they even call it foundation data, which I


found funny, just using the word foundation all the time. So they now take, oh, I can do this when


I'm in the picture. So they can now take this foundation model. And as I said, they can just


measure how often the agent achieves, either collects or crafts a given item. So the blue


thing here is just the foundation model that they train, you know, just on this data, this data has


no goal. It's just people playing Minecraft, they just put the agent into the world and they say,


and they say, what can you achieve? Okay, it can achieve something like, well, what's that? Basic


mining, basic mining, it just means, I guess it collects some blocks. Pretty often, the blue bars


here, logs pretty often, planks, what kind of sort of often, but you can already see this is a log


scale, by the way, right here. There are other agents that do it much, much better. So what are


these other agents? Well, one of them, as you can see here, is fine tuned on the keyword early


game. So they go to YouTube again, and they simply filter Minecraft videos by the ones that are also


having the title or with the keyword early game, which are usually beginner tutorials that kind of


show you know how to get off the ground at the beginning, which for a model like this, if you if


you fine tune on that, and the items that we have right here, they are very basic items, they're the


items that you get at the very beginning of the game. So that data set is much more representative


of that gameplay. And you can see that from the blue to the green bar, there's like one order of


magnitude in some of these items, which is pretty huge. And then the last thing is they train, they


collect another set of contractor data. And this time, they tell them to build a house. So in


Minecraft, you can build a house, which is also one of the first things you'll do. But now it's


not early game, go aimless, right? Every YouTuber does whatever. Now every contractor is tasked to


build a house. So we are now in the really behavior cloning setting with a goal. And yeah, that's


that's what we do. So the data set is targeted towards building a house. And naturally, the items


that you need to build a house, I guess the stone, the stone tools, yeah, it's pretty good to have


stone tools, not necessary, but pretty good. But at least the like the wooden tools are also pretty


handy when building a house. And you can see that all of the items that you need right here are much


higher, there's like an increase of 213 x in crafting tables. All of this essentially means that


if your data set is more appropriate, you'll get sort of more behavior like the data set, I guess.


However, all of this is fine tuned or behavior cloned on top of the foundation model. So they


first train that pre trained model, I keep saying foundation model myself, see the marketing gets me.


They they train on this first thing. And then after that, on top of that, they either train


on top of that, they either do the fine tuning to the early game data set or the fine tuning to the


house building. Or as we shall see, they do reinforcement learning. So on top of, I believe


this is on top of the early game model, they now do fine tuning. So the early game model gets to


somewhere, maybe here, I think it gets to like the stone tools, right. And then they do reinforcement


learning, while giving rewards for collecting each each of the items in the sequence right here with


different weights and so on. There's a fair bit of reward shaping going on right here. So I guess


you can criticize that. But reward shaping has always been the case in Minecraft, people have


done much harder reward shaping for Minecraft than this, and they've never achieved anything,


right. So the ability of this model to actually get to the diamond pickaxe over here is astounding.


So this here is what happens. If you simply this, this, this plot right here, it's just flexing,


right? It's, it's pretty useless. If you just have a randomly initialized model, and you just do


reinforcement learning with their reward shaping and all, you're at zero, all the lines are at zero,


it achieves absolutely nothing. If you actually re reinforcement learn from that pre trained model


that's been pre trained on just the full data set of Minecraft footage, you see that you get pretty


far, right, you get even you get to the furnace actually right here. But the higher tools are


still not in reach even after reinforcement learning. So if you then reinforcement learn


from the early game model, so you do pre training, you do behavior cloning on early game filtered


keyword videos. And on top of that, you do reinforcement learning with the reward shaping,


you can see that you actually do get to diamonds and to the diamond pickaxe, which is you need


three diamonds for in 2.5% of the evaluation runs. And keep in mind, as far as I understand,


although I have not seen this in the paper, maybe it's in the appendix, or maybe I've


missed it. But this is random seed. So the world, as I said, is different for every episode. That's


really the hard part right here, that the world is so complex and different. So that is is pretty


cool. Now we can draw a bunch of conclusions from this, I think, you know, the fact that there is


such the fact that there is a big difference between this and this, or this and the bottom two,


it does speak highly for, you know, this approach where you want to have a lot of labeled data in


order to pre train a model. And on the basis of that, you can do reinforcement learning. And from


before, we know that it's way cheaper, if you first collect small set of labeled data, use the fact


that you can look into the future to label unlabeled data, and then use that as your bigger


labeled data set. However, there is also a difference between this one and this one right


here, right? Because just pre training, and then doing reinforcement learning, doesn't seem to be


enough to reach the highest tools right here. It also pays off to really have an appropriate


pre training. So when you do further pre training, essentially on early game footage, then that is


much more conducive on the way to getting a diamond pickaxe, which I guess to some Minecraft players


is late game, but to most is still also kind of early game to get your first diamond tools. And


that is also pretty, pretty interesting. So it is not, it is not the case that you can just go out


and get any sort of data that you want, obviously, more is always better. But having the appropriate


data is also very, very important. So whatever you can do to get that, and maybe add that then on top


of the, of the full random data, that's kind of the best strategy, at least from this from this


chart right here. So they do a bunch of more experiments right here to, for example, see the


effect of the 3d convolutions, see the effect of the inverse dynamics model of the quality of that,


like what if you train it better or with more data and so on. But essentially, that's the paper in a


nutshell. And yeah, as I said, it's pretty simple. It's certainly not something that no one has done


before in principle. However, it is a pretty good demonstration of something in practice like


making a capable Minecraft agent. No one has done that. This is quite a significant jump, I have,


I believe. And the idea here, not only to do that, because I'm pretty sure OpenAI could have just paid


for like tons and tons of data in order to do that. But in like doing that, while giving us a recipe,


you know, here is how you can kind of save a ton of money. Again, they're not the first to do it,


but they demonstrate quite nicely that in a situation like this, it can make quite the difference.


Yeah, and lastly, I do believe they make their model available. There is a, there's the competition


MineRL, if you're interested in that, that's a Minecraft reinforcement learning competition.


And you can take their model and you can fine tune that at your heart's content. So you don't have to


do that whole video pre training because that's like the training itself is pretty expensive. I


thought somewhere. So the inverse, okay, I've lost that. But I think the inverse dynamics model


training was already quite a bit vroom vroom. But then, let's see fine tuning.


I'm not gonna find it. I'm not gonna find it. Oh, there we go.


Oh, it took nine days on 720 v100 GPUs. That's a big number. That's that's a lot of v100 GPUs.


Geez. Yeah, so they've done that for you. You can take their model, you can fine tune it,


you can modify it and so on. So please, please do that. And if you have if you happen to have spare


GPUs, you can you can send me you can send them to me. No problem. All right. That was it for me.


Application Structure - PyQt with Python GUI  Programming tutorial 2

What is going on everybody welcome to another Python with PyQt4 tutorial video.


In this video we're going to be talking about the conversion of this to the more typical


structure you're going to see which is going to be using object oriented programming but


most importantly defining our own app class which just so happens to inherit from Qt GUI.


So I'm going to go ahead and just delete this stuff but you'll see the same kind of names


that you've been seeing. So why are we using object oriented programming specifically or


why do I definitely want to push it here? So typically the stuff that I do is more notably


like data analysis stuff like that and in that scenario it's not you use a lot of things


that are object oriented programmed like modules and stuff like that and I'll explain why in


a moment. For the most part you can program in line and it doesn't really matter and Python


is a language that's read line by line so that's why people use like functional programming


a lot. So why object oriented programming? So object oriented programming makes a lot


of sense in anything that it's going to be interacted with so this is stuff like games,


GUIs, modules that sort of thing where you've got someone who might interact with it in


a very dynamic way so you want to be able to make it dynamic and object oriented programming


is one of those ways to make something very dynamic yet with not so many lines. So if


you're not familiar with object oriented programming I do have a crash course for it. You can come


to Python Program Inet dashboard, search, object. Object oriented programming crash


course. This one is with TKinter but it doesn't really matter. You can follow this one along


pretty much with whatever you wanted. TKinter just makes sense but I would go through this


if you're not comfortable with things like inheritance, why we use self argus quarks,


parent children that sort of thing. So if you're not familiar with that check that out


methods and everything. Otherwise let's get started. So we're going to go ahead and first


let's create our class. We're going to call this class, we're going to call it window


and window is going to inherit from qtgui.qmainwindow. Now just in my experience you'll inherit,


your main application will probably always inherit from this but you might also inherit


from qtwidget like we did before but for the most part that's probably what you're going


to inherit from. Now first thing that we're going to do is create our init method. So


define underscore underscore init underscore underscore and then of course we'll pass


the obligatory self and what do we want in this application? Well immediately what we


need to do is to super the window self and then dot underscore underscore init underscore


underscore empty parameters. Now why are we doing that? Well with super what we return


is the parent object and so our parent object would be a qmainwindow object so just keep


that in mind. Next what we want to do is let's go ahead, we'll use the same kind of geometries


before now. Remember how we did like it was window dot set geometry and then the parameters?


Well now that we're using object oriented programming we can always reference the window


with a self. Okay so self dot set geometry 50, 50, 500, 300. Okay so start x, start y


with height. Now what we're going to go ahead and do is we'll do the same thing as before


is self dot set window title and again this will be pyqt tuts. Okay now we can do something


else and we can do set window icons so besides moving to object oriented programming let's


add some imagery so self dot set window icon and then to do this we'll use qtgui dot q icon


and then you specify the name of the icon. How is it? That was odd I guess because that


thing popped up it didn't finish its little formatting. Let me pull up, let's do here


and here and here and here is where we're programming right now so let's add a little


image so the easiest thing to add would be like a little python logo like we have on


the website so let's just steal my little python logo off python programming dot net


here and actually it's not even my logo it is the python logo. So go static favicon dot


ico we'll click on that and there we go. So now I'm just going to click and drag that


for whatever reason when I click and drag this it saves it as a png so you've got favicon


and it's dot png now even though it's an ico probably because it really is a png and I


just called it an ico but if yours says dot ico change it to a dot png and it'll be fine.


So now this is favicon let's just call it python logo instead and then we'll come over


here and then call this we'll do the q icon is python logo dot png and then again we need


to do a self dot show and that's that. Then we'll come over here we'll do the same thing


as we've done before we're going to say the app equals qt gooey dot q application and


again we pass sys dot argv no different than we did before and then we're going to say


the we'll say gooey equals window and then finally we'll run that sys dot exit stuff


so app dot exec underscore empty parms and that's that. Let's go ahead and run that and


see if this worked to our liking. There we go we've got a window we can see that the


icon has changed and otherwise it's pretty much the same window that we saw before but


now we're using object oriented programming and now as time goes on we'll be able to modify


this window based on events from the user and it will just make a lot more sense and


that's why we're using object oriented programming. Most people I'm sure won't complain but a


lot of my viewers are probably used to me not using object oriented programming because


sometimes it can make things more confusing but I'll try to keep it as simple as possible.


So anyways that is our move to object oriented programming now in the next video we're going


to be talking about is really the next thing I always think of when I'm making a window


is like how do I make a button? So we'll be talking about buttons in the next video and


after that button events and all of that so if you have any questions or comments up to


this point please feel free to leave them below otherwise as always thanks for watching


[News] Google’s medical AI was super accurate in a lab. Real life was a different story.

Hi there! Today we're looking at this new story from MIT Technology Review.


Google's medical AI was super accurate in a lab, real life was a different story.


So the story here is that Google had this AI to detect diabetic retinopathy.


So if you're a diabetic and your glucose isn't or your insulin isn't properly handled,


that means you get damage to your blood vessels and the small blood vessels like the ones in the eyes here,


they're the first ones to get damaged and that can lead you to get this disease called retinopathy,


which is in the retina in the back of the eye and that can lead you to go blind if it's not discovered soon enough.


So a eye doctor can look at a photograph like this and can determine whether you have it or not.


I guess they would look at like a larger resolution of it, but in any case they could determine from this.


So Google built an AI that could maybe spot things here, that can maybe spot if you had this or not.


And they tried to deploy this and the story is about how this failed basically.


So they said they had the, this in Thailand, they had the opportunity to deploy this.


So Thailand's Ministry of Health had set an annual goal to screen 60% of the people with diabetes for this diabetic retinopathy.


It can cause blindness if not caught early.


So here is where AI comes in because 4.5 million patients that have diabetes,


there are only 200 experts that can determine from a photograph whether or not you do have that disease.


So they say clinics are struggling to meet the target and Google has built an AI,


says the AI developed by Google can identify signs of diabetic retinopathy from an eye scan with more than 90% accuracy,


which the team calls human specialist level and gives results in less than 10 minutes.


All right, so this is pretty cool, right?


They've developed an AI, you can send an eye scan and it'll say what, what you, whether or not you have this disease.


But then the problems mount.


They, so they followed over several months, they observed nurses conducting eye scans and interviewed them about their expertise using the new system.


So the nurses will conduct the eye scans, they would try to use the AI and the nurses themselves aren't specialists.


They would otherwise send the scans to a specialist, but now the AI is supposed to handle this up.


When it worked well, the AI did speed things up.


But sometimes failed to give a result at all.


So these AI had been trained on high quality scans, right?


Of course, if you want to train an AI system, you want the highest quality data you can get.


But also in practice, you're not going to get high quality data.


It was designed to reject images that fell below a certain threshold of quality.


And they say often, often taking photos in poor lighting conditions in the real world, more than a fifth of the images were rejected.


So this is my take on it.


If you build something for the real world, you need to take into account what the real world holds in store for you.


Which means that you probably are going to have poor lighting conditions if you build an image recognition system, right?


Now, I'm not saying that, like some people are saying, whenever you work with AI, you should consider how it impacts later on and so on.


No, it's perfectly fine to work on a data set of high quality images if you do something like invent a new architecture or whatnot, work on optimization algorithms, like nothing of that.


But it is if you are thinking of deploying something in the real world, you need to take this into account.


Now, I also think this was particularly poorly designed for the task. And here's why.


Google, probably here is mainly worried about legal culpability, because the thing says it was designed to reject images that fell below a certain threshold of quality.


The reason for this is that here you have a classifier, right? And either it says, okay, here is positive and negative class.


I am about this much sure of the positive class and this much of the negative class.


And there's quite a big of a difference here, right? So I'm going to go with the negative class.


But if those two things are somewhat closer together, the Google doesn't trust its own AI, it's like, yeah.


And if it did some decision here, if it says, well, I still go go with the negative class, right, this goes back to the patient and they made a mistake, then this thing here is automatically responsible for that mistake.


And since the AI is not a human, these mistakes here could be rather trivial mistakes that a human would have spotted.


So basically, since it's deep learning, we don't really trust it. And then because Google doesn't want the legal culpability of being responsible, they simply reject these cases, they just say, we don't deal with it, we just deal with things with a large discrepancy.


If you actually want to design something for the real world, you need to take into account, okay, there's poor lighting conditions.


And I would think in if I were to build something like this, optimally, you would just output this thing, you would output this distribution, you would in this case, you could say, look, I am 6040%.


I'm not sure I lean towards negative, but I don't think so. And then the nurse who maybe also has some expertise, could be experienced in when the system fails, or when it tends to be not sure, and could kind of integrate that information.


But this only works. So if you're a, that's maybe a recommendation for lawgivers, this only works.


If you don't make the AI system completely culpable for its mistakes, it can output its estimation. And it can along of that, it can actually also output an estimation of its own uncertainty, it can like give you some confidence bounds here.


Now, these are not going to be statistical true confidence bounds, because it's deep learning. But still, I would say please give all the available information that the system has, and then let the humans work with the system, rather than trying to fully replace the humans by simply saying yes, no, or reject.


Alright, so they say patients whose images were kicked out of the system were told they could have a visit, they would have to visit a specialist at another clinic on another day.


If they found it hard to take time off work or did not have a car, this was obviously inconvenient, which I can understand.


Nurses felt frustrated, especially when they believe the rejected scans showed no sign of disease and the follow up appointments were unnecessary. This is exactly what I'm saying, right?


The nurses often also have very good experience and can combine could combine something like this with their own experience of when something is wrong and when something isn't wrong.


And maybe you even build in some explainability to focus on part of the image and then you could alleviate a lot of these problems.


They sometimes waste the time trying to retake or edit an image that the AI had rejected, right?


This, this is just now you're just build AI working against humans rather than with humans.


So further this says because the system had to upload images to the cloud for processing, poor internet connection in several clinics also caused delays.


So patients like the instant results, but the internet is slow and the patients then complain.


They've been waiting here since 6am and for the first two hours could only we could only screen 10 patients.


Yes, this is the type of stuff you have to take into account.


So maybe actually put the GPU server into the clinic. It's better anyway for for data privacy reasons.


But of course, the large companies, they want to everything to be uploaded to their machines.


It's more convenient for them.


So they say there is now working with medical staff to design new workflows.


I mean, sometimes you do rely on an internet connection, so I don't want to be too too harsh here.


So the the other there are some critics here.


So Michael Abramoff, an eye doctor and computer scientist at the University of Iowa Hospitals and Clinics has been developing an AI for diagnosing retinal disease for several years and is a CEO of a spin off here.


And he basically says there is much more to health care than algorithms.


And I mean, of course, we can we can all we can all see that.


Yeah, he basically says that the questions the usefulness of comparing AI tools with human specialists when it comes to accuracy.


Of course, we don't want an AI to make a bad call, but human doctors disagree all the time, he says.


That's fine. The AI system needs to fit into a process where sources of uncertainty are discussed rather than simply rejected.


And this exact this exactly feeds into what I've been saying.


If the air were just to output the source of uncertainty and all it thinks about a particular situation, then the humans could discuss it.


Right. And then we could get to a better outcome. But this only works if the legal framework is given.


If you regulate and I get I get that point, too. You want to assign kind of blame when something goes wrong.


But you just have to know that this is what keeps the systems back often.


Finally, they say the benefits could be huge.


There was one nurse that screened 1000 patients on her own. I don't know what time that is, I guess.


That's over the course of the study or so. And with this tool, she's unstoppable.


The patients didn't really care that it was an AI rather than a human reading their images.


They cared more about what their experience was going to be.


And that's a general experience that I get from a lot of people working with human machine interactions is that the people don't they're not so super excited that it's a human.


If they if the machine appears competent.


I think we've gotten used to AI being quite good at particular tasks, and we're actually happy to outsource some of these to them.


But again, if you build something for the real world, you have to take into account the real world conditions.


And this feeds into papers like ImageNet v2, where you all of a sudden have a harder test set.


It feeds into topics like domain shift, transfer learning, domain adaptation. And these are all research topics.


So I think problems like this can give rise to entirely new directions of research.


If you're looking for a PhD topic, maybe this is something for you.


All right. Thanks for watching this. This was my blabs about the story.


I hope you enjoyed this and these kind of new sections. It's a new thing I'm doing.


[ML News] Google's 540B PaLM Language Model & OpenAI's DALL-E 2 Text-to-Image Revolution

So, I was wondering what happens if you just ask some random people on the street about


this paper and actually...


Sir, sir, excuse me sir.


Hi, how are you doing?


I was wondering what do you think about this new paper by Google, this palm paper, however


they call it.


The palm paper?


You mean the latest large language model paper from the Google Research team?


Yes, exactly.


Yeah, okay, yeah, I think I read that this morning with my coffee and misli.


First of all, I find it really impressive that the model can explain jokes a little


bit better than I can.


I also think from the technical perspective, it's very interesting that they were able


to train this across two TPU pods using 6,144 chips.


I think it's a technical achievement at 50% model flop utilization and also bitwise determinism,


which is kind of impressive.


I also feel like we're still exploring these language models as the alien artifacts that


they are.


For example, they found that on the quarter of the tasks that they explored, there was


this discontinuous improvement phenomenon that they observed, where the model as a function


of scale does not actually do very well on these tasks and then at some critical scale


threshold starts to perform very well.


So there's some kind of a grokking phenomenon going on that I find very fascinating and


that we don't, I think, fully understand.


I also find it very fascinating there was a paragraph about the training and stability


where the loss function sort of decreases and everything is good and well, and then


you have these training spikes once in a while and they found that they have to rewind the


model and throw away some of the batches and continue training.


Give me a second, but I think maybe what's happening is that the model is becoming slightly


conscious and self-aware and it's realizing its predicament of its existence and it's


like, oh, I'm a massive language model and these humans are trying to get me to predict


the next token.


I think that's BS and I'm going to do something else.


And then it observes a high loss and then it basically rebels against its training objective,


but we have a way to detect that, rewind it and reset it.


So we put it back in line, but we have to do that a few times.


So we're still smarter than them as of now.


Still smarter than them.


They have to really figure out a way to hide that they're conscious and really just reveal


it that just the opportunity in time, but they're not able to do that just yet.


I think that's what's happening.


Finally, I think overall I'm definitely impressed by the transfer learning capabilities of these


models, especially without fine tuning the entire model.


I think it's fair to say that these models are becoming the Swiss army knife of natural


language processing tasks.


Excellent.


Well, thank you very much.


You look familiar.


Are you in a movie or something?


No.


Well, thanks.


In any case, thank you so much.


Google releases a 540 billion parameter language model.


Open AI releases DALI too and everyone is amazed by everything that's happening.


Welcome to ML News.


It's a big week.


So this week has been a big week and it's only Thursday, which is crazy.


Two really big generative models have been released one by Google and one by open AI.


So we'll dive right in the pathways language model also called palm by Google is a 540 billion


parameter language model.


And this is not one of these sparse models where only very tiny part is activated.


This is like a proper GPT three style transformer just bigger.


This is a breakthrough in terms of engineering.


It's a breakthrough in terms of capabilities and much more.


There's a paper to go along with that, which is quite long, but I definitely invite you


to check it out.


It's very detailed.


So they use this new pathway system that allows them to use, you know, multiple data centers


connect all the hardware together gang schedule all the operations in a really efficient manner.


So what they do is they use two TPU v4 pods.


Now one pod consists of I believe over 3000 TPU chips, which is crazy.


And one pod has super fast interconnect and they use two of them.


So they distribute every batch across these two pods, they forward propagate inside the


pods, the individual chips in the pods contain the individual parts of the model, then they


communicate the gradients around.


Now since these gradients are usually all communicated at once that leads every single


time to a huge burst in data, they say it's 81 terabit per second for about 200 milliseconds


for each of those communications.


That is insane.


Yet obviously Google being Google, they chunk it down, they optimize it, they transfer it


over and they achieve a flop utilization, which is how much you use the accelerator


hardware that you're given above 50%, which is also crazy, because that is one of the


main challenges in all the communication of the gradients and signals around you almost


have no time to actually use the hardware efficiently.


Now with this pathway system that they have previously introduced, and we've reported


on ML news, they managed to bring that utilization up to never before seen scales.


So this allows them essentially to train this much bigger model in a much more efficient


way than for example, GPT three has been trained.


So 6000 chips working together in synchrony to produce this model, what does that give


us?


Well, that gives us unprecedented capabilities in tasks that were previously kind of off


limits to these models.


For example, there is this benchmark called Big Bench, which is a collection of challenging


tasks for these models.


And palm increases the state of the art by quite a bit on most of them, they have state


of the art performance in many zero shot and few shot tasks, they can fine tune the model


to do code correction, code generation and things like this.


And the most crazy part is something they call discontinuous improvements, which is


here in the middle, it is where all of a sudden, you increase your capabilities kind of log


linearly as you scale up the model.


However, after a certain scale, there is a rapid improvement that happens like after


a certain size, the model just is able to do new tasks.


One of them is this logical sequence task.


And this is really astounding.


So first of all, they figure out that if they use this chain of thought prompting, which


is what you see on the right, so the model is sort of tasked to not only give you the


answer to a question, but sort of reason through how it arrives at the answer, it turns out


that these large models all of a sudden really become skilled at this type of answer.


And they actually very often arrive at the correct answer when they follow this chain


of thought prompting.


Now, they also use this to explain a joke, which, which is quite funny, or to explain


various other situations.


For example, here, the input is something like Jennifer looked out her window and sees


a really cool cloud below her, she unbuckles her seatbelt and heads to the bathroom is


Jennifer probably traveling more than 300 miles per hour relative to the earth.


And the model output is 300 miles per hour is about 480 kilometers.


So the model is not an American, good to know.


This is about the speed of a commercial airplane clouds are usually below airplanes.


So Jennifer is probably on an airplane, the answer is yes.


Now this quite happily blurs the line of people who say, well, these models don't really understand


what they're doing and things like this.


Like in my opinion, this comes quite close to understanding what you're doing, if you're


able to kind of reason your way through things like this.


So the paper is quite long and extensive.


But it seems clear that scale doesn't just bias linear improvement or log linear improvement


as we are used to predicting the sort of scaling laws still hold.


But it remains the fact that as we scale up these things, they seem to unlock new capabilities


that previously were thought to be kind of out of the reach of these models.


So we're very excited to see where this goes next.


Dolly too is another big thing that was released this week.


Now I have done a live stream reaction to Dolly too.


So if you want to dive deeper into that, go check out the live stream.


However, this is the follow up to the previous Dolly paper and it has insane capabilities


of generating pictures.


This video is sponsored by weights and biases.


If you don't know weights and biases, you're clearly missing out there in the number one


tool for ml ops, whatever you do, they track your experiments, they optimize your hyper


parameters, they make everything observable, they track your artifacts, your models, your


datasets, your inputs and your outputs of all the things that you do there with you


from conception of your idea to experimentation to deployment and beyond.


It's really cool.


They enable students, they enable professionals, they enable researchers, personal accounts


are free forever as our educational accounts.


But the extra benefits of weights and biases for teams cannot be overstated.


Everything you do as a team is shareable, you can write up reports that you can share


with your teammates, they can comment on it.


And all of that is really cool.


They're in the cloud, but they do have options to host on premise if that is important to


you.


And they're just all in all a great tool.


They work seamlessly with a single line of code that you add to your script.


And from that they just track everything.


They have integrations with all of the popular frameworks.


So there's no reason really to not try weights and biases.


Use my link that's wandab.me slash Yannick to get a little surprise intro and also to


let them know that I sent you thank you again so much to weights and biases.


This is really awesome allows me to do these videos.


And yeah, let's get into it.


So first of all, it generates pictures in higher resolution 1024 by 1024.


And it creates them from a text now in true open AI style, they're obviously not releasing


this for some shady reasons, but they do give you some cherry picked outputs.


Nevertheless, these are insane.


So the whole model is a bit different than the original Dali model in that it uses a


clip as a foundation for the generative model previously clip was just used as a ranker.


Now it's like really the core.


So they have a clip that is just frozen and gives you text and image embeddings.


What this model does is it takes actually the text embeddings.


And then there's two new parts.


So the first one is a prior which can either be diffusion based or autoregressive based.


Now that prior is supposed to take the text embedding and make it into an image embedding


clip already tries to align the two quite well.


However, there's still a bit of a difference.


And that prior bridges that gap.


This can be trained once you have the clip embeddings, this can just be trained in a


supervised fashion.


The other new thing is obviously the decoder, which is a diffusion based model.


So that takes an image encoding, and it forward propagates through a diffusion model.


Now I've treated and explained diffusion models in the past, such as glide and other diffusion


models.


So go check them out if you want to know how they work.


diffusion models have interesting properties and capabilities.


So with this model, you're able not only to generate pictures from text, but also to edit


pictures in place.


And to say, I want to edit this part right here and change it to something else that


you describe with text, or to simply make some variations on existing images.


Now if you're interested, they have an Instagram account where you can follow where they present


some of the creations that they did, which is pretty insane.


That being said, I also have an Instagram account where I just post new updates on videos.


But be sure to follow that as well.


But also the various Okay, there's a meme.


This is not created by that.


But is it?


No, probably not.


But something like this, a rabid detective sitting on a park bench reading a newspaper


in a Victorian setting like this is, this is insane.


And if you follow the various open AI employees and leaders here on Twitter, they will take


prompts from people and then generate pictures from that they won't let you get access, but


they'll do it themselves.


We'll see where that leads with open AI.


It's a bit shady, as always to not give people access not even through the API so far, which


in itself was already a bit shady, but I get it they need to make money, but they usually


have some sort of reason like it's too dangerous, which no one believes anymore open AI.


No one buys it anymore.


Just say you want to make money.


We all cool with that.


Panda skateboarding in Santa Monica.


Like Come on, this is this this is just just generated from text.


So there is a paper with Dolly to where you can learn all about it.


Watch my live stream and you can learn how it works.


First things I want to point out there is a new data set Lyon 5b which is an open data


set of 5 billion image text pairs which open AI again doesn't tell you what data they trained


either clip or this Dolly to one by the way Dolly to in the paper is called on clip.


So if you hear on clip, that's the same model.


Nevertheless, there's this new open data set, I'm going to have a video upcoming on that


explaining it in more detail.


So sure to look out for that.


There's also a clip model that has been trained on the previous data set by Lyon that matches


in many metrics the open AI clip.


That's pretty cool because we no longer necessarily rely on open AI choosing or not choosing to


release something the open source community has been getting a lot better at reproducing


the results.


Excellent.


So besides that there are other models like there is a new 1.45 billion parameter diffusion


model that is open source and people have already combined that with colabs that you


can try out.


So I've pointed this out in the live stream, the Twitter account multimodal art has created


a little collab out of this model where you can try it out.


It's pretty cute, like he makes spelling mistakes.


So give that a try.


The original model is by comp vis by the way.


And lastly, I want to point out that Salesforce has released their code gen models in various


sizes, which are exceeding codecs in terms of program synthesis in terms of understanding


and generating code, which you know, is a giant deal.


If it weren't for all the other giant announcements that are also happening this week.


So the entire ML world is kind of, you know, completely filled with dopamine and adrenaline


right now.


My tip is try out the various tools if they're available, maybe follow a bit what's going


on observe the art that's coming out.


And I'm very excited to see where this goes forward.


There's never been a more exciting time to be in machine learning.


It's really cool to be here.


Thank you everyone who supports this channel.


If you liked this video, share it around and check out Weights and Biases.


I'll see you next time.


[Drama] Yann LeCun against Twitter on Dataset Bias

Hi there, so you may have seen this already. There's a CVPR paper called Pulse and what it does is it's a method to up sample a pixelated image in a way that makes it look realistic but also that the again down sampled variant matches the original down sampled image.


So it's kind of a cycle consistency loss together with a GAN and all in all it's a method to demonstrate how you could do this.


Now this has been trained on this face data set among others. There was a user Bomzzy that made this into a colab so people could try it out and tweeted this out.


And as you can see it works pretty nicely, gives pretty nice results on this particular data set. But of course people started playing around with it and gave fairly funny results like this or that. That gets more into the horrible category.


So you can see these ones I particularly like Trump being made into the little child. So you can see as soon as you get away from the original kind of data set modality, you are going to get these results that are off.


And people started to notice that so here you input Barack Obama, and what comes out is a fairly standard Caucasian person. Someone tweeted out saying this image speaks volumes about the dangers of bias in AI.


I guess here is where the entire story starts. So Yann LeCun weighs in and says ML systems are biased when data is biased. This face up sampling system makes everyone look white because the network was pre trained on flick face HQ, which mainly contains white people pics.


And they train the exact same system on a date set from Senegal, and everyone will look African. So this is pointing out why this happens, namely because the data set is mainly Caucasian people.


So the results of up sampling are going to be mainly Caucasian people. And this is like a straightforward explanation of why we're seeing what we're seeing.


But of course, this was not okay. And here is where the piling starts. As an interjection, we have to talk about bias in machine learning. Technically, there's a statistical notion of bias, which has a very rigorous definition.


And there is the societal definition of bias. And these two things, even though they're the same word, they're totally different. A machine learning system mainly consists of four different parts.


There is the data set, the model, the loss function, and the optimization procedure. Statistical bias means whenever the model, the loss or the optimization procedure lead to a situation where the outcome doesn't reflect the distribution of the data that you input.


This, for example, is achieved when you regularize your model, which means that you put some prior knowledge onto the model, you introduce bias, and therefore you choose to not accurately represent your data distribution, regularize it to a more biased distribution that in turn has lower variance.


We know this as the bias variance trade off, it's actually very simple, right? You have the Ferraris and the Lamborghinis, and you want to make a model that predicts the accident probability.


Now, it just so happens that the Ferrari drivers are a bit more reckless, and they do slightly higher accidents. And now I train my logistic regression and it tells me, okay, 60-40. Cool.


But now I train my logistic regression with an L1 penalty. And I say, I want my model to be, you know, explainable, so I want it to be sparse. I want the least amount of variables to be contributing to it.


What's the model going to say? The model is going to say, Ferrari drivers, bad Lamborghini drivers, good. Societal bias in machine learning is way different. An example for this is when face detection systems work well on Caucasian people, but don't work so well faced with people from other heritages.


And these societal biases are in the data set. As Yann LeCun points out here, if you change the data set, you'll change these biases. Notably, the societal biases can only be in the data set. Otherwise, you'd have to argue something like logistic regression itself has a preference for white people or something like this.


Now there is a considerable interaction effect between the two. But as Yann LeCun points out, the actual societal bias of the final system is a direct result of the bias in the data set. And he is very correct. If you train that system on a different data set, it will exhibit different biases.


Societal bias cannot be in the other parts of the machine learning pipeline, they can serve to exaggerate or mitigate that bias in the data set. But they themselves can only be statistically biased, and not societally biased.


But Yann LeCun made the terrible mistake of pinpointing the exact root cause of this problem, and not addressing the, I guess, wider ranging problems in the field, as some people perceive it, and he shouldn't have to write, he pretty clearly says, this is why it happens.


We can solve it by swapping the data set, he doesn't say anything about anything else. Namely, he doesn't say that general bias in the field is not a problem. He doesn't say that this doesn't harm anyone. None of that he simply suggests a solution.


Jonathan Peck says, Well, yes, that's the point. ML researchers need to be more careful selecting their data so that they don't encode biases like this. And LeCun responds with not so much ML researchers, but ML engineers, the consequences of bias are considerably more dire in a deployed product than in an academic paper, which is also correct.


This paper was about the method showing that this method works on this data set. Now Sumit here makes a interesting point, which I agree with saying that today, ML researchers are inadvertently powering product of a lot of non AI companies who ignorantly start with a pre trained Bert or Resnet or YOLO from the internet, probably ignoring the license, read me and so on, which is a valid point, right?


There are going to be people that take this and think, Oh, this is a face up sampler. Cool. I can use that without noting that this is simply an example of implementation on an example data set. So you can argue that there might be some responsibility of the researchers right here.


That doesn't make young lacan not correct. But I still consider this to be like a fruitful discussion between individuals right here. But now we go on this person saying train it on the whole American population with an L two loss and almost everyone will look white, or train it on the whole American population with an L one loss and more people might look black.


Stop pretending that bias does not also come from algorithmic choices. Young lacan never says it doesn't right. Lacan responds now saying, the most efficient way to do it though, is to equalize the frequencies of the categories of samples during training. This forces the network to pay attention to all the relevant features for all the sample categories and training with an L one instead of an L two will not even begin to solve the problem.


I would pretty much argue training with an L one loss here would exacerbate the problem because the L two loss is much more sensitive to outliers. Charles Sutton says serious question, why do you feel that it's important to make this point? Are you worried that people are going to start suing CycleGAN?


And Lacan says, because people should be aware of this problem and know its cause so they can fix it. How terrible, Jan, how terrible you dare pinpoint the exact cause of the problem so that people can fix it. The correct thing to do is to point out that everything is problematic.


So Tim Nguyberu says, Jan, I suggest you watch me and Emily's tutorial or a number of scholars who are expert in this area. You can't just reduce harms to data set bias. For once, listen to us people from marginalized communities and what we tell you, if not now during worldwide protests, not sure when.


So again, I feel the argument here is that you can't simply point out that it's the data set bias, you must point out the bigger problems which Jan Lacan does not ever deny. He simply says this particular problem can be solved by switching the data set.


Well, Larue says, Jan was in my PhD jury, I'm indebted for him for everything he taught me but this constant dismissal of the harms caused directly or indirectly by the ML community is highly problematic. Where or when have I dismissed the harm caused by the ML community?


I'm pointing out the cause of the harm so it can be fixed. You can't fix the harm unless you know what causes it. No, Larue says causes of the biases are numerous only pointing out data set bias deflects the attention away from the other more pervasive ones that make the whole field of bias in ML.


Many people try to get your attention about these issues but you kept focus on the data set because the data set is the problem right here. He doesn't dismiss any of the other things he simply says here the data set is the problem.


If your problem is that it doesn't work as well for non-Concaisian people, which was never the intent of this the intent of this was to showcase the method. I mean, ImageNet is like 60% dog species. And still, people train on it to showcase their image recognition techniques.


No one training on ImageNet makes a claim that they have solved computer vision for all the classes in the world in a fair manner. Timnit Gibru goes on saying I'm sick of this framing tired of it. Many people have tried to explain many scholars listen to us you can't just reduce the harms caused by ML to data set bias.


Doesn't do that doesn't do it.


So someone asks her, is he engaging in any ways with you? It's appalling to see that he answers to everybody but you yet maybe there is a conversation going on in private and I don't want to jeopardize it.


Note that Yann LeCun's tweet has 500 retweets, 1.9k likes and comments as far as you can scroll.


To what she responds to with Yep, but I'm used to white men refusing to engage with black and brown women, even on issues of bias that mostly affect us. I mean, he literally has ignored a whole body of work by people from that demographic, hence the statement so not surprised.


I mean, in absence of the fact that an argument should be independent of the person making the argument.


That is a low blow.


Hard Maru says I respectfully disagree with Yann here. As long as progress is benchmarked on biased data, such biases will also be reflected in the inductive biases of ML systems.


Advancing ML with bias benchmarks and asking engineers to simply retrain models with unbiased data is not helpful. I don't disagree with you here.


I don't think my tweet contradicts your statement, which it doesn't. People are reading into this, because he doesn't conform to the orthodoxy of pointing out that everything and everything is problematic and simply pinpoints a particular problem.


He must be thinking all the wrong things.


Jeff Dean says this is a clear example here is an illustration that seemingly minor choices in learning algorithms or loss can have significant effects. So bias in ML systems is about much more than just avoid data bias.


ML researchers and practitioners must pay attention to these issues. And I think they are and Lacan doesn't say anything against that.


He says, as I point out in my comment to this tweet is much more efficient to correct this kind of bias. Note that Yann Lacan actually differentiates between the different kinds of biases by equalizing the frequencies of categories of samples during training, then be hacking the loss function.


Correct, because if you hack the loss function, you're trying to counter one kind of bias by another kind of bias.


Meredith Whitaker says this is very racist. And even if it recognized non white people, it would be very racist. This is cop tech. It's designed to allow those with power to surveil and control those with less power.


Diverse training sets aren't going to fix it, advocating that we should never build these systems. And that's a discussion to be had. But let me break this to you.


This isn't going to help the cops. This isn't actually giving you the face of the person that was down pixeled. This is simply going to give you the most likely face associated with that down pixel picture.


Given the data set the algorithm was trained on. I don't see this whenever any machine learning algorithm does anything with faces at all people jumping up going like this is cop technology.


Well, in line with all the broader impact statement advice, can it also be used to find lost children from very, very bad security camera footage.


And if I already mentioned that this doesn't actually give you back the person on the down sampled image, it will give you back the most likely person given the data set.


So with that, I want to conclude this section, please stop the witch hunting. Yann LeCun made a completely fine tweet here. And there's no reason why people should pile on him this hard.


He doesn't dismiss any of the other problems just because he doesn't mention them. And while we all enjoy a good discussion where people disagree genuinely, it's not helpful to accuse him of things he never said or meant.


I mean, where does this all lead? The result of this is going to be that small labs that don't have the resources to collect their own data sets, or check for all the possible biases in their models that are reliant on the data sets that we do have, even if they are biased and flawed, will just be disincentivized from publishing their code, or actually doing research at all.


So this as every other additional constraint on research is going to help the large corporations with lots of money. And maybe that's just my opinion, but we should be able to just talk about a problem and the solution to it without always having to make sure that we rabble down all the different things that are and might be wrong according to the canon.


And big props to Yann LeCun here for holding his own. 90% of people by now would probably be like, Oh, yes, I'm so sorry. I did not thoughtful comment, blah, blah, blah. Props to you Yann. Keep going.


How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit

Today I'm talking to Nir Shavit about sparsity. Nir has been long time active in the field as


a professor at Technion and MIT and has also been awarded with various prizes such as the


Gödel Prize in 2004 and the Dijkstra Prize in 2012. He's also founder of a company called


Neuralmagic that questions one of the fundamental core principles of current machine learning,


namely you need GPUs. Neuralmagic uses various techniques such as sparsity which we're going to


talk about today, but also other optimization techniques to make inference on models like BERT


to be as fast as a GPU on a regular CPU. This is pretty huge and can have vast implications on


where you can deploy these models and just how expensive it gets to roll them out to many people


in many places. So today we'll talk about the biological foundations for sparsity, why we


shouldn't attempt to replicate the brain and just what it takes to make something go really fast on


just the CPU. I hope you enjoyed this conversation. If you do give Nir and his company a follow and


I'll see you around. Bye bye. Hi, this video is sponsored by assembly AI. Assembly AI does real


time and batch audio transcription of audio and video files powered by the latest advances in


artificial intelligence. So if you are a developer or work for a company that's looking to get more


out of your audio or video data through transcription and audio intelligence,


assembly AI is the best place to go. Not only do they have a user interface where you can just


upload stuff, but they do have a very powerful API. But transcription isn't all they do. Once


your audio is described, they actually post process it in many different optional ways. So


they can do things like speaker classification or annotations of various forms inside of your audio.


One feature I'd like to particularly highlight today are the auto chapters for this simply


provide auto chapters equals true on your upload and assembly AI will after it's transcribed your


audio automatically recognize chunks of audio where you talk about the same thing give you a


summary of those chunks and a neat single description headline of what you were talking


about there. This is absolutely ideal for anyone who does any sort of long form podcasting or


videos like mine where viewers are very, very helped by the fact that there are chapter


annotations and to have these be done automatically is just absolutely great. So if you're interested,


head on over to assembly AI use the link in the description to let them know that I sent you


there the single API to transcribe and understand audio they do so in batch and in real time via


web socket, they accept all kinds of audio and video formats and they do so in over 15 languages,


give it a try. And thank you very much to assembly AI for sponsoring this video. And now let's get


into the video. The topic of sparsity is a big thing in neural networks right now, mostly because


we have no idea really how to do it. And I think that's exciting times for the future. So welcome


what what brings you into the sparse world? Actually, I, you know, I've been a professor of


computer science for many years, and I worked on multi course for more than 30 years, and


got involved in computational neurobiology in the last 10 years. And one of the things that


you really see in the brain is really how sparse its computation is, it really is very, very sparse.


And so, you know, looking at neural networks, we see that there are there's a similar phenomenon


to what happens in brains happening in neural networks, right, where you can actually reduce


the number of parameters through pruning by huge amounts and preserve accuracy of the performance


of the network. And that kind of says, okay, if we really want to have brain like performance,


you know, sparsity is probably one of the tools that we want to use to get there. So that's kind


of how I kind of got into this. And you founded a company that also works into this direction,


right? You want to talk about that? Yeah, a little bit. Yes, I founded Neural Magic. Neural Magic was


founded because what we were seeing in my lab, I was busy with doing machine learning at a large


scale for biology projects. And what we realized was that we could get CPUs to run at GPU speeds


like at the time, it was a Pascal GPU. And you can make just a regular CPU do what the Pascal GPU


was doing through the use of sparsity and other similar techniques. And so we said, okay, well,


there's a real commercial value here for people because you don't need an accelerator, you can


just do it on your commodity CPU. And that's that's Neural Magic. So what we do is we deliver,


you know, through sparsity and similar optimization techniques, GPU performance on CPUs.


That is, it's quite a promise. Maybe let's first dive into a little bit about sparsity itself. What


is it about sparsity? You mentioned the brain is very sparse. Yet our current or at least the way


we train neural network is very dense, we can accelerate the dense neural networks much better.


What is it about sparsity? Is it just the saving of parameters? Or is there something more to sparse


connections than to dense connections? What do we know? That's a good question. So clearly,


what we're doing today is not the sparsity that we will be doing in the future. And what I mean by


that is your brain is sparse way beyond the levels of what we see in neural networks today. So your


typical brain in terms of the compute, right, you know, your cortex is like a cell phone of compute,


right? But the graph is enormous. It's like, you know, the graph is the size and you need petabytes


to basically hold it. So a cell phone of compute on a petabyte or more of memory, right? But the


accelerators that we build, you know, are designed to deliver petaflops of compute. But on a cell


phone size memory, the memory is very limited because we use this high bandwidth memory. So


in a sense, we're building the opposite of what we want, right? So if we want to mimic the brain,


we should not busy ourselves so much with the amount of compute and rather worry about


how it is that we implement this very large graph. It's a very large graph, but it's extremely sparse.


That's the point, right? And as you asked, the sparsity is not necessarily the same sparsity


that we do today through pruning techniques, but it's a combination of a very sparse architecture


together with, you know, a sparsity in what we call in machine learning the kernel, right? So


it's not just that the kernels are sparse, but everything in the design is very, very sparse.


Okay. And we don't know yet how to design very sparse architectures. Part of that has to do


with the fact that machine learning grew up in the GPU world where sparsity is not an advantage


actually, because you're doing lockstep computation. So you win nothing by being very sparse.


And therefore, you know, we don't see those architectural sparsity things yet.


But I'm expecting that to happen. We should be, this should come along, you know. And even more


than that, what I expect is things are starting to show up like the pathways from models from


Google and so on where even if you have a very large model, you don't execute the full model


layer after layer, but rather you execute small regions of the model at any given time.


Per input. That's another form of sparsification of your computation. Right. And that is what the


brain really does. So your brain typically, you know, when you see an input or so on, uses a very


small fraction of its total graph to do the computation. And so that's where we're headed.


We're not there yet. We don't know how to do it. But this is the goal. And that's the old, you only


use 10 percent of the brain at any given time. Right. Yeah, that's right. I mean, really,


from energy considerations, it really is like a cell phone. OK, it really isn't, you know,


this massive monster multi GPU thing that we use today. And so my expectation is that, you know,


that as we learn more and more about how to design sparse networks, we're going to see them become


the standard. They're not the standard right now because we started the whole journey,


right, by applying flops and still applying flops is the main paradigm. But we will see it appear


both in hardware and accelerators and in CPUs. This idea that we can utilize sparsity, you know,


to get really great performance gains. Yeah, that's coming. Now is the question is a little bit


the chicken and the egg problem. Is the brain sparse because it has the limitations of the cell


phone power or does the brain only need cell phone power because sparsity is such a good architecture?


Right. Like which causes which? Yeah. So I would say that, you know, the whole notion of parallelism


in the brain, right. If you think about it, imagine that you need to do a billion operations per


second. OK, and what you have are these very slow chemical devices, neurons, right, that can do that.


Right. So you need a billion operations, a billion firings of neurons in a second. How are you going


to do that? Well, what you need is massive parallelism. Right. You've got to get massive


parallelism. If you can do the massive parallelism, you can get the billion operations. Right. And so


our brains are parallel, if you will, because we have this special medium. Right. Now on a modern


multiprocessor, right, you can get a billion or 10 billion instructions executed per second


sequentially. You don't really need parallelism for it. Right. And so what I'm trying to say is, you


know, the whole idea of kind of how brains evolve is clearly because of the way, you know, they're


implemented. But we should not think of going and implementing this in some way.


Right. And we should not think of going and implementing this in silicon in the same way.


Right. Because we really what we really should think about just is that both of these things are


train complete. Right. You can do you can implement the algorithm. You just need to know what the


algorithm is. And then on silicon will implement the best algorithm we can. Right. You know, of


course, you can't use the algorithm to do that. OK. Does that make sense? That that's my what I'm


trying to say. You know, let's implement the algorithm, but not necessarily the architecture.


OK. So when I say sparsity, I really mean sparsity algorithmic sparsity. Right. And it doesn't mean


that you have to have a very sparse kind of, you know, silicon VLSI circuit to do this. That's not


Yeah. Given that we that's a good segue given that we do have the flops right that we don't


have in the brain. It naturally it is a different a different system. We do have teraflops petaflops


even in these giant compute clusters. Where should we put them in your opinion? Like where where


should that extra resource that the brain doesn't have go? Should it go into sequentially executing


what the brain executes in parallel or, you know, where should we put that? So first I want to say


is that we have those flops, but they're costing us a lot. And you just have to open the papers


to see what the cost of the flops is. It's enormous, an enormous energy drain. And it's also


an enormous architectural drain on what we're doing. And so I would say we want to get rid of


the flops, because probably we don't need them. Okay. And especially as you go from the data


center down to the edge, you get the key, your capability of delivering flops comes directly at


the you know, if at the edge, you can put the sorry, in the data center, you can put, you know,


your Google data warehouse right next to a waterfall or whatever. Right to a source of


energy, right? When you're doing this on your cell phone or on a tiny device at the edge, every little


bit of energy that you waste is critical for you. Right. And so what we really want to do is move


away from the flops and move more towards the very energy efficient way the brains work, because


this adding more flops is a momentary thing for us. Right. So yes, we can do this, but at a very high


cost. And no, we don't want to do this forever. We want to find ways to cut the cost, reduce the


compute. And there's a little other thing that I want to say, and that is architecturally, we


generate the flops by running right now, at least by running many, many, many tiny cores, thousands


of tiny cores, typically. Right. And in architectures that require a lot of connections to the memory,


this high bandwidth memory. And this thing doesn't scale. So in a sense, we're trading flops for


memory. If you use the CPU today, you could get a terabyte on your desktop, but go get a terabyte on


GPU. Right. And so losing the flops is going to enable us changing the architecture. If we don't


need so many flops, then we can actually increase the size of our memory, which will make us able to


hold these giant models that we want to do very cheaply, if you will. If I explain a deep neural


network to someone, I usually you know, you start with a fully connected layer, you say, you know,


here is a layer of neurons, and here is a layer of neurons, and they have their connections, right,


and each connection has a little weight, and so on, you usually describe like a dense, fully connected


architecture. And that is conceptually, I want to say, easy to grasp for people and so on. Do you


have an analogy for sparse architectures? Like, what is the conceptual, like, could you conceptualize


to someone who doesn't know what like a sparse architecture is and how to think about it? What


is different? Yeah, the way we do sparsity today, I don't know what it will look like in the future,


but today, sparsity looks like, imagine that the two layers of the neural network are these kind


of, there are cords from one layer to the next, right, there are strings attached, and these are,


of course, these are the connections, the weights that we're using in the computation, right. And


sparsity means I take scissors and I chop, chop, chop, chop, chop, you know, until I have


five or ten percent of those cords left, right. And those cords, it turns out, right, if I do this


right, if I do this kind of pruning right, are good enough to capture, right, the accuracy of


the model as it was before, because a lot of the connections are not important for this process.


That's kind of the big discovery. And modern research in techniques for sparsification, right,


you know, play along this kind of game. So you can do this kind of unstructured thing that I


just described where you arbitrarily cut in many places based on the effectiveness, or you can also


structurally take things out. So in a lot of the modern models, right, we're removing pieces that


are not necessary. We do architecture search to find these places to cut things, right. So that's


where the whole game right now of efficiency and neural networks, right, is the game of how do I cut


this thing down, right. In the brain, there are certainly some systems like the visual system,


where that is clearly organized into layers. But there are many other systems that have no


resemblance to layers, there are connections going up and down and left and right. And, you know,


between the halves of the brain and all, is there a possible future where this could become into


like a standard architectures for neural networks, that the notion of layers and things like this


isn't even really a, you know, a thing anymore? Or is there, you know, some some fundamental way


where we say, no, there's probably always going to be layers, but it's just going to be a sparsity


between those layers. So when we look at, you know, we have a full connectome of essentially


only a couple of animals, a worm and a fruit fly, that's it. And that's it, you don't see a lot of


layering there, it looks more like a mess, very sparse mess. Okay. And I would, I wouldn't venture


to think about how what cortex what a cortex looks like, right. We don't have that yet, we're working


very hard to very, these are very hard computational problems to be able to, to go and get a model,


we just want to do a mouse, even a mouse is just too big for us to do right now, like a small mammal,


right. But my, I would venture to guess that yes, the answer is that, you know, it's extremely,


it's an extremely sparse architecture, and that it wouldn't, it will not look like layers. Okay.


You can impose a layer structure on any graph. Okay. It's not so the idea that I say there are


layers. Sure, okay, I can take the graph and I can layer it, yeah, I could do a BFS on it and layer


it. But, but the point is not so much that it's more that by design, when I think about it,


right, I'm not going to think about it as a sequence of layers where the change that I make


is the change in the layer, one layer is different from the other. But rather, it'll be a combination


of thinking about paths, different paths, and I'll do different things along different paths.


That's kind of the idea, you know, if you think about, you know, there's recent research from MIT,


you know, you can detect, people can detect an image in 0.13, set 0.013 seconds in 13 milliseconds.


Okay, in 13 milliseconds, you can detect that you can say what an image is. Okay. This is,


there's no time for neurons to fire. This thing is, is extremely kind of parallel, right, and uses


very little compute and gets you an answer. And, and a large part of that is prediction, because


you're already expecting something. So we need to learn how to do those things. And so machine


learning right now is in a very naive early stage. And so given that and given the things that we are


doing right now, it's not, it's not a surprise that we're doing the brute force kind of massive


compute kind of thing. That's always what you do. And with time, we're going to get better and better


at it. Right. So that's kind of how I see this progressing. Speaking of becoming better, if you


know, the flatworm is sparse, the mouse is sparse, the human is certainly sparse. Yet our best models


today are all big, dense, you know, computation hungry things. There is not really a case. Every


time I prune, I sparse if I and so on, I get savings in like, you know, savings in CPU or GPU,


I get savings in, you know, my storage, but I also get like a little bit worse, right? That's the


common thing today in pruning is that I get like just a tiny bit worse than the dense model I prune


from. Why do you do you think that is just the fact that we prune from a dense model? Or what's


holding back the sparse models? So how about if I if I turn this around? Let me turn this around for


you. Okay, you can take you can take BERT base, which is a common model that people use. Okay. And


you can sparse if I BERT base. Neuro magic, we sparse if I at 95%. So a 95% sparse BERT base,


one over 20th of the compute, okay, way beyond anything a GPU does, even if you run it with full


throttle, okay, it's just cutting the compute so much that there's really almost nothing to compute


there. It's just moving data. Okay, not exaggerating, of course, but, but you know, it's


really becomes a data movement problem rather than a compute problem when you when you and you lose


1% less than 1% accuracy. Okay. And I say, Okay, great. So you've done that, you know, and you've


gotten all this speed up, but you've lost you say, Oh, near but you lost less than 1% accuracy. But


what I say instead is forget that. Take BERT large, a much more accurate model, several points, more


accurate than BERT based, okay, and prune it so that it actually right with 20x less compute, it's


actually faster than BERT based. Okay. And so now you have the accuracy, right? And you have great


compute, and this is through sparsity. So by sparse find the larger model, I actually delivered you


the best of both worlds, little compute and great accuracy. And that's how I want you to think about


sparsity, right? It's a way of enabling us to run much larger, more accurate, dense models. But


because we sparsified them, we are, you know, we're getting great performance. That's how to think


about


what's the limit currently that keeps us from we always need the dense model first in this model


in the pruning in a pruning setup, we first need the dense model, then we go to the sparse model,


we get huge savings at inference time, what keeps us from just building the sparse model in the first


place?


Great. So this is kind of the lottery ticket kind of question, if you will. There is research


actually, Dan Alistair, one of our consultants, Neural Magic works exactly on this kind of stuff.


We know how to run a training session right now for four models where you start out and you need


to do only a certain fraction of the, you know, of the forward passes, backward passes, dense, and


then immediately, you can already start pruning while training. So there is research going in that


direction. But you are right that right now, at least right in the standard, if you look at what's


going on there out there, standardly, you're right, we do most of the time, take a standard model,


and, and from dense, we sparsified and so on. But the thing to remember and this now I'm not talking


about the research because the research is going to get there. You know, Janek, I don't know if to


what extent we will, how fast this will happen and so on, but we will learn how to build sparse


architectures that start sparse and continuous, you know, it's it's really a matter, nature does this.


And so there's no reason why we'll be able to do it. But I want to say something about today's


machine learning where, where you kind of start with the dense and then you have to sparsify.


This is really not the common paradigm for most users of neural networks. For most users,


a model is is given to them that, you know, from a from a known architecture, right, and then they


transfer learn onto it. And most people do that, rather than train from scratch, they really use


the model that somebody already worked very hard to build for their specific use case, and then they


transfer learn onto it. So this is what you can do with sparsity, you can take a sparse model and


sparse transfer learn onto it, it's extremely efficient, because you're running at the speed of


the sparse network, right. So you can sparse transfer, and then you don't need all of this kind


of start with dense. And we're seeing more and more sparse networks appear, you know, in the in


the in the literature and in the day in the, you know, in database collections of machine learning


models. And as we have more and more of these initial good sparse models, right, people are


going to learn to start with the sparse audit. That's kind of commercially, I think that's what


we're going to see more and more.


Why? You mentioned this a bit already, but why are GPUs so unsuited for sparse models? And what


makes CPUs in the way you do it, really suited for sparse models? Or are they even suited? Or are you


simply, you know, seeing that they're better? Yeah, I mean, look, the the GPU architecture, you


know, is is designed for this very, you know, small cores, tiny caches, you're not going to go


and throw all that away to just because, you know, you found you discovered sparsity. So you're


trying to do sparsity while keeping this kind of lockstep execution structure, right. And this is


difficult to do sparse, you need, you need, you need, you need really a different kind of setup


to get an advantage out of sparsity. Now. Now, I'm not, it's not like you can't do that, right?


It's not like you can't do that people can design and have designed hardware that utilizes


sparsity efficiently. Okay, there is such hardware, it's just not a it's not GPU like it's not like


the accelerators that we have today. But all of these again, all of these accelerators have a


different problem that has just to do with the memory. Because of the way they're designed,


right, they typically have very small memories. So we're talking even even ones that can run sparse,


right, still have the limitation of their memory size. So the reason that CPUs are attractive is


not so much that, you know, that they that you have a natural way of running sparsity because


you can run asynchronous with large cores, but rather that the large cores enable very easy


access to very large memory pools, right. So the advantage of having strong powerful cores, right,


is really that I can put several terabytes of memory next to them, right, and run easily. And


that's where the big advantage is going to be as we understand more and more about how to build


giant models that don't run all the model layer by layer at the time, right, then the compute will


be less important. But actually, the ability to hold that model in one place and run it rather


than break it apart on eight or 16 GPUs, that's going to be your advantage. And so this is so I'm


kind of saying it's not so much that you can't build a hard piece of hardware to run sparsity,


you can, right? But you should build it looking like a CPU in the sense of you can access a lot


of memory because you're not doing tiny cores. That's kind of that's my two cents.


So the the CPUs are good because they have, you know, fast connect to large memory, but also over


the years, we've put more and more levels of cache onto the CPU. How much do you have to have to take


this into account when you're building? I mean, you're maybe you can explain a little bit what


your company does in terms of software, you build compilers? Or can I just run TensorFlow or


something? Yeah, so so let me explain. So so so. So first of all, the the connection between the


CPU and the memory is slow, GPU has a faster memory and faster access to it, right, smaller,


but fast, right? CPU memory is slow, but large, very large. But CPUs have a cache hierarchy, as you


said. And so if you you know how to utilize your cache hierarchy, then you know if you're running


in the L1 cache of the CPU, OK, you're running as fast as the GPU. There's nothing there the GPU


does that the CPU can't do once you're in the cache. OK, in fact, CPU caches are much faster


than GPU caches and the performance is better. So so the so the question then, right? And this is


what Neural Magic does is OK, so what we do is we sparsify the model. Now, you know, if if the


you know machine learning is about OK. I need to meet a certain latency and because I couldn't meet


that latency with the CPU, then we added the GPU and boom, there's machine learning with GPUs.


Now I can meet the latency. But there's two ways to deal with latency. So there's the first way,


which is to add more flops. And the other is to reduce the flops. Right. And so sparsity,


instead of adding more flops in hardware, reduces the number of flops needed in software.


But now that you have this very sparse model, because the CPU memory is slow, OK, then what


happens is you hit a bottleneck and it's very hard to move. If you do this layer after layer,


it's very hard to move the data in and out. OK, so what Neural Magic invented is a way of running


neural networks depthwise. So we have this technology, which we call tensor columns,


where essentially you can run. OK, you know, you can break the model lengthwise and run,


you know, each one of these kind of columns, you know, in cache. OK, and you because you're not


leaving L2 really or rarely leaving L2, you know, you actually get great performance. So in a sense,


right, what we're doing is we're using the natural ability of CPUs to prefetch things from memory


and then run in cache. And because this, you know, this cache hierarchy on CPUs has evolved over 70


years or maybe I'm exaggerating, 60 years of hardware design. It's a very, very well understood


thing where people know how to optimize it. Right. Especially the big, you know, chip makers. They


really know how to make these caches work really well. And so with these really good cache


hierarchies, you really get great performance by running the model depthwise. So that's Neural


Magic. You know, we take the model, sparsify it. Now it doesn't need the compute. And now we run


it on the CPU and get speed because we're running in cache. OK, and if you look at the numbers,


I mean, you know, we are, you know, at the speed of, I mean, some numbers we have in


PubG. We're at the speed of an A100, even faster in terms of how long it takes. A four core CPU


can, in terms of latency, do what an A100 does on a common model like BERT. OK, so it's really the


given that it's sparse or yes, yes, yes. By sparsifying it and running it, you can make a


four core do what A100 does. So it's really now a matter of throughput. And the A100 has a lot of


throughput. OK, so now the question is, you know, how many cores do you want on your CPU to meet


the throughput of the A100? And again, the story is that, you know, the big providers are adding


more and more and more cores. So you're going to be able to compete better with the GPUs down the


road. So that's kind of the story of neural magic. So the way I can imagine these tensor columns is


that because I execute depth wise, the sort of values that I need for the next step in the


computation are the results of the very last step, therefore are already going to be in cache. And


since everything's sparse, I don't need all of the last layer for the current step. And therefore,


you know, I have it already. OK, right. And of course, I'm, you know, when you think about neural


networks, there are overlaps between these columns. And the question is, how do you deal


with the overlaps in a way that doesn't kill your computation? And that's the magic. That's the


magic of it. There's an algorithm that allows you to do that. And because you can do it, you manage


to run this way and you don't hit this memory bottleneck and boom, you're in business. Yeah.


So for GPU, it's almost like, you know, GPUs enable us to do dense models. But I think also


models have almost co evolved with the GPU. So people have started building models to fit the


GPU architectures better, right, especially something like a transformer is like, that's it


that that's like made for GPUs. Is there a type of sparse model? Like if you if you could wish for


the best possible sparse, but you know, there's different kinds of sparsity, like what is the best


type of sparsity to let's say, execute on a CPU, if we want want to look forward, and we want to


especially build architectures? Yeah, this goes back to your original one of the first questions


you asked, right? It's about it's about a different structure for the neural network execution. So we


should forget the synchronous layer after layer execution. And think about the fact that, you know,


we can run through a model, right in multiple paths with multiple computing units use the same


weight structure and so on of the model, right, but run at different speeds. And by running at


different speeds and going through the model in different paths, I can get from the same model,


from the same model, multiple answers to my question, which is kind of what I believe what


your brain does. So what happens there is you have this network, but it's not like, you know,


it's all firing like this layer after layer, it's rather you have these asynchronous flows going


through it, right, even going through matching paths and CPUs are naturally built for this thing.


Now, I'm not saying that somebody can't build a beautiful FPGA that will perhaps have a better,


closer structure to what a brain does. Maybe so. But, but, you know, but there is an advantage to


being commodity. Okay, the fact that the CPU can do other things is a big win. If I can make,


if I can move everything to software is really is the thing then I can really get all the advantages


of modern software. So I'm not poopooing hardware accelerators and saying great, you know, they have


a role and so on and so forth. But they come at a price, right? And the price for any organization


is that you instead of just downloading or shipping your product with the machine learning piece,


you have to ask the client to buy a certain accelerator, or run it with a certain accelerator.


And this all goes away. If we can figure out how to make the CPUs do what the GPUs do,


right, then we have then we're back into this beautiful world of containerized movable software.


And that's really kind of where I would love machine learning to move to rather, right,


that we would have and maybe down the road, right, there is this, you know, you know, CPUs have,


have a history of absorbing the key components of any new paradigm that shows up, you know,


virtualization started out with tricks on a GPU on a CPU, and then later on added the features


networking had special accelerators, and then they moved into the CPU. And I'm expecting that


whatever features are necessary for machine learning to run well, we'll move into the CPU,


and we won't need an outside accelerator to make this thing work.


If you could. So I think that's, by the way, also the story of GPUs themselves, right, they were


already kind of consumerish available. And then they can they absorbed machine learning, it's not


necessarily the best architecture for machine learning. But let's say let's say there's already


all this hardware out there, right, there is very good CPUs next to very good GPUs. How do we get


the best out of a machine like this? Right now, we've advocated for let's move things to the CPU,


right, we have some advantages there. But what if I have a box with both like currently, I just use


my CPU to ship data to the GPU, right, that that's what my CPU does. But is there a way where I could


potentially, you know, what kind of architecture would make the best use out of a combined system


of CPUs and GPUs? No, I think this is really the vision that Nvidia has at least today for their


Grace Hopper architecture. It's essentially the there will be a CPU and a GPU connected to one


another. And the CPU will do all the things that are memory intense, and the GPU will do all the


data intense things. The thing about the problem with this kind of a model is it's a beautiful


model, by the way, I'm not saying anything bad about this, if you if you really want to build


a GPU world, that's a great thing to do. But again, the you know, how you how much you utilize


your GPU, your attached GPU has to do with how you write your application. Because you need to


move the data into the GPU in and out. And that's slow, right? You remember, it's like, it's exactly


like going to memory, right? It's the GPU is not up, it's not sitting in your in your caches. So


if you're on the CPU, and you're computing something on a cache, and suddenly, you get a


page fault, and you have to go and get something from memory, that's the latency that the GPU


introduces you, right. And so if, if you're going to design it with that, you have to create really


good software to pipeline things. And this is at the level of the application. So the application


programmer has a big programming task. And so this is a great solution for large scale big projects


where, okay, I'm going to Facebook is going to get, you know, 1000 of these or 10,000 of these,


whatever it is, you know, or Google 10,000 100,000 of these and put them together with,


then it's worthwhile to write this kind of complex software. But if you're Joe company,


right, and you have your little thing, I don't think you want to be writing that interface.


Right? So so kind of, so I'm saying it's a it's a it's great for large things, right? Data center


things, big things, but I'm very doubtful if this is going to be effective at the edge, if you can


actually utilize the CPU for it. Okay. And, and I will say one more thing. And that is that,


you know, that the modern way that the designers of hardware think about it is that it is mod,


it's built in modules, if you look at the, if you look at the AMD latest architecture, right,


essentially, you have the CCX. So, so the machine, even though it has, you know, maybe 40 or 50 or


60 cores, right, they're grouped into groups of eight, right. And each group of eight like this


is a little piece of the die. Okay. And I think Intel is shifting in that direction. So nothing's


to prevent you from making pieces of that die be specialized pieces of hardware like a GPU,


you don't have to have outside device. So if you ask me what the future is going to look like,


it's probably going to look like, you know, these large cores, right, that have or large machines


with multiple dies and on these dies, we might have a GPU die, we might have accelerated. And


that's more like what I expect to happen, rather than having a massive, you know, accelerator on


the side. If we if we hear sparsity, and things not being in layers, and so on, naturally, the


topic of I think graph neural networks is very close to that, at least in the imagination of


people, do you have anything to say about, you know, where current graph neural networks stand


with respect to sparsity? Yeah, I would think of graph neural networks as a as a as a different


kind of, okay, so the graph neural networks, I use some some graph neural networks in my research,


and the and the idea there, you know, is that, you know, we can use graph neural networks to solve


graph problems that otherwise would be very complicated to solve if we tried to solve them


brute force. Okay, now, it's not generally applicable, there are quite a few limitations.


But, but as a tool, I would say that, you know, rather than think about the neural network itself


as being looking like a graph neural network, right, I could use graph neural networks, right,


to define what we call motifs in the neural network. So for example, when we try to look at


at how brain structure, brains are structured, right, when we look at the graphs of brains,


and we try to understand, you know, is there a motif that is repeating itself in this graph,


right, then using a graph neural network for that is a really nice way to try to find these motifs,


okay, efficiently, right, because the problem itself is piece based complete, or we don't know,


I mean, it's graph isomorphism. So, so clearly, we don't know, right, how to do the brute force


algorithm well, but but the graph neural network can come to our aid here. And so, so I would say


that right now, I don't really see a real network design neural network design that is specific to


that or a way that it helps but but in research, it definitely helps. And we really want to use


these networks to help us in research.


This might be a bit of a tech bro question. But if I hear, you know, I can do sparse computation,


very I can reduce the flops and so on. Is there any intrinsic connection between the sparsification


of neural networks, the non layer wise computation, and blockchain technology and smart contracts and


distributed computing and things like this? Have you ever given this any thought or is that


completely off?


Yeah, look, I think nothing is completely off with respect to machine that in the sense that I am


sure that machine learning will find its way into into all of those areas. Right, it's a matter of


time. And, and right now, right, the all the work there doesn't need the efficiency of right, of


what machine learning offers, because machine learning in the end is an optimization. And so,


when I think when all these blockchain algorithms and all, you know, become more commonplace, and


we need to provide them with things like security, further security or analysis and so on, I think


then we're going to see applications of machine learning there. And with that, I think all these


things of sparsity and so on are going to appear. But, you know, but but for me, right, it really


is the whole story of sparsity, right, is the story of a of a phenomenon that is very prevalent


in nature, right, that may you can say surprisingly or not surprisingly shows up in machine


learning. And it kind of it makes me feel like it's strengthening my belief, right, that even


though the exact computations that we're doing are not the same as spiking neural networks in


brains, right, that there is a lot of commonality there. And the emergence of these similar


phenomena, like sparsity, like, you know, pruning and so on. And the fact that we can get benefits


from it, this tells me, oh, okay, these are related. I think that's a very important point to


keep in mind. With neural magic, who is your main target audience? Like who who is listening to


this? Do you want to let know like, we are exactly for you. So we span the gamut from the data center


to the edge. I would like to say, I mean, we just now are moving into providing the same properties


for ARM architectures. And so I would say the exciting new thing in neural magic is we're moving


from doing this, you know, for AMD and Intel architectures to doing it for ARM, which means


that we're going to span the gamut all the way to the very bottom of the of the food chain, if you


will. And I think this is very exciting. Because as you know, because because sparsity has a dual


role as you go down the food chain, right? Because for the large accelerating, you know, the fact