Can I parse this out from the middle of the model somewhere, when the model isn't explicitly trained to give me this, if I can, then I can really make the point that the model does something like this and learn something like this from data.


Deep Differential System Stability - Learning advanced computations from examples (Paper Explained)

Whereas if I can't, that will be more of an evidence that the model is simply sort of pattern matching close enough seen examples in the training data.


So that's a bit of my of my criticism right here is that they they show it works, which is pretty cool.


But then they, they don't do the sort of interesting experiments of these of this introspection right here, which is a bit sad, but you know, they leave it for future research, which I guess is going to be themselves.


And that's how you make two papers.


So no, I don't want to be too critical. It's a very cool paper.


And I invite you to check it out and leave a like and subscribe and leave a comment of what you think of this kind of research of this paper, and whether or not you think I'm totally wrong.


Neural Networks from Scratch - P.9 Introducing Optimization and derivatives

Welcome everybody to part 9 of the neural networks from scratch series.


In this part we're going to be introducing the concept of optimization, why it's hard,


and then we're going to be introducing the calculus that we'll be using to overcome this


challenge.


For the following code you can feel free to write it out if you want by pausing the video,


but it won't be necessary so you can just sit back and watch.


For this we're going to be using a simpler data set to start which is nearly solvable


with just some vertical lines and then later in the video we'll get back to our spiral


data set.


For solving this with a neural network we know that we can adjust weights and biases


and that the weights and biases impact things in different ways, but the question is how


do we adjust these so as to improve our model which in our case means to decrease the loss.


Doing this successfully would almost certainly require us to have some sort of idea of what


adjusting each of these parameters individually might do.


Let's get some intuition of the complexity of this problem.


In an effort to avoid the calculus you might wonder what about just randomly searching


or some halfway intelligent random search.


So we'll make a model, define the loss, and set some variables to remember our lowest


found loss and the weights and biases that got us to that point.


Then we'll go through 100,000 iterations randomly choosing combinations of weights and biases


for both hidden layers.


As a reminder np.random.randn will produce a standard normal distribution of values for


whatever shape you pass in the parameters.


So in this case if we just look at np.random.randn with parameters 2,3 we get a 2x3 array of


random values and a standard normal distribution.


We'll continue this for each of the layer weights and biases.


Each time you run the random.randn function you get varying results so this is how we'll


somewhat randomly hunt for weight and bias combinations.


With this we'll do a forward pass, calculate the loss, and then if that loss is less than


the previously lowest seen loss we'll update our best weights and biases along with updating


our lowest seen loss.


If we run this we will indeed see that the loss did decline some but only just barely


value wise and accuracy does not improve at all.


So what could we do instead?


We see that randomly searching for weight and bias combinations is probably going to


take far too long.


Again the problem that we're trying to solve here is an extremely simple problem.


This should be nearly instantly solvable so randomly searching for weight and bias combinations


is just going to be unacceptable on any decent problem.


Maybe rather than randomly searching for totally new combinations every single time we could


instead make random tweaks to the current best known weights and biases.


All we have to change to do this is change the assignment to instead being a plus equals.


If this adjustment decreases loss then we'll keep it and we'll continue to try adjustments


from this new point otherwise the weights and biases stay exactly where they were before.


Now when we run with this concept we can see that we've actually come up with an okay solution


for this problem.


We see continued progress for quite a while and we end with about 0.17 loss and 93% accuracy.


So again all we did here was rather than just randomly picking combinations of weights and


biases we just randomly tweaked from the last best known combination of weights and biases


for the lowest lost value.


So have we found a simpler way of doing optimization of neural networks?


Is calculus not required at all?


Well remember that we're using that more basic vertical data set and that this data set could


be easily solved with just some perfectly vertical lines and this would probably indeed


get you 93% accuracy.


So this is really just a data set that we don't even need deep learning at all to solve.


So now let's change from the vertical data set back to the spiral data set.


This data set still has clear and obvious classes but is not solvable by just some vertical


lines.


We need a nonlinear solution here.


With the semi-random logical-ish tweaking from the best weights and biases code that


we used before we can run this and see that the outcome while it does learn is still a


loss that's greater than 1.0 and an accuracy that floats around 40%.


It's of course still possible that we could continue tweaking the logic that we're using


here for semi-random optimization but even this spiral data set is actually a very basic


data set for neural networks and the complexity of a model required to learn this would be


a pretty small model.


But the problem with randomly searching is the search space, even if it's a very small


model, the search space is still theoretically infinite.


So this is still a huge problem with random search.


Another issue here is that we're treating each of the weights and biases identically


when it's likely that certain weights and certain biases might be more influential and


impactful than others.


Also in the end it's likely we reached what's known as a local minimum.


This is a concept we'll talk more on later but definitely going to continue being a problem.


We would have to add some further logic to help us get out of this.


It turns out that hard problems are hard to solve and this is why we turn to calculus


which can help us to intelligently navigate to a solution.


The first step towards this navigation is understanding the impact of a variable on


some function.


In our case that's going to be the impact of the weights and the biases on the loss


per each sample since samples too can impact the model differently.


While the loss function is a function that we apply to a model's output, not the weights


and biases, the weights and biases are directly influencing that model's output.


So it's really the impact of the weights and biases on the loss function that we're


interested in.


Now let's start with a simple function to understand what we mean by impact.


This function takes an input x and multiplies it by 2.


Now let's add some code to visualize this and we'll create some inputs x using Python's


range and then y will be the output.


We can also graph this to get an idea for what it looks like.


So with f of x equals 2x or y equals 2x, what's the impact of x on y?


In this case it should be fairly obvious, it's 2.


y will always be double whatever x is.


Another way to describe this impact in the case of a linear function like this is the


slope which is defined as the change in y over the change in x or delta y over delta


x.


With a linear function we can calculate the slope from any two points.


Let's say x equals 0 and x equals 1.


So that means point 1 would be 2x or 2 times 0 so y would be 0.


For point 2 that would also be 2x or 2 times 1 so y would be 2.


The delta for y in this case is from 0 to 2 so that's 2.


The delta for x is from 0 to 1 so it's 1.


So the slope is 2 divided by 1 meaning the slope is 2.


In Python we can calculate this by indexing y at index 1 minus y at the index of 0 dividing


that by x at the index of 1 minus x at the index of 0 giving us again the same answer,


the slope is 2.


But what happens when we have a nonlinear function like 2x squared which makes a graph


like this?


Depending on which two points that we use the calculation for the slope is going to


vary.


So how do we measure the impact that x has on y in the case of a nonlinear function like


this?


Calculus proposes that we use the slope of the tangent line.


The tangent line is the quote-unquote instantaneous slope between two points that are quote-unquote


infinitely close on a curve.


This instantaneous slope is also the derivative and it should be noted that this can only


be done on a continuous curve so no sharp corners or anything like that.


At least at the moment we have no idea how to get these two infinitely close points on


a curve but we could approximate this by simply adding a very small delta from some sample


point.


Let's imagine that we'll use a delta of 0.0001.


This means that 0.1 would be 1 and 0.2 will be 1 plus that tiny delta.


Now we'll calculate the slope between these two very close points which gives us an approximate


derivative of about 4.


We'll soon learn that the actual derivative for 2x squared at x equals 1 is indeed exactly


The small difference here is because we're using an approximation technique.


This is what's known as numerical differentiation and it produces the numerical derivative.


It's a solution where we just calculate the derivative which is the tangent line's slope


by using two points that are infinitely or at least in our case sufficiently close.


The further away the two points are for this calculation on a curve the larger the inaccuracy


is likely to be so this is why we want to use two extremely close points.


To help with visualizing this in Python we'll modify our code slightly so that our graph


is created with much smaller steps at a time creating a smoother curve.


Finally, for us to calculate this line we need one more element, the y intercept or


the b in the equation for a line which is y equals mx plus b.


The slope of a line is handled in the mx part and for now we're just saying the slope m


is 1 for example but we also need to know how to calculate the b variable if we want


to visualize things.


Since the equation for a line is y equals mx plus b if we already know y and m but we


want to solve for b we can use basic algebra to say b equals y minus mx or using our code


as variable names b equals y2 minus the approximate derivative times x2 so the derivative is the


slope.


Applying this to our existing code we can now calculate and draw a section of that tangent


line which you can see as a green line on the graph on the right.


Taking this a step further we can add a list of colors, move up the approximate tangent


line function and add the approximate derivative and b as parameters.


Then we'll create a loop of five iterations with a 4i in range 5 tabbing over the calculating


and graphing code.


We'll then replace x1 to be whatever i is.


Then we're going to add a scatter so we can mark each point that we'll be doing this calculation


of numerical differentiation with a dot colored uniquely for that specific point and then


finally we'll be modifying our plot function to update with the new parameters for that


approximate tangent line function and we'll be adding colors.


We can then run this calculating the tangent lines and the slopes of those tangent lines


for each point along the way using numerical differentiation.


So again what's the point of all this?


In order to intelligently know how to adjust parameters in a function to make some desired


change we need to be able to have some way of calculating the impact of those parameters


on the output of the function.


Now a slight issue for us is that the actual function in a neural network is not so simple


as having a single variable.


Often times we're attempting to adjust millions or more parameters.


Calculating derivatives using numerical differentiation would require multiple forward passes for


a single weight or bias adjustment per sample.


This is because for each sample we'll do a forward pass calculating the loss at the end.


Then for each weight and bias individually one at a time we'll add that small delta,


do another forward pass calculating the loss again.


Then we'd revert that parameter back to the original state, choose a new one, add the


delta, repeat the forward pass and continue this for every single weight and bias in the


network.


Note that here we're only displaying the deltas and passes for each weight but this would


also be required for each individual bias too.


And again we'd need to be doing this process not only for every single weight and bias


individually but we'd need to be doing all of that per each sample.


This would still most likely be a faster and more successful approach than doing things


randomly but we can still do a little better.


We're definitely on the right track with derivatives but using numerical derivatives in this way


would be similar to brute forcing the problem or something like that.


It would work but it would be slower than ideal.


For multivariate functions instead of doing something like multiple forward passes for


our neural network we can use partial derivatives which are indeed coming up.


But first we need to cover the analytical derivative which is the subject of the next


tutorial.


As you may be aware if you are enjoying the content and want another medium to absorb


the material with you can check out the Neural Networks From Scratch book which is fully


complete for making, optimizing, and testing both classification and regression neural


networks.


You can get the eBook, hardcover, or softcover at nnfs.io.


We've now sold over 10,000 copies at the time of this recording which is pretty staggering.


Thank you to all of you who have chosen to spend your time and or money on learning this


NeurIPS 19 Poster Session

Hi there, we are here at the NURBS 2019 poster session, one of the poster sessions specifically.


NeurIPS 19 Poster Session

There are two poster sessions a day, three days, so this is day two of the first poster


NeurIPS 19 Poster Session

session.


NeurIPS 19 Poster Session

It's technically lunchtime, so most people are out, but you can see there's still so


NeurIPS 19 Poster Session

many people here.


NeurIPS 19 Poster Session

There are about 250 posters in this room, and every poster has a ball of people around


NeurIPS 19 Poster Session

it.


NeurIPS 19 Poster Session

This is not peak time.


NeurIPS 19 Poster Session

Yesterday they didn't even let people into this room.


NeurIPS 19 Poster Session

That's the only reason you come to the conference to actually talk to the people doing the work,


NeurIPS 19 Poster Session

but it's almost impossible because they're constantly trying to explain their work to


NeurIPS 19 Poster Session

about 20 people at a time, asking any meaningful questions, getting into a conversation.


NeurIPS 19 Poster Session

It's almost impossible.


NeurIPS 19 Poster Session

It's about 10 degrees warmer in here than outside.


NeurIPS 19 Poster Session

It is sweaty, it smells, it's absolutely beautiful.


NeurIPS 19 Poster Session

I don't know, there is a kind of a feeling in the air that this is a bubble.


NeurIPS 19 Poster Session

Just the sheer amount of people attending this is crazy.


NeurIPS 19 Poster Session

I don't know what this looks like in a few years.


NeurIPS 19 Poster Session

Maybe this is peak, or maybe it's just going to grow and grow and grow.


NeurIPS 19 Poster Session

I don't know.


NeurIPS 19 Poster Session

You can see what it looks like, and maybe I've described well what it feels like to


NeurIPS 19 Poster Session

be here.


NeurIPS 19 Poster Session

With that, I am going to dive in.


NeurIPS 19 Poster Session


All about AI Accelerators: GPU, TPU, Dataflow, Near-Memory, Optical, Neuromorphic & more (w/ Author)

Hello there, today I'm talking to Adi Fuchs, who is an expert in AI acceleration technology.


We talk about a whole bunch of things in this interview, but it is a little bit of a special


thing because it's not about a paper or anything, but it is about a series of blog posts that


Adi has authored.


I am very much a noob in the AI accelerator field, so I thought it'd be really cool to


talk to someone who really know what they're talking about, who are in this industry and


explain everything from very technical to very noobish for me.


We go over a whole bunch of things like why do we even need accelerators, what are the


reasons behind it, why are GPUs here and why are they good for AI, up to very, very modern


approaches to AI accelerations, TPUs and beyond that.


If you're interested in this, watch the interview, it was very cool, I learned a lot and I hope


you do too.


Without further ado, have fun.


Hello everyone, today I have Adi Fuchs with me right here.


He is the author of a series on Medium called AI Accelerators and I have noticed in the


last few years and certainly months that I have no clue about hardware.


My conception of hardware is something that goes vvvvvvvv and if I want a neural network


I need like a GPU that goes vvvvvvv and then there's TPUs and then there's IPUs and there's


lots of stuff, but I never had any clue what any of it meant.


So this article series was really valuable to me and I thought maybe it's valuable to


some of you too.


So Adi, thank you very much for being here.


Yeah, thanks for having me and thanks for the kind introduction.


Can you tell us a little bit about how, what your background is in this space?


Why did you decide to write a series like this and why did you think that you had the


knowledge to do so?


Well, so I've been back and forth between I would say industry and academia.


I've been working for several hardware and software companies, you know, Philips, I also


worked for Mellanox, I also worked for Apple for some short period and I've been back and


forth.


I did my master's back in Israel and then I did my PhD at the US at the Princeton University


and I always, you know, my studies have been mainly focused on computer architecture.


You know, more recently my experience has been with computer architectures, processor


architectures in general.


There's a lot of software going on into it but, you know, from the architectural perspective


is how you can design systems that are, that can execute these applications very efficiently


and there's a myriad way of actually doing so.


So after my studies, I started working for one of the big companies in the landscape


and I said, actually when I graduated, I had, when I graduated my PhD, I always had in the


back of my mind that AI and machine learning and deep learning, all that has been very,


very exciting.


You know, I took just like one or two classes but I didn't really have any extensive experience


in it but I do feel like I do, I was able to see that potential and I wanted to say,


okay, one of the natural things for me after I graduate would be to work for one of those


companies that are developing hardware for AI but, you know, the story goes well beyond


just hardware.


You know, people right now understand that they need to develop smart systems, smart


software.


It needs to be a full stack view, just going beyond just like you said, the GPU that goes


vvvvv or the TPU or the underlying processor or whatnot.


So the landscape seemed to be very exciting.


It's rapidly evolving.


There are a lot of solutions out there and I thought that, you know, as a hobby, what


I did, it's just started as a hobby and you're just observing what people are doing, trying


to look at the competitive landscape and try to see if there's anything that could be interesting


for someone that wants to know more about that world, either be it a research scientist


that wants to know a little bit of what's going on under the hood or people that are


hardware engineers that wants to know a little bit more about, you know, the high level motivation


for why people are doing AI accelerators.


So I was hoping that I will be able to create something like that, that will be able to


contribute to several types of people, I would say.


Very cool.


You, so my question is a little bit, why, what does it even mean to build hardware for


something?


Like, obviously, you know, we have computers and I can, you know, I can do pretty much


anything with a computer.


What does it mean to say, make hardware for AI?


You have this term of user to hardware expressiveness.


What does that mean?


So I would say it's, as I said, there is, it's more of my term and lack of a better


term.


I would say that probably people have several either academic or industry more accurate


ways to depict this is that the user knows on the high level what they're doing, what


they want to do, what type of models they want to explore and how they translate it


to high level code, you know, like cafe, PyTorch, TensorFlow and all that.


So the research scientist has the big model that they want to explore.


But under the hood, there is what the hardware understand it, what it can execute.


So if you look at it, you can see that there is a lot of layers that you need to go to


get you need to lower from the high level code all the way to the, you know, to the


bits that are basically executing on, you know, the electrons that are flowing.


And it gets really, really complex because you need to have a full stack view and really


know whatever crazy idea that the user is doing and whatever.


And the last low level detail of everything that your hardware basically can execute.


You know, it are degrees of parallelism, how it accesses the memory, be it DRAM, high bandwidth


memories, HBMs.


There's a lot of things that are going on.


What are your precisions?


Are you doing FP32?


Are you doing FP16, BF16?


Are you doing integers?


What is your bit width?


And there are a lot of details that someone needs to understand in order to build a full


flesh, fully capable compiler stack that you can basically write whatever you can think


of and it'll out of the box be not only working because as you said, you can basically compute


everything right there.


I don't know, Church Turing thesis, a computer is a computer, but there is a difference between


just solving the problem mathematically or accurately and actually doing it in a performant


fashion because you can either solve a single problem and it will take a month to run or


you can solve the same problem and it will be more efficient.


It can take, I don't know, like a few hours or even a few minutes.


So that's the idea of user to hardware expressiveness.


The user can think of whatever and the hardware can execute whatever and you need to bridge


that cemented gap between them.


And okay, let's say we agree that we need to build hardware for AI.


You go through a little bit of the history of that, I guess, starting with what everyone


knows, which is kind of Moore's law that processors or number of transistors increased over time


in an exponential fashion, but then you go into some less known laws like Dennard scaling,


all of this leading up to saying, you know, we've reached the end of clock frequency.


I think this is also known.


What's also known is probably that we have replaced essentially speed with number of


cores and we're going to parallelism now you draw an excellent comparison to GPUs here.


GPUs being the current super many core architectures or not current, but in the history, they had


more cores.


What makes GPUs so attractive for AI in the first place?


Yes.


So this I think this goes back a little bit to more of the intro.


You know, you're just saying hardware and you're saying computer, but the fact that


you can compute things at certain speeds have been key enablers.


I go in the introduction, I'm talking about AlexNet, right?


You see in the AlexNet paper, they say in the abstract, we were able to develop a GPU


implementation and efficient GPU implementation that allows us to number to crunch a lot of


data and train a lot of data within a reasonable timeframe and get a super fancy model that


can run efficiently and within reasonable times.


And that basically was a key enabler.


What I didn't even mention is that for example, for natural language processing, the same


story happened.


If you look at the attention is all you need paper, they were able to say in the abstract,


we were able to train it on GPU for three and a half days, which was order of magnitude


faster than previous solution, you know, all those LSTMs and RNNs that have this inherent


sequential part that we were able to devise a new architecture that is able to run on


hardware.


And just by being able to harness the power of GPUs, we were able to run and it basically


unlocked our capabilities.


So the ability of hardware has been, the role of hardware has been very significant and


basically being the key enabler of AI capabilities.


And that's why I think this series is very important.


Going back to our discussion, you know, trying to talk about frequency, it's good to know


about the history because when you're talking about AI accelerators is essentially why do


we need accelerators?


And why now?


So as you can see, as we said at the beginning, there was frequency.


We were able to get our circuitry going faster.


You can say that, okay, we have, back at the nineties, you can have like this 486 going


at 33 megahertz all the way to like a hundred megahertz.


Then came the Pentiums and people will say, yeah, I have like 300 megahertz and then you


go to like a gigahertz.


And then ultimately going to the Pentium four with like three or four gigahertz back at


the time, you know, during that time, people understood that because you're not able to


do the NART scaling, you know, that the NART scaling, what I mentioned there is the actual


real problem, you know, going beyond Moore's law.


The NART scaling says that it's not only that you can have smaller transistors, they can


also go faster and you can cram more transistors and you can have like, if your dimension scales


by K, you can have K to the squared number of transistors, each one will be K faster.


And the key enabler there was that you were able to, you know, to lower the voltage by


that factor.


The thing is, back at the 2000, the voltage stopped scaling at the rate that you were


able to increase the frequency.


So you can get faster circuitry, but your power density essentially increases.


And that's where you can see that the graph that increases and then people say, okay,


we cannot have faster transistors.


So that was the first stage in the evolution cannot have faster transistors.


You can see like the green dotted dot is basically plateauing and say we cannot, so the implication


is that we cannot have a single task going faster, but as Moore's law saying, we can


still have more transistors.


They just cannot go faster.


So instead of having one task going fast, we're going to have multiple tasks going at


the same speed.


So instead of, you know, increasing the frequency twice, we'll have twice the number of cores.


And depending on how we can map the problem, how efficiently we can map the problem, we'll


be able to still get 2x by essentially parallelizing.


And that was phase two, which is essentially the multi-core era.


So you're able to cram more transistors, they'll be able to getting on the same silicon wafer


or the same silicon die.


You'll be able to get twice as many cores.


And as you can see here, the green line, especially for GPUs as the main beneficent, you're saying


let's develop these instead of having this design, which is the CPU, which has all sorts


of very sophisticated mechanisms like stuff that there are branch predictors, pre-fetches


and all these speculative things that are saying we can execute an instruction, but


this will take too long.


We can do out of order execution, but doing all sorts of tricks to make a single stream


of instruction go fast.


Instead of it, let's do, let's re-devise our software a little bit and break these, the


stream of instruction to several independent stream of instructions that are called threads.


And we're going to be able to run them hopefully in a perfectly parallel fashion on different


what we call cores and each core will execute its own stream of instructions.


So essentially we'll break up one task into multiple subtasks.


And by that, we'll be able to still get the same degree of speed up.


If we'll be able to get it to be able to get like 2x tasks, we'll be able to get a speed


up of 2x.


Obviously there's a lot of difficulties, but that's the main idea.


So we'll be able to, so eventually if we have enough parallelism, we'll be able to get to


hundreds or even thousands of cores and we'll be able to get hundreds of thousands of speed


up compared to our regular tasks.


But at the mid, I would say the beginning of the 2000, around 2010 and 2011, there were


two different works that highlighted the same phenomenon.


It's meaning that because the NART scaling, again, we're not able to scale the voltage,


just having transistors powered, not even doing computation, it doesn't matter even


at what speed, just having them powered on will increase our power density, meaning Moore's


law is still working.


We can still shrink down the transistors.


We can still cram more and more cores into the same silicon square, square millimeter,


you know, in the same skeleton area.


We'll be able to get more cores, but the power at that time will not remain constant.


So the power also increases.


So that will be unsustainable.


And this has created the phenomenon that these works are talking about that is called either


the utilization wall or dark silicon.


It means that, you know, you can have, let's say a million, it doesn't matter that you're


going to have a million core with micro transistors.


It means that not all cores can be turned on at the same time, meaning for the purpose


of your computation, you're going to remain under a fixed budget just due to power constraints.


So basically what it means that you're not going to be able to get more transistors.


And at this point,


And so the power constraints are mainly due to us not being able to cool down a, like


a thing that consumes more power.


Yeah.


Or what are the constraints there?


So the constraints is that the power density, the power density, you know, the watt per


millimeter square just starts growing exponentially as you start exponentially cramming more transistors


because the power per transistor stops scaling.


It remains constant.


So you'll have thousand X transistors, you'll have thousand X the power.


And that creates a problem that will be unsustainable and that will require cooling that either


does not exist or is super expensive to manufacture.


So and that created a problem that essentially says that, okay, we're not going to be able


to get more transistors.


So if you're not going to be able to get more transistors, then came the notion of building


accelerators, meaning that instead of having a single piece of silicon solving a wide range


of problems, you're going to be focused on a little bit of a narrow scope of certain


applications.


And those application needs to have some properties.


So and that's the idea.


If we're not going to get more transistors, we're going to be able to create smart purpose


built circuitry with purpose built compute and memory and communication that is basically


targeting specific problems.


You can see an example like video encoders, Bitcoin miners and AI.


So you can see there if you look at more general purpose processors, if you can look at power


efficiency or even performance, you can see that the general purpose processor does fairly


well for a wide application range.


But those accelerators are, for example, for FFT or graphs or matrix multiply, they're


really good at a certain task, but they do really poorly on something else.


For example, you cannot run your operating system or it wouldn't be recommended for you


to run your operating system on an AI accelerator.


Well, wait, wait, wait, just wait.


Like the community is going to figure it out.


You just need to scale enough.


But I guess I think from this point on, it's sort of common, let's say common knowledge


again that GPUs were purpose built for graphics, but inherently that meant kind of matrix multiply


things together.


And then on the other hand, deep neural networks just by happenstance, by being convnets or


feed forward networks, also using a lot of matrix multiplies.


And I guess that was just, you know, how the universe works.


These things came together, and that was just a really neat fit.


And the point though, is the GPUs weren't made for AI in the first place, even though


it seems to be like a really good application for them.


How you know what's what's GPUs are good for AI, but what can be even better, right?


Like in which in which places are GPUs still suboptimal for the AI things that we are doing?


Well, it really depends on your applications demands and the application scopes.


For example, you can see like in the map that you're showing here, you can see that GPUs


are really good at flexibility, and they're really good in having matrix multiply, as


you can say, linear algebra is something that GPUs do pretty well.


And if you can map a lot of these problems, like a lot of cons and, you know, recommender


models and all that, you can map them into a GPU and do dense linear algebra pretty well.


That will give you a fairly good boost.


But if you would could devise a certain, you know, if you would go all the way to the efficiency


and doing something really, really specialized, you'll be able to say, let's develop an accelerator


that just does ResNet, for example, that'll be really, really contrived to collapse to


a certain a certain type of network.


Theoretically, everything will be hardwired, even the weights and everything will be perfectly,


perfectly fit for that, but it would not be able to execute anything else.


So if you if you would be Yeah, it'll be very, very bad in doing other more general purpose


AI.


So that comes the question, you know, what, how can you trade flexibility for efficiency?


For example, one of the things that some of the companies are that are not GPU based companies


are tackling are these big, these large language models, for example, those GPT threes and


all that.


And GPUs, if you look at the A100s, you can see that GPUs from the from I would say that


it was a conscious engineering decision for Nvidia to go for high bandwidth models, high


bandwidth memories, I'm sorry, that are basically fast memories, but they're limited in capacity.


Alternatively, you can go for something else, you can go for a slower DRAM based memory.


So HBMs are fast, but they're limited in capacity.


And DRAMs are huge and have like, terabytes versus, you know, dozens of gigabytes.


And if your model requires terabytes of data, you would need hundreds or even 1000s of GPUs


just to be able to have the same to do everything in memory, you know, to have the same to


map the memory space of your model.


And that would be something that, you know, I'm not saying that GPUs can do but it would


require a lot of GPUs turned on and a lot of power and a lot of communication going


from different from different GPU systems to be able to train a single, you know, like


hundreds or hundreds of billions of parameter model.


I mean, that's exactly what we see, right?


Okay.


So yeah, I guess we can we can just dive into what kind of, you know, hardware that goes


beyond GPUs exist.


That is to say, in part three, okay, in part three of your series, you go into a little


bit of the architectural, sorry, foundations, and you describe kind of what what exists,


you know, what instruction sets are, what kind of models exist, for example, reconfigurable


processors, you, you, you make sort of a good, very extensive background overview, which


we're going to skip right now, just due to time, I just found this very, very funny.


I guess that's why you posted it here.


So there is, this is a single instruction on that I can use on an Intel processor that


computes approximations to the reciprocal square root with less than two to the negative


28, the relative error of the PAC double precision floating point values from these things, and


stores the result in that thing with right mask k1.


That is excellent.


Like I need I need that instruction every day.


Yeah.


But yeah, depending on the way that this is basically showing how you can devise when


you look at a processor, you know, the traditional, the traditional model of processor is called


a for Neumann model.


It's you're saying that you're you have a processor, your processor accesses the memory,


your processor fetches an instruction from the memory, it decodes the instruction and


says, Oh, yeah, we should do this and that.


So this instruction accesses the memory and loads.


It fetches the next instruction and all that.


So the instructions are basically built from an ISA, which is the instruction set architecture,


which you can think about it as the vocabulary in which the the processor says that the processor


supports some processors support x86, some processors support arm.


And so which, which is I would say, like, the x86 is an example of what we call a complex


instruction set computing or CISC and arm is the risk.


So there was a trade off between, you know, how much you're going to be able to, to have


a single instruction, you know, compact nicely, which will take less memory.


So you're going to have a large vocabulary to express more complex computation versus


the risk, the reduced instruction set computer like arm that is going to be basically be


translated to a lot of a lot of micro instructions that will be simpler.


So that was an ongoing discussion.


But you know, this, you know, this gives a background of how basically a processor works.


So there are a lot of concepts that I showed at the part three that were basically used


as the background for part four.


You know, historically, I wrote part four as the combination of part three and part


four, but someone said, but you know, a lot of people just advise me that this is just


going to be super long.


So I needed to break it down.


So


Yeah, so if anyone if anyone wants wants the background, this article is is really nice


on sort of the foundations of all of this, if you if you want that.


And I think people can relate a little bit because in NLP, you have this whole tokenization


problem of you know, how big do you make your vocabulary and if you make it too small, you're


gonna have to break down stuff into smaller pieces and so on.


Just I think it's, it's approximately the same concept right here, you're trading essentially


memory for for for speed.


And also the thing is that you need a difficult, you need a very smart compiler to look at


your code and say, okay, these sequence of if for example, if you're writing in C, so


these sequence of instructions are going to be translated all to that single instruction.


And that way, you'll have a smart and very, very complex compiler will be able to map


your sequence of operation into that.


Sometimes it works.


And sometimes you're just going to have like these ghost instructions that no one's really


going to use.


So here in part four, I think that that is it is the longest part and you dive into the


various companies startups that exist today, building AI, AI accelerators or AI hardware


in any form.


And it is we have to say that you are associated with one of those companies.


We're not going to say which one though, obviously with the best one.


But I felt I felt reading the article that there was no there was no, I didn't feel any


favoritism so I was I was pretty happy to see that.


Now we have a lot of them even discussed in your articles.


Do you maybe have some that you want to, you know, want to highlight in particular to just


maybe show the diversity of the field and where it's going?


Yes.


So while there are a lot of solutions out there, I would say most of them stem from


a handful of a few architectural ideas that were highlighted in part three.


So I would say that there is originally there's the GPU with the CUDA that has dense linear


algebra that is basically has this model, this execution model, single instruction multiple


thread.


It's the idea of the classical Phan-Noy-Mun model, you have instructions, they're translated


to processor level ISA that the instruction set architecture that Nvidia GPUs understand.


And it's being parallelized and it and you know, it has all these, you know, systolic


like execution.


And a systolic array is an idea that dates back to the 1970s, where you're going to have


a single piece of hardware that is really good in doing matrix multiply because the


data when you're doing matrix multiply the data from the A and the B matrix is basically


flowing like that.


And if you have a very smart circuitry like that, which is in a sense, a smart arc accelerator


like engine just for matrix multiply, it'll be able to carry out matrix multiply really


efficiently.


So yeah, so the GPUs have that.


And you can say that there are some other companies that I would say that are in the


camp of VLI, a combination of what we call a VLIW, a very large instruction word, where


you're going to have a heterogeneous array of compute machines like a memory compute


machine, a vector compute machine, a matrix multiply, and maybe, you know, some sort of


a linear compute machine for your reuse or or tangents operators and whatnot.


And you have a static compiler that basically creates this huge instruction that says, okay,


this data goes to the vector unit, this data goes to the matrix multiply, and this data


goes to the vector unit, and you're able to, and you know the timing of all these units,


and you'll be able to have a smart compiler that statically creates this single word that


is going to be fed to all of them.


So you can have at compile time, a smart compiler that will be able to efficiently schedule


these different data or operands to these machines, and they will be able to get really


efficient execution.


So for, I would say the systolic slash VLIW camp, I would say things that are, I would


arguably the most famous example is the Google's TPU that was presented at, I would say, mid


2017 at a conference called ISCA, the International Symposium of Computer Architecture, which


is the biggest computer architecture conference.


So they showed a model that is basically the TPU is based on a big systolic array execution


with a linear unit, and this smart memory and everything is being fed and they have


a smart compiler that translates AI code for, that is able to execute DNNs, these deep neural


nets, and that was the first time, arguably the most famous non-GPU AI accelerator that


was presented.


So you can have, you have the Google TPU.


You also have a startup that is called Grok.


Some of its founding members were part of the Google TPU team.


There were architects at Google that took parts of, that took some of the ideas of Google's


TPU and created a more commercialized accelerator for deep neural nets.


And also there is Habana.


So I would say Google, Grok, and Habana are, I would say, the camp VLIW plus systolic array


accelerators.


I understand this correctly.


Essentially they have a chip or a board and that has many different, let's say, sub chips


on it.


One is really good at matrix multiplying.


One is really good at doing ReLU, one is really good at whatever, softmax.


So kind of all these operations that we need in AI, they have like specially sub chips


for and then they have a very smart, essentially router that says, okay, you go here, you go


here, you go here.


So I could compute, let's say, I could compute the last layers ReLU at the same time or the


last batches ReLU at the same time that I compute this layers forward through a linear


layer.


Is that, am I approximately correct?


Yeah, this is essentially like you're basically pipelining it.


So if you have like one thing that needs to ReLU and then one thing that needs the matrix


multiply for the conv operation, then it needs to ReLU and then you can feed the next sample


or whatnot that uses a matrix multiply while the other one is already doing ReLU.


So you can do like sort of a pipeline execution and by that you're basically filling up your


compute machines, right?


And by that you're getting better utilization because you're using all of your hardware


at a single point and everybody's happy and your architecture is perfectly balanced because


your compiler is smart enough to understand the program.


So essentially, we're saying we want the purpose built hardware like the unit that just does


ReLU because that's way better than having a CPU do ReLU.


But in order to have the flexibility, we have a bunch of them on a chip and then we have


a router and the compiler that knows how to use that router and the pipeline.


Okay, excellent.


So but that it seems really it's it seems like just from for me now it seems a little


bit still in the spirit of like a GPU of what you said that you you essentially have this


von Neumann model except here, there's sort of pipelining added, there is distribution


to different subunits added, right, but it's still these kind of instructions that are


in sequence and the compiler needs to understand how to translate a program into that.


And as I understand, the other companies here, they're trying to go sort of bit more out


of like out of that paradigm.


Is that correct?


So I would say that the other big directions that companies are doing is the data flow


directions.


So some companies are combining two elements.


One is called reconfigurability.


And the other one is called data flow.


So the reconfigurable data flow, I think that 10th Torrent are doing it.


I think that Samba Nova is doing it.


And then finally, there was a company called Wave Computing that did it.


And there is another company, there was another company called Simple Machines that are doing


it.


So the idea of reconfigurable data flow is that, first of all, if you look at a PyTorch


or TensorFlow or Keras or a cafe program and an AI deep learning application, you can see


that there are different layers, and they're communicating with each other.


So you have a known, a predetermined set of operands.


And you know how the data is basically being communicated between different parts of your


graph.


So in the underlying computation, the underlying computation is basically constructing of a


computation graph.


What does that mean?


Like you can see over there, you have your layer.


And from that, you have another layer that does ReLU, and then you feed it to another


conv layer or ways and do that.


So you have basically something that is not instruction level, but basically more of the


way that your data, you can see that your data is basically flowing between different


layers.


So the idea is that instead of having that data, that program, that data flow communication


graph, you flatten it to the classic von Neumann model, then you try to reparalyze it.


You can start off from this data flow model, from this data flow graph, and you can basically


statically map it via a node.


Again, you need a smart compiler to do that as well.


You need to map it to your existing, to a specialized hardware that is capable of executing


data flow, meaning you can have a compute element that does multiply in here and you


can have another one that does add in here and you can have, you can basically break


down your dense linear algebra to compute unit and you can feed them to other compute


unit instead of breaking down your computation to micro unit, like saying, oh, here's an


add, then, oh, you need to multiply and all that.


So it would be more natural to look at the compute, looking at the computation graph


as a data flow graph and map it to the hardware and you can start it instead of going back


and forth, flattening it to the von Neumann and then reparalyzing it to the von Neumann.


So these companies bets are that this model is more natural.


It's more hardware friendly and ultimately you can have, you can get a better game because


you're able to have a better, more complex understanding of the graph.


You can look at different elements in your graph.


You can have a smart compiler that fully understands your hardware.


It knows the underlying, the number of compute elements and what each compute element in


your processor, in your accelerator is doing.


And from that, it will create a mapping that will essentially go be very static and your


data is just going to flow instead of you needing to manually orchestrated and breaking


it down to instructions.


So one of the main selling points of the existing landscape like GPUs is that GPUs are, they


have a very mature software stack and they're very flexible.


They can program everything from that von Neumann model.


If you can create a flexible enough architecture, you'll be able to basically handle new models


because the main challenge for you to build an accelerator company is that it takes two


or three years to take out a chip.


You need to think about your idea, you need to think about your architecture, all of what


you can execute, and you need to be generic enough because within two or three years,


it's possible that your application has completely shifted away.


And if you look at the mapping of specialized accelerators, if you're here, but your application


space has moved here, you're not going to be able to execute it efficiently.


So you need to be very open minded and you need to be very mindful about being flexible


enough to support this.


One of the main challenges for that is the ability to create a smart enough software


stack that will be able to execute it.


So it's not a trivial task.


So you can take the wave computing case as an example.


Wave computing was a company that was really revolutionary.


They were able to present a commercialized accelerator that does reconfigurable data


flow at the beginning of 2017.


So they had a fancy hardware with 15,000 cores running at 6.7 gigahertz with a lot of engineering


complexity that is able to have both slow memory and fast memory and all that.


But from what I understood that the CEO interviewed and say, OK, we were not able to succeed in


it because it was so complex that going from the basic cases where we were able to showcase


a few kernels, trying to generalize that to more complex and real world application, we


found that our hardware software stack had to solve intractable problems.


And that would become unreasonable.


So I would say that their problem was that they were way, way ahead of the curve.


People are just exploring these problems and they were not able to estimate those difficulties.


They were pioneers, but ultimately it didn't pan out so great for them because eventually


they filed for bankruptcy.


There's also this concept of in-memory compute or near-memory compute.


What is that about?


So there are several notions of how close the compute and your memory should be.


So one form of near-memory compute is saying that you have your memory model and from that


you're loading it to what we call a software control scratchpad memory.


You have small, fast memories.


You can think of it as a processor cache, but they're software control.


Traditionally a processor cache like in the Von Neumann model is basically trying, it


has a heuristic of saving the most recent accesses just because this is the hot data.


And a software-defined scratchpad memory is something that is more compiler controlled


that you know how you're going to be able to access.


One of the guiding principles of devising an accelerator is that you're basically able


to anticipate how your memory and data accesses are going to be like.


You're going to have a handful of basic computational structures that you're going to iterate


over a lot of data and it's going to be really recurring.


That's one of the things that enable you to develop an accelerator in the first place.


So a scratchpad memory is a fairly small and fast memory.


It can be kilobytes, like a megabyte of data that is really close and it sits within the


same piece of, not even a piece of silicon, but within the same core within that piece


of silicon and you'll be able to communicate that data fast.


It will take like one or two clock cycles.


Another approach would be a processor and memory approach.


That's when the processing element sits really close to the actual memory model.


If you're going to manufacture something like a DRAM or something that is called memoristers,


which are memory-based resistors, you're going to be able to manufacture a memory module


that is going to have logic elements inside of it.


You can see those examples like Mythic or one of those companies that are developing


what we call the processor and memory is the idea that you can look at deep learning computation


and you can look at the dot product and from that you can do analog computation and that


will be fairly, fairly complex.


But the idea is that you don't really need to fetch back and forth data from the memory


because it's all within like this special circuitry that sits within your memory module


and you're saving a lot of that energy going back and forth from the memory chip and into


a different chip, which is the compute memory, the compute processing element.


So it's essentially like having a lot of, like given that we already have a lot of cores


that we also have like lots and lots of registers at those cores, but the registers aren't just


for temporary data, but they are actually the memory.


Yeah.


Okay.


So in a sense, you can think about it as the difficulty is that you needed to really change


the memory that you're manufacturing.


That's something that not a lot of companies are doing, but it's a promising direction


because if you have something that is less depending on your transistors, so it's less


prone to the failures of Moore's law.


The end of Moore's law might not be the bottleneck for some of these modules, but there are other


things like you can see that there's like an analog to digital converter, which could


be power hungry and that creates a slew of analog compute problems.


There are also a bit more, let's say, call them esoteric things that you, all of these


were already esoteric to me, but there are more esoteric things.


There's like optical computing and neuromorphic computing and things like this.


Do you have any favorites there or anything that you think is promising and not buzzwordy?


I think that Light Matter is a company that was founded by a few MIT graduates and they


have this idea that light, that representing analog computation via light could be more


efficient than using it, but then expressing it through the digital domain.


It's an interesting problem.


I'm not really versed on the different types of difficulties there, but it's sort of like


thinking about an analog neuromorphic model where the brain acts basically like on analog


pulses.


So this is a little bit more trying to mimic the way that the brain works than you would


go traditional artificial neural networks where you're going to have a BF16 represent


your weights and you can say that is closer to reality and it's also more energy efficient.


But you can say that these are more advanced technologies.


So I would say that they probably have their own set of challenges.


And you never know which one of these technologies will prevail and be the winner.


And what is neuromorphic computing?


I think of the neuromorphic computing as the way that we know it is the form of analog


computing.


You're going to have data over here, you're going to have like the weights that are sitting


within in your memory and your activation is going to be coming from that memory from


as inputs to that memory, you're going to be able to do an analog addition.


And instead of doing that dot product between the weights, you're going to have a single


dot product doing vectorized compute in an analog fashion and you're going to be using


analog circuitry to compute the results.


So it's more of a, I would say it's more similar in theory to the spiking neural network model


where you're going to have like your brain act on electric pulses.


So that's what these circuit, that's what these solutions are trying to mimic conceptually.


And eventually if you look at hardware from the grand scheme of things, you have those


accelerators, these accelerators are good at doing AI.


But if you really want to get into the definitions, you can go, you can look at the Ian Goodfellow's


deep learning book.


It's not really AI.


There's a Venn diagram where AI and inside of it there is machine learning and then there


is present work presentation learning and then there's deep learning.


And from within that deep learning, you can say that these accelerators are good at a


subset of deep learning and a subset of ML that is good at doing matrix multiplication.


They're really good at doing things like conv and transformers.


But is that a general solution to AI?


No one really knows.


You know, you can say that the interesting thing is that because the hardware was a key


enabler, it's also sort of used as a limiter to what you can achieve.


People are saying, is attention all you need?


Is conv all you need?


Could be, but one thing is for sure is that it consists of most of what your hardware


can do.


Your hardware is really good at transformers and attention and cons.


But is that how intelligence really work?


Maybe there is a huge slew of applications that can mimic more human intelligence that


cannot be efficiently ran on hardware accelerators the way that they're built today.


And we're not going to be able to explore it just because we don't have the hardware


for it and we don't have a way to run it efficiently.


So it's an interesting problem.


There is this concept.


People say this, right?


This is a sentiment that's echoed throughout the community that, for example, graph neural


networks, we don't have good hardware for graph neural networks, and therefore, probably


we're not going to explore them as much, which also means that hardware manufacturers, since


we can't demonstrate that graph neural networks are really good, won't build graph neural


network chips.


Do you see this?


Do you see it generally going, let's say, more and more converging on some applications?


Or do you think, okay, we'll discard some of the applications, but also the ones we


have will sort of morph and develop into different variants and so on?


Like how do you see the hardware, essentially the expensiveness of manufacturing hardware's


effect on the diversity of the ideas in the field?


Do you think there is hope to increase diversity even with the cost of hardware?


It's an interesting question.


I would say, obviously, money makes the world go round.


If there is money within these applications, you're going to be able to build the hardware


for it.


The thing is, like we said earlier, hardware has been a key enabler for what you can achieve.


And basically, if you cannot run your application on hardware, it will be hard to create that


ecosystem for that application to be able to justify building specialized hardware for


it.


So it's a bit of a chicken and an egg problem.


If I were to develop an accelerator for a non-Euclidean set of problems, I would first


need to look for the applications for it.


I will need to be looking for that justification for it simply because if I'm a startup company,


I'm going to have to need funding for it.


But if you don't have people that are exploring it just because there's no hardware for it,


you won't be able to find that justification.


So as I said, maybe attention is all you need.


Maybe a con is all you need.


For surely, it's most of what we have right now.


And it would be interesting to see.


I would say that, as I said in the final thoughts, I would think that in the next two or three


years or so, things are going to become clearer and architectures are going to be able to


stabilize just because we understand the problem better.


It will take us four or five years to really converge to a set of common practices in the


way that we're developing hardware, the way we're developing software libraries, and the


way that we're developing compilers.


We're going to be able to have this, I would say, three or four stable software stacks


that are really good at the conv and transformer games.


Will there be other models to create other stacks?


Sure.


But if I were to start a startup today, it will be really hard for me to go for the conv


and the transformers just because this is a saturated field and people are doing it


fairly well and you're basically almost maximizing what you can do in your hardware.


Yeah, you have the last saying here in your final thoughts is, everything old is new again.


Do you want to explain what that's about?


Yes.


So there are a lot of, it seems like there's a bit of, you can say that on one hand, these


models have been the most popular models, those key enablers, those AlexNet and those


ResNets, those attentions and BERTs and the GPT-3s, they all originated in academic papers,


right?


But in the hardware field, things are, there's a little bit more of a disconnect, I would


say that there are a lot of papers, there are dozens of papers presenting new ideas


every year and the top conferences are the ISCA, HPCA, ASPLUS and Micro.


But eventually you can see that all these fundamental, all these accelerators were basically


using ideas originated like 30, 40 years ago.


Processing and memories was, I would say in the 1980s, VLIW again, the 1980s, systolic


arrays, the 1970s, data flow programming is the 1970s.


Processing and memory also like 1970s.


So it's a bit of conservatism because as you can say that a company building hardware


knows at least in the older days where it was hard to get money funding for it, you


would need to really, really justify and really go for these well hashed out ideas before


you would go for those wildcard ideas.


And once you have that, you might be able to explore more revolutionary ideas.


Unfortunately, I think that at this point, a lot of your architectural foundations are


already established.


So you won't be able to explore this crazy accelerators or those things that are really,


really out there.


You'll be able to somewhat integrate it into your existing architecture, but it would be


very daring to go and break your entire architecture completely.


And especially in a very competitive landscape, you might not be able to go for that risk.


You would be surprised, but there are many people in the AI community that say that all


the AI ideas have been had in the 80s and 90s as well.


And there's essentially nothing new under the sun.


But it's a debated position.


It's a debated position.


Well, I would say that for one thing for sure, going back to the attention is all you need


and con is all you need and essentially is what you got.


A lot of these, you know, the basic computational structures are already there.


You know, people are building on the baseline of these architectures simply because, you


know, one, for me as a hardware architect, from my perspective, this is what the hardware


can do.


It even goes back to this academic notion of accelerators.


There's a work called Stream Data Flow Acceleration that was presented in ISCA of 2017 that there


are saying, OK, the acceleratable domains need to, you know, they need to fulfill certain


properties.


They need to have like a fairly confined control flow.


They need to be like fairly repetitive.


You need to know how the data reuse.


You need to know a lot of how your computation patterns behave.


So if you're not going to be able to build an accelerator that completely breaks out


from this common wisdom and breaks out this template, you might not be able to have an


AI model that behaves that way.


Is it true or not?


Could be or could be not.


Maybe we will find out that our existing patterns are fulfilling enough.


I would say that there are a lot of problems within even within the existing architectures


that we were able to fully explore.


Cool.


Is there anything else you'd like to want to give people on the way?


I guess there's not an easy way to necessarily get into, you know, hardware yourself at home


or something.


But if if people want to want to dive, they can certainly go to your articles, which I


think are great.


I will obviously link them in the video description.


Is there any any message you want to get out there regarding this?


I would say, you know, I cannot really say anything about looking at the blog.


Try to look at high level overviews of how hardware and software behaves.


It's really tightly coupled today.


It's a really exciting time to be either in AI or in hardware because it's a really great


opportunity from many aspects historically that you can explore hardware either as a


research scientist, as a data scientist or even a computer scientist.


It's really good to see how all these pieces pan out.


Start looking at the high level overviews and then just deep dive into any of them.


You know, open a computer architecture book.


You know, the old ideas are already there.


Try to look at, you know, the high level white papers from the big companies, you know, the


Googles and the NVIDIAs, you know, and some of the accelerator companies.


Try to understand how your software behaves.


And you might find out that it's really great that you can execute your models much faster


than you have anticipated because if it's going to take for you three days to train


your model versus if it's going to take you three hours to train your model, that's going


to be a key enabler to a lot of your capabilities.


So just try to do all those tweaks.


Try to understand the common practices.


Try to follow programming books and rules and best practices.


And you might find out that you're going to be able to be a kick-ass data scientist.


Excellent.


Well, Adi, it was a great pleasure having you here.


This was, I learned a lot.


Like really, I had no clue before this.


So thank you very much for these articles and thanks for being here.


AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control (Paper Explained)

Hey, yo, where's my money? Well, give me my money. All right, we're going to get into


this video in a second. Today we're going to look at AMP adversarial motion priors for


stylized physics based character control by Xuebin Peng, Cema, Pieter Abbeel, Sergei Levine


and Anju Kanazawa. And this paper is in the domain of control and reinforcement learning,


but it's with a little bit of a twist. So on the high level, this paper trains an agent,


a physical agent, as you can see here, to perform some sort of goal in the case on the


right, it's walking up to a target and punching the target. But to do so in a certain style,


and the style is provided by an expert data set or a demonstration data set. So the technique


that the paper presents mixes two things, it mixes goal achieving reinforcement learning,


and it also mixes adherence to a given style. And the adherence to a given style, that's


going to be the adversarial part right here, because that's learned in an adversarial way.


The mixture of the two at the end looks pretty, pretty cool. So the setup right here is a


setup of goal achieving and imitation learning as we have already outlined. And the way it


works is the following, there is going to be a task and the task can be, you have to


reach a goal, the task can be you have to punch something, you have to overcome some


obstacles and then reach a goal, any anything like this is a task. So the goals are fairly


high level, and they are given, obviously by a reward function. So you place the agent


in an environment and there is a reward function. By the way, the agent here is as we already


also said, is this sort of physical agent that is going to have some sort of a 3d structure.


There is going to be joints that it can move. There's a joint here and one here usually.


So and there's a head. The agent is this physical thing and it's in a physics simulation and


each one of these joints, it can move kind of independently, sometimes free as a as a


ball, sometimes it's restricted. It's modeled very much like a human there are other I believe


other models such as a T Rex, which of course work differently. But you have this agent


and the agent is supposed to reach a goal like somewhere over here, there's a little


flag to the goal. And the way the agent can interact with the world is by putting force


on any of these joints. So it can move these joints in pre specified ways. And that constitutes


the actions. So the agent will observe the state and the state here is given mostly by


it can observe how all the joints are currently the velocity of the of the joints or of the


of the individual parts of itself in relation to itself. So it can sort of feel itself.


And it also knows in which direction and generally how far away the target that it needs to reach


is. So that's the observation space, the action spaces, it can affect these joints. And the


reward function is often modeled in accordance with the goal. So the reward function for


walking to some goal might simply be you get reward if you are closer to the goal. Okay,


so this encourages the agent to go over there. So we work with quite dense rewards right


here. Because I guess the fundamental problems of reinforcement learning aren't exactly the


point here. The point here is, can you teach these things to achieve a goal while maintaining


a certain style? Now, this is the the task and the environment. In addition to that,


you do get a data set. And the data set is demonstrations of a certain nature. So this


is not necessarily demonstrations of how to reach the goal, it can be any sort of demonstrations.


So usually when people do sort of imitation learning or learning from demonstrations,


there is a bit there are some requirements, if you want to do pure learning from demonstration,


of course, the demonstrations need to be how to achieve the goal. And that we don't we


don't have that here. In other cases, you do need the sort of policy or the action of


whoever performed the data set, we also don't need that here, our goal is simply going to


be we have to reach the task while while sort of adhering to the data set in a way. And


this way, we're going to define in a second. So the data set, you can imagine, I think


there is a good demonstration down here, you can imagine the the data set to give you sort


of the style of movement. So in one data set, you can have running movements and walking


movements. And in another data set, you could have these movements that were just the these


actors walk like zombies. And the goal here is to combine the style of the data set with


reaching the goal. Okay, so the combination would look like a zombie walking to the goal,


which adheres to the zombie walk in the data set, and the goal and specified by the task.


Okay, naturally, you're, you're going to model this as two different reward signals. So there's


the reward signals of how much you reach the goal. And there is the reward signal of how


well you adhere to this style in the data set. The reward goal right here is modeled


by classic reinforcement learning. So this is very much very, very classic. Where do


we have it? So you would simply train, I don't even think it's it says here, it's update


G and D, yada, yada, yada. So this is a policy gradient method reinforcement learning, which


means that you do have a policy function, which takes in a state and maybe a history.


And it will give you an it will give you an action. And with that, you also train a value


function that takes a state and will give you a value for that state. Now, the value


function is purely for training the agent, because you do a do advantage estimation with


this value function. But essentially, this is a standard policy gradient method that


you train this part is lower part of the this lower part of the thing on sorry, you actually


trained the whole thing on this reward. But the bottom part you can imagine is it a reward


comes from reaching a goal. The top part gives also gives you a reward. Okay. And yes, I


want to reiterate both of these rewards are used to train the policy and the value in


a policy gradient fashion. So both rewards ultimately, are in this standard advantage


estimation reinforcement learning setting. However, the top reward is calculated differently


than simply do you reach the goal, the top reward is a measure of how close you are in


style to the data set. And that's given by this motion prior. And the motion prior is


given by a GAN by a generative adversarial network. And I'm trying to, to find the formula


here. I think this here is the the best description of it, though it's just a formula. So a generative


adversarial model, I'm pretty sure you're, you're all aware, there is a data set right


here. There is a generator right here, the generator gets some random noise as an input,


it outputs a sample x from the data set, you get a sample x prime or a mini batch. And


then both of these or the these either of these goes into the discriminator model. And


the discriminator has to decide for any sample, is it real? Or is it fake? So the way this


generative adversarial network approaches the problem of specifying which motions are


real and which ones are not, is by looking at transitions. So the data set here is not


images or so like you're used to in a regular GAN. But the data set is transitions. What


does that mean? So in every situation, your humanoid or whatnot is here, and the goal


is over here. And this is one state, this is s. And then the agent takes an action,


okay, the action could be please lift one leg. And how does that evolve? So the new


agent would be kind of here shifting the weight a little bit, and lifting one leg, okay. So


this would be one action, which would lead to a new state s prime. So you have three


quantities, you have the state, you have the action that the agent took, and you have the


new state s prime. Now, you could parameterize the transition either using state and action,


or state and next state, the paper here does state and next state for the reason that in


the data set, in the data set that you get, right here, you do not have the action available,


you can probably guess it, but you do have the state and the next state. This data set


can come from anywhere it can come from human demonstration, it can come from keyframes


made by a 3d artist, or maybe another agent that has already solved the problem. Therefore,


you don't always have the actions available. So a transition is going to be specified by


a state and a next state. And the transitions from the data set are transitions that you


observe in the real world. So these are state next state pairs that you observe in the real


world. And the generator, the generator essentially outputs state next state pairs. Now this generator


isn't a generator in a like in a classic adversarial network. But this here is generated


by your policy interacting with the environment, right? So here's your policy, it interacts


with the environment. And the environment gives you the state and in the next step,


it gives you the next state, right? So by interacting with your environment, you do


get state next state pairs, these are essentially your generated pairs. And the discriminator


is trained to discriminate between whether or not a transition is from the real data


set, or whether it has been generated by your agent. Now, of course, this whole system isn't


back propagatable. And that's why you do train it using reinforcement learning. So the reward


the usual back propagation signal that you would have in a generator right here, you


can't do that. That's why you simply take the output here, the loss of the discriminator


as a reward for the for the policy right here. So in this case, the policy using policy gradient


is trying to fool the discriminator into thinking it into it thinking that the transitions that


it generates come from a real data set, while the discriminator at the same time is always


trained to differentiate between the true data set and the transitions that the policy


generates. Alright, so that gives you a reward signal for the policy. And the other reward


signal comes simply from the environment, as we've already stated. So these two rewards


are then combined with each other and used to train the policy. The discriminator itself,


as we already seen is trained. So this thing here is actually the discriminator, this more


motion prior is trained one hand from the data set. And on the other hand, from the


from the policy, generating actions, and generating transitions through the environment. Alright,


I hope that is a bit clear right here. So there are many components to this, but two


are important. The policy, which tries to at the same time reach a goal and fool the


discriminator. Those are two rewards, there are two rewards are combined. And on the other


hand, the discriminator itself simply gets transitions from the data set and gets transitions


from the policy environment interaction, and tries to train itself to pull the two apart.


So it's a it's a classic two player game. And yeah, that that is what you're used to


from a GAN. Alright, and that's essentially it for this thing. Here is the algorithm,


we generally initialize every thing there is a replay buffer like in a classic reinforcement


learning, which stabilizes training quite a bit. I also mentioned the value function,


which is used for the advantage estimates of policy gradient. So you for m steps, you


collect trajectories using the policy you already have, then you feed the transitions


to the discriminator right here. Now, this here is a feature function of the state. So


you only they have special feature functions, which make the this problem easier. There's


a lot of expert knowledge going into how you build the features, how you represent the


environment and so on. So it's not quite trivial, but I don't I don't want to go too much into


that. You do calculate the style reward according to equation seven, equation seven is simply


the discriminator. It's not the discriminator loss. So the discriminator loss is actually


is this thing right here. They do use a square loss for the discriminator instead of a classic


GAN loss. So the classic GAN loss would be this thing up here, where it's log D minus


log one minus D. Yet they use this square loss that they found to work a lot better


or least square loss, you can see the discriminator is trained to be close to one if the data


comes from the real data set, which is capital M here, and it's trained to be negative one


when it comes from the policy. Okay, so nothing stops the discriminator from spitting out


any number like 15 or three, it's just trained in a least squares fashion to go to these


numbers, which gives you a better gradient. So for this, for these continuous control


problems, often, you have to go to least squares objectives, because which number is being


output is often quite important rather than just a classification. And even here where


it is actually a classification loss, right, which is surprising, but cool. And then the


reward, you know, given a transition is calculated as so this is clipped at zero. So this is


also between zero and one, as you can see here, if the discriminator says one, the reward


is the highest, the reward is actually one. And when is the discriminator one, the discriminator


is one if it thinks that the reward sorry, that the transition comes from the real data


set. So if the policy manages to produce a transition that the discriminator things comes


from the real data set, it gets maximum reward, okay. And if it also reaches the goal, it


gets maximum reward from that part of the reward signal to so the general encouragement


that we give the policy is, you should reach the goal in a matter that's consistent with


the data set. So it should probably pick out things that do both, right? It could try to,


it could try to switch between the two modes, like, okay, let's do a little bit of data


set, let's do a little bit of goal reaching. But it's probably better if it actually picks


things from the data set or behaviors from the data set that also reach the goal in a


matter consistent with the reward with the task reward. So the algorithm just to finish


it goes on. And it says, Okay, so this is the style reward, the true reward is given


by a mixture, a weighted mixture between the style and the task reward and the weights


you have to specify. And then we simply store these, this trajectory in our replay buffer.


And then we use the replay buffer to update the discriminator. And we also use the replay


buffer to update the value function and the trajectory. According to policy gradient,


they point out a few things that are important right here to their algorithm. One of them


they find very important is this gradient penalty. So GAN training can be a bit unstable.


And these gradient penalties, they are a way to stabilize this training. And they found


that simply penalizing the norm of the gradient as it comes out of the discriminator is stabilizing


the training right here. So this is one thing they've they helped they, this is one thing


that they claim is helping them a lot to actually converge. And this tells you a little bit


that it's still quite, quite finicky. They talk a lot about the representation of the


actions right here, the policy here in network architecture, the policy and value and discriminator


functions, they are very simple multi layer perceptron. So you can see like the mean,


the mean of the policy function is specified by a fully connected network with two hidden


layers consisting of 1024 and two 512 ReLU consistent ReLU. Okay, I guess that's a fully


connected layer with a ReLU non linearity, followed by linear output. So the networks


aren't super complicated right here. What's more complicated is the training procedure,


the loss, the regularization constants and the reward engineering. So there is a lot


of reward engineering happening right here. And that's what you find in the appendix.


So the reward, for example, for going and punching something is is threefold. So if


you are far away, it's one reward, if you're close, it's a different reward. And if that


target has been hit, it's a different reward, right? I guess the top line makes sense.


But the others are sort of reward shaping the behavior one. So you want the the agent


to kind of approach the target fast, but then kind of slow down. And also, you know, if


you look at something like dribbling, where there is a ball involved, there is a lot of


reward shaping going on, even in in target location, there is a lot of reward shaping


going on where you sort of encourage the agent to have certain velocities and so on. So this


is important because of the experimental results that they show. And that's where we go back


to the video. Where's the video? Right here. So keep in mind, their point is you're able


to reach a goal in the style of the data set. So this is the simplest task they have, it's


called target heading. And the goal is simply to walk or to go in a given direction at a


certain speed, okay. And the example clips they have are displayed on the right. So the


example clips are of someone walking and of someone running. Yet there is not really a


transition in the data set from walking to running. And the the agent learns to this


transition by itself. So their point is always look, we have kind of simple things in the


data set, we have the individual parts in the data set that the agent should do. But


we never have the combination of all the things. And to kind of stitch these parts together.


That's the powerful thing about this method, which is pretty cool. So here, you can see


at the top right, there is a target speed. And all of these three agents are trained


agents. And the in the same manner, right, and they're all told to reach that given target


speed. However, the agent on the left only has been provided with a data set of people


just walking the date agent in the middle the same but it has only received a data set


of just agents running. So no walking. And on the right, this agent has received a data


set of agents walking and running. So you can see that as the target speed changes,


the like if it's fast, the walker is not able to keep up when it's slow, the runner is not


able to slow down. However, the agent that has the full data set available can not only


match the speed and change its style according to the speed, it can it also learns the transitions


from one to the other. And this these transitions are not in the data set itself. Okay, so the


cool part about this method is it can sort of stitch together the appropriate behaviors


from the data set. Even if you don't provide these specifically to solve the task.


The Yeah, this is the T-Rex. I think this is just to show that you don't have used motion


capture, but you can use it, you can learn from a provided data set of keyframe animation.


And you can also see the there is nothing in the data set about reaching a goal, there's


just kind of demonstrations of the T-Rex walking. And the method is able to adapt this walking


style in concordance with reaching a goal. So you can see that the turning is much like


the turning in the example clips. Whereas if you've ever seen things like this without


without the the examples, these policies that these things come up with are quite weird.


So here's a failure case. And so the difference between this method and other methods is other


methods such as this motion tracking in the middle, what they try to do is they try to


match a given behavior from the data set as closely as possible. So this it's called motion


tracking. Now there is a some sophistication to it more than I'm saying right here. But


essentially, you have a front flip on the left, and then the motion tracking algorithm


tries to learn a policy such that the the behavior is followed as closely as possible.


Now again, this is really good when you have the exact demonstration available from what


you want to do. It's not so good if you if what you have available as demonstrations


is not isn't really what you want to do is just sort of some demonstrations. But there


are failure cases, of course, if you want to copy exactly. So if you want to do a front


flip, and by the way, the reward function here is how closely you match the motion from


the reference motion. So that's the reward function. However, motion tracking does more


than that motion tracking really tries to track the motion itself. While this method


here would only get the reward of tracking the motion. And you can see it doesn't manage


to to actually learn. It more like doesn't try it tries to not fail it. So it reaches


the same end position and that sort of good enough for it. So there is a Yeah, there is


a trade off right here. It's probably also given by how much you weigh the different


components. So here you have a data set of agents walking and agents waving. And then


what you want to do is you want to have a agent that walks in a direction while they


wave the arm, or why they they lift the arm or something. So at the left, you can see


if you only have a data set, if you only have a data set of the waving agents, it's really


struggling moving forward, right that the walking learns it has no demonstration of


walking. So that's a struggle. If you only have the walking demonstration in the middle,


then it doesn't really track the arm movement where it should even though there is a reward


for it, right? Only Yeah, on the right, I mean, this is somewhat somewhat, but it is


kind of able to, to interpolate. So if you if you want to check out this video, there


is another one that actually explains the paper in a short form. This is from from SIGGRAPH,


go check it out. They do have more sophisticated behaviors. So on the bottom here, you can,


for example, see the obstacle, run, leap and roll. So the data set contains demonstrations


from all of those things, but not the things in conjunction with each other. In this here,


at least what they describe in the text in this, this right here, what they have in the


data set is demonstrations of walking and demonstrations of getting up from the ground.


And whenever so the agent learns that whenever it falls over right here, that it can get


up faster if it kind of does this rolling motion right here. So this was nowhere in


the data set. But because the agent wants to go to a get up state, both because that


will go it that will make it go towards a goal. And also because that matches behavior


in the data set, it will learn this rolling motion as it falls down in order to get up


again. So that is that's pretty cool. Also in this strike and punch example, the data


set apparently only contains agents walking or agents punching, it never contains agents


walking and then punching. So the transition that you saw at the beginning is a learned


behavior that wasn't in the data set. So that's, I think, it's a it's a pretty cool application


of and a combination of two things of adversarial learning and of, of learning, sorry, not from


demonstration, because that's adversarial learning of learning to reach a goal. And


it's a good Yeah, it's a good demonstration of how you can combine the two, they have


a lot of ablations, where they sort of show that the impact of the data set makes a big


difference. I mean, you've seen this in the demonstrations. But also here, you can see


that again, in a graphical form. So the locomotion data set contains both demonstrations of walking


and running, while the walk or the run data set only contains demonstrations of either.


And the here is the target speed versus the average speed that the agent does. Now, if


you only have a walking data set, the agent, no matter the target speeds, the agent will


always kind of stick to walking. And if you have the running data set, it can run faster


up here. But if you want it to slow down, it can't really run slower than you require.


Only when the data set contains both things, can it transition between the two and actually


match the running or walking. So what do we think of this? My opinion is it's probably


it's very cool. And it is a it's a good way of sort of bringing demonstrations into the


picture without manually like tracking the demonstrations or copying exactly. So you


just give some suggestions to the algorithm of what it could do. And you do that in form


of a data set, which is something that I you know, like, because it's not as invasive as


telling the agent, you know, you need to match the joint movements and so on of the of the


agent. This enables demonstrations to come in that are of a much broader range, not necessarily


reach the goal, not necessarily even have a goal in mind. So that's cool. On the other


hand, I think it's pretty finicky, because you have to strike the trade off parameter


between the two rewards quite cleanly, or clearly for your goal. Because we've already


seen right at some point, the agent won't reach the goal anymore. If, if this reward


here, if the reward of the style is too high, we already saw this, if you have a data set


of just running, the agent will simply neglect the goal, it won't go slower than you know,


the kind of the slowest run or demonstration or a little bit slower than that, it just


won't change its policy because it needs to match the data set. And this balance seems


to be quite, quite a important hyper parameter. And that also makes the provided data set


here quite an important thing to to have available. So which data set you provide is also quite


important. And lastly, the tasks themselves are the reward of the goal directed task nature,


or in this paper, extremely engineered. And that's what I want to come back here lastly


too. So what they tout, for example, in this walk and punch thing, they say, Oh, when the


agent is far away, it runs towards the target. But if it's close, it only it slows down.


And then when it's really close, it punches the target. And it sort of learns to combine


these different skills. But and which is cool, right, because the transition wasn't in the


data set. But a big part of it combining these skills is because in the reward, you make


the reward different, whether the agent is far away, or whether it's near, you can see


that right here. So these things are reward shaped to a high degree to encourage these


kinds of transitions to happen, which I think is not really practical in a lot of settings.


So it's still to be seen how much this is of practical value in other reinforcement


learning tasks, where you don't have that available. And also in other reinforcement


learning tasks, where maybe the reward is more sparse, and how that affects this thing,


because essentially, if the reward is much more sparse and irregular, now you have a


problem because now the style signal is much more prominent. And that's not necessarily


solved by simply reweighing the style signal. So I'm excited to see what comes out of this


line of work next. It's a pretty cool line, as I already said, it's a good application


of GANs in a different field than images. And with that, let me know what you think


Movement Pruning: Adaptive Sparsity by Fine-Tuning (Paper Explained)

Hi there, today we're looking at movement pruning, adaptive sparsity by fine-tuning,


by Victor Sun, Thomas Wolfe, and Alexander M. Rush of Hugging Face and Cornell University.


On a high level, this paper proposes that you should, if you have a transfer learning


objective and you want to do pruning, you should not do pruning by weight magnitude,


you should do pruning by how much the weights move during the transfer learning.


This yields better results in the very sparse model regimes and is specifically relevant


to current NLP transfer learning tasks such as BERT models.


So if you like content like this, consider subscribing and sharing it to your friends


and as always leave a comment if you have anything to say on this.


Alright let's dive in.


So they say magnitude pruning is a widely used strategy for reducing model size in pure


supervised learning.


So what is magnitude pruning?


Now if I have a neural network, let's say I have a convolutional neural network and


I input my little cat right here and I have a bunch of layers, right?


And now if we look at these layers, each of these layers is going to be made up of these


units of the neurons and the next layer is also made up of these neurons.


Now what kind of neural network that is, is not that important but what is important is


that you have these connections from neuron to neuron and in let's say a fully connected


network every neuron is connected to every other neuron.


In a CNN that would be slightly different but in essence you have a lot of connections


here and these are usually called weights.


So these are the weights.


Now the problem is if I train like this giant neural networks and I want to ship them for


example to mobile devices to my customers then they won't be able to download gigabytes


of models or even like hundreds of megabytes of models, just not possible.


So what we want to do is we want to prune this model which means we want to remove parts


of these weights, a lot of these weights but we don't want to lose accuracy of the network.


So imagine I have a network and that's trained, it's an image classifier, it's here, it's


cats or dogs and I have it trained to a good accuracy.


I want to delete these weights but I want to retain the performance and these methods


are called pruning.


Now what people do is usually they sort of go in a stepwise fashion they say well first


of all I don't need some of these and then they delete some and then they sort of retrain


the pruned network and after that they go again and they say well I don't really need


that one and they don't really need that one.


So they do it in this stepwise fashion until the network is of the size that they want


and the hope is that you don't lose too much accuracy.


So the current the question is how do you select which weights you need and which ones


you don't need and usually this is done by so-called magnitude pruning which means that


look at the weights and the weights they'll have some distribution.


There will be very negative so here is very negative weights and here is very large positive


weights and what you'll say is that okay probably the weights that are very large they contribute


a lot to the signal of the network within the network and the weights that are quite


small they're you know since there's all this noise and stuff they're probably not that


important so I'm going to cut off basically right here and everything that's in here I'm


going to delete.