the authors here propose to simply count the number of word occurrences of state


State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication

of the art, case insensitive, very important in the text, right? It stands


to reason that a higher state of the art count is preferable, of course. All right,


so the second thing, so this might be a bit controversial, the second thing,


significance, and they now make the claim significance is measured by


efficacy. So they simply, the efficacy term, so if your paper is effective at


achieving its goal, you can also say it's significant for the community, because


again, significance should, like, if you have state of the art, then your paper is


significant. If you do not have state of the art, then your paper is obviously not


significant, because why should it matter if you don't have state of the art in a


given task? It's useless. All right, so we weigh it twice, that's pretty good. And


then novelty. Now here they take much of the same approach. They say the authors


probably state this, so how much they use the word novel in their manuscript will


dictate. So here they say, okay, they novel, wow, this is failing me. Hello. How


much they use the word novel in the text will probably be an indication. I


don't think so, though they do do the smart thing of they include, they include


the works, they include the related work section from this, sorry, they exclude the


related work section. They say we make the key observation that individuals best


place to make the judgment are the authors themselves, since they have


likely read at least one of the works cited in the bibliography. I don't agree


here. I think a better method would be to simply count the number of references,


and the lower the amount of references to related work, the higher the novelty,


right? Because if you think, right, if these are current papers, right,


and you place, and your paper is here, right, you'll have a lot of related work,


right? So it's not as novel if you're like way out here, you'll have maybe one


or two related works. So it's way more novel if you have less references.


So this would be my kind of criticism of this, so this novelty thing here, I


think this term should be replaced by kind of a graph centrality measure, or


something the account of how many references you have would be enough,


probably. Alright, so they define their score. Their score, as we saw, is the SOTA


term weighted twice, a geometric mean between that and the novelty term, which


I've criticized, and they add the suffix out of 10, because out of 10 score is


pretty interpretable. So they divide by 10 here. So, sorry, yeah, they say that


here, that we attach a suffix out of 10, because that's easy to interpret, and as


you saw in the kind of archive implementation right here, sorry, this


will be then easy to integrate right here. So they even give code, right? They


give code in the paper themselves of how to implement this. It's pretty


easy, and I think, yeah, even though it's a quite a short paper, it's


thorough, and it's a good new method, and I think this could revolutionize


publishing. And they even, so as a bit of a bonus, they give the official


pronunciation of state-of-the-art reviewing, which is something like


state-of-the-art reviewing, pretty smooth. And yeah, with that I hope you enjoyed


this, and if the authors could just be a little more subtle next time, that would


Sparse Expert Models (Switch Transformers, GLAM, and more... w/ the Authors)

Hello, today I'm having an interview about the topic of sparse experts. Now, ironically,


the people are absolute experts in this type of models. These models, they are huge, they're


usually language models, but they don't have to be they're usually transformers, but they don't


have to be what they do have in common is this notion of sparse experts. These models go up to


the trillions of parameters, and they achieve this via sparsity. Now I want to do a very,


very brief introduction of what sparse expert models are. And then we'll dive into the interview


right away, because I don't want to keep it from you. So let's look at a transformer model. Usually,


I have some sort of an input that is tokens, a sequence of tokens, which are represented here


by circles. And what I'm going to do with these tokens is I'm going to alternatingly push them


through different layers. Now one big layer type that is common in transformers is the attention


layer. We're not going to talk about the attention layer today, all you have to know is that it takes


in a sequence of tokens, and it outputs a sequence of tokens again, ideally the same amount as went


in, which I failed to draw here. The other very common big type of layer in these transformers is


what's called the feed forward layer. Now the feed forward layer is just a linear layer and every


token goes through this linear layer by itself. So every token individually goes through the same


transformation. And thus as we do this with all tokens, again, we end up with a sequence of as


many tokens as we input. Now a sparse expert model isn't very different than this, the attention


layers commonly aren't really touched. So that works just the same. However, in the feed forward


layer, we see a big difference. Notably, we don't only have one feed forward layer, we have many.


So here is feed forward one, here is feed forward two, here is feed forward three, and here is feed


forward four, each one representing a different individual linear transformation of a token. Now


when we talk about sparse experts, these things here are called the experts. They're called the


experts because they're thought to specialize in very specific tasks. And the goal in sparse expert


models is to route the tokens to the corresponding correct experts. So every token goes through


what's known as a routing function. We're going to talk about this routing function in the interview,


but in essence, it is a very simple, usually something like a linear function or a simple


transformation that decides to which of the experts any given token is routed. So sometimes


even in sparse expert models, a token is routed to multiple experts. But in the newest iterations,


the tokens are simply routed to one single experts and none of the other. Usually this is done, as I


said, by some sort of a linear transformation, followed by a softmax to decide where the token


goes. So every token would be assigned to one expert. And that gives the possibility of scaling


these models up dramatically. Not only do you save a lot of compute, because the tokens only go to


one place, ergo, you only need to compute that one thing for that particular token. But also,


there's the opportunity to massively shard and parallelize these different experts across


different machines, as you only need to route the token to one place, that means you dramatically


reduce these big all to all reductions, they still happen, but not as much. So as I already said,


the biggest models have trillions of parameters, you need to take a little bit of care of how you


then aggregate the tokens once they come out of the experts. So essentially, what you want to do


is you want to carry over the likelihood from the routing function up here. But this is a minor


detail, a minor details are important, but you know, so I know it doesn't look like much, but


these sparse expert models really have the potential to massively scale up our current


efforts in AI. And I have no doubt that they're going to play a role in the near future, when


we're looking at bigger and bigger models, because at some point, the purely dense models will reach


sort of the limit of what's physically doable. And then it's a good opportunity that we have models


that can go even larger. Alright, so without further ado, let's jump into the interview. I hope


you're enjoying yourself. If you do have any sort of comments, please leave a comment, share the


video around if you like it, and I'll see you around. Bye bye. Hello, everyone, my guests today


are William Fedis and Barrett Zoff, who are engineers and researchers at Google, Google Brain,


and have been diving into large models, specifically sparse expert models, which are


models that, well, feature this notion of experts, and also have a notion of sparsity. And hopefully


today, we'll discover what this is all about. Specifically, we'll talk broadly about three


papers in a long line of work. One is the Switch Transformers paper, which was really, I believe,


one of the first papers that just had like massive amounts of parameter. Was that like trillion,


probably trillion parameters? It was big. 1.6 trillion parameters. That's right. Yeah, yeah,


insane. And then there's there's GLAM, which demonstrated really nice scaling laws with these


sparse experts. And more recently, there is designing effective sparse expert models,


which as far as I can see, is also a bit of a of a maybe a summary recommendations, more of a what


we learned type of thing. So William and Barrett, welcome to the channel. Thanks so much for being


here. Yeah, thanks for having us. So can you give us just a little bit of context what


you mean when you say sparse expert models? Yeah, sure. So this is a great question, especially


since the word sparsity crops up in like many different aspects of deep learning, whether it's


like sparse attention or various other sparse paradigms. So yeah, sparsity in our case means


that each input can get different subsets of parameters. So that's kind of like the main


sparsity that we're talking about here. And it's like, you know, it's a very natural concept,


right? Like normally in like a dense transformer, for example, you have, you know, a word embedding


and, you know, any word will have the same parameters and compute applied to it.


And in sparse models, typically what happens is you have the same amount of compute,


but you can have different subsets of the model parameters be like, you know, acting on the model


inputs. And what does that mean in practice? So we're talking mainly about, let's say,


transformer models here. No, is that a good characterization of things? Or do you see sparse


expert models in a more general sense? Yeah, I mean, these things actually almost sort of like


cropped up originally as almost like in the context of like ensemble type methods where you


have a bunch of like almost like fully independent models. And then you're sort of using these as


like, you know, each model is an expert. But the common paradigm as of like 2022 is sort of experts


as a layer. So this was like really popularized by Noam Chazir's work in 2017, outrageously large


models. And in that context, they were actually inserting it in between LSTM layers, which is


like the prevailing like recurrent architecture at the time. Most of the things just because like


the world has sort of shifted towards transformers in it seems like almost all modalities now,


we're often thinking about experts as a layer inside transformers. Typically, we're sort of


doing this at the feed forward. So these blocks that just sort of independently apply on the


different like tokens, but we've also kind of considered it in self attention layers. It's just


sort of like a very general concept. But yeah, typically in transformers. So you have this


notion of an expert, which you say is is sort of a specialized function or something like this. And


then there's often this thing called a router. How does information find its way through these


experts? What are the general principles in that? And why would I even consider doing something like


this? Yeah, so great question. So yeah, so you have this figure up here. And so one thing to notice


that basically, if you only have a single expert, it essentially reduces to just a normal dense


transformer. So the interpretation is pretty natural. And in almost all of the ways people are


doing sparse expert model nowadays, there's some notion of a learned mechanism that for you know,


embedding at the current layer, you figure out what expert you should send this representation to.


And this can be ranging from very simple to just like a simple softmax function over the total


number of experts to very complicated, linear programming type solutions that have a more


like globally optimal solution. So yeah, so this is kind of like the paradigm and I think it's a


pretty natural one. So even if you want to only, you know, yeah, apply one set of weights per


representation, now you have the option of just instead of always applying the same weight matrix,


now you can, you know, maybe have a selection of in this figure for different weight matrices. And


the way that, you know, we've done this in our work, and I think is the most common is just as a


single feed forward network. So you take your input representation, and then you just, you know,


apply it with something that's going to be like, you know, the model dimension by the number of


experts. And then you apply like a softmax function to get like a probability over all of the


different experts. In our switch transformer work, the routing was extremely simple, where it's just


like you just send it to the highest, like the highest expert with the highest probability.


And then, you know, you just simply route it to that expert, then the output of that computation


gets scaled by the router probability. So if it was like, oh, with 0.9 send it to experts,


it to expert two, then when you have the output of that computation, you scale it all by 0.9.


Do I remember correctly that there was some paper that it was this an older paper,


and this might be getting very technical for a second. But was there an older paper that said


something like you always needed to send it to at least two of these experts? Otherwise,


it's kind of unstable. Is that an older paper? Or a newer than yours?


It actually wasn't instability that they're clasping against. It was more this idea that


we're doing this like weird discretized operations. So instead of using like reinforcement learning to


sort of like update on the experts, we're kind of doing this like kind of hacky back propagation


through these like softmax operations, which had been masked. And the idea that top two or greater


was necessary because they were thinking, well, I'm creating a probability distribution for this


token for this word over the available experts. If I don't have at least two, I can't tell whether


expert I or J was sort of better for this one. So it's like, in order to have, you know, the hypothesis


was sort of like a useful gradient signal for the router, it has to know, well, should I have sent


it to I or J? And then we just sort of didn't follow convention and did one and it also seems


to work just fine. I think in part because you're sort of doing this sort of normalization. So you


can still get an up weighting or a down weighting if you select an expert. So it's like, oh, if that


expert selection worked out well for you or worked out poorly for you, you can then sort of adjust the


embedding for that expert. And then you at the next pass, if you saw that same token, you can


still do this softmax distribution. So you're kind of like up weighting or down weighting it.


So I think that's sort of like the gist of the mechanism. And this, I think this idea was at


least from 2017, it may have predated it. Could you maybe now that we're talking about history,


trace the evolution of this line of research a little bit? You already mentioned this existed as


sort of ensemble methods inside of it. I'm talking specifically about sparse experts within


transformers, which are the things that allow us to really scale up to these these giant models.


What's the what's sort of the line of research? What are the original things? I'm going to guess


this this work is among them. And what were the improvements that happened since then in this


field?


Right, do you want me to go you go for him? Yeah, so I mean, like, going back 30 years, like you


have like Jordans and Jacob. This obviously predates transformer because transformer was a 2017


development. So I mean, the concept is very, very old. I think it just kind of like resurged in


popularity. I'd say the first Yeah, the very first sort of use of mixture of experts and


transformer was left in it all in 2020. So this is G shard. And it just showed really remarkable


improvements in translation. What they were doing was, you know, analogous to switch transforming


these other works is they just sort of substitute these feedforward blocks with experts. And in that


case, sort of also similar with switch transformer, they had many, many experts, I think in that case,


it was thousands. And they were showing really significant improvements over state of the art


translation models. I think as the field has sort of evolved, as we've sort of like learned a bit


more about it, there seem to be this like kind of general trend of like, okay, cool, we can


pre train these models or like in the case of translation, there's no big distribution shift


when you're training to translate, you're also doing inference to translate. But in switch


transformer, we found okay, we'll pre train to, you know, improve the perplexity improve the


prediction of next token. And we were getting significant improvements. But then when we took


it under a data distribution shift to fine tuning, it was performing quite badly with many experts.


So I think there's been this trend to try to balance the computation and the parameters a bit


more. So I think some of the prevailing models have actually in transformers have actually gone


towards fewer experts. So 1632 64 experts, not thousands of experts. So that's kind of


like the lineage of mixture of experts and then like mixture of experts in the context of


transformers.


And what is so in that context? If one expert is the classic transformer model, and that seems to


not work as well as many experts, but too many don't work. What is the abstraction that I can


think of for an expert? Like what does an expert learn? What is an expert responsible for?


Approximately? Do you have any idea what happens? Like what? What? How does it make sense that the


optimal number is, let's say, a few dozen and not super many, but also not one?


Yeah, so great question. So yeah, there's like a few parts to this. So one, like, I think it's


really just like an empirical observation right now that, you know, 16 versus 64 versus, you know,


2048 versus 10,000. You know, like, it seems like the expert numbers in the middle, like, it's not


from the standpoint of like, on a per step basis, more experts typically don't make things worse.


Usually it's like better or about the same, but things start to level off. But it's very


inconvenient to have a lot of experts, because it's just like a huge memory footprint. The way


that the models are distributed, it's not really amenable towards typically unless you have like


tons of, you know, parallel cores going. So like, actually the observation where you kind of want


to actually have like a middle amount of experts is a lot of the times actually driven by just the


like practicality of then like training, serving these models. Yeah, in terms of like what these


models are actually learning, like intuitively. So we actually studied this in our most recent work,


kind of looking at, you know, each expert, what are they specializing in? What are they learning?


And interestingly, they kind of specialize in some shallow concepts, which you would think maybe


there would be like only really deep things going on. And it would be kind of hard to inspect them.


But you know, we noticed like, oh, there's like a punctuation expert or an expert that will, you


know, talk about, you know, like proper nouns, which we thought was pretty funny and maybe not


super intuitive for, you know, how. Yeah, actually, if you want, you can switch over to the recent


paper, and we actually have a figure which sort of shows some of these things. So you can kind of


like follow along and see how shallow these things actually are. Yeah. Yeah. So this would be,


this would be different. So you'd you found an expert or in this case, multiple experts that


that focused on the these sort of things. So there's conjunctions, punctuation, verb,


visual description, which is which is interesting, because that's kind of


I want to say like a higher level thing than just the punctuation, right? Counting numbers.


Yeah. How do you make sense of this stuff? Like, what's going on?


I Yeah, I mean, I think we were sort of expecting maybe like a higher level of description, but like,


or like, sort of like representation. It's, I think we've just started started to sort of like


crack and like, look into these models to actually see what's going on. That obviously, like one big


specialization that you're seeing here are these Sentinel tokens. To make sense of this,


we were sort of doing pre training where it sort of fill in the blank tasks. And a blank is sort of


represented by these like little sentinels. So like extra ID 10 represents like, you know,


the blank 10. And we often really frequently see experts sort of specializing on these blanks.


So that's sort of an interesting thing. And I think that also might segue into maybe you want


to actually give an example of like, you know, like, you know, like, you know, like,


you want to actually given this sort of like, you know, observe specialization, maybe you actually


want to make some experts higher capacity, or give them more compute to sort of do things that


might be harder. But honestly, I mean, this is still very early. It'd be interesting for sort


of like, you know, some of the interpretability lens that like entropic has on some of the recent


transformers to be applied to like some sparse expert models. Some questions we've kind of


received are, what is the interplay of expert specialization with sort of like self attention


specialization? And that's honestly completely open. I think we were just sort of putting this


table forth to the community to be like, well, we we started, it's not exactly what we would


have expected. But definitely kind of like a call to dig further and hopefully, like, you know,


further improve things. With the also, I believe that this was Oh, yeah, here already in switch


transformers, this ability to distribute these things across devices that comes naturally with,


with having sparse experts. So sparsity, meaning in this case, I only send stuff to one or a few


experts. And there, there came the ability to shard this across devices. How, like, how practical


is this really to like, what, when would I do something like this? At what point would it become


practical and useful and the best thing to do to communicate across devices for my experts?


Yeah, so really great question. And I actually think this is the reason why the method works


so well, actually. So the standard way I would say people are doing distributed training of these


models is they have, you know, either fully data parallelism, which means like, you know, each


machine has the same set of weights, but different slices of data or a blend of data and model


parallelism where it's like, you know, kind of a mix where certain like, you know, cores have


sometimes different weights or sometimes different data, and then you communicate stuff to make it,


you know, emulate like a full model. But I think experts, one really easy interpretation of this


is like, let's say you have a model and, you know, you're using data parallelism and you have


four different machines. A really natural way to overlay experts on this would be you just have one


expert per machine. And then, yeah, so this is like a really nice interpretation because then


when you have all of your, you know, local data per core, you'd have the router weights replicated,


but then you just figure out what expert they need to go to. And then that's when you kind of,


you know, shuffle all the tokens around to the machines, do all the computation, and then shuffle


them back. And this makes it really nice because then per machine, you actually never have any more


parameters than you would have had just with a dense transformer, but now you have experts.


So it's actually like a really nice way of kind of, you know, thinking about how to design the


models would be like, oh, you know, you have this many cores for data parallelism, just have that


many experts. And that's actually a paradigm that Liem and I use a lot when designing these models


as well. And yeah, I mean, I think as soon as you have this sort of like distributed model where


you're already going across accelerators and devices, you do already have these communication


patterns, right? Like you need to get activations to a certain place. You need to like get gradients


to a certain place. So you already have these sort of like all reduced communication collectives.


An expert model is going to introduce all to all communication patterns. So that can be like a more


expensive thing, especially based on like your topology and the bandwidth between all of your


networks or between all of your devices. But yeah, so I mean, this is something you sort of have to


like kind of empirically test, like, OK, how how much does this architecture kind of buy you in


terms of performance on your task versus the additional cost of all to all communication?


But you will be communicating across devices for these big models regardless to train them.


Yeah. So this is a good I guess a good segue because you can achieve these giant models


like trillions of parameters using these sparse expert models because naturally I can parallelize


these experts. It doesn't cost me really much more compute because any data point or any token only


goes to one single expert. There is always a bit of the let's say the question of how comparable


this is to the dense models. It was it was often I don't know if this is a latent feeling that I get


from the community, but people would rather have the 175 billion GPT-3 model compared to the switch


transformer, even if it is trillions of parameters. What is there some sort of division factor where


I could compare to a dense model or do you think that it's an entirely different nature of a


function that's computed here? Yeah. So this is a really great question. I think there's a lot of


different ways you have to kind of look at this to figure out if a sparse model is right for you.


So I think actually in a lot of applications, if it's like, hey, I want to train the model


with the smallest memory footprint so I can just be using it on the smallest amount of devices as


possible, a dense model will always be better. I think on a per parameter basis, dense models are


going to be performing better. So for those types of applications, I'm like, yeah, I don't think it


makes sense to be using sparse models. If you want to just train the best thing that you can fit onto


your local 2GPU machine or like a 10GPU machine and do really kind of low throughput feeding


in data to this, like not high or anything like that. I think sparse models are good where you're


going to be training a model and you're going to be hosting it on a lot of machines and you're


going to be having a lot of high throughput going through it. So a lot of queries, a lot of stuff


going through it because then things can be batched together and then the models actually become


pretty efficient. So I think that's kind of one lens to look at when you would want to use a


sparse versus dense model. And I think the kind of second lens is that for a given amount of GPU


or TPU hours on a compute cluster, what model will get you the best performance? And I think that's


the lens that we actually would spend a lot of time looking at for pre-training models in this


paper. Like, oh, you have 512 TPU chips and I give you X budget training hours, is a dense model or


a sparse model going to give you the best pre-training performance? And I think our


assessment was that, yeah, I think the actually the Pareto optimal model typically is a sparse


model in that setup. Yeah. And like comparing parameters, especially between a dense and a


sparse model is just totally incomparable. So using like GPT-3 and then like our largest


switch transformer model, it's just wildly different amount of computes in our case.


You can't infer that from the parameter budget. So I don't know what the compute ratio was between


the two, but far different. Our 1.6 trillion parameter model was actually only doing about


as much compute as a billion parameter model. So for each token, it was doing roughly a billion


parameters worth of flops. And whereas GPT-3 is doing 175 billion parameters worth of flops.


So you can sort of tune this and DeepMind has sort of also tried to come up with like a


characterization of scaling properties, far more robust than we've been able to do,


of sparse expert models and try to come up with like a dense model equivalent. So that might be


like an interesting word to sort of like refer to in the future. But really, it's just like,


practically speaking, it's like, okay, I give you these accelerators for this amount of time,


what's like the best model? So that's like probably the fairest comparison.


Have you seen this pathways paper?


Yes, definitely.


That came out, like, how does it how does it play into something like this? Is it going to make


is it going to make this easier? Is it going to make it superfluous? Like, how does the sort of


ability to schedule things heterogeneously across devices? Or does it does it enable new


possibilities in the sparse expert world?


Yeah, so great question. So, so one thing to note is like, okay, so typically, you have dense models


and a dense model, like every input will have the same amount of compute and parameters applied to


it. And sparse models, now you have the same amount of compute, but different parameters.


And I think the kind of natural next step that I think makes a lot of sense to both Liam and I is


that now for each input, you have a different amount of compute applied as well. And I think


pathways is really exciting, again, like you kind of mentioned for like the heterogeneous compute,


where we want to have inputs that might require, you know, different parameters and also different


amounts of compute. Yeah, and I think, you know, a framework like this is going to really open up,


like a lot of really exciting research avenues along that direction. And I think it feels like


a very natural interpretation for kind of where our models are headed for in the future.


Yeah, like right now, it's like our experts are all sort of completely homogenous. They're all the


same size, they do the same operations, pathways, you could be like, oh, this is like a recurrent


expert. This is a huge expert. There's a group of small experts, you could just be a lot more


flexible in design. And like, you know, sort of like alluding to that a little bit with when we're


still looking at the visualization, it's like, oh, wow, a really consistent thing are, you know,


experts that want to specialize in these like, fill in the blank tokens, these Sentinel tokens.


Perhaps that might be an avenue or an area where it's like, oh, let's dramatically increase the


compute here. This is, oh, hi, Kat. This is like an area where we like a lot of extra compute could


really be helpful. And there wasn't really an effective way to do this with the existing


infrastructures before PathWise. Is there is there a Yeah, sorry, that does


lost lost the train of thought. Explain to me a little bit how how GLAM improved upon


switch transformers? Like what's new? What's exciting there? Yeah, so I think GLAM so so one


also thing to note is like, there's kind of a right now division of two different types of model


classes in like language modeling space, I would say. So one is like these decoder only models,


where, you know, it's just, you know, a single set of parameters. And it's like, you're just


predicting the next token, like autoregressively. And this is like, you know, what GPT3 is. And this


is also the kind of architecture that GLAM studies these models in. So the other classes, these like


encoder decoder models like T5, this was also G shard, this is kind of what also we studied in


switch transformer in our most recent work as well. So I think GLAM did a few things. So one,


they really, I think, pushed the scale of these models. So like, well, you know, our original


models, which transformer had more parameters, like GLAM had, like much more compute applied


per token. And they studied these very extensively with decoder only language models. And yeah,


I think their main comparison point was to GPT3 as well. So they were studying a lot in the context,


the few shot and like one shot evaluations, whereas I think a lot of our work actually centered


around like fine tuning the models. But yeah, I think GLAM really like pushed the scale of these,


especially in these decoder only language models and showed that like, yeah, you know,


you can get as good a quality as GPT3 with like, you know, huge computational training savings as


well. And that did a really, a lot of really good work. And then the other one was like,


and that did a really a lot of really good work in that space.


Is there a functional difference between the sparse expert routing or anything around this


in in GLAM? Or is it mainly what you what you said with decoder only and applying more compute


scaling it up? So actually, there is a few differences that are more nuanced than technical.


But yeah, at a high level, you know, there's a routing function and they actually route each


token to two experts. And actually, there's like some of the differences in these models comes from


like how much buffer you give each token, each expert, because you know, you need to have like


fixed batch sizes for all the experts ahead of time. And so what can happen is like, you can't


guarantee that like, there's going to be perfect balancing among all the tokens getting sent to


experts. So like experts can overflow. And there's this key parameter that we call the capacity


factor. That's probably the single handedly most important parameter when designing a mixture of


expert models, because it just has such a huge impact on the communication costs,


computing, everything like that for how much buffer you should have. And yeah, I think a big


difference from GLAM versus our models is they actually use like a much larger capacity factor


than we've used in our other works. But yeah, the routing algorithm is essentially the same.


That is, yeah, I want to get a bit more into the routing algorithm in just a bit. But just to end


this with the last paper that we've previously looked at, was I right in saying that this is


much more often, let's say a general, almost like a review paper? Or how would you describe it?


Yeah, I mean, I think we we tried to make sure like, we're contextualizing a lot of the work. So


we try to make sure the related work was like, pretty inclusive, because I mean, I think the


field is really adjusted and improved a lot in the last two years. But I would sort of


characterize this paper as fixing the two big flaws from our first one from switch transformers.


The first was these models are unstable to train. So you'd be trying to train them to


all of a sudden the loss or just diverge, which thwarted a lot of our issues. Interestingly,


it doesn't seem like the instability arises from a lot of experts. We were consistently able to


train models like our trillion parameter model, for instance, with 1000s of experts, never really


hitting any unstable sections. Really, it kind of came from like high flops or high computation


expert models, even with like few experts, those were highly unstable. And then we had a lot of


problems with high computation expert models, even with like few experts, those were highly


unstable. And then the second thing that this paper sort of fixed was the sort of like poor


fine tuning quality. So we would sort of pre train a model, it would show like really significant


speed ups over a dense counterpart. But then when it came time to fine tuning, say I'm like


super glue or some like other task of interest, it would just be considerably worse. So I think


we're really trying to sort of like kind of patch up a couple of those issues, we identify them in


our first work.


Yeah, I'm always a bit intimidated when a paper has a table of index by itself. Can you can you


go?


This is something that Barry and I discussed. It's like, okay, should we break this up into


multiple papers? Or should this be one? Because, you know, this is like, you know, a lot of work


and you know, this is like something that we discussed, like, maybe in the future, we should


be producing like more bite sized pieces of work.


When you when you talk about fine tuning, can you go a bit into more detail? Like, what was


exactly the problem? How did you how did you also go about fixing it? So I'm not only interested


in, you know, how did how what's the final model like, but what does the process of debugging


something like this and then getting to an architecture or a solution that actually works


look like?


Yeah, I mean, it's sort of this like, very interesting problem of like, you want to,


there's really just like fundamental trade off. And whenever you're sort of doing this sort of


like large scale work, where you want to try to understand and characterize things at a smaller


scale, understand scaling properties, understand understand, like hyper parameter dependencies.


But then you also want to be consistently checking yourself at the largest scales. And


this sort of balance of like, okay, you have this much compute, you have this much time.


Where do you allocate it? Do you do a lot of small experiments? Or do you do a few big experiments?


It's kind of tricky. But I'd say part of our, like findings were the first one was like, okay,


well, characterization is we're not doing better on fine tuning, what's the cause? And it seemed


like perhaps our cause is not that of optimization, it's that of generalization. So if you scroll down


into section four, you can just click on the link, we might be, yeah, exactly. Yeah, so this is an


example that, you know, kind of supports a lot of the trends we're seeing. On the left is a small


superglue task. So this task has only 250 training sequences, so very small. And on the right is


record. So this has over 100,000 training examples. We're showing sparse models versus dense models


in the two things, in the two plots. Blue represents the sparse train eval, and you can see


just very quickly gets to 100%. And it outpaces, in both cases, the small task and the large task


outpaces the dense model getting to 100% train evaluation accuracy. But when in the small task,


we'll see the dense model in red actually outperforming the ultimate performance for the


sparse model in orange, whereas for the bigger task, the sparse model does well. And so we kind


of kept seeing this, like, you know, overfitting issues. And a lot of this was then led us to sort


of like investigate hyperparameters. And, you know, some of the hyperparameters can sort of be


adjusted in a way to make the model like less susceptible to overfitting. So you can use like


you can use like different dropout parameterizations, but also things like batch size and learning rate


can inject more noise, which can also be sort of like a counter to some like overfitting properties.


So we tried and then sort of consistent with this, like a lot of these things were sort of like,


you know, more exhaustive studies at say, a billion parameter scale. We then tried to continue


to sort of like fact check this against our larger model and make sure that these conclusions were


holding. So I think it was just sort of like, you know, the debugging process was, okay, what more


precisely is going wrong? And then like, what are our levers that we can sort of like pull in order


to try to like improve it? But, you know, a bit of art and science, really.


You, so you, is it you observed, okay, we are probably overfitting, because you saw the smaller


the tasks got sort of the worst the sparse models would ultimately perform on the validation set of


those tasks. Did you and you have it's not like quite like, yeah, it's not always like quite so


easy as that. But it's sort of like, you know, directionally, like, I think we have support of


the hypothesis, but it's not like every single small task does poorly in every large task is


great. Yeah, but I'd say directionally, it seems to be a phenomenon we've observed.


You have also a bunch of experiments down here where you actually where you investigate some of


these, for example, dropout probabilities, you also have expert dropout probability, which is


one of the questions I had in that you have a particular architecture, right with these with


these experts. And when I think about overfitting, when in regular transformers, I have kind of


handles, I can use adapter layers, I can only fine tune the head and so on. Did you ever


investigate maybe only fine tuning some of the experts? Like is that the keeping others constant?


Is that ever a thing? Like, would that work? Or, or, you know, can we can we make use somehow of


the fact that we have these different experts, and they're actually different functions?


Yeah, great question. And I think actually, if you scroll down, we did a very naive kind of version


of this not where we freeze different experts. But we you know, freeze all of the experts or maybe


only train all the experts and freeze all of the other parameters. I would say our findings were


this were surprising in a bad way. So nothing, nothing really works super well. So here you can


see like, and this is also we only studied this on super glue, right? So it's far from exhaustive,


but yeah, so one thing we tried was updating first all of the non mixture of expert parameters only.


And that actually performed about the same, which was kind of interesting. It's like, hey,


like actually freezing the mixture of expert weights like seem to perform about as well as


just like updating the whole model. Then when we started to, you know, update only the mixture of


expert weights and freeze all the other model parameters, like the performance was actually


really bad. And there's some we still fully don't understand what's going on here. We have like a


few kind of like half baked hypotheses, but yeah, then when we update only the attention parameters,


things are worse. And we found a slight boost updating only the feed forward network parameters


that weren't the mixture of expert layers. But yeah, overall, nothing worked that well. But yeah,


I think there might be some potential really interesting things of like, hey, maybe allowing


only, you know, a certain subset of experts to be fine tuned. We did spend a little bit of time


actually studying like pruning off experts during fine tuning. So like for a specific fine tuning


task, if your pre-trained model has like 64 experts, can you just take like a subset of like


two, four, eight or 16 of them? Yeah. And we also didn't really get that good of signal with this


as well. Also, some of your recommendations, they actually would be compatible with expert


models too. So you're free to just like fine tune like the top like top logit layer or like


you could add in adapter layers. Yeah, we didn't do anything like really funky like you were


suggesting like, oh, we're only going to expert like update experts like three, eight and 14 or


something. Yeah, my intuition is that probably wouldn't work well. But I mean, I've been proven


wrong many times. Yeah, we tried some like other things that didn't make it to this table or these


plots. And yeah, again, we didn't really see like a significant boost. That said, if you are only


updating like a fraction of the parameters that you get some memory savings. So, you know, some


nice things. Cool. I guess, you know, there's almost an infinite number of things one could


try with these things like distilling experts, like distilling multiple experts into a single


expert. So you have another expert that's again free to do some new tasks once you know that


two experts are converging something like, yeah, it's really interesting, right? A lot of a lot of


we're adding a new experts on the fly. Yeah, a lot of possibilities. And that brings me a bit to


this routing function that we talked about before and at the beginning, which seems to me is a


really crucial part of the system. Yet, as you said before, very often, I've just seen this being


implemented quite simplistically. Maybe there's a linear transform and then a softmax or something


like this, maybe not even maybe there is some sort of a, you know, you have some fixed keys for all


of the experts and then you route according to that. Do you like my intuition would be that this


could be a powerful handle on what's, you know, on my performance downstream this this routing


function, especially also making this different during inference. You know, any any number of


things doing a Monte Carlo tree search at inference time to be as accurate as possible, kind


of something. Do you have an idea on what the power of the routing function in these sparse models


is and how does it work currently? Like what's the most the latest and greatest and how good is it?


Yeah, so this is a really good question, actually, and something we've actually spent a lot of time


about. So I would say actually, in this project, probably the thing I maybe spent the most time


with is trying out different routing algorithms and routing parameterizations. But we ended up


going with the default thing, which I also think says something a little bit about the results of


it. Yeah, so I would say my intuition is that the model actually works surprisingly well with a lot


of different ways you can route the tokens. So, like, you know, we tried a lot of other routing


algorithms. We tried making like the routing network larger. We tried, like, you know,


some fancier ways of actually figuring out where you should send the token to. We tried, you know,


using additional information of like, oh, when you're routing this current representation,


you have access to whether or not like it was routed or like where it was routed before in


previous layers using like word embedding information too. But yeah, I think overall,


it seemed to be, you know, kind of insensitive. We actually did find like one or two methods that


improved things, but they could only be used in certain situations. So it was a bit trickier to


just like replace everything. The current routing algorithm we're using is basically what the


original one was doing, I think in Shazer et al. in 2017 when these kind of things were like really


introduced into the LSTM language models. And I think, you know, our newer work and then also


GLAM as well, we're using these kind of routing algorithms too. Yeah, and also like one kind of


like detail here, it's like, so right now we're sort of splitting out this little box and we're


like, oh, this is the router. It's not really an accurate characterization. It's like, yes, okay,


you're mapping some vector into a vector that has like the same like length as number of experts.


But if you just don't update that matrix, it still works fine, right? Because now just the


represent like the weight matrices below you are just sort of adapting and just piping whatever


activation they need, right? If you freeze the grade, if you stop a gradient through that,


then it's like catastrophically bad. But yeah, I mean, I've also sort of been surprised by the


relative insensitivity to the routing algorithm. Like we've seen like, you know, maybe some small


boost here and there, but it hasn't been super significant. I think you'd probably have a better


sort of like a bigger significance by actually just sort of fundamentally changing like the


architecture. Like maybe there's like some wildly different approach for sort of sparse models that


we're not considering. Maybe we're in some sort of like local min and like these small tweaks,


I'm like, oh, okay, precisely how are we doing this? Maybe it doesn't matter as much. And DeepMind


also explored some other kind of interesting routing algorithms. Like you sort of alluded to


fixed routing algorithms where it's just like, you're not even learning. They've also tried


RL based routing algorithms. And I think it had like actually similar scaling properties.


So again, kind of corroborating what Barrett is saying, it's just like a lot of these things when


we're kind of doing this like per token routing haven't really moved the needle substantially.


That's been our luck. Yeah. And I think another important trend actually is that when we were


experimenting with a lot of these different routing algorithms, we actually found that they


did help models. And maybe when you had like a 1 billion parameter dense model-ish size,


but then like as we scaled up the models, like actually a lot of the times, sometimes the


differences would just like wash away as well. So it's kind of this interesting effect of when


more scale is increased, like it maybe becomes a little bit less insensitive to some of these


decisions. Yeah, I can totally see that essentially that the rest of the network adjusts, especially


if everything is trainable. What I would be excited about maybe is to somehow at inference


time doing something smarter than, because at training time I can adjust to everything, right?


But at inference time, maybe there's something that I could do, especially with regards to


domain shift, domain adaptation, anything like this where I could tweak routing in some way,


but I guess that's also up for future work. So there's a little bit of this, not tweaking the


routing algorithm, but tweaking the capacity factor hyperparameter I mentioned a while ago.


So this is basically the parameter that's going to dictate how many tokens are being dropped.


And one cool thing you can do is you can have some capacity factor during training,


but then at eval time, depending on if you want to use more or less compute, you can use more or


less compute. You can be either dropping more or less tokens and either increase or decrease the


performance, which is pretty cool. And the model is actually pretty robust to having that train


from train and evaluation time. So that's actually kind of like a good lever for depending on if you


want to use more or less compute during evaluation. I think we have a pretty good overview now. I want


to get a little bit into just the future prospects maybe also of this. We already talked about,


now with pathways, we could have heterogeneous things. Could this be pushed to some sort of


limit? Whenever I see a distributed system, I immediately think distributed, maybe not even in


a data center, but across users, across networks. Is there applications to maybe, what was it called?


Federated, some kind of federated computing, some kind of federated learning where I could


somehow contribute with my maybe confidential data, but I could still contribute to a whole


compute process. I'm going to say the B word. Is there an application for blockchain


distribution, something like this? Do you think about sort of the higher degrees of distribution


here? Do you want me to go for it? Yeah, go for it. Me personally, I haven't spent a ton of time


thinking about this, but I do think it's very interesting. There definitely seems to be a lot


of really open problems around this, especially given the growing amount of fragmented compute,


fragmented devices. There's so much compute around here. How can you effectively utilize all of this,


utilize different data and stuff? I think it's a super cool. I think it was going to require a lot


of really interesting research because right now the way we're currently training these models is


it's all synchronized lockstep typically. You're doing like, oh, after each batch you do these


gradients, you send the gradients around and everything. But I think actually maybe the future


of these models when you're really allowing them to be distributed across very different types of


computing, everything might actually now introduce asynchronous training as kind of the new paradigm.


So I think that's a really exciting space, but yeah, I haven't spent too much time thinking about


it personally. Yeah. And I think as it pertains to say blockchain or something,


like I think one problem with these expert models as designed in this way are these all to all


communications. So over this sort of like decentralized like peer to peer network where


it's like nodes are like really far apart, inconsistent sort of bandwidth and stuff,


that could be really tough if sort of your experts were sort of distributed among like many different


nodes in this sort of like unreliable network where nodes are kind of coming and going. Like


right now all our systems are in this sort of like very constrained fault intolerant area where it's


like, oh, all highly internet work chips that are highly reliable. And then so like blockchain would


just have like a whole different set of like kind of problems that you'd have to sort of address,


like unreliability and some of these other areas. Not to say, I think it just like requires some


like additional kind of research, like just sort of adopting the model as is, I think would pretty


poorly map on that kind of computing infrastructure. But I think there's something there that could be


done. Is there work on because I see these works mostly here in NLP, yet transformers kind of


taking over the rest of the world. Is there work on how these experts sparse expert transformers


behave in vision in reinforcement learning speech, whatever? Yeah, yeah, great question. So


absolutely, actually, there's been some really good work applying these models to like the IT based,


like image classification and stuff. And there, it's actually really nice, because then you can


leverage all of the niceties around like people figuring out how to get these working really well


in transformers and kind of, you know, nicely map it over as well. I've all yeah, there's also been


some good work using these in speech as well. Liam, any other things to add on top of that?


Yeah, some I used to do reinforcement learning more full time and some colleagues kind of reached


out about doing like sparse expert models for RL. I haven't seen I'm not familiar with some work.


But, you know, that might be sort of like another interesting avenue. But like, for sure. So


language, vision, speech. I don't know if there's been any video work yet. Yeah, but like high data,


a lot of throughput, those would be like, you know, really good areas. I think video would be


also really promising. Yeah, I really like also the I feel like it feels very natural in these


high dimensionality spaces that you really might want different parameters to be applied, like when


you have a video, like one, I think you don't want to be applying all the same amount of compute to


every frame. But then on top of that, I could see like, actually, you really want to have different


parameters applying to different, you know, things going on in the video, because it's just gonna be


like, wildly different stuff happening. So yeah, I think I think I'm very excited about these models


for video as well. Do you imagine that these models will just essentially right now, they're


competition to dense models, they are they're competing, you're tracking Pareto frontiers,


how much compute, how well are they doing, tackling very much the same tasks? Do you think this will


go on? Like, do you do you think these models might overtake dense models, if we figure out how to


handle them correctly? Or is it is it more like, there's a killer app for each one of them?


Yeah, I think in Oh, go ahead. Yeah, I mean, I honestly think that the future is going to be


adaptive. Like, I don't think there's any way that like, in 10 years, our models are treating all


examples coming in with like the same parameters over and over again, and the same amount of


compute, it may not be this precise sort of like sparsity regime, or it may not be the precise sort


of adaptive computation, kind of like paradigms that have been put forth. But I view this sort of


kind of work of like sparsity, adaptive computation, as kind of like inevitable,


like, I don't think it's going to be considered like competition, it's just going to be sort of


like, integrated into a lot of like leading models. That's, that's my expectation, I'd be


really shocked in like 10 years, we're training like a 100 trillion parameter dense model, and


it's just kind of doing the same thing, like, over and over again, for no matter what comes in,


just seems really strange to me.


What's the future for your particular research? Like, where do you see where do you see yourself


going in the next, maybe not the next paper that you haven't published yet, but


maybe a bit broader timescale? Like what excites you? And what are your next plans here?


Yeah, great question. I mean, I think the thing that really excites me is like what we were kind


of talking about earlier of each input getting a different amount of compute applied. Like,


I think right now the models are working well for each input getting different parameters. And I


think, you know, coupling this with like, adaptive amounts of computation is like, I think,


really where I want to be spending time thinking about in the next, you know, upcoming years.


Is there? Yeah, I don't know, is you have something like Ponder, there's PonderNet and so on,


there's these recursive architectures, or recurrent architectures that, that sort of decide themselves


when to when to exit? Would that be one thing? Or do you simply imagine that each expert is kind of


one is the buff expert, and one is the lean expert, and then the routing function is the


expert, and then the routing function essentially takes care of the different amount of compute?


Yeah, I don't know. This is a great question. I think, I don't know, I could see either approach


potentially working, or maybe you actually want combinations or potentially something completely


new. Yeah, it feels like the space is still, you know, very exciting. And there's like a lot of


really interesting different verticals being pushed. So the space still feels like, you know,


pretty young to me. Okay, last question from my side, what's the connection of this to something


like capsules? I don't know if you've ever thought about the the connection there. But


with capsules, I always think this is abstract, very abstract things, very high level ideas flying


around. And you here have something like very practical, you know, very on the metal thing.


Yeah, there seems to be quite some commonalities. Is there, is that something that ever


came up to you or? In the two years of doing sparsity research, this is literally the first


time. I actually should be going back to that work. I feel like capsules like had a lot of like


really interesting conceptions. But maybe like you're kind of alluding to it didn't like map


super well to the metal. So maybe that sort of like hindered it's like it's used, whereas this


is just like highly motivated from like an engineering perspective. We've had like some


questions like, Oh, what is like the neuroscientific kind of motivation of our work? And it's like,


it's really engineering kind of driven. So it's like, okay, what what will be fast on our existing


hardware? But yeah, I will revisit capsules and kind of see like, Oh, okay, how could we actually


map this a little bit better to the hardware? And like, you know, I think that could be like,


you know, an interesting source of ideas. Is there any any last thing you want to get out to viewers


that they should take away from this work? Any any way that a regular person can get into this type


of research? Anything like this? Yes, a great question. So actually, one thing we tried to show


in our switch transformer work is that these models work pretty well, even if you only have two experts.


So I definitely I don't want people to think that, you know, you really need a supercomputer to run


the models or to, you know, get benefits from having experts, even having I think, as little


as two experts and running models could lead to developing really interesting research ideas,


improving the performance and everything like that. So yeah, I definitely hope that, you know,


more people can continue to experiment and push for these models. Yeah, and then I would say,


like, another interesting trend that I've been following is sort of in parallel to


sparsity in these, like, you know, really large models is the idea of like, well, what if we just


sort of like have the model sort of offload and like, sort of do look ups or, you know, look at


documents and retrieval type methods? I think this is sort of like a very interesting area. And I'd


love to see like, kind of head to head comparisons of like, okay, do we want to try to encapsulate


the knowledge into parameters? Or do we want to just like, keep it sort of like, you know,


parametric, non parametric type thing? And we keep the information kind of written in docs, or,


like, what does the interplay look like? I think that's sort of like another really interesting


avenue, like, kind of comparing these things. Awesome. Yeah, it sounds really cool. I'm excited


to see what the future of these models bring. Yeah, Barrett and William, thank you so much for


being here. This was a lot of fun. I hope to see you again soon. Yeah, cool. Thanks for having us.


Kivy with Python tutorial Part 12 - Adding better Navigation

Well everyone, welcome to another Kiwi Basics tutorial video.


In this video I'm going to be showing you guys how we can get, say from our paint screen,


back to our main screen.


So before it was really easy because we had just gigantic buttons, but now we want to


actually impose another button on this screen, maybe a button here that says go back home.


Okay, and it looks better than that hopefully.


And we can click that and actually go back home.


So that's our goal, and so that's what we're going to be doing in this video.


So anyways, let's go ahead and get started.


So we don't actually have to change really anything here.


You might notice that I've changed, I lost my mouse, okay.


I've changed this to main three now because this is another video for me, but you might


be following along in the same file the whole time.


So here is my main three script, and now to add a button we actually don't really need


to touch anything more in here.


So I'm just going to come down here and within another screen we have painter, and now what


we can do though is we can also, see the painter is basically just a widget, okay.


So we load in painter and then we'll load in a button, easy enough.


This button is going to have a bunch of parameters.


So the first parameter will be, let's say color, the color of the button 0101 for lovely


green, font size we'll say 25, size hint, this is basically just the size of the button


according to the size of the page, so 0.3 and 0.2.


Then we're going to add some text on this button, the text is going to say back home,


and then we're going to say on underscore release, and on release what do we want to


do, basically the same thing as this button so I'm just actually going to copy and paste,


and instead of other we want to go back to main, so you might remember this from a couple


videos ago, and then we're going to position this using post hint, and in fact I forgot


to add our float layout, anyway, post hint and we'll add this in for right, oops, colon


1, and then top colon 1, add your quotes here, picture spacing issues.


Okay, now what we also want to do is our button is just a widget, or well, okay, the painter


is just a widget, and so sort of is the button, but we're going to come down here and now


we're going to reference float layout, colon, and then within float layout we're going to


put the painter and the button, the painter widget and our button, so let's go ahead and


save that now, and let's run that and see how we do.


Next screen, now we've got our paint, we'll just paint high, and then we can click back


home, we head back home, we can go back to next screen and our stuff's even still there,


fancy, fancy, fancy.


Okay, well, that's basically it for adding the button to this screen, just wanted to


show a real simple example of how to do it.


Sometimes people ask me to give them some challenges, so I will give you a challenge,


and that will be now to add a clear button to the screen, and there's even some code


online as far as how to do the clear, you can go onto the Kiwi website, and they actually


have a paint application and they have a clear button there, you probably won't be able to


superimpose that clear button really simply, but there is a way to do it, so I encourage


you to try to do it, if you can figure it out, please try to post, or please post it


here and share with the others if you figure out a good way to do it.


So anyways, that's it for this video, if you guys have any questions or comments, please


feel free to leave them below, otherwise, as always, thanks for watching, thanks for


Models - Django Web Development with Python p.2

What's going on everybody welcome to part two of the web development with Django and Python knowledge transference


Okay in this video we're going to be talking about is models we're just going to start getting into models and how we can interact with them


but pretty much models with Django is where you're getting almost all of the value from the Django web framework is coming from models


and all basically the abstraction from those models so this is another one where I think you might want to spend a little more time


making sure you understand it really well before you move on to the next topic so with that let's get into it


so again models are just the way that your website or your app gets to interact with the information stored within your database


so for example on our website we might want to store like our well for sure you're going to store like users to your database


you're going to store so like for a user you might you might store like a username the actual person's name an email a password all that kind of stuff


and then in our case too we would like to likely store our actual tutorials to the database we will get to users I promise


but that's a more complex issue because you got to do the registration login and sessions and hashing passwords all kinds of stuff and forms and stuff like that


so for now we're going to try to keep it more simple we're going to focus first on just a simpler tutorial model


so tutorial model what kind of aspects or attributes would a tutorial have we've got like the tutorial content


so this is just you know the actual body of the tutorial but then you've got like maybe a tutorial title and then the published date


maybe a playlist like or a series that it's a part of and then maybe like a category or something like that we probably won't do all those yet


but eventually that might be what you would use now in terms of other like if you were just trying to come up with some way to store these things to your database


you would really want to spend a lot of time before you build your actual database out you want to spend a lot of time thinking ahead like what might I need in this database


because you're not going to want to migrate a database but with Django it really is so simple that you don't need to think about everything in advance


it's not going to cost you anything really to later decide I want to have this column or I want to have this other attribute or I don't really need this one anymore


so I'm going to get rid of it that kind of stuff it just it's not a big deal with Django so anyways let's get into it so with models we're just going to navigate into the app that we're curious about


so in our case that's main we've been working there I'm going to close these just to avoid clutter and we're going to open up models.py


so in here you've really only got this one import and that's just import models and all this is for is every model is going to inherit from Django's base model


so what we're going to do is we're just going to define a new class and we're going to call this class tutorial and like I said it inherits from models.model


and all that is all that means inheritance if you want you can go to Python Perlman net go to the intermediate tutorials you can learn more about inheritance


but basically we are extending this model so models.model already has a bunch of different attributes and things associated with it


and then we can just say the little things that we want to change so like what columns and stuff like that


but honestly if you don't know what this means or you don't care to know it's not that big of a deal you should just know that you have to pass that


but if you want to learn more it's just inheritance and you can look more into it so anyways let me zoom in a little bit here


okay so what we're going to do now is just basically specify the columns that we want in our tutorials


so you know maybe you're going to have a tutorial title okay and that tutorial title would be models.carfield and we'll give this a max length of 200


what I want to look grab for you guys is I'd like to show you the documentation so let me think here Django model fields


hopefully that will take us where we want to be let me just do a quick search for carfield looks like it's probably on this page yeah


okay so I will put a link to the text based version of the tutorial for sure in the description otherwise you can just do the same google search I just did


and find yourself here but obviously this is quite a large page there are many different fields that you can have I'll explain the ones I'm going to use


but just know that many many more exist and you should just come here to learn about at least all of your options but the ones where most people are going to use like carfield


a text input or something like text field and then there's maybe like a date input type and then maybe like numbers only stuff like that pretty much people are going to use all the same keys and you probably won't need 90% of the available keys


so anyways back to where we were so this is a carfield max length is 200 then we're going to add a tutorial content and that will be models


models dot text field and then no parameters so then immediately the astute among us would be like whoa whoa whoa whoa what's the difference between a carfield and a text field so a carfield or a carefield or whatever you want to call it charfield


I have heard people call that a charfield anyways whatever you want to call it the carfield is basically if you want to set a maximum and generally a carfield is going to be a shorter thing whereas a text field is like a blob although I bet I would be surprised let's see if we search blob


oh really text blob no okay anyway I thought they would probably have a unique one for blob but maybe that's what it maps to an SQL or something as just like a text blob or something anyway text field is like what you would use for a tutorial or an article or something of not finite length


so anyways next thing we're going to do tutorial content let's do tutorial published that's going to be models dot date time field and then we're going to have this called date published published okay so that's enough information also we can do various overrides like I was talking about


so we can override the string method so we could return tutorial title and in fact first of all this would need to be self self my eye is super itchy self so all we're doing this for is because generally if we display this object in like print out the object let's say it's going to look kind of ugly it's going to be like something like tutorial object blah blah blah


and instead what we should say is as we iterate over it maybe we want to know the title or something like that like so it just it just makes it a little better to read. So okay so we've got our model now anytime you create a model basically a model is going to map to a database table.


So this is our so we've got our database already basically are you already should have a database I'm trying to decide where it would be probably back here yeah here's your database right here DB sq lite three.


The default is sq lite later you can change that if you go to my site settings dot pie.


And then somewhere in here.


It'll be somewhere in here.


Anyways over here probably yeah so this will be your database engine you could change it sq lite is great for development and tutorials like we're doing right now you can scale an sq lite database actually quite well so I wouldn't you know the conventional Internet wisdom will tell you are never use sq lite in production.


Maybe sq lite is actually pretty darn powerful it's just that my sq l or something like that or even some of these other databases is probably a better choice in your deployment but.


We can worry even talk about that way later down the road sq lite three is nice because it comes with python nobody has to do installation and everybody can follow along no matter the operating system so we'll keep sq lite three and anyways you already have your table.


So what this is going to do is or I'm sorry you already have your database what this is going to do is create a new table tutorial and then all of these things become your look like your columns in that table.


And then it automatically just by default gives you a primary key so anybody who's familiar with making databases and stuff is like wait wait wait.


You will already have a primary key is just default done for you but you could also if you wanted set one of these to be your primary key if you so chose.


But we're not going to focus on that for now so every time you make a new model that's a new table therefore you have to do two things with Django one you have to do a thing called make migrations and all that does is kind of prepare your migrations.


Then you have to actually migrate so it's two steps so as you've seen already when we run our server I'm assuming one is still running nope I guess I'm not running the server anymore.


So python manage dot pi run server it was giving us a message before let's see what it does this time okay so it says you know you we need to probably run a migrate.


This is really only like because we just made this website that we want to run this so what I'm going to do I guess cd what do we call this my site right my site yes so I think it's in here so what we can do is I just I hate that I'm just going to do it in order that you're supposed to do it will do the other migrations at the same time.


Because there's only one time you'll run that migrate first so what we're going to say here is basically every time you add or change so if you add another model or you want to change a pre-existing model you have to make migrations then migrate and that's basically it unless you happen to hit an error.


So we've made this new model so the first thing I want us to do is make migrations so we're just going to say python manage dot pi make migrations and it says no changes detected so what happened here the reason why I really did do this on purpose you're going to hit this and if I did it didn't hit it for you you would be stumped or at least if you're like me and you're slow you would be stumped for a few minutes.


So what happens is anytime you start a new app we were able to point to that new app but we haven't like installed that new app so first we have to install that new app in order for it to add tables to our database.


I'm not really sure the design reason why we can like have like URLs be pointed to it and all that we could do that but we can't do models I don't know anyways that's the way it is and I guess because if you you're explicitly pointing but anyway I don't know it feels funky to me.


But anytime you create a new app you have to also install that app if you want to use models so let's go ahead and install it we're going to come over here to my site settings dot pi actually was already open and then it should be near the top basically where we.


Yeah so right right here this is where you're going to add any new apps that you want to add to your project so let's just go ahead and do it so in this case it's actually main dot apps dot main config.


That's what you need to add you should immediately ask yourself does main dot apps main config already exist well we can actually check that really quick main apps boom here we are main config name equals main great so it already exists sounds good.


So as long as settings is saved we'll come back over here and let's try to make migrations again sure enough we made a migration so all this says is it just kind of gives us a quick.


Summary of what it's done for us so it's just saying hey we're going to make a basically we're adding a new model if you were going to change things that would say hey we're changing something to something and blah blah blah and all that also you can come into.


Your directory here and then now we've got this migrations and you can kind of see like what its plan is for the future if you open that I don't know why I should have stayed in some line but anyways if you want you can check that out you probably will never check it out.


But if you wanted to you sure could the other thing you could do is actually get the SQL for it so if you want to do this you could say Python manage dot pie SQL migrate main and then you get the ID so in this case it was 0 0 0 1.


Enter here and this just tells you the exact SQL should you be curious that will be executed maybe if you like you know SQL a lot you'll you just you just want to see how they're doing it or something but again you're probably never doing that.


Really all you're going to do is make a new model you run the make migrations that like Python manage up high.


Make migrations we can run it again it's just gonna say nothing new then all you're going to run is Python manage dot pie migrate.


So this time it did quite a few things only because we just never ran that migrate that it was asking us to run I personally found that to be kind of confusing that we initially ran migrate and then later we make a model and then you got to run make migrations and then migrate again like I just really stumped me.


All you need to know you do something with a model make migrations migrate done it's not complicated I promise so our model is completed there now we're ready to start interacting with it so the next thing that I think is kind of cool that I don't think you're going to need it this early in but I do think it's it's just a cool aspect to Django that again if you're doing things on your own you would probably never make for yourself but it's pretty cool that we have this.


So one thing that we can say is Python manage dot pie shell and this gives us like an interactive shell to play with our website so rather than like if you're trying to do something like for example let's say we make some tutorials and we want to iterate over the tutorial but something's going wrong and especially like later on when we have like foreign keys that are pointing to different tables and we're trying to figure out okay how can we pull and how do we write the statement that we want to write and if you're trying to like build a view.


And a modern yeah if you're trying to do like views dot pie and then put it to a template and then like navigate to that page and reload that page just to see if it worked very tedious so instead you could use something like the shell so just real quick we can do something like this we can say from main from man main dot models we want to import our tutorial model.


Then what we could say is like tutorial dot objects dot all and this is the exact same syntax that you would actually use in like for example your views dot pie you wouldn't say from main it would be a relative import but anyways you would you would do something very similar to this at least and actually tutorial objects dot all this is something that you would do.


So because you can do like dot all you can filter you can get you can do all kinds of cool stuff all of which I think all of which will end up actually using so anyways in this case we see there's nothing here there are no tutorial objects all we did was just create the table the database so we would need to like add a tutorial in here so we can definitely do that we have tutorial title content and published so we real quickly we can just add one in so we can say we can create this like new tutorial object.


We can say that is equal to a tutorial and then we'll say tutorial title equals to be can anybody guess where this is going tutorial content equals or not to be.


And finally we should have done goofed I don't know if I can get away with this because what we want to do is actually use a date time here.


I think I'm gonna let it throw a syntax error and then I'll just use the up arrow because I guess I could copy it too but I'm not really worried about that then what we're going to say is tutorial underscore published equals and then I'm gonna say time zone dot now this is going to fail because we don't have time zone.


But that's okay cool so let's bring in time zone really quick that's from Django dot utils import time zone and then I'll just run that again so now we've got this new tutorial object and again if you're familiar with SQL you kind of you can build an insertion but then at the end of the day you've got to commit it so we're going to go ahead and new tutorial dot save because commit is a horrible word and nobody understands.


It doesn't make sense to a newcomer anyways so new tutorial dot save now we actually have it so now if I just up arrow all the way to destroy objects to all boom we've got one later we can iterate over that so for T in tutorial dot objects dot all dot all one two three four prints T dot tutorial title for example.


Whoops did I let's see what did I do wrong for T in tutorial T dot tutorial what did I do wrong there?


I feel like I've got to be like totally blind here for T oh capital T tutorial is my issue there.


I was like when did I use tutorial okay print T dot tutorial title did it okay great so again this is the same sort of syntax and logic that you would use to actually iterate over your objects instead of use dot pie before you might pass it to a template but also you could run this in a template which probably means nothing to you at the moment.


But later it will anyways pretty cool that Django basically we've abstracted quite a bit away from writing our own SQL queries and all that and later if we wanted to change things and we will probably change things it's like super simple to go about changing things.


So I think at this point I'm going to leave it here and I think in the next tutorial we'll talk about is likely the admin page whenever we did this we also brought in our admin so we just need to create an admin user account and I can show you guys the admin page which is again one of the reasons why your models are just incredible.


That we have in the admin page from Django is just exquisite so I'm really excited to show you guys that's again one of the coolest things that you get from Django and yeah so questions comments concerns whatever feel free to leave those below.


Quick shout out to my most recent channel members Rodolfo, Sidkay, Jiwan Sun and Oli thank you guys very much for your support I really appreciate it lets me do this kind of stuff and I truly love doing this.


So that's it for now if you didn't know how to become a channel member you click that beautiful beautiful blue button there.


Q Learning Intro/Table - Reinforcement Learning p.1

What is going on everybody and welcome to a


Reinforcement learning tutorial series in this video we're going to be starting off with


Q learning now every time I do a new series people are like what are the prerequisites really for any of the series there aren't


Any just anytime we hit a concept or topic you don't understand look it up


In this case though there really shouldn't be anything if you know basic python


You should be able to follow along with the Q learning tutorials pretty well now


We started getting into deep Q learning


That changes now you need to know tensor flow and keras and stuff so so just keep that in mind


But at least a start pretty much anybody should be able to follow along here so first of all what actually is Q learning?


The idea of Q learning is to have these like Q values for every action you could possibly take given a state


okay, and


over time we want to be able to update those Q values in such a way that


Running through a chain of actions will produce a good result, okay?


now that's done by


Rewarding some sort of agent as they get through this environment and the idea is to kind of reward for the long-term goal


Rather than in any immediate short-term actions now Q learning specifically is what's called a model free learning


The idea is that the the Q learning model that we rewrite basically the Q learning algorithm is applicable to


Like any environment. It's not really environment specific so the code that we write


Should be mostly applicable to


Any other environment that we might?


Want to use it in as long as that environments simple enough so


Q learning in the way that you know what we're about to work with is good for really basic environments


But it's not going to do anything really complicated for that you're going to have to start getting into deep Q learning or something like that


So anyways, I think the best way to learn is by doing so let's jump in so we're going to need a couple of things


Right out of the gate


so go ahead and


Pip install Jim I should already have it


And then also go ahead and pip install


If you can type on like me install matplotlib


Grab that as well. We're going to be using that in series, and I might forget to tell you to get it


But it should be it so


So I'm going to go ahead and go into Q learning here and just edit this file should be empty for now


Don't need that right now


cool


So the first thing we're going to do is just kind of poke around get an idea for this environment


So even though it doesn't this environment like doesn't matter much to our AI


It matters to you to understand how Q learning actually works


So what I'm going to go ahead and do is fix this first of all there we go


import


Import Jim and then what we're going to say is the environment is a gym dot make and the environment


We're going to use here is mountain car Dash


V zero that's title cased mountain car


Anytime you have an environment the first thing we're going to do is reset


And then we are ready to actually iterate through this environment, so what we're going to say is


Done equals false and then while not


Done we want to step through this environment now


To do a step we need an action so for now. I'm going to say action equals to now again


For your benefit. I think it's useful to understand how the environment works, but in terms of


your


Agent it doesn't need to actually know but


For your own information mountain car this environment has three actions that you can take now in a little bit


I'll show you you can actually pull all of these gym environments to figure out how many actions are there?


But in this case just for simplicity's sake I'm just going to tell you there's three actions and


Action zero is like push this car left action one is do nothing action two is push the car, right?


So I'm just going to say basically push the car right is our action and we're just going to do that every time so we're


Going to say m dot step


we're going to say action and


Then end dot render and in fact I think I'll already populate this that way we don't have to worry well


We'll hit an error if I don't do this so the next thing we're going to say is new underscore state


comma


reward comma done comma underscore equals m dot step action


So every time we step with an action we get a new state from the environment so the state is


The things that were like sensing from the environment basically in this case the state is two values its position and velocity


But again like I can I'm going to continue repeating


Your model it doesn't really matter all that much what these values actually are as long as they're meaningful


Obviously we don't want meaningless data here, but to the environment or to the agent it really doesn't matter


But just for your own curiosity sake


The values that we are getting from here our position and velocity of this car


And the dot render so that'll show it and then if done is done great and then if we are done


We're going to do an end dot close. I'm going to save that I'm going to run it from terminal


python


Q learn one pie and we should get great


So as you can see this car is trying to get up this mountain for this hill


I'm not sure I'd call that a mountain, but


Anyway, it just clearly doesn't quite have the horsepower and instead what it needs to do is build momentum by going up here swinging down


Here and then building up and and the goal is to eventually make it


to this yellow flag area


so


Okay, so how do we do that? It's actually really really simple. We just we just do this duh. I mean


obviously


So this is the actual function here for


calculating these Q values


But don't worry. You don't actually need to know how to read that or write from it. We're going to convert that to python in due time


But basically the way that this is going to work is


We want to create this this q table


It's this large table that given any combination of states we can just look it up on the table, right?


so given any combination of state of


Position and velocity for every combination of position and velocity


Velocity we were we just want to look up on this table


Hey, what is what what are the three in this case q values at this position and velocity and?


As long as we're trying to exploit the environment. We're just going to pick the one with the maximum the largest Q value


that said when we


Initialize this Q table we just initialize with random values


Which also means those values are random and useless so so actually initially our agent is going to do it


We call explore a lot which means do random stuff


And then slowly update those Q values as time goes on so


Anyway, that's a lot of information to throw at you, so


So I think the first thing we're going to do is build the Q table


And then go from there, so


So first of all let me just


print


We'll print new state save that did I actually I think I closed the console


Q learn I'm just going to run it again, and we're just printing out now the states so as you can see here


We've got all these states being printed out now


This is the first issue that we're going to come across with Q learning and that is that these values like


That's out to what is that eight decimal places


That's a lot of decimal places. Can you imagine every combination of these two values out to eight decimal places?


That's a lot


So that would make for a huge table that's just going to blow out our memory not only is it going to blow out our


Memory but the amount of time it's going to take to learn on a table of that size in the amount of episodes


We'd have to like run through to do so I mean by the time we were done quantum computing would be like a huge thing


already, so


So we don't want to wait that long so now what we need to do is is convert these


continuous values to what are referred to as


Discrete values basically we want to bucket this information into buckets of some sort of size now


What size those buckets are is?


Is going to be a yet another one of many variables that we're going to set and likely need to tweak as time goes on


so so with that let's go ahead and


Get started here so first of all and


So basically at this point we can start pulling some information from this environment one of the things that we can do is print


env dot


conservation underscore space dot hi, and then let's just print the low real quick hi low


and


Then one of the other things that we can print is because I just I just want to point this out


I think we'll end up using it anyways, but it's uh


Yeah, it's action underscore space dot n and that'll tell us how many actions we can actually take


so say if we'll just run that in here so as you can see this is the high value for all of the actions or


I'm sorry all of the observations possible now


Another thing to note is you might actually have reinforcement learning


Engaging in some sort of environment where you actually don't know


These values and you actually might just engage in this environment randomly for a while before you even start to learn these values


so in this case the environment is gracious enough to us to report these values


But we might actually not know and then here with actions chances are you'll know your actions in advance no matter no matter what?


I can think of a few use cases for like reinforcement learning at least where you wouldn't but probably not Q learning, but anyway


We can pull the environment of any of these gym environments


This way to know exactly how many actions are possible


Okay, so the next thing that we want to do is


Create we want to figure out how we're going to create this queue table so


So we want this queue table to be of a size that's at least manageable, so how we're gonna make that size


I'm just gonna say


Discrete


Observation observation space or OS


Size and in this case we we're gonna say 2020 now


I'm gonna change this in just a second. I am kind of on the fence as far as like hard coding this


I think that a real our


Reinforcement learning agent would not have this hard coded because it would change by environment


But you won't necessarily always want these values to be the same number like this one


Maybe you want 50 and then this one should be 20 and so on


But for now what I'm gonna say is just gonna be 20


Times


the len of


Any observation space thing so I'm just gonna pick one at random, so this is two


So it's just 20 times two so it's a 20 by 20, okay?


I'm only doing that just because I really want this code to be at least


functional on every environment possible


So I just don't want to hard code that because some environments won't only have two observation values. They might have five ten a hundred


Hopefully not but maybe so anyway cool so alright, so we've got the discrete observation space size


So with that entire range so from 0.6 to negative


1.2. We want to separate that range into 20 chunks or 20 buckets or


20 discrete values okay, however you want to call it


That's what we want to do and then we want to do the same thing for the range of 0.07 to negative


0.07 we want 20 chunks and buckets that way we don't just have this infinitely continuous range


That would be forever for us to learn


So now we need to know how big is that range how big are those chunks?


You know like each step through those chunks, so how big is that number so now we're gonna say discrete os


when size is equal to n


And


Really are you not gonna help me


Observation space dot high I was really hoping it would just like autocomplete that entire thing for me


anyway minus m dot observation space dot low and


Then I'll add the spaces around that I'm pretty sure that would be pep 8


Then what we want to say is divided by discrete OS size


That's freaking huge


And then what I'm gonna do is I'm just gonna print this out just so you can see what I'm talking about so now print


Not in all caps though


print


discrete OS when size


We'll run that real quickly and you can see now you okay so so for positions each bucket will be of a range that is


0.09 long and then over here 0.007 long


For the velocity chunks and


Knowing that I would say I'm just gonna guess that's probably granular enough and in fact. I'm not guessing already know it's gonna work, so


So it will be but sometimes you might find that you have to tweak these values and again like I said sometimes


They won't necessarily be the same between two so I just I made this change because I want it to work across all


Environments and by work. I mean just the code should run


But really you would take your cue learning scripts and make them far more


dynamic and


Probably also run like a few maybe even a few hundred episodes to


tweak these numbers a bit


but


Because this is not our stopping point


I'm not gonna spend a bunch of time doing that like we're gonna get into deep Q learning and stuff


So I don't I don't want to spend too much time there


We will spend a little bit of time talking about how we can optimize these numbers


But yeah anyways moving on so discrete win size is now known


So finally we can create


The cue table so again. I'm gonna open up paint because paint is love and paint is life


The the objective at the end of the day for us is to have this like massive cue table, okay?


And so I'm just gonna draw


Just the best table you've probably ever seen in your life


And as you've seen we have three three actions, right?


You've got action zero action one and action two right and then over here. You've got


every possible combination of


Well now it's only 20 by 20, so it's a much smaller set of you know it's only 400 things right


But you know this would be I don't know combination one combination two


combination three and then where wherever the agent finds itself whatever


combination of


Position and full on velocity that it finds itself in it's gonna come to this table