framework or kind of you know resource specifications you have but then kind of really pushing on how far you can you can take that thing and some of the methods and techniques that have been developed especially in the last year or two of kind of increased efficiencies by factors of maybe five to ten so there's I think a lot of promise there from you know really pushing even with a fixed size and many of those still fit on single GPUs yeah thanks cool yeah so yeah I guess given it seems like the class has gone over transforms a few


times I won't do the super detailed version here so yeah I guess we'll just kind of look through that real quickly so we've kind of talked right now so far about standard mostly standard language models and kind of using different architectures you know character level aren't ends and 2ls TM or double all those teams and transfer based language models and they're always kind of trained with the standard auto regressive left right or in the case of Elmo adding a backwards right-left language model and you know that's nice

L11 Language Models -- guest instructor: Alec Radford (OpenAI) --- Deep Unsupervised Learning SP20


it's a it's a clear framework it allows you to compete probabilities easily it allows you to sample kind of in just a iterated fashion it's not the fastest but it's quite simple to do you just feed in the sample in the distribution over the next word and then you feed that as a new input and conditioned on it and then resample and so it's it's it's a very clean in like general framework but it may actually not be all that optimal so it's cool and exciting to see some of the things that these language models are doing and some of the work as I was


just mentioning has really pushed farther by walking away from that very explicit like left/right auto regressive language modeling strategy so this is a common leaderboard it's called the glute benchmark and it combines a set of like nine tasks together and this is this was pretty important for the field to kind of standardize on the set of tasks people reported on as you can imagine especially early on when the research is kind of scattered and not all that standardized you kind of see you know people picking their favorite benchmarks


I was totally guilty of this myself I really cared about seven classification just because I happen to you know find that to be an interesting task and worked on it a lot and so you know my favorite report is that a classification someone else is to report on you know in tailmon and someone else report on crushing answering you gotta have this lack of commonality and comparison points so the blue benchmark came in and said we're going to standardize we're gonna focus on 7th level comparison tasks primarily and we're going to kind


of have a suite of them and we're basically gonna say hey you should report on all of them so you can't hide your bad results on one and this helped drive a lot of progress too so this is a screenshot of kind of where this leaderboard has gone showing all these new improvements and methods so there's the BIOS TM Elmo baseline autumn that we mentioned and GT one would have slotted in slightly above that but then there's at now ranked 20 from Jacob Devlin and crew and then there's Facebook ai's Roberta as another big jump and so we


saw it like on the average metric here which kind of just averages the performance across all these different tasks it went from 70 to 80 between the biles team elmo bench baseline - yeah - burn so that was a big jump there and then an almost equally sized jump happened with Berk - Roberta which we'll talk about in a bit and then there's newer things like Elektra and t5 so this kind of whole area has really you know really exploded in the last year two years in terms of the amount of teams and basically every


major research lab Microsoft like Stanford and why you like pretty much you know AI too you see a huge amount of people you know I do every everyone everywhere has been kind of pushing what they can do on this this kind of benchmark and really seeing a lot of progress so we're going to go through kind of each of some of these improvements that these are highlighted select a few there's many others so sorry if I dropped here you briefly was a cool recording spec on so an SST too is like synth analysis like we mentioned before so it's kind of a


you know again a diverse suite of tasks here so how do we kind of what are these big improvements we're seeing beyond the standard left-right language models and there's one more point Domanick which is there is a human baseline here and it's slotted in actually in the middle it's in 12th place now so what does it mean like are these models actually better than people and you know the answer really is no and it's complicated and confusing and we'll chat about this a bit more later and supervised learning is always playing tricks on you but you


know now these models are like really really you know went from like in the last two or three years because of leveraging unsupervised pre training and kind of scalable methods to really make quite a lot of progress in this space very quickly so this is Bert so what Bert does is it basically finds a very great way to hybridize a language model objective with kind of the importance of like bidirectionality so again you know by default we have this like left/right auto regressive factorization where we say given the


previous words predict the next word in the language model that's like what GPT one does and so what we see with that is you're not able to exploit context for the right you're not able to see you know you by masking the model you have to prevent it from being able to just look at the next word and say well I see my sequence that it's cat so I'll just learn to copy it over in predict cat so that has a major limitation and when we when we release GP t1 we actually like weren't able to do well on or you know we found that some of the


question-answering datasets we just couldn't do as long because we weren't able to exploit bidirectional context whereas elmo was the old bi-directional image model and with an LST M and you know because they trained afford one of their backward one and the average of the representations that works quite well for shell models and that gets you that my directional context and that can help a ton and you know they performed us I still have performed this on some data sets because of that and then Bert basically figures out how to have


bi-directional context within a self attention model and the way they do this is they change the objective so they're no longer doing this like standard you know maximum likelihood training on like just the data distribution they use this kind of proxy tasks called mast language modeling so again you know at the bottom here we could see left-right LM is like the cat sat on the and then you blink out a word and it's supposed to predict math right language modeling would be we'd go the other way around and same at the onset


cat mask and predict you know there and so what master LUN does is it just takes your input sequence and it's it corrupts a few locations 15% in the case of bird and it trains the model to predict what's at those masked locations so you know might in in this case there's no like left requirement to write requirement it just randomly selects 15% and this allows you to have bi-directional like representations you can't leak the word because it's masked in the inputs whereas for a standard left-right right-left Ella you kind of


hide that this is probably one detail of self attention models that you have that you have the self attention matrix and that kind of defines the connectivity pattern between different locations in your sequence of inputs that you're processing and so you used masked self attention matrix for standard left/right language modeling or right left wing which following where you masked the upper triangle and that prevents you from that future blinking and so you know we say bi-directional context that corresponds to training the same self


attention transformer basically except you don't longer have this masking to prevent future locations like J you know after I being able to have I look at and attend to position J after I so that that's kind of the architectural detail that corresponds to this change and it kind of makes sense that having that ability to look at both sides of context helps with disambiguation it helps with information processing and you know information flow through the model because you know the model can like query back and you know for things like


question answering for instance if you have a if you have the question after the context you can't update the representations of the context you know in a left-right Auto regressive model after you've seen the question because they're mass then they're hidden from it so the model isn't doing any you know right context dependent processing but in bird it can actually you know bidirectionally attend and quickly passed information forward and backward and this is just what you see if anyone who actually does like


self attention architecture from scratch on a supervised test they always use bi-directional or almost always instead of no mess mess self attention matrices and so this turns out to have a huge boost on glue so that that bump I believe between GP t1 and Burt was like they went from I will GPT one had like an average of 78 or something or sorry excuse me I think this got reworked and sorry we excluded WN a lie it was like a bump of like five five plus percent so it basically got a double the Headroom on gbg1 and they show with very careful


controls that for the exact same model in the exact same setting you know it does look quite a bit better so bidirectionality makes sense also for sentence comparison tasks like entailment where you have two sentences you're comparing really want them all to be able to compute them and tend back and forth between them and you know look at one and then the other that just seems like correct behavior to do whereas 251 would just go left right and then you'd be done so yeah Burt ended up being kind of the thing after Elmo Elmo


kicked it off especially in the research side and got a lot of people to start investing in this space bird is kind of the thing that moved this to the point where suddenly it was like you know ready for like more commercialization or you know production ready basically and so this is now deployed in Google search and its really like kind of showing up everywhere you know if you go to basically any leaderboard burg variants are often very near the top now and pretty much most NLP tasks and just like GPT one they use the same architecture


everywhere they remove the need for kind of having these tests specific modules on top and so this was another like incredibly strong step so you know that was Bert I guess there's one more point to make which is because it's predicting these masking tokens it's only predicting like you know you have to set that mass percentage and by default it's often said to like 15 percent so you should understand that like your left/right model it actually predicts a lot more words because it'll predict the full sequence within a single fordpass


whereas by default you'd have to run a Bert you know model like six times two on average see every predict every token so it turns out that they often learn a bit slower early but then they just keep training and they begin to learn how to use the bidirectional representations to their benefit and then they continue to outperform left/right language models now the problem is you can't sample from it and it's no longer quite as clear that it's like you know you can't compute a correct I'm normally like correctly normalized probability over


the sequence without a lot of work there's some research and to figure out how to do this with clever methods but kind of it removes some of the elegance of like sampling and having easy density or probability estimates for kind of trading off this representation capability so Roberta is if we go back to this leaderboard the next big jump up from 80 point five to eighty eight point one you know kind of you know like as a benchmark or you know important event it kind of you know solidly is above the supposed human average baselines here so


what is Roberta Roberta is a very well executed engineering refinement on Burt it's it's a good example of how so often in this field kind of you know the second pass at an approach with maybe the same very similar model architecture algorithm I've can just by careful engineering and fine tuning and tweaking still have tons of extra Headroom to it so they better tune the hyper parameters they remove a few hacks that the original Burt had so for instance the original point for computational reasons predicted most of its training on a


relatively short context length and I believe out of 28 tokens and then right at the end of training double that two times two up to 512 tokens for prediction and so they just train at 512 the whole way through it's the same model capacity it has the same runtime per sequence length but they just have you know they spend the pre-training compute to buy that and when you're thinking about deploying the system you know one of the important criteria to realize is especially when you're talking about a system that might get


deployed broadly and used across the you know across the world and many different companies once it's released most of the compute is going into inference time it's not actually going into training time and so that means that if you have a method of getting further performance improvements by spending more flops at pre training time often it can be quite worth it from like kind of a full ecosystem view of where is the compute being spent this is one of the counterintuitive things I think about how you think about these systems so


they also do better data generation it turned out that the original bird kind of from a simplicity perspective cache the masking so they only actually masked the sequences once and they always print mask location and so you can simply change that to an online setting where you keep sampling the mask and that helps with overfitting and they also use a more flexible vocab scheme these kind of a full BP scheme that can do kind of full utf-8 byte sequences so you can handle any string at least with the standard byte sequence representation and then


they just train longer with more compute so as we mentioned before bird is only predicting like one of six tokens on average so that just means it's under train for the coolant amount of time and you can actually just keep training it longer with more GPUs and continue to see higher and higher performance and so I mentioned Roberta Bert was on the leaderboards everywhere well now about you know eight months later it's Roberta everywhere on the leaderboards and that's still true today largely except for a few like targeted things is I


think if you go to the our general PD leaderboard you're gonna find that model in the first place is some variants of like a row burger or something so that's like an example again of where you know it's not you know like there's no super clever new algorithm or approach or you know and even for bird it's a pretty you know it's a pretty precise refinement of previous of like gt1 but it can have a huge impact when it's just well executed and you know is I think somewhat you know exciting from one view where it's


like okay we're kind of really finding that there's a lot of fertile ground here and with like kind of the right tweaks and you know clever insights we can continue to make further progress so this is where a lecture comes in and this is like I think one of the ones that first shows kind of another new interesting algorithmic potential improvement and someone excitingly shows that it's much more efficient so we mentioned the masking for bird so there's actually this kind of gap here which is the problem is when you


were training you're masking all these input sequences and you know you may sample the masquerades so you you're kind of crunching your inputs but then when you want to train a test time or when you want to transfer to some downstream task it doesn't make sense to corrupt the inputs right because if you were doing some analysis and you mask the token you know this was a mask movie you don't know if it's going to be a great movie or a terrible movie in that mask location so bird just kind of like as a few tricks to minimize this impact


but if the other day it's kind of this train test gap where you trained it with one distribution with mask inputs and then you want to test it and and and predict with it on a different one and it turns out though that gap actually looks to contribute to some performance issues the other gap is again it's only predicting 15% of tokens so it may also be learning slower than it could because he would have to F crop six times to see the same same predicted segment of data so when Elektra does is a very clever hybrid system so they have


a bird or basically a mini bird inside of it so it's the standard math language modeling technique and then you sample from it and you say well you know for that first word that we've asked what do you think is the right word so sample from it's uh its distribution over next tokens and then you're going to feed it into this discriminator which is the actual Electra model and what its job is to do is to predict whether or not the token at any given location is the original token or a replaced token so if the generator gets it wrong again it


might sample something kind of reasonable and kind of correct like cooked verse eight the job of the discriminator or the Electra system is to just estimate is this the correct one or wrong not so it's just a binary classification task but it's done at every location it's basically saying was this input corrupted and that allows it to you know one it has a natural distribution and it may be because this math language model could be quite good a lot closer to the real input distribution so you don't have this


shock when you transfer it for your downstream tasks and you also speed things up because you're taking a loss and propagating a gradient for every location because you're always estimating is it the correct one or the wrong one and that's like still can be a difficult task for every location rather than like kind of the degenerate thing for like 80 percent of tokens which is just like the egg and B function for birth and so when we look across the board here we see that this model kind of the standard models and you get like


glove or selma over spurt and they're all kind of like smashed up right here on 0 because and your TP one starts to move over and then you get kind of you know roberto scales with more and more compute and then the graph keeps going it is hidden you can see that electro is kind of across the board can be quite a lot more efficient and often by like factors of 5 for kind of equivalent performance on a data set so that's quite exciting and in the limit they show that for instance in Elektra a small electrical model so quite a lot


smaller than even a GPT one by exploiting bidirectionality in this dense training function can actually outperform GPT one in two days on a single be 100 whereas gbg1 was 25 days on 8p 6000 partially this is because I have P 16 verse F P 32 but it really shows like how you know I think unfortunately some people have and it kind of makes sense because I've talked about the importance of scale what not you know some people have kind of like written this whole subfield office like whoever has the most GPUs is gonna win and you know oh


it's all just training bigger models and maybe you know as a new when you're a grad student or as a hobbyist I don't have access to the resources to do interesting work in this space but if paper like electro it's really exciting because it shows that you know a single commercial GPU can actually still have very interesting results in this space nominally they still run the foreign version of the model on a TPU pod but you know there's here you're already having last year's model being beaten in a day or two on a single GPU next year


so I think that's a very exciting point in this is from Clarke it all blew the system for Google Clarion I think it's Kevin sorry his first name yeah so it's it's really exciting work here there's this final one kind of this is like kind of the deluxe result coming out of space from Colin Rafal and collaborators at Google and this is like kind of after the first crazy year of like well there's you know bird and now there's Roberta and others you know like you know all these things coming out one after the other every few months bumping


up the leaderboard this is the paper that like took a step back and kind of more systematically studying the space analyzed it used a lot of compute to do it but kind of really brought a lot of things together and kind of very carefully curated it's a it's a treasure trove of information for this space it's 50 pages long there's pages and pages of table so with hundreds of numbers and them so it can take a while to work through it but I really recommend it is like one of the ways to like get up to speed on this whole area and all the


techniques and all the different ways so they again systematically study this so their standard language modeling objectives there's bird style masking there's there's there's their own kind of things like spam based extensions of bird and then they also look at differences in the architecture so there's your standard left/right language model there's encoder decoders which could have like bi-directional encoder that processes like the previous sentence kind of skip thought style and then a autoregressive decoder and then there's


a corporate hybrid called a prefix L M which is a single well you could have untied weights but you think of it as like a partial and tying of the masking and a self attention matrix where you allow some part of the sequence to do bi-directional attention like in the past and then you switch over at some point to doing auto regressive like language modeling so you can get the benefits potentially of for past contexts doing bidirectional representations or in the limit if your downstream task is always just going to


be bi-directional you can just run it in purely bi-directional mode so it's kind of trading a hybrid system and I think that was also a quite clever improvement they had the other thing I really like about this paper is it goes even farther in terms of elegance of kind of this shared framework for doing all tasks and all predictions so kind of one of the trends has been moving away from these custom architectures to the kind of shared pre-trained models that are a little bit more monolithic and can be used across a wide range of tasks with


high performance and so tv5 you know typically like you know what do t1 and Bert the only difference we would do is we still flawed in the linear classifier at the end you like predict which of the right classes it's correct so what t5 says instead is and this is extra something that Brian McCann and collaborators at Salesforce introduced about two years ago is they basically say we're gonna phrase everything is like pure natural language pure question-answering or something so we're going to give the model like you know a


command or a prompt as the prefix like translate in English sentence to German and then it'll give it the English that is good and then t5 just through natural language rests that - you know dust is good or something and you know for all of these it does this basically so for the coolest sentences it'll predict the language phrase not acceptable and for you know STS STS B here's a kind of almost silly version where it's a it's a continuous valued sentence similarity prediction task and then they just have an output discreet token 3.8 so it has


the but you know because it's pre-trained it's learned kind of the continuum of numbers and the similarities between them but it's kind of like this for me to see a regression test reframed as discrete token prediction and you know again it's quite general you can do summarize and everything and so we saw a little bit of this what's like when we were probing with like where Schwartz's work just using natural language like probabilities from language model or like some of those zero shot transfers transform like GPT one and so


like to five really goes through and shows yeah you can actually exploit the natural language that it's learned and that helps with the transfer and helps with the fine-tuning tasks potentially so yeah to five it's a really good kind of overview of like all the work in this space and then they also just threw a big model on it to at the end in that gets you another bump on those leaderboards that we were talking about so that's that in fourth place though now so others have done some more things on top of it


so that's kind of I think the core like set of literature I wanted to cover here and ideas at this point we've kind of gone through the history of kind of language models and how they've been adapted and used across kind of you know the winding history here of how kind of NLP like kind of really took off with these and supervised and self supervised methods and kind of figured out how to use them and all these different papers that kind of found pieces of the puzzle and propose different methods that it or didn't work and kind of combined well


with other you know modeling improvements and everything I think it's a really cool story I'm excited that I was able to chat through that with y'all today the last bit here now we still have about fifteen minutes left but we should maybe leave a little bit for questions at the end to is kind of just a bit of more high-level thoughts on this is an unsupervised learning course why do we need it you know what's wrong with the current paradigm of supervised learning and I'm sure you see motivation you know and there's been great discussion


already on this topic here but kind of I would like to share a bit of my own like thoughts and opinions here so you know I think a motivating thing here again kind of we've had this thread running through a lot of the discussions so far on this in this talk has been how well does supervised learning work and you know what should we expect of it and so kind of concurrent with some of the stuff taking off in the last few years was kind of a lot of work that started critically evaluating kind of deep learning for supervised NLP and so you


know for a national ears in France for instance this is a three-way classification task and even before pre training really took off so isom is using just word vectors and it was a very you know well design architecture they nominally got to you know average human accuracy of a I believe a single Turk worker it may have been an unsound three so it's like whoa is this done did we already hit human accuracy and you know like I think everyone kind of knew well no because clearly these models are you know still making weird mistakes and


this is kind of where in the last few years there's been a lot of great work starting to really quantify these concepts of how robust our models how well do they work kind of distribution kind of pressuring and challenging the standards supervised learning paradigm of you know training on an iid training set and evaluating on another ID split held out data and basically showing that that's no longer sufficient and something's going wrong somewhere in supervised learning that means this is a being too fortunate to algorithms and


you know not being fortunate enough to humans and so this is a great paper from a Sutra and I believe called annotation artifacts of natural image inference data and so when you do hear people talking about these models are exploiting statistical artifacts and Maya sees of the train distribution you know this is a paper that really nailed that down and showed it quite conclusively so you know they kind of start from a high level well how were these datasets created these supervised data sets you know admittedly they're


kind of artificial you're paying people to label these tasks they're not natural instances of the task it kind of what people can come up with on the top of their head or you know they can have very good experimental methodology in data sets like a semi multi know why are some of the best we've got in terms of like very good set ups you know curated by people who really know what they're doing but you still run into the issue of like well you've got to have a human generate an example and you know maybe they're less creative than you think


so the data is actually drawn from a much more narrow distribution that it should be and so this paper kind of went through critically and showed a lot of these artifacts actually showing up so you know a worker would be told make a you know a negative or a contradictory label and so they would just be like oh I'll just slap a knot and top with all copy the sentence you know and it's not quite this bad but it gives you the idea of what's going on is they'll copy the premise sentences the hypothesis and they'll just put a knot in it or they'll


you know to have entailment they'll just restate the sentence in a more generic or abstract way and so you might go from you know you know you know a dog is you know playing to an animal is playing or a pet is playing or stuff like that or you'll add some kind of like super information like tall or sad or popular to hint at the neutral class which is like well it might be true or am I not but it's not clear that way and so what they showed is somewhat disturbingly if you only trained to model on the hypothesis sentence so the second


sentence and again semantically this task is defined as the logic correlation between two sentences but have you trained a model only on the second sentence to predict which of the classes it would be it actually did it basically got half of them right you know it went from 33 to 66 percent or so it was a large bump and you know by default we know that model can't be doing the true task because it's just predicting you know given only at the random second sentence so this is a great example of where you can see that standard


supervised learning might be picking up on these spurious correlations or artifacts and you know assume when you evaluate it on a new card set which is the set of answers that the model that only looks in the sentence evidence can get right drops quickly from like 16 percentage points from 80 to 88 to 72 percent and this shows up across the board there's now probably a dozen papers in this space if not more that show that kind of these analyzed systems that you know nominally we're supposed to have human level accuracy actually


are not consistent or not robust or not systematically generalizing so you know this is another one from Glockner that you know very carefully constructs kind of these probe sets so it's like permuting objects and the sentences are permuting you know synonyms or antonyms and you know on these probes they show that you know actually again drops quite a lot and then a final point here is on distributional robustness so this is a paper from Devon called learning evaluating general linguistic intelligence and so what they showed is


those near sort of question-answering models which again on squad it's you take a wikipedia sentence and you take burden which we've already talked about how much of an improvement it's had and you know how you know it's improved scores a ton so you take that sentence that question answering well that's trainable capilla and you just run out on a different data sets it's still question answering except maybe we run it on like trivia like trivia factoids that are sourced from like google search results or maybe


we run it in a more conversational framework with kind of you know to to people asking questions between each other and we see that just like a gracie's can crater or you know f ones basically actually metric here so it's you know the same task and we know that people when you ask them a question on one task force on another they're gonna do about the same you know maybe yes one task is a little bit harder than the other but you don't see them like suddenly you know lose half their accuracy this is again just hit set some


of the distribution a lot more muscles tissues and brittleness we're seeing and again this is still some of the best stuff we've got it's combining supervised learning and unsupervised learning but there's hints as we're going to go through here that these self supervised methods and unsupervised pre training is really helping with the robustness we're still not there yet but we're making progress and all that's being driven by moving away from a purely supervised learning framework to moving to these like hybrid Android training and pre


training methods so as I mentioned like there's a lot of things that could be going on current techniques are brittle they're memorizing and so generalizing they're exploiting spurious correlations you know also stop learning once you want to get to you know memorizing your training set you just wall turns off because the gradient ties ISM training lost goes to zero so it just kind of feels incorrect so there's like a lot of different routes we could go down to make progress we can do better models and architectures we could do more data


we can go down different paths all together so obviously since I'm kind of talking about unsupervised learning in a supervised learning class I'm gonna talk about how that's a very exciting one but you know we could always keep working in the supervised learning paradigm and just say well we're gonna have better models and we're gonna get more data we're some kind of purses problems in the same way and so this was like kind of what I'd say a lot of like early deep learning was really highlighting was kind of you know we were working on


supervised learning datasets we kind of were seeing you know these new architectures that were exploiting priors and inductive biases of the data domain we're really helping a ton so on images you know this is the grand story of we added you know comets and they are a great fit for the domain and they kind of cleverly quote encode you know all these equivariance and translation and you know shared weights and all this structure and that helps a ton with their accuracies and then we just use a large supervised


notice and let HDD figure it all out for us and so this kind of led to I think of mindset of heavily emphasizing architecture engineering you know there's a very large design space here someone cynically it allows for a lot of different papers to be written and you know you can really kind of combine and contrast like all these building blocks like we really like playing with these blocks and you know a lot of really good work has been done that like does empirically push the state of the art by exploiting you know properties of domains and you


know an example of that is this diagram on the left so does anyone want to guess the name of this model well sorry because it's kind of hypothetical it's called a simple model so this has got six different color embeddings and you know there's screws and character models and by attentions and MLPs and you know it starts to get quite complex when you're really all you've got is inductive biases and kind of the standard supervised learning datasets so it's a heroic effort but you're kind of exploiting more and more


details and getting more and more complex to make progress when you kind of have locked in these other constraints like the dataset size and the paradigm of training and on the right is another one that I think is like almost looks like it's like you know some like pentagram or something you know they look like kind of these very cool like architectures and they're very quite fun to look at and kind of look through all the work that's been done on creating these systems and again like we said there's all these different


methods of having a deck Tobias and it can really help a lot and so they're all important and very impactful and please don't take this as like criticising kind of the standard approach of like iterating and hell climbing on supervised learning with like better and better architectures but I think it's a bit like this where really when you treat a data set in isolation if we come back to how people learn and experience the world it's so varied it's so diverse there's so much experience in information and knowledge you're


leveraging before you ever saw this data set and some machine learning models when they're started in isolation on a supervisor zone you get a set by itself are kind of like you know that supervised data set is like a peak in a very big space it's a small peak and we can add more and more data and make that peak more you know taller and wider and that might help with robustness and generalization but at the other day it's kind of a little bit futile I think you know the real way to solve these tasks or at least the way that people do it is


they don't sit down and memorize you know a million different examples they somehow learn a much more general set of tasks behavior and transfer of knowledge and information instead of like just becoming a master at a very specific isolated domain you know we're amazing because of our Gen not because of our you know or well we were amazing for both because we can do incredible things in specific domains but at least machine learning is starting to see that I'm very targeted supervised data says you can do it


models that do a bit better and so then there's papers that even on architecture engineering show kind of somewhat critically that some of these you know fancy or new architectures that we saw them don't quite improve as much as you think or with more careful oblations don't show much of a benefit so you know there's a one of the famous examples here is they took a baseline Alice T and gave it some love this is kind of a common story for language modeling and show that it was about performing kind of a lot of new recent state-of-the-art


models if you just have careful comparisons in careful trying to me so you know maybe we need to back off and rethink beyond just pure supervised learning on test specific datasets and you know I think one of the reasons to frame this is the largest supervised days so you know basically in the world what I'm aware of publicly is gft 300 million actually there's a Facebook one that I haven't talked about their Instagram pre training but this work this was true a little while ago so there's a straight a million images


18,000 classes if you if you do like a very simple like loose bound on how much information content you get you have like log 18,000 bits per image and you have 300 million of them so that ends up with about getting 300 50 megabytes of constraint on the function you can learn so this is the world's biggest data set and in terms of the correct function that we're trying to approximate with supervised learning you know we only are able to pump about from this kind of slightly naive in toyish view about 530 megabytes of information into the system


from the supervision here but you know like like trying to connect this back to everything we've been talking about today there's you know terabytes and petabytes of actual raw natural language on the internet so if we figure out how to exploit all that information in some reasonable way there's a hell of a lot horror there that we should hopefully and again we're gonna be a lot less efficient you know gold labeled supervised data you know per bit is probably helping far more and less specify and learn a task but we only


have a little bit of it it's like you know yawns kickin ology of the cherry on top first kind of you know everything else we need to be able to do and I kind of tried to take the supervised learning approach for language for I spent most of 2015 myself building what I hoped would be an image time for text it was a very large weekly supervised data set where we basically did classification over edit communities and we built like on a 50 million training examples over a thousand communities we turned our Nan's to


predict everything and we're hoping they would learn useful features and representations kind of skip thought style it was pretty concurrent work at the time except we were going with the supervisor route instead the unsupervised route and the sad thing was the unsupervised model beat us so skip thought vectors was beating you know just by doing a large bottle objective was beating this system that we built with like your middle e weekly supervised data but we were like oh yeah this is the gold label so you know these


are the right things to predict they're semantically aligned with classification and this kind of really made me quite confused and kind of skeptical so what's going on in his face because you or at least supervised learning and got me really excited more on the gener of long ago and some of us I'm excited because we just weren't seeing the supervised learning pull through here because it's just I think is a little bit too weak of a supervision source and a little bit too specific so like again the big question I think is a lot more you know


in terms of like novel research frontiers how do we go from kind of isolated peaks of competency that you can very quickly you know fall down if you change the problem just a little bit you know quickly collapse in terms of task mastery how do we go to systems that perform you know and then much more general robust kind of you know maybe they're not nearly as good in terms of competency on any given specific task but how do they perform much more broadly across the board and again so this this is an example of kind of the


classic architecture engineered approach like one of the kind of you know incredibly well done versions here that's exploiting so much information with inductive biases is using a word net which is like that great hand curated data set and so we see that it like gets and you know because it's able to exploit all this site information of you know helping with like learning oh these are in it you know synonyms or antonyms or this is you know more abstract or less abstract you know a child or a parent and in terms of like


semantic hierarchy of a different word so you can see how that's bringing in the information that should help with generalization and so it actually does better on those kind of systemic evals so this is one way of like widening that peak and someone excitedly though if we just slot GPT one in as well it performs just as well on the more robust transfer setting so there we didn't have to you know manually curate that the relations between words or build Ward Annette we kind of let a language model figure it out and so I think this really again is


one of the proof points that you know some supervisory training is really figuring out the same relation is the same kind of connecting the concepts connecting them and helping with robustness and generalization and there's some new work from Tim Hendricks this week that I've which I have put in these slides showing that Berkeley as follows are much more robust a t-distribution than classic purely supervisor models with like LS TNS or cnn's so I think that's starting to get much more well empirically founded than kind


of me spouting off one or two numbers from like the models I know so kind of at the high level takeaway here kind of this is just a hurrah message for everyone taking this course is you know I really think that one of the most promising methods of moving forward here is in terms of like really lying tasks and robust systems that actually you know perform the things we want them to is we need to move away from standard supervised learning instead of manually specifying what to predict through the creation of large supervised data sets


we need to figure out how to learn from and predict everything out there and you know one of the ways that you can think of this is like every time we build a data set we're sitting the importance of everything in that data set to one and everything else in the world and all the other useful information may be out there is set to zero so like when you start with a model from scratch you should really get in that supervised learning as well as head and be like oh it's almost a hopeless task you know they know so little and we've


hidden so much from them when we only give them this one canonical gold standard data set and of course they're gonna cheat however they can because they're you know great at optimizing the objectives we give them but if they don't have the foundations with which to truly you know build off of all they can do is exploit clever spurious correlations so yeah I think this kind of comes together with all the work we've been chatting about of like a potential recipe for and you know I think this is getting proved out with t5


and all the future work here of how to kind of combine a bunch of pieces together we need high capacity and flexible model classes so they can handle a lot of different tasks we need algorithms for extracting information running the structure across many different domains so this would be basically you know a lot of things we talked about it turned out language modeling you actually just worked really well as one of these it's an incredibly old idea but that algorithm just or you know method just worked quite well in terms of people


there's a lot of different clever approaches to specify and proxy tasks but this simple one has gone it's quite far and you're unfortunately still going to need because these are dumb models that don't you know have anywhere near the robustness or generality of humans you're going to need a lot of data tiling everything but at least it'll be unsupervised and so we have it available and you're going to need unfortunately at least to get the you know the soda grind a little more you can in fed some amount of compute with which to learn


them but again that may produce a model that's actually quite small and efficient to run a test time and I think that's one of the hopeful direction is going for is you you know train these big models and you know Google or Facebook or open the I you know burns the GPU years to to get that model but then you know you're able to distill it and prune in and release it and then it can still run on your own laptop and or on you know a single GPU and you know that means that there's downstream tasks that you may want to investigate or you


know build models on are much more efficient because you've amortized all this compute that went into pre training and now you're able to you know use that during the fine-tuning so it may actually be that like bird is actually you know though it took a ton of compute to train bird and Roberto may actually have reduced the overall volume of compute needed to achieve a given level of result and may actually widen the amount of usefulness and test that can be tackled in the field because it can transfer and you know been and be


beneficial to everything downstream and you know I think it's very reasonable that some people in the field kind of look at all this coming together and are like well you know I don't find that satisfying and so I think that's a valid view and so you know maybe backing up and working towards you know more grounded learning and there's lots of really interesting work in this space now of you know moving towards reinforcement learning and granted learning with you know multimodal agents and all this kind of stuff that connects


to you know more what feels like you know true learning about the world instead of just seeing abstract bits of text you know that I think that's a very valid approach but right now you know we've been just seeing that it's been driving a good chunk of empirical progress over the last few years you know there's a whole other set of methods here that's multitask learning and I think that that's actually been showing a lot of promise in the last when I made these slides this slide last year I think I was a little bit more


pessimistic on it and there's actually been a lot of good work like I'm tdmn and others that's been making progress on on this set of methods but it still kind of relies on us building a data set so for multi task learning you train on a bunch of different tests together and you kind of hope that you get transferred nationally between them but often they're all supervised tasks and t5 is a good paper actually like really talk through the nuances of what's the test learning for gendered pre training and one of the surprising things they


share now is when you do it well and you kind of exactly emulate the pre trained and fine-tuned framework if you do multitask pre-training followed by supervised fine-tuning you still need the uh you still need the unsupervised objective of like math slang which blah blah like but you can get rid of or you can at least find very similar performance in many cases compared to having to do the giant pre-training on you know the full internet for instance so they're still having room left and it's actually improving these methods


and the final one here to just chat a little bit about is some of the fall work we did it and they're open ion gt2 and this is kind of like what I've been chatting through here is kind of like a lot of the motivation that went to this project so we collected more data compared to GPT one and we collected much more diverse and heterogeneous data so we're hoping that we have models that would generalize better and see a much broader set of tasks so it's 40 gigabytes of text 10 billion to go cans 8 million webpages we scale up the


models just because we kind of saw those trend lines and you know I think there's a lot of reasonable arguments for why you just need bigger models to handle complex tasks and it's just a language model which predicts everything so immediately it still left right on aggressive model so it has some drawbacks compared to things like Bert but it's just a language model and so what we focused on in this case was purely how well this mala could do across you know many different tasks in at zero shot setting so we we never


fine-tuned it because you know supervised learning is tricky and it learns to exploit spurious correlations and dependencies so we're only ever saying well you did all your pre training work and we had you predict a bunch of words how well can you handle this new data distribution you've not only never seen before I mean really you know we trained on a lot of data so we actually see a bit of a lot of data distributions you're not letting it like specifically turn specific tasks but that specific label we're just saying run it and see what I


can do and we show that like it actually begins to do something particularly as you scale the model across a wide range of canonical and LP tasks so it's purely unsupervised there's no you know there's no direct human labeling or supervision going on here but this model can actually you know you can feed it a paragraph in the mask of question and you get transfer and linkage well can give the right answer sometimes often they're just matching kind of old baselines and they still have a huge gap to the you know human


performance but I feel like this is a much better measure of potentially what the like underlying performance of these systems might be because we're not doing supervised training here and you know yeah and surprisingly we know our models are still worse than people so that kind of shows up here but it also shows a promising trend line where in some cases like there's very domain-specific algorithms for unsupervised translation middlee it's been a year so that speech should be up here now is there like some great follow up work from Ferran well


there's pushing and supervising on team farther but this is just a language model with no real customization and we're just seeing that it begins to do translation between the cumulation French you know you can tack a tldr on the end of a document and get something like summarization it's pretty garbage on the official metrics because it's only barely matching read three random sentences from the article but kind of quantitatively and qualitatively if you ask people which you prefer it looks a lot better than these numbers show


because this is a kind of very coarse evaluation metric and then the final thing here is like question answering so it's kind of shows domain knowledge and kind of like kind of world knowledge and potentially a lot of factoid information and this one we unsurprisingly see a really strong scaling curve with model capacity so like how is this working how does this kind of unsupervised system it's just a language model begin to translation question answering and reading comprehension well if we go through an inspector data said it turns


out there's actually just like kind of natural occurrences of tasks and you're turning them all to predict the next words so that's you know it's easy an English sentences then you're like then it happens to just have you know inside of the middle of this article that someone wrote a training example of English to French so it's a much more natural way of learning and when you have very large data sets you just actually begin to have a non-trivial data and so you see for translation for summarization like if we just like crap


through the data set how many times does TLDR up here well there's not a thousand training examples in quotes here and how many times does like someone asked her who what where when how why question well there's six million of those so we're kind of seeing that these kind of systems that you know don't make assumptions and silly about any specific task we kind of try to predict everything kind of really begin to make some progress I mean again like one of those areas we saw the most on is this question answering an open domain


question answering where you're just asking like what is the capital of you know Paris or in what year was star wars released and you know I think that this kind of gives you a very clear picture of why supervised learning with like kind of the standard approach just is never going to really be able to solve this kind of task so on the x-axis we have a number of training examples seen and again this is log scale and yeah if you start with a Randal initialize what model there's no way it's going to be able to do question answering I don't


open domain you know there's no way it can have the information for you know what is the capital of Paris until it's seen that training example and there's very little generalization there you just need so much data to approach this from a naive supervised learning approach whereas we have bigger models that have more capacity you know in the limit they very quickly began to do non trivially on these data sets and then they kind of fine tune in and learn how to better extract the information that's somehow contained within the weights to


various degrees so again this red baseline here is completely randomly emitted and these are basically random guessing numbers the entire way through you know those data sets doll is 20,000 labeled examples but as we try bigger and bigger language models we see that they really begin to make personís and t5 I think has continued pushing this quite a lot farther to where they're actually sometimes matching with only a neural model that's never looking at documents with like the actual factoids in them it just from its parameters is


able to answer quite competitively on some of these tasks yeah we're pretty much into a conversational period at this point but um you know some of the takeaways I would kind of say from this and kind of you know really pushing on language models for a few years here's performances you know not usually limited by something single paper fixes this is a very long history of you know I think we probably talked about 25 papers during the trajectory of research here and usually it's always someone chipping away on one


specific access you know diminishing returns basically mean there's always some other bottleneck so if you scale to compute but not the data you'll get back there if you scale you know the parameter size you'll just need more computer or if you scale you know the Moloch caste but don't increase that is that it'll just over fit or you could try to scale via like you know human intuition and you can use fancier models but maybe that's just more difficult to train so kind of I tell you that like you know particularly if you have a


little bit more of an engineering mindset here kind of the pragmatic approach of kind of pushing on all these axes together may allow kind of for a larger effect size to show up than pushing on anyone in isolation this is an unfortunate tension I think in research and science where you often want to you know microscopically measure effect sizes and walk controlled oblations and experiments in isolation but you know if you change a few things together you might actually see morgan outsized effect because that's like one


of the things we do we typically to where we get more data get more you know a bigger model through everything cut it together let me try to see if that really pushes toward qualitatively different behavior maybe yeah I mean I really could transition in a question period at any point now you know there's a little bit more advice at the end just saying that don't work on large scale models particularly you know like as things like a lecture so show you can work on the smaller models and see the same effects showing up they're not


going to have the same accuracy curves but you know we know from scaling laws and kind of all those trend lines that if you start seeing an effect that's robust at small scale probably fingers crossed it'll also hold at larger scale so you can do a lot more more rapid development and you know and this I think works quite well you know you should try ten as many or ten times as many times models that are just ten times smaller each and you know that way you can run 10 times as many experiments in parallel this is still you know a


large research field so there's a lot of things you got to try and you know beat all the behaviors in a paper like GP d2 which kind of I feel like gets pointed to as a canonical like the computer Big Data kind of thing they still show up on models you can train on a single desktop it you know it takes a week to about to see the hints of that middle E but you know gbg small you can train quite well and I've got a week on like a for GPU setup and then after you get proofs of concept on like your algorithm or your idea then you can scale up if


you can have the computer resources you're going to get a you know a low enough time on the cluster or thanks and kind of the same strategy used back to the day with like the seventh unit where the initial proofs of concept were 512 dimensional as teams that took a day or two on standard hardware and then you know for the final version then we kicked off a big run with the model that took 16 times the computer and you know how do you not go insane when you wait for a model to train from month well we like to do this thing at opening eye


where you boot your big model over before you go on vacation so you put that before winter break and you just let her train over the break and luckily fortunately you're in the set of sight machine for the whole time but don't stare at that graph every day you won't make nearly as much progress if you're just staring every day at that number but often models surprise you when you give them more time to learn so you know when you're really trying to push that result at the end it's a really good idea to try that if it's available in


the option yeah and again one of the other spreading things just about this field has been how far we've gotten pushing often where the developers of one paper or modeling architecture let me just push them log probability or the you know type 1 evals and then someone else coming along in another paper and showed oh this thing's actually great a type 2 evals so I think that's you know really reassuring and you know I'd often say that you could work on one or the other in isolation and often you see things that robustly


scale or contribute on both sides you know there's some gotcha as always with scaling you know things break when you at some point you can't extrapolate too far and things just change so you've got to watch out a bit for that you know for like a model like 2 PT 2 real on one of my collaborators was like we were originally trying to train these deeper bigger models and they just weren't working better and we had to fix an initialization technique and rearm came up with this and it helped you know continue scaling so when you see you're


scaling like not happening in the way you'd expect or the try mods kind of suggest it's also in some that something's wrong you need to like tweak it or fine-tune it or come up with like to actually do the clever work I don't do much of that myself to fix it up and try to keep making progress yeah and then the other thing is just like writing efficient and smart code these days luckily hardware is proving and for the same price point so with things like FP 16 1/2 precision compute if you switch over to that with


things like with example being GPT one the original version took 25 days and FP 32 and one generation older hardware and then the same next generation hardware where you you know a lot of people did a great job optimizing this to Scott grey in particular it's amazing GPU engineer and researcher and opening I we worked with some blocks part part the box of our spork is basically his work and he was able to optimize these down by almost order of magnitude on just the next generations hardware from a lot of you know great improvements across the


field so often if you write efficient code and use all the right tricks in terms of accelerating your models you can ring a lot out of the same you know the same level of hardware and just be efficient about that we have a library called the block sparse library that can help with that and provides a lot of these opsin honestly also libraries like right origin are doing a great job merging these in providing their own ops kind of more integrated into these these kind of wrappers so that's I think exciting for


the fellows at all yeah you know in terms of sweet spots for computer you know for 20 ATI desktops still can do along with space they just cost a fair amount of money and then you know your standard 80 100 bucks on a cloud provider is a very is a medium scale compute platform you know papers like electro can do a lot which 'single be 100 and I mean a 20 ATI is basically the cheapy 100 for 4 or 5 times less oh yeah that's about it honestly I think we have where you have about 15 minutes left for questions and you know I have a few more


random slides everyone this this is really great Alec thank you so much let's see if people have some questions hey Alec yeah how question so I was wondering if you could give you a views on like what do you see 0 shot language modeling something that could could could be production quality performance over time or do you think it's always gonna be lower than a collecting supervises and fine tuning some big current model just try to understand like the space between GPT and like Bert like models yeah oh yeah


so you know right now it is absolutely garbage from a production perspective like GPT - I mean well okay there's hints of life there you know for reading comprehension it's it's matching some of the original neural supervise baselines so I'd say there's hints of life there we're still talking about you know you need to do a lot more research and if you looked at kind of those scaling laws for like what kind of you know GPT - looked like like if you draw those out there's still quite a lot of order of magnitudes left to go so from a


pragmatic or practical perspective it's not really there right now and that might be the scary answer which is you know our models do rely on exploiting and you know I don't override this view but you know like it may just be to actually do these tests correctly you do just need you know much more compute and something like the zero shot setting so it's kind of like working I think I see it kind of is like working with you know like letting shoes or something like resistance training I think it's a fascinating research area to push on


because it does have some of these exciting qualities from like maybe representing the you know much more difficult and hopefully much more true representation of test performance but yeah it still has a long way to go so I think it's a fascinating research direction but here's a lot of pushing to be done on that thank you and yeah I think for a pragmatic perspective like you said you know you really should find tune on some supervised data and you know like I mentioned Burke models are still showing quite good robustness out


of distribution there I don't think there's been any good work comparing pure zero shot to learning of a task to like supervise the fine-tuning of a pre train model but I think we're talking about something that's like a few years out at least thank you thank you I saw a question here earlier you motivated Ellen's by comparing probabilities of pairs of strings to exact knowledge such as cats at first cat sets has this intuition comparing sentences I guess with exact knowledge been used for training general models a text or


Maxo like that ubiquitous so maybe this is about a comparative or contrastive method for training generative models where you compare sentences and know that like one should have higher probability than the other there was one one paper for reps in tation learning perspective which it's not quite the generative model side but it's representation learning CPC you know is that whole family of contrast methods is dominating you know unsupervised learning for image representations so it's somewhat of a contrast where in NLP


we haven't seen it really tick off yet so I think it's a very exciting research direction in the original CPC paper actually had some results that were promising on natural language but they you know like the original CPC paper in general we're exciting but nowhere near stay the art and a lot of the refinements in the last year or two on the image side really pushed that quite far I think you might have had a lecture just on that or about two so it would be very cool to see if someone could do that kind of similarly for natural


language but if it was about kind of exploiting more structured knowledge about like differences and encoding that into the generative model there is some pretty interesting work on this particularly from some more the like linguistic heavy folks and field of combining kind of hybrid systems of you know neural and with like kind of something like grammar constraints or the like and it's you know I'd say it's primarily focused a little bit more on you know the settings where you might expect encoding that inductive bias to


help which is like smaller data sets but you know at least personally my kind of I find some degree of at least from a pragmatic perspective a lot of current language modeling benchmarks I think are quite artificial because they work with such small amounts of data which from pragmatic perspective just doesn't make sense because there's all of what could be out there there's it's so easy to just write a scrape or your stuff or download a shard of comic roll and that's more data than you basically ever going to need to work with or you know


be able to process and so I think at least from a pragmatic perspective we should really be for how to use the large volumes of data we have you know I think it's a very valid other approach to push on data in isolation and you know how how much you know how data efficient we can get with limited set of data but I think it's probably add just to farm in extreme when you have you know only a million words of training data and things like country things so Alec a follow-up question on that it seems like one way to to learn languages read the


entire internet right another way to learn language is the way I think most people learn language which is absolutely you kind of I don't know how many words or how large the data set would be that somebody encounters by the time maybe they're six years old and they can speak pretty well maybe at that point they have any notion of kind of how much data is required in that context compared to how much data is required here oh it's it's awful at least for you know in for neural models I think it's um yeah for like a six-year-old child I


think it's maybe you know I just bashed on 1 million words being unrealistic but I think it's about one to ten million so you know compared to GPT to being ten billion tokens there's orders there's three orders of magnitude at least of headroom there potentially and i think that again understandably motivates why a lot of people do work on that city but my guess would be that to really make progress in that setting a lot of that is because of transfer between modalities and you know actually you know interacting with very


high quality sources of supervision like other people and you know being you're grounded agent that interacts with you know video and audio and like i think that that research is very interesting longer-term and you know we're probably going to saturate kind of what we can do with these ungrounded giant systems in the next few years or maybe it's even already starting in the last year so that's like very i think exciting next round of work and clearly like the numbers just show there's a huge amount of room to go got it thank you makes


that work well for when you are laying apply to like other abilities like video surgeon educators okay yeah so genetic did is actually a great example there there's a really I think Joshua Meyer and collaborators between I think was it proud and it's an NYU team in slush fair I think Rob Fergus is not working a lot on this so they took Bert and they applied it to protein sequences or I think sorry amino acid sequences and probably I don't have strong bio background but much of bio background but they were showing that


the same methods or you know learning like a lot of the structure in those different domains so like kind of the central unit analysis or the sentiment you know example I gave for pure language there was also another paper from I believe church the church lab at Harvard where they took like literally my code and ran it over amino acid sequences and we're showing that there was like instead of a central unit there was like a like a beta sheet unit or so current course finding like secondary or tertiary structure of proteins the


models were having units that were like understanding you know or even though these are like very nonparametric kind of abstract models that just like you know have a bunch of parameters that just factorize a probability distribution they're somehow learning the structure of the domain or hints of that so I think that's very exciting and that's another line of work I think given how exciting this stuff has been for MLP and how much of an impact it's made over the last few years whether it could work in other domains would be


quite interesting you know there's definitely differences so for video I think video just needs so much compute that it's like still maybe quite a few years off just because of the volume of data and you know the amount of compute that might be necessary but maybe I'm just being cynical there whereas I'm images you know there's a weird contrast which is like I mentioned the contrast in methods are doing quite well and if you just run a generative model where you know actually okay that's not quite right there's one paper from deep mind


called big by again where they took it immediately you know pretty different generative model and they were showing that those are starting to learn quite good representations or images at least from the standards of unsupervised learning still being crushed by the latest moco's or sim clears but they're you know they're quite promising and you know showing a kind of a foothold of this generative model kind of approach in other domains and maybe you know one more piece of context to shine on there I think there's some one of a nicety to


language because it's produced by you know people it kind of is naturally designed to be very clean and very high-level and yeah it removes all the noise so when I think we run and try to train the same generative models or approaches in domains like images or video it may just be that like when you're dealing with raw natural audio signals are you know sorry not raw natural signals they have so much noise like particularly a likelihood based generative model is just like spending so much effort and capacity trying to predict all that


noise and the you know this noise to signal ratio it's just a lot cooler that that just like makes it a much more difficult task right now yeah you know it's it's I think it's a very interesting research question so Alec we're about out of time here give any closing thoughts oh yeah let's wrap it up we're mostly there I guess you know what one thing again is like you know III one of the things that I really enjoyed about being able to have the opportunity to this talk was kind of going through and showing that full history here and


kind of you know I think it's it's a great example of how there's so many pieces that built on top of each other and you know there's so many different authors and so many different institutions that really contributed to this and you know even given that open area there's been a lot of climbers that have pushed on this stuff over the last few years and you know it really you see it just evolved like so many different pieces of the research with all the different you know things being brought to bear new models new datasets you know


hello welcome to lecture 8 of deep and squeezed learning today we are going to talk about the strengths and weaknesses of various narrative models and representation learning methods that we've seen so far so the brain has 10 to the power 14 synapses and we only live for 10 to power 9 seconds and so we have a lot more parameters then then I'm the data we ingest so this motivates that we should do a lot on scores learning because in order to provide sufficient fodder for the number of parameters that we have in our brain we should be able

L8 Round-up of Strengths and Weaknesses of Unsupervised Learning Methods -- UC Berkeley SP20


to predict a lot more bits from the data that we ingest which is 10 to the power 5 order magnitude smaller right so this was a statement made by Jeff Fenton in this 2014 in a reddit so firstly you summary of the course so far we've looked at or aggressive Morrow's fix learn and pick so CNN picked two skin and paws glass pixel snail we looked at four models really only P family of models and also the connection between Auto regressive flows and in with Auto regressive flows next we covered latent variable models models with approximate


density estimates using the variational lower bound and various associations of that like the VA e importance rated our encoder VQ BAE pixel BAE and so forth we also then jumped into a different class of jeredy models that don't work with the likelihood principle the impasse density models against energy based models in the moment matching principle and finally we questioned the idea of like whether we even need to learn generative models if all we care about is extracting useful features from unlabeled data and that God isn't with


this topic because house provides representation and we saw that with the right kind of simple cognitive principles and a lot of data and compute we can learn really useful representations of unlabeled images that are competitive with supervisor plantations so represent so let's let's look at auto regressive models used in the in 2015 the main paper was Bush with which introduced this idea of masked or encoder for density estimation and it was able to produce these I'm miss digits which were reasonable looking but very jittery and


this idea was extended to much stronger architect more expressive architectures well-suited for image modeling like masking illusions this individual introduced in the pixel or an analytics is seen and family of models and you certainly started seeing generative models working for higher dimensional and much more diverse assi Commission ad so these are samples from image net 64 by 64 you can see that the the structure across 4,000 pixels is pretty coherent but the color is not that good and therefore you're not actually able to


identify any visible class from imagenet but this was a big big jump from the quality you saw and made and this idea of mass convolutions it has also been applied for one dimensional data like audio and in order to model long-range cover in audio samples the idea of using dilated combinations was introduced and this was also applied for a text-to-speech system where you're going to convert linguistic and text features to raw audio and that can be used in any Indonesian assistant like the Google assistant and this was the Wayman


architecture that was commercially deployed after after a year and the same idea of using mass conversions with Auto regressive pixel level modeling has also been applied for Rio prediction why are you looking at the pass frames and encoding them with a convolutional STM and then you're taking the embedded representation as a conditioning information for a pixel scene and decoder that generates the next frame pixel by pixel and it's able to produce coherent video look like a robot moving out to Tehran so over time


the order aggressive modeling community has expanded further and further in terms of the level of engineering and architectural innovation and on the left you can see if the subscale pixel networks which have very coherent samples because of the clever conditioning can assume to use on the right you see hierarchical auto regressive image models with auxillary decoders where the idea of using latent space auto regressive models was introduced by quantizing representations or encoders and and modeling pixel CNN


the latent space which is also similar to the vqv a a idea that you seen in the VA a lecture so apart from images and audio and video auto regressive models have had immense success in language and these are samples from GPT to riots it would actually produce a coherent story about unicorns and like a like a story of how unicorn skin when their own language and also talks about a scientist who is able to observe all this phenomenon and this shows that language modeling at the level of a paragraph or even multiple


paragraphs as possible by just training large models which used to order aggressive structures this slide shows the evolution of language models over time we're on the first you see Shannon's three grand models which I reasonably good but not super coherent across the full sentence and then Ilya sutskever is model of using an RNN is able to produce a couple of sentences but not completely making sense and then over time they using bigger LSD and bigger transformers you ended up with the quality that's UPD to experts right now so all these huge


advances have been possible due to multiple reasons and let's go through them quickly the first thing is just being able to train with larger batch sizes because of more computer availability and training with larger bad sites then we stabilizes the training of these models and optimizes these losses much better making the models wider making a modest deeper figuring our clever race condition your next year you're building a conditional class conditional or audio condition or text condition model the figuring out


ways to get the conditioning information cleverly is very useful pre-processing like in wavenet we use some new law pre-processing to quantize continuous audio into discrete entities are for example in pixels you're actually using categorical information for modeling rather than rather than using gaussians so these are these are and in language you using by parent coding which is pre trained on a huge corpus and therefore your mod in on modeling neither at the character level or at the word level but your modeling in the sub word level and


that's much more useful for generalization and also building more efficient models compute power and as we progress in the last two three years we just have at we just where access to a lot more compute like TPU so I like big GPU rigs which have lots GPUs connected really with a really fast interconnect and therefore be able to train data data parallel model is much better and we're to train see that several weeks or basic training are usually producing much better results and also making fewer assumptions about the whole problem like


before trying the idea of this predicting categorical distributions for every pixel why would he want to imagine that pixels are definitely gonna be modeled with calcium's instead of categorical distributions like indy really doesn't make any sense but then practically it's better for a neural network to work with cross entropy losses there are also been architectural advances that made all these was much better so mass conversions were applied in the original Pisa CNN but as transformers and dilated communist art


exists the samples just got much better with more coherent structure across long range dependencies and and making the whole modeling problem look more like supervised learning helps a lot and therefore relying relying heavily on oh they'll be here crossing will be lost and optimizes that have been much better tuned for this loss ensures that generative modeling can also benefit from all this but engineering advancements so now what's the future for our regressive models we're only scratching the surface of what's


possible and and once we have motor pilot training we'll be able to realize a lot more for instance be able to train trillion parameter models on all of the Internet's text and that that way we could compress all the Internet's text into a giant neural network that can be a like a know-it-all language model and secondly we can figure out ways to Train one single model for multiple modalities just even bigger generative model they could work at a video level on YouTube or image level Instagram text level Cabiria so that way it's able to


probably correlate information across multiple entities and chameleons for expansion so for all these kind of modeling requires hardware and software advances from auto pilot training we should it's also possible to make or aggressive models more useful by figuring out faster ways to sample with better low-level primitives at the CUDA that will like for instance fast kernels and and better act like for example wave are an N uses all these mechanisms for production components and doesn't need to be distilled into something like a


parallel bayonet this work as a standalone auto regressive model and still be deployed on an Android phone hybrid models with much weaker or aggressive structure but that can be trained on a large escape could be revisited and and of course all these architectural innovations that help in long-range dependencies would always help in you know as you keep moving to bigger image this or a video or something like that these kind of ideas should up a lot so like a summary of auto regressive model could be that it


is an active topic but a lot of cutting-edge to us and there's a lot of moscow for a new engineering and creative architecture design and larger models and data sets are clearly needed to you know realize the full potential of these class of models and standalone they are very successful across all modalities without any conditioning information like class labels so that's that's like a very appealing property of these models every Universal in that sense and also they can work without much engineering for sampling time so


that makes them really look creative but but but nevertheless for production if you you should really cut down on the sounding time to be useful and so innovating on the low-level primitives was very important so that said there are a lot of negatives for aggressive modeling one is you don't extract any representation there is no bottleneck structure and sampling times not good for deployment it's not particularly usable for downstream tasks like for instance a language Maru you need to sample multiple times to see coherent samples


so you can't just roll out a language model that's a software and there are no interpolations that you can see to visualize what the models actually learning and every time you sample it's going to take a long time to produce like a diverse set of samples so that's it about auto regressive models now let's look at flow models in flow models it all started with the nice architecture by loaned in and those the model was already producing very good digits on the endless data set and on the T of tedious it was producing


reasonable phases but it really was bad on see far and SPH India said the samples were very blurry but it all improved with the real end we'd be architecture which introduced other kinds of flows and rational room to make the models better and then the glow model from King model was published where the real and Ruby model was taken to another level by making it prettiest much larger images and overdone in our lab called flow pass class advanced the likelihood scores for flow based models to competitive scores that with that of


autoregressive models for the first time and this was done by this architecture engineering and scale so this shows the power of flow models of potential they have in terms of closing the gap in density estimation between autoregressive models without having the powerful or aggressive structure but at the same time being really fast with sampling and also potentially useful for inference so given all these practices there's a lot of future work left in terms of how to learn the masks how do you actually completely close the gap


with our regressive models whether you want to use very expressive fluids but very few or whether you want to use shallow flows which are not particularly expressive but then keep on stacking them so that you can get a very expressive compose model how do you use multi scale losses for a trait and how do you trade off between your density estimates and your sample quality and how to use the representations you derive at various levels of the flow model for downstream tasks all these are like fundamental advances think about


for flow models and also how do how do you carefully initialize so that flow models can train very fast so in terms of core achievements that you can aim for you can aim for producing low level samples which are truer models that have way fewer parameters the globe uses half a billion parameters for all the celebrity faces and that's unlikely a scale and how do you make it work potentially for even larger images how do you do dimensionality reduction with flows and think about other other flow models like conditional flow models and


you know how do you actually close the cap and sample quality de Gans and also close the likely skoros gap between autoregressive models so the models would provide the pathway to do both and it's it's interesting to think about how to do all these things together so the negative of flow model says you expect to have the same dimension at every layer every stack of the flow and so it's unlikely to scale if your data is getting bigger and higher dimensional and unless you innovate on how to do dimensional reduction sauce it's unlike it'd be


useful and you really need to carefully initialize and use things like AK norm for good numbers so that's that's another negative because it may not be directly usable for another modality or another data set or another kind of architecture so let's look at late engraver models will see the various different be strengths and weaknesses and what have been some visible successes in bas it all started with the original Emnes modeling by dirk Kingma where you could see various types of digits and strokes and the slopes of the strokes


and shades across multiple digits and then it got extended to much better more powerful data sets like Elsa in bedrooms by pix ovae and also image not 64 by 64 creating much better global sound globally more coherent samples 10 pixel CNN because of modeling latent structure and then there's the latent variable models innovation in terms of using hierarchical models and multi stack using hierarchical Laden inference and producing really high quality sound really faces on par with slow models so there are well-known applications of V


like sketch iron and role models and BW is used for modeling visual concepts and there are applications like deep mines jeredy cry networks which does view synthesis of a separate view by taking in two provided views and embedding into a latent rifle and interpolating the lane space for a query view across across multiple possibilities and therefore you can just collect data in a completely new environment from first-person vision you can you can keep a track of all their poses when you're recording things and then in principle


you could figure out how a particular scene looks like from any other viewpoint and therefore reconstruct the entire room or entire environment completely through this kind of a synthesis model that has rational inference so we have practically used in these kind of architectures and there are lots of advantages of EA's you get a compressed bottleneck representation you can get approximate density estimates you can interpolate and visualize what the model learns you can potentially get disentangle representations where


different readings correspond to different aspects of data and it is like a model that allows you to do all these things together at once like you basically can sample so you are a gyrator model you have a density estimate so you can use for our distribution detection as a density model you have latent variables so you you do representation learning and you also have a bottleneck representation so you are able to reduce the dimensionality of your original data set so a VA is the only model that lets you do all these four things together and


that makes it very appealing that said there are disadvantages you often end up with Lurie samples and assumption of a factorize Gaussian for the posterior or for the decoder this may be very limiting and you need more powerful decoders or more powerful posteriors and large scale successes are still yet to be shown and even though people have tried to like get more interpretable more disentangling variables by prioritizing the KL term over the reconstruction term the last it's still only work on toy problems and they may


actually be better ways to do representation learning or generation or like yeah interpolation in some form hierarchical Layton's individually so expecting for one model to all of them well may be truly hard and so a we may not be the state-of-the-art models on anything but maybe a model that lets you do all all that it all these things recently well in using a single single single modeling framework so that's that that's the that's what you lose when you want is everything within one model so that these are the disadvantages to me


but there's obviously scope for future work you can but you can use bigger decoders more powerful posteriors you can think about how to do hierarchical Leyton's to learn covers and fine-grained features and discrete Leyton's like weak uva and also large scale training like slow models have been done like glow or focus bus so next let's cover implicit models but we look at general adversarial networks and just just basically what what's happening ganz though we also covered moment matching energy based models in


class the Gann samples the quality of Gann samples has dramatically advanced from the primitive samples that you saw in the original Gann where you saw X reasonably looking good faces but then the c4 samples it's not pretty cooing too critical in terms of what is the object or class of C far that's been captured but it certainly looked different from Larry BAE samples at the time next you saw DC Gann which clearly advanced some the some quality of dance to a state where again to assign you a looking much and much more exciting than


any other model because the samples were much sharper and all these bedrooms were very high dimensional and then recently again giving again has been taken over by began stag and classic models were clearly careful attention to detail in terms of architecture design and also really really large-scale training like large pad sizes and a lot of stabilization tricks can produce these amazing photorealistic samples that you've already seen plenty of times in the class so I'm not going to go over them in terms of future work for Ganz I


think I think it's really hard to bet against cans to say hey this is work cans weakened its most likely that if you put sufficient effort in engineering you can get it again to function well on those things as well but but nevertheless there's still more progress we made an unconditional cans more collapse and also more complex scenes and video generation will be cool for instance will be nice to get a model that works on real driving data where and a lot of pedestrians are walking and then you want to be able to simulate future


you have to keep track of multiple people multiple objects multiple cars road signs and so forth so it's a very complicated jeredy modeling problem and it'll be interesting to see it ganz which are known to identify only a few cues in your dataset would they still work in such complex settings where you need to keep track of multiple things at once so future work in terms of modeling you can like think of more purchasable Lipsius knows better conditioning tricks like how to feed noise if your various levels like for instance stai again basically


innovated their batch or instance normalization of how to design better architectures working on sampling and down something ops to use how do you how to do channels of sampling and done something without introducing a lot of parameters what is the right objective function for your discriminator and how to scale and train ganz in a stable manner for like larger problems and how to preserve it at various different levels like how do I instance noise a feature noise so that it can stabilize the training of the discriminator much


better so all those things are very very interesting and think about in terms of negatives again scans if one could say there's plenty of engineering details and it's hard to clearly identify which is the most important core component that helps you reproduce these high-quality images and it's also very time consuming to ablate for these details so and and and and and it's very clear we need to improve on the sample diversity but then we also don't have very good metrics for evaluations so we need to work with what we have and even


though it may seem like we're improving a lot on the current metrics we use for again evaluations objectively the sample diversity is not a spurious likelihood based models so how do we actually come up with better valuation measures also one thing to think about with all these aspects like good evaluations good metrics relations these are not particularly specific to the scans these can be said for any any any kind of model as with any other model so if you were to make a choice between Ganon or density model


one would imagine you need a lot of engineering details for Ganz but it's not particularly true even for density models the architectural engineering has been comparable level of detail and you know trickery that you need for Ganz and secondly there is a lot of attempted theoretically understanding Ganz so the trade-off between having blurry samples versus of being okay with mode collapse is basically the same trade-off that you make when you care more about compression at the cost of sample quality was this you wanting to have


really good samples at the cost of missing some modes so it's basically which direction of kale that you care about and the reverse direction you care about more if you don't want any spurious sample but the forward direction you care about more if you really want to make sure that your modeling is good and you're not going to make any mistakes even though your you're not gonna miss out anything you in there you may make some mistakes at some of some of the points so mostly apart from the fact that they can produce amazing samples cans are


popular because they can work with much less compute for instance in order to generate a 1 megapixel image for an auto regressive model or even a Leighton space our aggressive model you need to use at least 512 course or TPU to do that because you need such large pad sizes whereas for gans you can make it work with a single V 100 GPU and then so there so that's that's one reason why gangs are clearly preferred over than 10 C models because I'm amount of time taking the train as a sample and you can also see better interpolations and


better conditional generation in cans so this dis leads to adoption by people who are more interested in art and fine tuning to like interesting artistic datasets you're not particularly machine learning relevant and that's one of the other reasons again a speaker plot so on the bright side we can think about how like many technological advances have been possible without the correct science and so ganz can we consider in that way as well and this is a slide from young Conan the epistemology of deep learning where explains that