feedback often involved and sometimes your ensemble in predictions of three five or more laborers it's often a few hundred thousand examples is like a big data set especially for NLP in computer vision you sometimes see you know things like imagenet where they push that to a million or ten million but those are kind of afar outliers and you know very many canonical NLP datasets might only have five or 10,000 labeled examples so there's not really a lot of machine learning great data out there at least compared to what kind of the current


learning complexities and efficiencies of current models are you know one of the primary criticisms of modern supervised learning deep learning in particular is how data intensive it is so we really have to get that number down and this lecture is basically going to be discussing all the variety of methods that have been developed for using natural language that kind of is available beyond kind of the machine learning great data and unsupervised or scalable self supervised methods for hoping to somehow pre-trained do some

L11 Language Models -- guest instructor: Alec Radford (OpenAI) --- Deep Unsupervised Learning SP20


auxilary objective or tasks or you know hand design some method that allows you to improve performance once you flip the switch and go to a supervised learning on the standard machine learning great data or in the limit as we'll talk later get rid of the need entirely to have a classic supervised learning data side and potentially begin to learn tasks in a purely unsupervised way and evaluate them in a like zero shot setting so there's a variety of methods this lecture is going to focus primarily on auto regressive maximum-likelihood


language models they're kind of the core and I think they're the most common uniting thread that kind of carries the early days this field through to kind of the current modern methods but I want to you know make clear at the front that there's many proxy objectives and tasks that have been designed in actual image processing to somehow you know do something before the thing you care about in order to do better on my thing you care about and there's quite a lot and in particular in the last year or two we've now seen that area really kind


of grow dramatically and in many cases they now I'll perform the standard language model based methods that I kind of will as the core of the presentation and we'll talk more about the details of the differences as we get to those parts so some more motivation in intro as we've kind of going so I think I think one of the ways to think about this is like what do we do with the Internet so you know the wild Internet appears and you can either have your glowing brain ask representation on the left we can laugh at or we can make it you know how messy


and random and weird and difficult it might be for algorithms to learn from it on the right so that's good old Geocities and so you know there's a lot of skepticism I think about kind of these approaches that might kind of at the highest level look kind of silly or kind of whimsical to be like let's just throw an algorithm at the internet and see what comes out the other end but I think that's actually kind of one of the like one seven summaries of basically what modern NLP has been seeing a lot of success from and you know I think one of


the reasons why is just the Internet is so big there's so much data on it and we're starting to see some very exciting methods of learning from this kind of messy large-scale and curated data and so there's a great tweet from from an LP researcher just kind of showing just how is are they big and you know kind of just massive the Internet is where you can go and find an article about how to open doors and you know there's often a lot of arguments saying that oh you know we're not going to you know and it feels


wrong in the limit to be like yes let's just throw algorithms at the internet and see what happens like that doesn't match human experience that doesn't match kind of the grounded embodied agents that you know we think of you know intelligent systems and instead is this kind of just like processing bits or abstract tokens and so there's a lot of skepticism about this approach but I think that just quantities of scale and other methods play very well with current techniques and you know you see lots of arguments about things like oh


there's this long tail and we're never going to be able to deal with composition and really it's just maybe brute force can get us surprisingly far in the in the new term not saying that these methods or techniques are and I'll be all but at least today there's I think strong evidence that we shouldn't dismiss this somewhat silly approach at a high level so let's start with kind of I think what would be the like simplest starting point that we can convert from this kind of high-level idea into something that


looks like a machine learning algorithm so we process a bunch of texts on the internet let's say and we're going to build this matrix called the word word co-occurrence matrix and so what we can kind of think of is it's a square matrix where the ith entry corresponds to for a given word like water the count of another word and whether they co-occur with each other so it might be you have to define when a co-occurrence is so that just means that the two happened to be present together and you might define a window of this for instance they both


occur in the same sentence or within five words of each other or in the limit you can go quite far with like just happen to occur in the same document on the internet and so you're just gonna brute force kind of countless it's just counting that's all it is we're just going over you know tons and tons of text and we're just building up this this table basically so just a lookup table and it just tells you oh the word steam and water co-occur 250 times or you know the word steam is just in the data set 3 to 24 times total or you know


the words hot and water you know 19500 forty times so that's all we're doing and this is a way you know one this is incredibly scalable you can just run a spark job over the entire internet with this kind of system you can quickly get this giant table and it's you know I'm not computationally intensive it's just counting and processing and tokenization this thing can be run on a common desktop and get very far and it's simple it's just counting so how good is counting a bunch of stuff like we're we're talking about something incredibly


basic it's just kind of how often do these two things occur together and I think you know one of one of the big takeaways that I'm gonna have a lot of during this presentation is just how far these simple methods that are scalable and with large amounts of data can get so this is a great example of a paper called combining retrieval statistics and inference to answer elementary science questions it's from Clark at all AI - from 2016 and what they do is they take the same data structure this word word co-occurrence matrix


they did let me start with the task so the task is elementary science questions so it's just I believe through 5th grade kind of you know elementary school kind of simple settings questions so they're multiple choice therefore no possible answers and there are these kind of simple things like a student crumpled up a flat sheet of paper into ramble what property the one who changed hardness color master shape or you know what property of a mirror makes it possible for a student to see an image in it is it volume magnetism reflectiveness or


connectivity so this is a kind of thing that like you know again it's pretty basic in terms of like the high levels they're you know relatively simple facts and they don't require all that much in the form of reasoning or comprehension but there's still the kind of thing that we do give to is you know kids learning about the world and so you might think that like oh you know this is the kind of thing where to understand a mirror you really need to you know exist in the world and to you know learn about all these properties or to have a teacher


and how are we gonna get there we're just kind of this brute force thing that just counts a bunch of words and puts them into a table and then starts looking them up and you know the takeaway here is that it can work surprisingly well so you can't quite pass these examples so the specific solver that we're gonna use talking about in a second is the PMI solver and that gets to about 60% but random guesses 25% so we basically almost you know have the error rate and get to addy with just this very dumb brute brute


force approach so what actually is this solver they call it the point-wise mutual information solver and what you can think of it as is it just scores all of these possible answers so we have this sentence of context of you know the question and then we have you know four possible answers so what we do is we loop over the basically the sentence and we just look for the word toward co-occurrences and we just keep counting them up and we use this scoring formula which is the log of a ratio between two probabilities the first the P of XY is


the joint which is basically the co-occurrence and so that gets you that count that's basically looking it up directly from that table the IJ entry for XY and then you normal by this kind of baseline assumption which is that the words should not Co occur more than by chance so that would be just their gonna depend of probabilities multiplied together as you can imagine those may be quite small and product into two gether makes them even smaller but some words co-occur together so a mirror occurs with reflective or


you know electricity occurs with lightning or you know crumpled up might co-occur with like hardness and so that's all this method does is it kind of just says these basic associations between words and that can get you surprisingly far it doesn't feel like real learning you know maybe and it does it's definitely not very human-like but it's just an example of kind of the power of basic methods and how something that you know doesn't involve any you know you know intelligence or hand waving that we might make about you know


complicated systems it's just a big lookup table you know a smart job but you might run on the internet and it can give you surprisingly far so there's a problem with working with these word to word co-occurrence matrices they're huge so let's say we have a million word vocabulary so we have a million words by a million words just to have the full version naively and then you might store it with in 32 hopefully you don't need them in 64 so that's four bytes so storing this whole matrix in memory in a dense representation is four terabytes


you know that's still huge for today most machines don't have that much memory in them so and you know if we were to kind of like start working with like how do we use this system or how do we kind of make it more general you know we just have this matrix and there's you can definitely design hand-coded algorithms to let go look up entries and query on it and we see that they can get quite far but you know we'd like to do more and how does this slot into NLP more broadly so we want to come up with a more compact but faithful


representation of the relations between the words and the information they represent and we could just say that we really just want to find a way of representing this J co-occurrence matrix as something more like what we know from deep learning and machine learning in general so here's the algorithm called glove from and in 10 Li the Stanford NLP in 2014 so we take that matrix of word word co-occurrences like I mentioned it's cheap so you can run this thing when we like a trillion tokens and each entry X I X IJ would be the count of word I


co-occurring context for J and what we're going to do instead is we're going to you know learning an approximation of this full matrix and the way we're going to do it is we're going to say we're going to redefined a word as a low dimensional or at least compared to you know a million by a million matrix much more low dimensional vector so we're gonna learn a dense distributed representation of a word and all we're gonna say is this very simple model such that we're trying to predict the log prob or the or the log co-occurrence


counts of the X IJ entry and then we're going to do it is we're gonna look up the rector representation of word I and the vector representation of word J we're just gonna say their dot product should be proportional to the log occurrence count and that's all this is and so it's really simple and you can just use a weighted like square to error loss so that's what this this FX IJ is a basically a weighting function to account for the fact that some words are way more common and you don't want to over train this thing on like those


words and you might also want to like clip because you might have like extremely long tail frequency distributions and things like that so but at the other day you just have there your ID WJ and you had some bias terms and you're just trying to compare that to the log of the rock codes count so this allows us to go from that giant m by m matrix which might be a million by a million to an M by n matrix where there's M words and each is an N dimensional vector and often this turns out that these can approximate that full


co-occurrence matrix quite well and they're much much smaller dimensionality so they might be just 300 dimensions and you know there's a question of what does this thing learn and how does it approximate that but empirically it just cannot compress it quite well and this might make sense because you can imagine that so many many words just never occur with each other all that often and in fact simple sparse storage of that full matrix can get a lot smaller already but then we work mostly with them distributed representations these days


in deep learning so we're going smash it into the framework we know there's another word of this yep the question so do you still have to first build the full matrix and then you run this or so this is a way of having had the full matrix you then run this as a way of like kind of compressing or we representing the matrix chronic Thanks mm-hmm so now as an example where you don't have to build that full matrix so there's another variant of very similar kind of and I think usually a more well-known version of kind of an


algorithms class called word effect and so weird effect is instead a kind of predictive framework where instead of saying we've got this kind of you know abstract like co-occurrence matrix then we're going to try to like compress it and we represent it as word vectors we're gonna just work with natural sequence of text so you might have you know a five-word sentence like the cat sat on the mat and what you're gonna do is there's going to be a model that's trained to take a local context window like you know the cat said maybe two


words of past context in two words of next context we're going to do an incredibly simple linear operation like summing them and then we're just going to try to predict that word in the center so this is called the continuous bag of words representation continuous because it's a distributed representation bag of words because the operation that composes the context is just sum or a bag and then we just predict the output and we can parameterize that as like the log probability of the word in the center of the context and there's the inverse


version of this which is the skip grand model which given a central word of context tries to predict the window and so this uses kind of the more standard approach of like online training and it just streams over a bunch of examples of text you can use mini-batch training it looks like your standard algorithms now the same way I mentioned some tricks with like the using the log co-occurrence or a real waiting function you need those same kind of things here again many words span many different ranges of frequencies where you might


have words like the be literally 7% of all your data so you if you now usually train or direct algorithm without subsampling or resampling based on the frequency distribution seven percent of your computes going to modeling the word though and then you know some important word in New York City or something or phrase is just basically lost in the noise so we use a real weighting function I believe it's the inverse fifth root so it just works and that just heavily truncates the frequency distribution so they're basically doing the same thing


today this is a predictive framework where it's it takes in a sequence and it tries to predict some subset of that sequence with a very simple linear model and you just have word of the same word event betting table we talked about but they both do about the same thing and they're kind of the canonical first round of distributed or scalable kind of unsupervised cell supervised representations for a multi again there's there's no you know human supervision classically involved in these algorithms they just kind of have


this automated procedure to just churn through large amounts of data and you know we're Tyvek came out of google in like 2013 and you know one of the first things that is written on a big city you cluster with like a very efficient C++ implementation and shove a bunch of words through it and it works really well and so let's kind of talk about what this does so for this graph I'm gonna talk about how I'm gonna interrupt for a moment if you go back yep so on the Left it's the word words are represented by vectors and then your


average and you're supposed to get two vector representing the middle word on the right where did he embedding slid they're the same embeddings wte so they're both inputs and targets so you you would basically slice out some word WT from from your from your list you would then also pull a sequence of context to be predicted like the were before and the word after and then you would have the same prediction objective like the whole or all of that word at that location and there's other approximations that kind of just


embossing over right now how to do this efficiently because computing a full normalization of the predictions over like a full million size vocabulary is very expensive so you often you can use a tree structure or a sub sampling algorithm where you might normalize over only a randomly selected subset and you can weight that subset and things like this all production negative sampling is a prediction some kind of inner product between WT & WT - - or yeah so that would be how you'd get the logit for the log problems it's a


dumb profit as well yeah sorry I should've been clear about that operation thank you cool thanks Alec so yeah what do we do with these things so this is where kind of a lot of the first wave of kind of modern you know modern modern is a contentious word but kind of NLP starting to leverage large-scale and supervised data started figuring out how to use these things so these examples on the left are with glove and what we see is kind of a suite of tasks so there's the Stanford sentiment tree bank which is predicting for a sentence of a movie


review is it a positive review that they like the movie or is it a negative review you know like movie IMDB is another central analysis data set but it's a paragraph of context t-rex six and 50 are classifying kind of types of questions like who what where when and SLI is a much fancier thing of logical entailment so it's kind of measuring the relation between two sentences a premise sentence and a hypothesis sentence and you're basically trying to say given the premise does the following sentence follow logically from


it it being tailed is it kind of irrelevant or containing information that's maybe correct but maybe not wrong which would be a neutral or is it actually a contradiction with the previous sentence so you know it might be the first sentence is like you know a woman is walking a dog and then the second sentence is like a man is playing with a cat and that would just be a contradiction of the first sentence so that's s Noy and it's some sensible objective and it's kind of this more complex operation because it's doing


logical reasoning supposedly and it's doing it over semantic concepts like you might need to know the relations between playing an instrument or you know that saxophone is an instrument so that if the premise is a man playing saxophone you need to know that the hypothesis might be you know in tailing it if it's the man is playing a musical instrument so that one has like kind of an interesting relation to some more semantic content and the final example here is squad which is answering dataset so you get a paragraph


from Wikipedia and you have to predict you know given a question what the answer is from that paragraph and so for all of these data sets again this is a pretty broad suite of tasks you see multiple absolute percentage performance jumps from slotting in word vectors compared to randomly initialized components of the models that were used to predict the so you can always do random initialization kind of standard canonical thing and deep learning or you could use these pre trained vectors and so they really do seem to help in terms


of data efficiency and you can see in some cases like for question answering that you can get a 10% plus absolute improvement here for glove glove plus code is another thing which we'll come to in a bit and you know why might these be helping so much so that's the kind of empirical data well on the right here we kind of have some of the work that God did to kind of inspect the properties of these word of vectors so they would for instance have a query vector like the word frog and then they would show all of the different possible nearest words


in terms of just cosine similarity to that first word so you can see that you know immediately it's the plural version of it frog two frogs and you know toad is very similar to frog Ronna is like I guess more scientific you name and then you get slightly farther on things like wizard so you can see how that can simplify the problem space if we have a distributed model and we have an input that's asking a question about a frog if we don't have any knowledge of the structure of language or the relations between the word frog and toad it's you


know naive or basically impossible for that model to then potentially generalize the same question asked about a toad instead but if we have this dense distributed representation that is bringing together these words kind of into this similar feature space then you might expect that well if the you know representation frog is very similar the representation for toad the model might just be able to generalize and handle that and you know there's even more relations and properties which is beyond just similarity in that embedding space


you can also get very interesting relations like the concept of like kind of you know like the CEO to a company might all be expressed in kind of the same same subspace or the same direction in the embedding space or connecting a zip code to its city and you can see kind of how you know how could this be happening well co-occurrence is kind of get you this don't they you know Honolulu if you want to predict this random number you've got to eventually figure out that oh these you know do occur together because they are


the code of one to the other and so yeah you you get a lot of structure even though again all we were doing or you know one of the views of all these algorithms are doing is they're just processing very local relations in a very simple fashion and it's just scaleable and simple but it can work quite well as a starting point now you know these aren't the end of obviously it's only thirty minutes in two or three hour lecture so there's a long way to go so kind of these are all cool and whatnot and they really did


drive the first few years of modern deep learning NLP and helping to move these models to much higher performance but kind of what might be the issues with them so you know obviously languages a lot more than just the counts of words it has a ton of structure on top of than in addition to words and and furthermore context is very important and these kind of fix static representations of words that we're learning are just insufficient in many cases so you might have for instance three different sentences I went to the riverbank I made


it withdrawal from the bank or I wouldn't Bank on it and all of them have the word Bank being used in a very different context and you know basically representing a different thing it's a now and a reverb or you know it or just a phrase of expression and so you really need to learn how to do more complex things but if you're just counting whether two words happen to occur in the sentence or you know in word to vector kind of looking at a very short window and just using an averaging operator you you you can't really model all that much


complexity so we need to do more and you know there's also just kind of the design space right now you have this million by 300 dimensional matrix so it's like word vectors and then the question is just what do we do with that and you know obviously we figured out quite a lot of ways to use them but there's a lot that's still up to the practitioner and this often involves a lot of tests specific models slapped on top of it and that's where a lot of the first few years of research and NLP for deep learning went was kind of designing


all these tests specific models a model for doing question answering a model for doing summarization a model for doing sentiment analysis and they would all kind of take this common input of the word vectors and slap them in but then there was a huge amount of design on top of that and these models got progressively more and more complex with more and more details and so you can kind of think this does like well we only really did the first step sure learning word vectors is great but they're really kind of like learning


just the edge detectors in computer vision and they get us something but we know like in you know debugging for computer vision there's a lot more that goes into comment than just some edge detectors at the beginning system and that's true for an LP as well so there's a lot more going on in language beyond just these input representations so kind of how do we get there well we're going to take a little bit of a detour into the history of language models and kind of walk through how this kind of method and kind of set of generative models


kind of ended up providing one of the methods for moving beyond just word vectors and kind of introducing the second wave of modern NLP methods that use unsupervised or self supervised methods so fun overview real quickly is kind of seventy years of history here on one slide where we kind of are looking at a language model what is a language model well it models language and it's a generative model so hopefully depending on how nicely it's set up we can draw samples from it to understand kind of what distribution it's actually learned


and how well it's actually approximated the real distribution of language so without getting in the details of how you sample you can kind of see this kind of list here so very early there's this thing called a three gram model from Claude Shannon himself in the 1950s and this kind of still makes basically gibberish they also point to ninety-nine point six billion dollars from two hundred four six three percent of interest rate stores as Mexico and Brazil in market conditions well that's basically gibberish but notice that


there's still a bit of like local correlation and structure it says a lot of numbers and then it mentions interest rates after six point three percent or six three percent and that's like all kind of right and you can see how there's the tiniest bit of structure in there beyond just like what it would look like I could be just drew words independently according to their frequencies and then there's been a lot of investment in this kind of field in area over the last few years so Ilya sutskever in 2011 kind of


introduced a character RNN for the task and so here anytime there's a prompt it's highlighted in yellow which means it's a manually specified kind of prefix and then you condition on that and you sample from that so the meaning of life is the tradition of ancient human reproduction that's almost a sentence it is less favorable to the good boy for when to remove her bigger so it quickly fell apart in the second part but it almost got something there and it's still gibberish but it at least shows another hint of structure and then


there's a Rafale or falls paper from 2016 which is basically a much bigger word level version of that RNN from 2011 and just kind of use a scale and a lot more data and here's a sample drawn from it with even more new technologies coming onto the market quickly during the past three years the increasing number of companies was now Ted called the ever-changing and ever changing environmental challenges online so that's basically a sentence at this point there's a weird thing where it repeats itself with ever-changing and


ever changing but we've now got a phrase you know multiple phrases or clauses and kind of longer term structure there so that's a big amount of progress and again as we talked about with word vectors a lot of their failure is that they don't exploit contexts and they're kind of these isolated representations of only single words so the fact that these language models were starting to learn context as you looked at and inspected their samples is kind of a clue that they're going in the right direction towards some of the


functionality behaviors we might want in a natural image processing so then the next major step came in 2017 2018 with the introduction of the transformer based architecture we'll talk a little bit about that later if that's appropriate but its handles long term dependencies much better through self intention mechanisms and then you start to see potentially multiple sentences that kind of flow together and then the final one here is GPT - which can kind of take potentially a pretty low probability or difficult to understand


prompt that you know probably isn't in the training data like scientist discovering a herd of unicorns living in a you know remote previously unexplored Valley in the Andes Mountains and they're able to speak English and then it can write something that looks like a news article on the top of that this does cherry-picked all howl like 20 times i sat there till I got a good one but it's progress and most of these are cherry picked so it's cherry picks again cherry fix all the way down and yeah at that point you basically have something that


just reads like a full news article and it keeps characters and names persistent and you know pulls information from the source sentence over you know multiple paragraphs and this is all a lot of progress being driven in the last few years so kind of now that we have just like a look at the cool samples let's like get into the details here so this is going to be about statistical or probabilistic language modeling and kind of the way we formulate this or is we interpret language as a high dimensional discrete data distribution that we want


to model and kind of the set up since this is statistical method is we're going to observe a bunch of strings of language and the framing here with a probabilistic language model is we want to learn a function that can just compute the probability or density of new sentences so we want to be able to compute Oh what is the probability of the sentence is it going to rain today and we're just going to give it a bunch of strings and somehow we're going to design a model that can compute this quantity so what does it mean to compute


the probability of a string you know what what should the probability of that sum is the cat sat on the mat B well you know there's some people who kind of think that this might not be the most well defined concept or there's a lot of reason for skepticism potentially Noam Chomsky in 1969 has a very famous quote but it must be recognized that the notion of the probability of a sentence is an entirely useless one under any known interpretation of this term of this term so some people were quite skeptical to be fair this is well before


that kind of any of the modern renaissance and he goes on to kind of explain a bit more that there's quite likely that statistical methods can work but it's a good example of kind of where we're coming from and some of the contrast in the field so let's instead kind of like talk about why this concept might be useful like why do we want to know what the probability of this is and this is where I think it begins to see the connection between oh what does the generative and how like we end up using it or why might actually learn useful


functionality for downstream tasks or for transfer and so you know we could compare for instance the probability of the sentence the cat sat on the mat so the probability of the sentence the cat sets on the mat and you know we would expect that let's say we somehow have the true function here we don't know how to learn it yet but we just assume we have like the ground truth of the probabilities of these two sentences well it should assign more probability to the grammatically correct one and that gives you something like grammar


and that's you know an important part of language is understanding its structure and what are the valid sentences or not but you know should the probability of the sentence the cat sets on the mat be zero well no because sometimes people fudge their keyboard or miss type it should be much lower but it shouldn't be all the way to zero for instance and then you can kind of get to more interesting sentences that you could query you could say you know what's the probability in the sentence the hyena stephannie met and compare that to the


sentence the cat sat on the mat and you know we would say well as a human being asked this you would say well hyenas you know are wild animals they don't often sit on mats unless they're at the zoo or something so this kind of shows how to do this to compute this probability correctly you would need to start to have world knowledge what is a common operator you know what is a common environment for a hyena you know what is even sitting on a mat mean and then you can ask other questions again you could start to get two conditional


probabilities too depending on how you set up this generative model is you might be able to query you know given that the prefix is the substring two plus two equals you know what should the probability of the completion for be it probably shouldn't be one because people sometimes joke that two plus two equals five but maybe if you had bit more context you would be able to disambiguate which of those two you might predict and then finally kind of coming back to some of the data sets or tests we've already mentioned if you


have the prefix that movie was terrible I'd rate it one star out of five you you really should know that that is a likely completion and so to do that completion and to generate that sentence and to know that string is likely you basically have to have that language model somehow I've learned what sentiment analysis is and what is a little you know a likely relation between the concept of like one star or five stars the kind of you know reception of the movie or the description of the movie before that and so with that one it kind


of becomes clear that in the limit these functions that these language models learn and compute should be quite useful traditionally we approach that as a supervised learning problem right we were gonna like oh let's go build a data set let's go collect you know a bunch of crowd laborers and have them assign ratings to a bunch of different movie reviews that's what the Stanford sentiment tree Bank is but in the limit this kind of unsupervised scalable method of just like fit a probability distribution to strings of language


should eventually maybe be able to handle a test like this without any of the classic supervised learning framework being used and yet so that kind of extends much more broadly those are kind of some you know canonical example or some toy examples but this actually can be quite useful and this is kind of where language models got their start in many cases 30 or 40 years ago or 20 years ago in kind of machine learning so they're often used for speech recognition and machine translation which again are traditionally approach to supervised


tasks with pairs of transcripts that are somehow aligned and you know a major promise is that we can somehow leverage these language models to you know really help with these problems and for speech recognition for instance you could prune the space of possible transcriptions from an acoustic model there's a variance example from Jeffrey Hanson of you know how to tell the difference between the sentence recognize each and recognize speech you know they're very similar from a you know a raw audio perspective but if you have context you


know that these can be quite different things and wrecking a nice Beach is just also a much less likely string than to recognize speech and for translation for instance you could rewrite possible translations based on a monolingual language model so if you have an English to French translation system and you have some proposal of the French translation you could say well hey language model that I've trained already how likely do you think the sentence is in French and there's a lot of work on integrating this directly into decoders


and using them as restoring mechanisms Oh statistical language model has really got their start often as in these tasks so let's move towards actually having a computational model of language so first maybe we'll do some pre-processing like lower case so we'll take some maybe messed up text and turn it into just all lowercase to simplify it well then you know maybe said a vocabulary size to just like make the distribution easier to handle to set it to like you know a million tokens or something so we might substitute a rare


word like countertop with like an unknown token just so we kind of don't have to deal with this like potentially open-ended probability of observing a new novel where I've never seen before and then finally we'll use something like a tokenizer which will take a input string and return a sequence of tokens so it'll chunk it into a sequence somehow with kind of some rules or logic and you know this is another example of classic and LP work on designing tokenizer x' so we might take you know the cats out of the man and choke it


into just the words and you know throw that punctuation on the end there and then you know because this is machine learning we basically are we representing these words as you know unique identifiers or indices and that's again a way to get a window into how a machine learning model really sees natural language you know we come in as humans with so much understanding in context and from like lived experience but you know if you try to train a naive supervised learning model and you started from random visualization it's a


lot harder to understand what 223 in 1924 742 followed by 101 23 etc is and I think this helps you get into the mindset of when people talk about machine learning models being spurious pattern matchers or just learning weird correlations that aren't true if you if you've looked at a bunch and it's like tried to do natural and processing tasks as a human where your inputs are represented in this format you'd probably be a lot worse than current machine learning models already are and it'd be understandable if you made kind


of mistakes as an algorithm trying to figure out how to make sense of any of this especially once you get to some more complicated task like do these two sentences logically reason or follow each other you could even just do a simpler thing like split it into spaces so there's a huge design space here and I'm just providing a few examples right now okay so there's character level there's byte level which would be kind of working on you know if you just work on characters how do you deal with non ASCII text or text in there you know non


a Roman numeral or sorry standard like lettering systems so you could work on like a standard encoding scheme like utf-8 byte stream you could also work on unicode symbols or code points and then there's kind of these middle grounds between word level and character level which would be something like by parent coding and this one actually turns out to be super important so I'm kind of just covering it as part of general NLP methods and it's used by quite a lot of methods in the space now so what this does it starts with the


character level vocabulary and it kind of just merges the two most common pairs of characters at a time so you might have T and H be the most common pair of words or characters and then you'll combine them into a new token called you have th and you'll merge that and you'll resub stitute it in all of your words and then you'll run this loop again and so if you run this and kind of just keep merging and merging and merging it learns basically a tree of merges that quickly pop out words full words like the and you know common endings like IMG


and he and two and this learns something that kind of lets us handle potentially the full distribution of of language while also having maybe the efficiency of representing semantic chunks like words instead of operating on these characters which might result in strings that are five or ten times or five times longer and require like much more compute and have much longer term dependencies that are difficult to handle then standard board models so if I clear encoding from recur Center is all over the place and is a very common


middle ground to back off to character level if you see something rare you don't know or to handle like all these different languages while still having some sort of like kind of sensible handling of common words and frequencies hey Alec is there a common kind of number of bite pairs that you want to end up with cuz it sounds like you start with byte level which is just 256 possibilities and then you could imagine that you can have many many by two pairs and sometimes it goes beyond pairs I think right where you recombine an


existing pair with an initial just a bite like the iron and the G yeah yeah when do you stop so yeah that's a good question usually you have a heuristic for merging only across or only within words so you won't merge across like word boundaries with like whitespace or things like that and that just helps with efficiency because otherwise you'll start wasting emerges on things like you know come in like pairs of you know maybe fill our words or stop words and the other thing is you you could just in the limit run


this all the way out to a full vocab but we often work on this kind of middle ground where you know to get good coverage of of natural language you often need a hundred thousand plus words and you know in the limit if you want to start having you know common names and places you need really like million sized vocabularies and that can just be incredibly competition expensive so you'll often stick this in a middle ground of like 32 K bps and you're absolutely right that it'll merge all the way up to a full word like you will


get things like you know neurobiology in the limit that would be merged all the way with BPA just by doing merges over and over again yep thank you cool so how do we compute the probability of extreme well the dumbest model is we can just assume a uniform like prior over tokens and assume all of our independence we just product they're probably independent probabilities together to compute for any arbitrary sequence dumbest model but we'll start somewhere all right so let's get rid of some of these dumb assumptions so we could


suddenly like say well we know some words are more common than others and that kind of word co-occurrence matrix has that diagonal term which is just the frequencies or counts of words so we could use that instead and you know that would just allow us to say well the word these really comments were going to send more probability mass to it and you know the word supercalifragilisticexpialidocious is just pretty rare so this would be called a unigram model where all we do is we just product proportional to the


probabilities of the tokens from like the empirical distribution and again we can estimate that just by counting a time we can then go a bit farther and start to exploit context so again we've talked before about how important context might eat and this is where you can start to see you language well begin to handle a potentially so you can say instead that we're instead of estimating just like the you know diagonal of that of that matrix we can use that full matrix basically and say well yeah then that you know we just saw the word the


how often is the word cat after the word though and so we can kind of conditioned on that previous token and you know use a modified version of that like look at our count table and start to handle a little bit of context so that's a bigram model by Gram language model but there's a problem of generalization here and this is where kind of counting methods eventually like hit their limit and yeah we can brute force them with all the data on the internet but at the other day they're not flexible enough so let's say you've


never seen a word before like self attention you can't assign zero probability to that if you're trying to optimize for like log loss or something because you just get an infinite loss and you know if we just start going to longer and longer strings this count method explodes and the observances of every substring get rare and rare and this just kind of hits a wall so in the like 80s and 90s the way we kind of handle this is we kind of accepted that we couldn't handle the longer term dependencies here and we kind of use


clever smoothing methods of mixture models where you might put a lot of your probability on you know the you know grammar by grammar trigram model which is more expressive but you'll smear probability backing off if you don't see a word for instance or don't have a match to that unigram model or uniform model in the limit and so this was kind of what you saw a language models in the 80s and 90s spend a lot of their time on is they kind of were these very they were still basically count tables and statistical count tables but they


optimized for kind of achieving something looking more like generalization of a simple form of kind of combining these mixture models and so this is a good review paper if you ever want to kind of go back through the history of this and all the different methods develop there you start to get things that look more like representation learning and even multi-layer models so they'll start doing things like clustering over parts of speech or substituting for that so it's a very hand engineered way of potentially adding expressiveness but


it's a good history of kind of where these methods came from so you know since we're talking about NLP and language models is one of the core workhorses here kind of how do you evaluate and interpret a language model well probabilities are often within running or of 0 since language is a huge discrete space in the sentence might you know or a document might just be very long and so the most common way of evaluating these models and saying how well does it do is we use a quantity that's not dependent on the length so we


use like the average negative log probability per token and you know this token definition again might be arbitrary character level or might be weird level and so if we're using character level we might convert from you know base e to base two and report bits per character or bits per byte you see a lot of common language modeling benchmarks work in this setting and word level language model is often exponentiate that quantity and report what they call the perplexity and set so yeah it's just giving you bigger numbers


and better improvements because you're working on expensive log scale so how do we ground these numbers they're kind of abstract or random quantities you know what is the difference between one point two three bits per character and one point two bits per character especially if you just spent pretty much your life working on a paper and that's the number you got out so you know it's important to understand these quantities our data set dependent it's really easy to guess all zeroes it's really hard to guess the


archive and you know you can start calibrating the scales by saying well random guessing would get you you know log two of you know one over 256 so eight bits per character and human estimates from not the best studies but the only ones we've got I've kind of tried to peg on people in the range of like zero point six to one point three bits per character and the best of the models now are often a little bit lower than one bit per character so that range probably is lower for humans and we're somewhere you know getting okay but not


matching humans on these kind of quantities and you know way of grinding perplexities is gonna use the same random baseline so it ends up just matching the vocabulary size for like a standard model so random guessing would be a plexi of 50k and one way of thinking about perplexity is as like a branching factor of language so flexi to the N is like the space of possible generations of length then how many your model might assign so you have a perplexity of 10 for a language model and you generate you know to two like


word sequence there might be a hundred kind of high probability events in that space and human level estimates again Europe between five and ten and an example though again is this is always data set dependent always problem dependent if you have a lot of well constrained context like in translation these numbers can be a lot lower and best models are often like three perplexity on translation so you're picking between maybe three as possible likely words and you know that kind of agrees with like maybe there's a few


different ways for a human translate to census but it's not a huge space by any means so evaluation type 2 is kind of what we talked about so that evaluation type 1 is very much the generative model perspective of like well how good of a probabilistic model is this and so type 2 is instead kind of transfer and the things we're really talking about and caring about more there's a lot of ways we could use these language models so you could say how well does a better language mall potentially improve the word error rate


for your speech recognition system or the blue score for your translation system or the Younger suit for your document classification and this is kind of where NLP has really taken off leveraging these language models and kind of the history of the last five years has been discovering more and more ways we could use smarter and smarter language models or better and better language models to do more and more things so let's go through kind of the history here of kind of the sequence of developing real context models models


that can generalize better than these kind of count based methods we've so far been kind of using it for all of our discussion so the first one here is surprisingly you know like honestly this paper is amazing if you go back and read it it's from yatra Ben geo and from 2003 and it has a ton of very modern things in it and has skipped connections like you see in things like resonates in 2003 you know it's learning distributed representations of words and this is kind of that core concept we mentioned right at the beginning of like


representing a word by a vector with learned values for each location this is like the paper that kind of really introduced this in the neural setting and they were doing like large-scale distributed asynchronous SGD on a cluster even back then in 2003 they had to do it because single mushy peas were so slow so this is like I think it's 64 128 CPU cluster and it would take them I think a month to train a model with like three layers and you know sixty hidden units so this is a still a and grand model but we're using a multi-layer


perceptron to compute the kind of conditioning on the context so instead of this kind of you know cow based method we have an MLP that looks at you know the index for reward t minus one the index forward to minus two and you know T minus let's say just a three word context so these three of these vectors can and together of you know the last three words seen we then run it through a hidden layer and then we feed it through a soft max to try to predict what the next word would be so this is a trigram language model still but we've changed


the model from a count based method to a distributed setting with an MLP and you know this was kind of the first paper that heroically showed that they could match the performance of some of those super optimized and grand models but again it took like ten days or a month and it was on a giant cluster so you know neural language models really had some catch up to play compared to these smart quick count methods and this is a lot of what took this so long was just unfortunately they do need a lot more compute so then the next major step


here was kind of moving away from these fixed context windows which are kind of unsatisfying we know that as humans we can look back in pieces of text and condition on multiple sentences but these kind of methods always so far have had fixed context windows and have only been able to process or condition on just the last few words so this is kind of where our n ends come in and Thomas week loves 2010 paper it's kind of the first modern deep learning version of this that kind of started working quite well so we


replaced that MLP with a recurrent neural network and that allows for handling potentially unbounded context now it handles that context in a learn fashion so you'll get an input word vector at one time step and you'll have this context buffer which is a learned memory state that the RNN kind of modifies and updates and you'll use that to kind of represent a running summary of everything you've seen that's important for predicting the next potential word this has potentially unbounded context but in practice we'll


train it with method methods like truncated back prop where we only update for and compute you know how to modify the the transition function of the state for up to like maybe 32 words or 64 words so it might be biased in that way but it's kind of just still can potentially learn to encode a lot of information into a learned memory system instead of kind of using like hard coded methods of just like keeping the explicit input presentations so here we get again like probably one of the first real language models where on that


previous one we were still using a type 1 evaluation we're just like how well can you predict the next word but here with my cloths paper they showed that if you ran this for a speech translation system you can get actually a much lower word error rate on you not only predict better and you start really improving over the the traditional like in ground based language models but if you look at this word error rate table here you actually see that it improves the speech recognition system so your transcriber will make much potentially


like you know over 1 to 2 so the k10 baseline here is 13.5 percent were word error rate so you messed up 13.5 percent words and using all these are nouns together you could actually reduce that by over two points and so you're talking about like a 20 percent error reduction which is quite significant yeah this is like a lot of early language models were actually published in speech conferences because this was such a important and exciting application of them to start with and again you don't need to collect


a bunch of speech transcription data here in the limit you could just run this thing over like New York Times articles and then use it to help potentially with your speech transcriber and that's where a lot of the power comes in from an unsupervised scaleable method and transfer capabilities so we already showed samples from this one but it's kind of a slightly different version where all these models so far have been operating on words and kind of pre-built tokenizer x' to split it off and chunk it and kind of fix


vocabularies the exciting thing with character level models so it's the same kind of architecture or recurrent Network it approximates a richer transition function where you might have a different set of weights with multiplicative interactions this was back when we thought optimization was hard so it's using second-order optimizers because RNs are scary and we still haven't gotten used to like just first-order methods working well and you know it begins to handle much longer short dependencies when you work on


character level you're you're you know suddenly talk about sequences that are five times longer so you start having models that maybe handle hundreds of time steps and that starts you know abstractly meaning maybe you could have a model that could actually parse a paragraph or parse a page and you know it wasn't a lot better than Engram models in terms of its perplexities but it was very easy to sample from and this was kind of one of the first I think demos that people might have seen online of the language model back on


some meats University of Toronto static website from like 20 2011 so the next male like not like quick question when you look at character level models versus word level models can you directly compare the perplexity uh if you're careful yes so you know it in the limit these are both just predicting a sequence and if you set it up correctly you could just like you know here would be the simplest way to do it with a character level model sorry I should clarify you can go one way so you can you can compute for a character or byte


level model the perplexities that it would assign to a ward level model but some word level models might have limitations like using unknown tokens or out of vocabulary that means they can't actually compute probabilities of arbitrary sentences whereas that someone in the expressive benefits of a character level model so the simplest way to do this would be you would convert the word level model like the token sequence of processed like let's just say it's split on spaces you'll just rejoin on spaces and then compute


the probabilities the the character level model assigns and you'll have an adjustment factor you could just sum the probabilities over the full sequence and then renormalize by the relevant metric and we'll actually be using that later to talk about how to compare different language almost more appropriately but again you need to have the expressivity to handle an arbitrary string to be able to compute this and you know old models because their computation is often worked with small vocabularies so they wouldn't truly be computing the


probability of arbitrary strings because they might normalize them in various ways got it thank you yep so the next step is kind of going to multi-layer ellis TMS and also introducing the LS TM again even though it came out in 2000 and kind of gotten realized primarily by you know one of the major people that we popularized it was Alex greys and kind of 2013 ish so this is gonna character level model except we now have a gated RNN which uses kind of these multiplicative gates and more complicated transition dynamics to


better store state and to help compared to like a multiplicative Arn in with kind of credit assignment and just trainability and you start to get the things that like can handle a kind of arbitrary arbitrary strings of text so you get you know something that's learning how to parse Wikipedia markdown or XML and Andre Carpathia kind of really popularized these models with like some blog post in 2015 showing that they're like you know work full of tech they work for XML they were Python programs you know they're


not generating valid things but they kind of can handle this they really have this flexibility of kind of you know feeling exciting from an unsupervised learning perspective you give them some data distribution of like Python programs or something and you just have a you know train over that and then you get something that looks like it's really drawn from that distribution so we kind of like talking out through like a lot of the early work here and although there was one example with the Thomas pickle off paper of Thomas paper


of actually having an application a lot of this was kind of just like competing for competing sake on type one evals or like look at the funny samples so again one of the very fascinating things about the last few years of NLP has been how we figured out how to really use these things much more broadly across the board and this is where I think it really starts to get exciting so one of the first papers to do this it was the skip thought vectors paper from Jimmy tauros and collaborators in 2015 and so what they did is they proposed learning


a RN and sequencing coder to provide context to in a language model and basically to learn how to use a sentence level feature extractor so what I mean by that is let's say we have a sentence I could see the cat on the steps what this model is trying to do is it first ingest this context sentence in the middle and they call it skip thought vectors because it's you can think of this is basically that skip grande model that was again you take a word in the center and then you predict the word before in the word after this is


generalizing it to sentences and it's using an RN end to kind of learn to summarize the context of the long sequence and handle kind of predicting complex dependencies between multiple words so we encode that center sentence with an RNN we iterate over it in the left-to-right fashion and then we have Anna linguish model that predicts the previous sentence so what might have happened before the sentence and then a language model that also predicts the suffix sentence that comes after it so what they then do is they say well


you know a model that does this test very well should learn to kind of summarize this sentence in the middle and for our nen's it's you know still these distributed representations so you have this state vector that's representing kind of an alert fashion all of the previous words you've seen so importantly that's now generalized from representations of single words to representations of sequences that can exploit context and potentially handle more complex properties and just big u8 meanings of words and they showed across


the board that these models healthily outperformed classic methods like so Cibao with word effect would be the simplest version so you know what does a document how do you represent a document with word effect well one of the future observations you could take is to just average the embeddings of each of the each of the words in the document and that would be what this like Cibao baseline here is on a bunch of different data sets and so you could instead say well we're gonna you know we somehow learned this sorry we somehow learned


this sequence extractor we could run it take its feature representation for each sentence and use that instead and you kind of see that these models you know if you use the combined skip of the like bi-directional models and using the forward and backward versions you can actually get these to start to outperform the words effect models kind of across the board and it wasn't really like this paper was kind of exciting especially from the breadth of things they do they have things like image captioning representations that they


learn and they kind of show you know with analysis methods like tease me that you see clustering according to classes you know it was kind of like on the edge where it showed some pretty exciting promise and it was you know a lot a lot stronger than potentially a super baseline but there were other still discriminative methods for like training models from scratch that were still matching it with like you know well-designed comment architectures or things like this so although this had be like a very exciting kind of oh it's a


learn feature extractor that's able to handle long term contexts and dependencies it it kind of worked but it wasn't like sweeping the sodas away it was you know exciting and honestly I think a lot of people ended up using it more as a language model where they saw some cool demos of having to generate multiple sentences but it never really quite you know blew everyone away from its quality so you know this is like a good early hit but it didn't quite you know it wasn't a homerun by any means and so this is where androids paper from 2015


semi-supervised sequence learning kind of comes at it from a slightly different angle so again for skip thought vectors we just used this vector representation as an input to a model and we fix the model itself and we just like train another model on top of this vector representation and it's a rebekah representation summarizing the whole sentence so maybe that's kind of a difficult test to summarize all the complexities of long sentences short sentences so what died all did instead was they said we'll take this language


model that we've learned and we're just going to fine-tune it directly we're not going to like precache the features like kind of words of Exile we're just gonna you know take it whatever parameters that language model learned predicting the sequence we're going to use that as an initialization point for training a supervised model for a downstream task and this is the one that started to get good results and they were showing compared to standard supervised learning on you know datasets with like 20,000 labeled examples and stuff like that


that these models could get quite far and so you see in the limit that you know if you you kind of have all of your different baselines here of you know word vectors feeding as inputs but then we could use like a sequence auto encoder or sequence language model and fine-tune that and you start getting quite large drops here and what's kind of cool here is these two different rows here one of these is pre-training only on the IMDB movie reviews so basically the same data set it's a two-stage algorithm and then this third table here


or this third row here is using a bunch of unlabeled Amazon reviews and that's you know starting to get towards transfer learning starting to get towards well we can run this thing over a lot of data and as we get more compute we can just get more data from the internet we can feed in more and we see that that actually improves things significantly over only using like the small standard supervised learning dataset in isolation some of this might have just been at the time that it was difficult to Train language models and


train Armand's back in the day but you know as we'll see for the rest of this lecture the methods have kind of continued to hold on top of this and continue to make progress this is the first one where it got a strong soda and you know there were strong bass lines before and people started like really I mean well to be fair it came out and not much work happened in the space for the next two years but it kind of and a lot of that was because it like really just killed it on these sentences datasets and not not as much elsewhere and this really


took some further work to kind of figure out how do we make this a generalizable approach that works kind of everywhere the same way that like plugging in word vectors does so moving back one one moment to a type one evil there was a followed paper or a neck in the next year that kind of really started to push on scale and compute used for training language models as we mentioned before they've kind of always been compute limited so this was a that Google paper that showed kind of the first language of all that could generate something


like a coherent sentence and a lot of it was using a larger data set so the billion word benchmark was a big data set at the time and they use a an HK hidden unit projection LST M which is basically a low rank factorization of like the transition transition matrix just to keep the parameter count down while keeping the state size hi it's character aware with some improvements that let it process the character level inputs so you kind of see on the right that this is starting get to be a kind of complex system and then they throw


you know a large vocabulary they throw a 32 King 4k 40s at it so 32 GPUs for three weeks and they kind of really got a huge improvement over the previous results and at this point those old Engram language models the old statistical methods were in the mid 40s or even in the 50s and 60s were hybrid systems and suddenly you're at like 23.7 so you basically have this metric you know again it's exponentiated so it's actually like a 20 percent reduction in like just actual log loss but you know you're starting to see a lot of


significant progress at this space just throwing scale at it and this has ended up being you know something that was really developed just to push it and say how far can we get you know sentence quality can we start to get something that looks like coherence and one of the surprising results is it turned out that this actually paved the way for further methods even though it was just designed to be a really good language model and just better predict the next word it ends up laying the foundations for talk about in a little bit called Elmo


that really was the first one to crack how do we use these Ellen's all over the place and start seeing it working for question answering and you know summarization and all these different domains so there's kind of a bit of the Tibbett here we're at an hour should we stop for a little bit or let me check out a stopping point I'm gonna go a bit farther we could go to about an hour 30 and stop for a little bit longer there is my kind of conference yeah so you know I've motivated scale a little bit so like I mentioned there's a whole


internet out there there's so much information and that perfect language model would you know basically from one view need to fit the Internet into its parameters given how big it is it's not surprising that we're going to need a big model to do that we're going to need a lot of compute pretentiously to do it to get as close as possible and for many of these tests we're talking about where you want to learn long term dependencies we want to learn complicated tasks you know they might be quite rare they also are quite difficult so you know the


closer you get the better you are to maybe learning real interesting behaviors first kind of a basic system that just like is locally plugging a few words together so another you know just vivid way of pointing this out is a small character RNN is basically gibberish you know this is what happens you know this can be a very good architecture but if you don't give it capacity it just can't really learn language you know there's so many words there's so many objects there's so many relations you really need a lot of


expressivity to handle all that complexity and you know another way of putting pointing this out is classic resources that were built by humans trying to map out kind of like the relations between all words in natural language you know build hierarchies over them so there's there's really heroic efforts here like wordnet they were larger than many of the language models we were still training especially a few years ago so it might have like five point five million relational features in this package and you know when you


have it zipped on disk or unzipped on disk it's like already 55 megabytes and you know a lot of common language model is especially early on we're only a few megabytes for parameters and so we know this is probably going to be very inefficient and you know we're probably going to need quite large models and right now you know the answer we have so far is to kind of address this facts with scale and you know hopefully we do find out more efficient and we'll talk a bit about that later too but right now you know kind of the


first dumb thing you try is brute force if we scale and you know another reason why this is worth investing in is it's now a very well validated empirical trend so across the bottom here is for both language modeling and and for like computer vision kind of the performance of models laid out on log scale plots where you see you have a large scale x-axis which might be the amount of words you train on so every new tic is a doubling of the data set size you know block scale is not great because it quickly gets inefficient but these


trends are incredibly linear they're very predictable so like it's almost like the natural kind of domain to think about is like what how does this look on a log scale and you see that again for language models on the right left and for like the performance of like captioning cysts or sorry image consecration systems on image net in the middle so these are quite consistent trends and they span now quite a few orders of magnitude so so far they've continued to improve from 6 million parameters up to 600 million on like


image net and you know data set sizes spanning that's probably over two to two orders of magnitude or two orders of magnitude there yeah and also computers becoming available as investment of more resources in Jim machine learning and AI and improvements in Hardware and distributive training have kind of allowed for you know even though there's there's this logarithmic or this heavy demand for additional compute to see kind of finite sized improvements at least as of yet kind of the industry as a whole has been developing techniques


and systems to keep providing that additional compute to keep these trend lines going so that was kind of just a quick digression on likewise scale might be important and it really intimately plays into like where these language models came from and how they kind of had their success so here's a like kind of a cute example looking at kind of starting to get away from just learning these kind of feature representations that could then be reused by downstream tasks towards maybe we can learn the tests themselves without having to have


standard human label feedback and kind of shared that intuition with like the talking about the you know computing the probability of the string I rate this you know one star out of five after seeing the prefix of the of the product review so this is a paper I did in a 20-17 which was like kind of a very targeted experiment here and one of the hypotheses I was working on was that maybe just data was the model neck you know our models are so inefficient that if we were able to just tile kind of in an unsupervised fashion the landscape of


one domain we might care about like product reviews we could maybe do quite well so we made a much larger dataset that's our rather we used an existing data set from from I think UCSD and Amazon in partnership which had 40 gigabytes of text so that was way bigger than that billion word benchmark and it's all in just one domain and we trained a byte level language model on this for you know a reasonable amount of computer month on for tiny Nexus the model ended up under fitting a lot but you know one of the most interesting


things about this is if we go and poke into that model and say well you have this hidden state that summarizes everything you've seen and we do probes over that we found that actually there was a single unit within this language model which very vividly indirectly just computes a running estimates of what is the sentiment of the characters I've seen so far in the review so you can see that you know as it turns on this is one of Michael Crichton's best books and so we have green colored as positive and red colored as negative so again there's


no supervised learning going on here this is all just unsupervised prediction of a byte stream it just sees a stream of bytes 40 billion in a row and they're all just you know numbers 0 to 256 and it somehow figures out in order to better predict this text you know it recovers this useful feature which is well as this review gonna be excited or you know dismissive and you know it can handle complexity where you know I can switch from a great start you know it's something where it's like you know you know here in the middle seriously the


screenplay and the directing were horrendous and then it suddenly drops off and it's you know performance analysis and starts going negative you know I can't fault the actors I know good novels especially are hard but this may be the absolute worst disparity in quality between a null and screen adaptation forever so it really does it and it turns out that if we just threshold on this unit so we're not even fitting parameters we're fitting one parameter it actually was matching these old words avec or by ground baselines and even things


like skip thought vectors and it's just a single unit in the model and we're just running it over this over the document and you know threshold the value at zero and so this is a histogram for positive reviews and negative reviews of what this system does so this is kind of showing I think in a very clean and pure way how you can really do some unsupervised representation learning here and start to learn something that really helps potentially with downstream tasks it's very hand engineered it was very targeted we knew that like you know


product reviews sentiment is a very important feature so we were kind of really hoping that something like this would happen and it would learn a really good representation but it was you know it's still like kind of shows a proof point when with limited scale but lots of data you can get something done here a follow-up work we did was with scott gray was pushing on kind of model signs again so we said maybe hidden state size is the bottleneck so again these standard LS teams and RNs summarize the entire past context as a fixed length


feature vector and so that might be for a standard model in a big model like 4096 units or that were false model was 8 K units and you know if we had like three hundred dimensional word vectors if you naively just concatenated them into your that state representation you could only handle like 30 in a row with like a you know an 8k or 9k state size that's only about a sentence or two so we thought that you know maybe it just turned out that models were really limited by their state size and so we pushed on these kind of blocks sparse


methods that kind of allowed us to train with much larger state sizes that we would factorize the weight matrices such that they would be represented kind of as this two layered system of having a dense sub dense block and a lot of sparse blocks that are pruned away and we saw that these were slightly efficient more efficient in terms of parameters and they also worked better on things like set analysis when evaluated by these linear models which is like a standard probing for how good of a feature representation have I


learned that's partially because when your model is just like lots of features and that's all they wear their expressiveness comes from you know when your summer ability is easier and high dimensional spaces and yeah I was this was kind of like explaining some of the history of I was pushing on trying to get these things to work and figure out how do I like really you know push their performance potentially and so this is like showing again that that performance analysis of these units learned by these models so this is how


that kind of representation evolves and we show kind of data efficiency on the x-axis here so in the limit we know there's that zero shot performance of fitting a threshold to zero examples and that actually turned out to be about about here on this graph if you use all the data to probe and find it but if you just fit kind of naively as you saw more and more data you you know could start with like in the limit only needing 10 labeled examples to beat some of the original supervised learning baselines which just train systems from scratch


there's this recurrent neural tenser Network paper from a searcher at all very early do planning work here with a you know really cool complex model and we were able to imagine it with just ten labeled examples whereas it was trained on all 8,000 in this case and then as we kind of keep adding more and more data we see that the representations learned by these language models can be quite powerful and you're kind of able to like quickly sweep through kind of in the limit you know if you don't have any pre training you started getting into these


increasingly complex and desperate desperate maybe a judgy word ensembles of 30 different models to hit sodas and then we're able to just use this model that exploits this unsupervised learning on a lot more data to push significantly higher and then that small world improvement with blocks bars had another large jump above that and so this is kind of one of the precursors that or kind of heralds what's about to happen on every task over the next few years this is 2017 as like this field really starts taking off so we mentioned this


kind of cool and interesting thing of learning a single feature within one of these networks that kind of really shows some representation learning going on so there's another really great paper I love here from Royce warts and collaborators in 2017 that I think again starts to speak to hey these language models that are you know recurrent networks or more expressive neural networks are really actually learning something interesting and beginning to be useful for downstream tasks that might be difficult so this is a data set


called the story closed task so what you do is you have a paragraph of context in this case Karen was assigned a roommate for her first year in college yeah they go to a music show together and it goes really well and then you turn you're trying to train a system to predict which is the right ending and which is the wrong end and so this fits very cleanly or this is what Rory was quite clever about was realizing that this fits very cleanly into the generative modeling framework you could say well what is the probability of the right


ending versus what is the probability of the wrong ending and again as we get better language models they should start to learn to exploit context and assign correct like you know the correct probabilities to these different strings and so very early work kind of took the classic supervised learning approach of just throwing you know a you know a model maybe with even word vectors pretty trained at the system and treating it as like a binary classification task but in this case the story close task it's difficult to


generate this data they only had 2,000 labeled examples so a purely supervised discriminatory system really couldn't get that far and they actually were basically not performing much better than random and so what Roy was able to show is that well if you exploit tons more additional data which was available of like training on small short stories and then you use this model to score the endings so it just produces a single scalar which is like the ratio of the probabilities is sinky my trick that we talked about before but commuted for a


language model where you say well what is the probability of the N being given the story and you normalize by the probability the ending in isolation and this this trick just helps a bit compared to just computing only the probability the ending given story that actually still works quite well but you get a fair amount more and so they were able to significantly improve the performance on this data set again in the limit just using that single feature the RN NLM features here they got a almost 10% jump in performance just by


using the generative model off the shelf there's no discernible it's not exploiting statistic you know spurious correlations here because it doesn't see any labels it's just fitting a threshold of what it already thinks is the right ending versus wrong I'm being you know another quick inner loop of scaling so these kind of all are happening Nestle together and I think this gives kind of a sense of how you know research feels often involved where you see these different authors and different people pushing down different


lines of work and then kind of things come together in exciting ways so this is work from Noma shows here really just pushing on maybe parameter counting the bottleneck you know maybe that's what's been holding back language models and so they really went crazy here and they they train models that have these what they call sparse again a mixture of experts layers so you have your standard LS teams and pink on top and bottom of this model and then in the middle you sandwich in what's called its mixture of experts layer and


what this has is it has a gaining network that decides to pick basically a two layer fully connected Network and it says which ones just slotted in for this given word so you think that you know maybe you want to memorize a lot of information and when you see you know they went to the city blank or something the mixture network and the gating Network will say oh I should load up like you know the expert that handles you know where cities are in the world or is this kind of just a hand wavy high-level intuition and particularly


when you train this thing at large scale because it's sparse only one experts being evaluated for any given location at a time so you can group these and you can have many of these experts being trained in parallel and so they're able to push to like you know an eye-popping 137 billion parameters in this language model it's all on this very specific sub module but it ends up being more computer ficient and it has like a lot of clever and very impressive system engineering work to handle how do you run this thing at scale and you know


have it be efficient when handling so many parameters there so now we come back to type to evals and kind of the standard slutted in and see how it does and this is like really the paper that kind of set this field off it's called a elmo from peter's at all this day i to work again and they or elmo is the name of them all but it's really about deep contextualized word representations and this is kind of where there's the clean mark between the word defect error and the like language model era and so the way era and so the way they do this is


they kind of cleverly say well what do word vectors do they slot in kind of as inputs and they rear nth you know this discrete identity tour ID identifier token of like you know word you know cab being ID 256 with a distributed representation as we discussed before things like contexts are missing in this case so this paper talks about how to use a language model to do the same thing they're substituting the input representation but instead what they're using is a deep bi-directional language model so this is kind of the schematic


here where they have a for Dallas TM that will first take in its own learn word representations and it right over the sentence in a left/right fashion and then they want to you know have context not just for sentence words that happened in the past but words that might be about to happen so they also run a backwards L s TM in the other direction from the right to the left and then they have this Bihar cable or sorry you sees me a deep model with multiple layers so they run two layers rails TM and then what they do is


they learn weighted averages of the word vector so maybe for some low-level tasks you only want those input representations but maybe for some tasks you really want that kind of long-range context and so you might want to use the higher-level layers and so then they rear nth instead of feeding in those kind of like one-to-one look up in a table what the word vector is they have this RNN language model that processes the sentence or a piece of text in both directions and it learns to you know reuse its hidden state representation as


the input to the model instead of the word vector representation so kind of seeing all those early results like skip thought vector showing that well you could learn and distribute representation of the sentence this one does it but it does it at a word level and it just cleanly slots in where word vectors used to go and so what this is quite nice by is it allows you to have very direct comparisons with prior work and across the board they basically show that like simple baseline models which substituted to use these representations


instead to reward effective representations we're outperforming very well-engineered very tuned state-of-the-art systems that were like squeezing as much performance as they could add award vectors and they're getting you know quite large numbers here where you see you know 10 20 % relative or sorry yet relative error improvements and importantly they kind of have that clean sweep of very many different tasks like question-answering entailments coreference ner so even classical tests like you know part of speech recognition like and you know


this kind of really just swept everything and it was very clean it kind of like made clear that okay you know it word vectors were great but it's time to you know here comes the new thing and you know the other very important and fascinating thing I find about this is this model was that language model that was developed or the limit of all they used for this system is that language model that were fall developed in 2016 at Google to just along with co-authors like Orioles that they really were just pushing on


four-plex keys they're just pushing on how well can we get a generative model this text and then you know two years later or you know two years later someone just was like wait a second this thing is learning amazing representations and you know those two works are separated by two years and completely different research labs and they just discovered that you know these language models are really doing something here yeah so that's kind of like really where things turned and you see you know again looking at data


efficiency that when you're at very low amounts of data you get huge improvements like 10 plus percent absolute improvements so that really feels like you know as you get more and more supervised data you can begin to overcome the limitations of you know training from scratch but in the limit you know you want to use as little data as possible you want to learn as quickly as possible so this is like very exciting and it's kind of like really got everyone to start stirring and paying attention to this field yeah so


final one before the break is kind of you could think of it as pretty much in the same vein as Elmo and what we did instead is we took a better language model again so transformers came out and we were really excited by their ability to handle longer range dependencies and they were also very computer efficient so you you could train them quite well and quite fast so we swap out the recurrent network or the LS TM in the language model for a transformer based language model and if we want we could talk a bit about a self attention and


transform based architectures in a bit but now just think of it as like we subbed in a different better architecture and it's slightly larger we use a similar data set of books it's the same data set that skip thought vectors introduced slash trained on and we we just fine-tune it the same way that Android I it all did and this exciting thing here is we saw that we no longer needed these tests specific architectures for each task so you know a lot of the cleanliness of Elmo was that because it was just substituting


the impact representation you could reuse all those and engineered architectures and often they would you know counteract for the issues of handling long term dependencies in or an end with like an attention layer the like but they still require you know that engineering of each these tasks for each of these different architectures which means that you're still leaving performance on the head room you know it's not like where you're initializing the middle layer features of a CNN instead of like the edge detectors of the lower


layers but then we still are sticking new layers on top so we were trying to like really kind of move towards a general-purpose framework that kind of a lot of Surrey is the same architecture everywhere and not have to have as much of these tests specific engineering which requires a lot of effort and time and grad student hours to like push those systems for so we have this transfer based language model and we kind of showed that for a fair variety of question of tasks primarily classification we kind of take the same


model and without having to modify it or introduce new layers we could just fine-tune it with only a linear classifier in text typed on top and we could across-the-board do quite well and in many cases we were outperforming ensembles the same way that elmo is doing before and using basically the same unified architecture to perform quite a lot of different tasks and the glue benchmark had recently come out as like kind of a standard multi test benchmark and this is kind of one of the first major ones to bump up accuracy


there and reduce the complexity of it and you know there's two particular things that I'd like to focus on for discussing some of the results from that paper which is if we oblate the number of features transferred we really see that this is a 12 transformer 12 self attention block model we really see that you need all those layers and the random initialization of higher layers was not working well at the time it may be you know as always that you figure out better initialization methods and you can close that gap but you see kind of


cleanly that we're transferring a deep heart you know a deep distributed representation and the you know the deeper it was the better it was generalizing and that seemed to hold true across multiple data sets and was a very clean kind of performance increase as you just transfer more and more of those blocks so Elmo is a 2 layer model and now we're going to like a 12 layer model and then this rightmost graph is really the one that I want to focus on and this kind of links together some of the hints and pieces we've been seeing


so far through like many of the different papers which is kind of this interesting behavior sometimes the language model is learning a supervised task or a task we kind of thought needed supervision to classically trained in the machine learning framework without any direct explicit labeling or supervision of it so what we did here is we took this transform language ball and we kind of design these heuristic ways of having it compute probabilities the same way that right where Schwartz was doing and we kind of started to extend that beyond


just you know the very specific thing like which of these two sentences is most likely so like for instance we could take a language model and do exactly that example at the beginning and ask it well you just saw a movie sentence review do you think the word very positive or very negative it's more likely after seeing this sentence so this would be this probe here which is sentiment analysis in blue and so we show over the course of training this language model we evaluate this kind of zero shot performance probe and we call


it zero shot because and this is a broader you know we didn't invent zero shot by any means but it just means evaluating on a task or data set or a class that we've never seen before and we haven't done standard supervised learning to update the representations or to train the model to do this and so we see that kind of as you train you steadily improve performance we've normalized test performance so that zero is random guessing and one was the overall state of the art do you still see across the board that these models


are you know nowhere near soda and often they're less than 50% of between soda and random guessing but they're showing clear and steady improvements and they're showing that even on tasks like question answering you could actually you know take a paragraph of like a question answering task and asked it well which of these answers do you think is more likely and you know there's no supervised training here it was trying to predict books and then you ask it like a 5th grade science question and it starts to sorry I shouldn't answer


promotoras it so much but you just probe it you know you can compute some conditional probabilities from it you start to see progress being made on you know some potentially quite far afield task the final point to make your to is self attention and transformers really seem to help a lot here where as we did the same exact model or you know it's equivalent size and similar compute with an LS TM and we were seeing that especially on the zero shot tasks sometimes it could do relatively well but on some of them especially ones that


handle long range dependencies you really need these self attention layers handle long term dependencies cool so I think it's we're at about half time for the lecture and I think that's probably good time for a break then fantastic Alec thank you let's let's take a break till 6.50 specific time well about eight minutes does that sound good yeah okay great and I'll pause the recording for a moment in here I'm sure if you have a certain like limitations on how large your Monica tree life is everything you need to run in like a


particular device and you can to like train a large model or a cartoon elephant this one are there any strategies yeah I mean so I admittedly have been emphasizing the need for scale but it's kind of a continuous spectrum thing and there's some work we'll be talking about later that kind of focuses on efficiency and kind of how far you can push models of a given capacity in size you know probably the answer here I think from a pragmatic perspective is to kind of use whatever is the largest thing you can fit into the given device