at least if you get them lots of resources can actually learn to compress any distribution so we call them Universal codes so there's one algorithm called problem if LZ somebody's and which works like this so I'll just try to very quickly describe it you're basically trying to so here's a long string that you're trying to compress and the way it works is that


when you try to compress but you're at some position in the file let let's see we're at this position of the file and we want to code the future so what you do is you basically try to find a string starting at this position eight which has already occurred in the past so we here we have this screen AAC and then we see out in the past AAC occurred so let's just store the index into the past

L10 Compression -- UC Berkeley, Spring 2020, CS294-158 Deep Unsupervised Learning


so this occurred one two three times steps into the past so let's just store this number three in the past and then we we also add into the next character which is B so that's that's basically how this works at this point C we see oh there's a string CAE in the future but of that string occurred in the past so let's just store the number three which indicates that you just need to jump


three into the past to just copy that string the past over so this is roughly how Lempel Ziv works you just look for matches that you're trying to compress and past and copy them over and so what why is this a good idea okay so why is this a good idea so just very roughly why this was a good idea if the source of symbols you see is independent then whatever symbol you're at right now will


reoccur that will will actually reoccur if you wait long enough and the reoccurrence time is as a geometric distribution so the average reoccurrence time is just one of the probability of the symbol so that's that and so the local civil rhythm says to just write down the time that the the write down the time that it that you have to look back to find the same symbol again so


that's going to take log t bits where T is the time so on average it's just log log T bits which is log 1 over P of X so you can see that this goes to the entropy of the source so this is an interesting algorithm it's basically nearest neighbors and it's saying that if you run if you just memorize tons of data over time and you run nearest neighbors then this is like the best


learning algorithm that you can or this learning algorithm does work um it just might take a very long time to learn and you can see that it does take very long time to learn because template matching does not generalize okay so that was simple as if so I'll conclude by talking just giving you a taste of some very recent research on on deep learning and compression so


by no means is this comprehensive or anything like that it's just to give you an idea of what might be out there so this the authors of EBA NS released some new work and I clear this year where they show that you can train a fully convolutional deep latent variable model on small images and just because it's fully convolutional you can just run it on large images make sure that this


works very well so this these are I think some of the best numbers on full resolution image net just by using this full accomplishment property these authors here describe a very intriguing alternative to bits back coding so they described what they called minimum minimal random code learning which is a coding scheme for latent variable models which achieves the bits that code link


without needing bits back so the way that works is that you know the encoder samples a lot of weights the number of leads and samples is 2 to the KL divergence between the encoder and D and and prior and then picks a random one I did just a uniform random one and the decoder can do the same thing if they share the same random number generator and so it turns out that this is a way


to basically get a sort of a low bias sample from cube by sampling a lot of these lanes and picking a random one and the number of bits you need to encode the index the the random one that you picked out of them it's just the KL divergence just log K so the visionary log K bits um so this achieves the bits back cut length without many bits back the trade-off is


computational complexity because the concurrency could have to collect a lot of samples and finally there's this other paper which has a very different flavor from the ones that we were talking about this is a paper about lossy compression where they come up with a recurrent encoder and decoder architecture for lossy compression on sequential data like videos say and the


way it works is quite interesting um just the very high-level idea is that the encoder simulates the decoder so normally you would think that the encoder and decoder just operate independently and the encoder you know just doesn't worry about what the decoder is doing but if there's this time structure then the encoder can simulate what the decoder is doing sort


of one time step behind and based on that you can send extra information that will help the decoder or just send the right information that will help the decoder reconstruct the data just in the right way and they show how to write down a neural network architecture that captures this idea and optimizes for the resulting code length in this end-to-end way so that's quite a cool idea right


yeah so that's that's all I have to say hopefully that was helpful that was great Jonathan what's it's all over time but I'm thinking maybe we can spend a couple more minutes if people have some questions didn't want to ask as we wrap up here anyway also value of lecturing and I also did a bunch of questions in the chat to be able to do that in parallel to you making progress


on lecture I had a question about ans so I still don't see what the connection between like oh it seems like ans was just like a little add-on to this lecture like what's the connection feels like I don't I don't really see why we need ans like why you can just use another yeah it's a there are ways of combining Ritz back with arithmetic coding it's just sort of the popular


recent thing to do is to combine bits back with ans and in the reason we do it is because you get a very clean algorithm that works very well so that's that's just them that was the motivation can't you use it sorry can't you use base back so do any encoding scheme yeah yeah you definitely can just particularly convenient because of the stack structure of ans and also because


the NS does work well in practice um to use it with that social practical reasons why a net using mmhmm yeah maybe also dumping in here if you look at the Brendan frame geoff hinton paper it managed to do compression with a VA e and arithmetic coding but it had a bunch of overhead that you encountered because I think they could a nice look cute and this back axle or like a stack and so


there's an overhead occurred then look at the Townsend on paper you can see how to make it all compatible through using a Dennis and get much better efficiency and compression efficiency tanam the previous paper that uses arithmetic coding another things that NS well invent it pretty late meaning compared to the other one it actually turns out and the ideas might be somewhat complex the


all right cool well thanks everybody um so I'm gonna give the second talk tonight which I'm not crazy about and and I don't want this pattern to to repeat but you know Andrew and I wanted to kick this series off and and felt like me talking twice or better than then not but we're gonna we're gonna get more diversity of folks if any of you want to give a talk yourselves you know

LSTM is dead. Long Live Transformers!


somebody who you think might that'd be awesome but a topic that I feel is important for practitioners to understand is a real sea change in natural language processing that's you know all of like 12 months old but is one these things I think is incredibly significant in in the field and that is the advance of the Transformers so the outline for this talk is to start out

LSTM is dead. Long Live Transformers!


with some background on natural language processing and sequence modeling and then talk about the LS TM why it's awesome and amazing but still not good enough and then go into Transformus and talk about how they work and why they're amazing so for background on natural language processing NLP I'm gonna be talking just about a subset of NLP which is the supervised learning a part of it so not

LSTM is dead. Long Live Transformers!


structured prediction sequence prediction but where you're taking the document as some input and trying to predict some fairly straightforward output about it like is this document spam right and so what this means is that you need to somehow take your document and represent it as a fixed-size vector because I'm not aware of any linear algebra that works on vectors of variable dimensionality and

LSTM is dead. Long Live Transformers!


the challenge with this is that documents are of variable length right so you have to come up with some way of taking that document and meaningfully encoding it into a fixed size vector right so the classic way of doing this is the bag of words right where you have one dimension per unique word in your vocabulary so English has I don't know about a hundred thousand words in the vocabulary

LSTM is dead. Long Live Transformers!


right and so you have a hundred thousand dimensional vector most of them are zero because most words are not present in your document and the ones that are have some value that's maybe account or tf-idf score or something like that and that is your vector and this naturally leads to sparse data where again it's mostly zero so you don't store the zeros because that's

LSTM is dead. Long Live Transformers!


computationally inefficient you store lists of a position value tuples or maybe just a list of positions and this makes the computation much cheaper and this works this works reasonably well a key limitation is that when you're looking an actual document order matters right these two documents mean completely different things right but a bag of words model will score them

LSTM is dead. Long Live Transformers!


identically every single time because they have the exact same vectors for for what words are present so the solution to that in this context is in grams you can have by grams which every pair of possible words or trigrams are for every combination of three words which would easily distinguish between those two but now you're up to what is that a quadrillion dimensional vector and you

LSTM is dead. Long Live Transformers!


can do it but you know you start running into all sorts of problems when you walk down that path so a in neural network land it's the natural way to just solve this problem is the R and n which is the recurrent neural network not the recursive neural network I've made that mistake but RN ends are a new approach to this which asked the question how do you calculate a function on a

LSTM is dead. Long Live Transformers!


variable-length set of input and they answer it using a for loop in math where they recursively define the output at any stage as a function of the inputs at the previous stages and the previous output and then for the purpose of supervised learning the final output is just the final hidden state here and so visually this looks like this activation which takes an input from the raw

LSTM is dead. Long Live Transformers!


document X and also itself in the previous time you can unroll this and visualize it as a very deep neural network where there the final answer the number you're looking at the end is this and it's this deep neural network that processes every one of the inputs along the way alright and the problem with this classic vanilla all right on this plane recurrent neural network is

LSTM is dead. Long Live Transformers!


vanishing an exploding gradients right so you take this recursive definition of the hidden state and you imagine what happens just three points in right and so you're calling this a function this a transformation over and over and over again on your data and classically in the vanilla want a case this is just some matrix multiplication some learned matrix W times your input X and so when

LSTM is dead. Long Live Transformers!


you go out and say a hundred words in you're taking that W vector W matrix and you're multiplying it a hundred times alright so in in simple math in in real number math we know that if you take any number less than one and raise it to a very high dimensional value sorry very high exponent you get some incredibly small number and if your number is slightly larger than one then it blows

LSTM is dead. Long Live Transformers!


up to something big and if you go if your X even higher if you have longer documents this gets even worse and in linear algebra this is about the same except you need to think about the eigenvalues of the matrix so the eigenvalues is say how much the matrix is going to grow or shrink vectors when the transformation is applied and if your eigenvalues are less

LSTM is dead. Long Live Transformers!


than one in this transformation you're going to get these gradients that go to zero as you use this matrix over and over again if they're greater than one then your gradients are going to explode all right and so this made vanilla RN ends extremely difficult to work with and basically just didn't work on anything but fairly short sequences all right so LST m to the rescue right so I

LSTM is dead. Long Live Transformers!


wrote this document a few years ago called the rise and fall and rise and fall of LST M so at least Tim came around in the dark ages and then it went into the AI winter it came back again for awhile but I think it's on its way out again now with with transformers so Ellis Tim to be clear is a kind of recurrent neural network it just houses more sophisticated cell inside and it

LSTM is dead. Long Live Transformers!


was invented originally in the dark ages on stone tablet that has been recovered into a PDF that you can access on Sep hawk right there's a server III kid but seven and you're gonna both grade I enjoy they're both quite a bit but they did a bunch of amazing work in the 90s that was really well ahead of its time and and often get neglected and forgotten as time goes on that's totally

LSTM is dead. Long Live Transformers!


not fair because they did an amazing research so the LST emcell looks like this it actually has two hidden states and the the input coming along the bottom and the output up the top again and these two hidden states and I'm not going to go into it in detail and you should totally look at Christopher ollas blog post if you want to dive into it but the key point is that these these

LSTM is dead. Long Live Transformers!


transformations these the matrix multiplies right and they are not applied recursively on the main hidden vector all you're doing is you're adding in or the forget gate yeah you actually don't really need it but you're adding in some some new number and so the OS TM is actually a lot like a res net it's a lot like a CNN resonate in that you're adding new values on to the activation

LSTM is dead. Long Live Transformers!


as you go through the layers right and so this solves the exploding and vanishing gradients problems however LST M is still pretty difficult to train because you still have these very long gradient paths even even without even with those residual connections you're still propagating gradients from the end all the way through this transformation cell over at the beginning and for a

LSTM is dead. Long Live Transformers!


long document this means very very deep networks that aren't just Toria Slee difficult to train and more importantly transfer learning never really worked on these LST M models right one of the great things about image net and cnn's is that you can train a convolutional net on millions of images in image net and take that neural network and fine-tune it for some new problem that

LSTM is dead. Long Live Transformers!


you have and the the starting state of the you mention at CNN gives you a great a great place to start from when you're looking for a new neural network and makes training on your own problem much he there was much less data that never really worked with Ellis Jim sometimes it did but it just wasn't very reliable which means that anytime you're using an LS TM you need a new label data set

LSTM is dead. Long Live Transformers!


that's specific to your task and that's expensive okay so this this changed dramatically just about a year ago when the burp model was was released so you'll hear people talk about Transformers and Muppets together and the reason for this is that the original paper on this technique that describes the network architecture it was called the transformer network and then the

LSTM is dead. Long Live Transformers!


Bert paper is a muppet news and Elmo paper and you know researchers just run with the joke um so this is just context you understand what people are talking about if they say well use them up in network so this I think it was the natural progression of the sequence of document models and it was the transformer model was first described about two and a half years ago in this

LSTM is dead. Long Live Transformers!


paper attention is all you need and this paper was addressing machine translation so think about taking a document in in English and converting it into French right and so the classic way to do this in neural network is encoder/decoder here's the full structure there's a lot going on here right so we're just going to focus on the encoder part because that's all you need for these supervised

LSTM is dead. Long Live Transformers!


learning problems the decoder is similar anyway so zooming in on the encoder part of it there's still quite a bit going on and so we're but basically there's three parts there's we're gonna talk about first we're going to talk about this attention part then we'll talk about the part of the bottom of the positional coding the top parts just not that hard it's just a simple fully connected layer

LSTM is dead. Long Live Transformers!


so the attention mechanism in the middle is the key to making this thing work on documents of variable lengths and the way they do that is by having an all-to-all comparison for every layer of the neural network it considers every pause for every output of the next layer considers every plausible input from the previous layer in this N squared way and it does this weighted sum of the

LSTM is dead. Long Live Transformers!


previous ones where the waiting is the learned function right and then it applies just a fully connected layer after it but it this is this is great for for a number of reasons one is that you can you can look at this thing and you can visually see what it's doing so here is this translation problem of converting from the English sentence the agreement on the European

LSTM is dead. Long Live Transformers!


Economic Area was signed in August 1992 and translate that into French my apologies la casa la zone economic European at this may and oh I forgot 1992 right and you can see the attention so as its generating lips as a generating the each token in the output it's it's starting with this whole thing's name button its generating is these output tokens one at a time and it

LSTM is dead. Long Live Transformers!


says okay first you got to translate the the way I do that it translates into la and all I'm doing is looking at this next I'll put a color all I'm doing is looking at agreement then sir is on la is the okay now interesting European Economic Area translates into zone economic European so the order is reversed right you can see the attention mechanism is reversed also or you can

LSTM is dead. Long Live Transformers!


see very clearly what this thing is doing as it's running along and the way it works in the attention are setting the transformer model the way they describe it is with query and key vectors so for every output position you generate a query and for every input you're considering you generate a key and then the relevant score is just the dot product of those two right and to

LSTM is dead. Long Live Transformers!


visualize that you first you combine the key the query and the key values and that gives you the relevant scores you you use the softmax normalize them and then you do a weighted average of the values the third version of each token to get your output now to explain this in a little bit more detail I'm going to go through it in pseudocode so this looks like Python it wouldn't actually

LSTM is dead. Long Live Transformers!


run but I think it's close enough to help people understand what's going on so you've got this attention function right and it takes as input a list of tensors I know you don't need to do that a list of 10 serious one per token on the input and then the first thing it does it goes through each everything in the sequence and it computes the query the key and the value by multiplying the

LSTM is dead. Long Live Transformers!


appropriate input vector by Q k and V which are these learned matrices right so it learns this transformation from the previous layer to whatever should be the query the key and the value at the at the next layer then it goes through this double nested loop alright so for every output token it figures out okay this is the query I'm working with and then it goes through

LSTM is dead. Long Live Transformers!


everything in the input and it multiplies that query with the the key from the possible key and it computes a whole bunch of relevant scores and then it normalizes these relevant scores using a soft Max which makes sure that they just all add up to one so you can sensibly can use that to compute a weighted sum of all of the values so you know you just go through for each output

LSTM is dead. Long Live Transformers!


you go through each of the each of the input tokens the value score which is calculated for them and you multiply it by the relevance this is just a floating point number from 0 to 1 and you get a weighted average which is the output and you return that so this is what's going on in the attention mechanism which can be which can be pretty confusing when you just look at it look at the diagram

LSTM is dead. Long Live Transformers!


that like that but I hope this I hope this explains it a little bit I'm sure we'll get some questions on this so relevant scores are interpretable as I say and and this is is super helpful right now the an innovation I think it was novel in the transformer paper is multi-headed attention and this is one of these really clever ID and important innovations that it's not actually all

LSTM is dead. Long Live Transformers!


that complicated at all I you just do that same thing that same attention mechanism eight times whatever whatever value of 8 you want to use and that lets the network learn eight different things to pay attention to so in the translation case it can learn an attention mechanism for grammar one for vocabulary one for gender one for kent's whatever it is right whatever the thing

LSTM is dead. Long Live Transformers!


needs to it can look at different parts of the input document for different purposes and do this at each layer right so you can kind of intuitively see how this would be a really flexible mechanism for for processing a document or any sequence okay so that is when the key things that enables the transfer model that's the multi-headed attention part of it now let's look down

LSTM is dead. Long Live Transformers!


here at the positional encoding which is which is critical and novel in a critical innovation that I think is incredibly clever so without this positional encoding attention mechanisms are just bags of words right there's nothing seeing what the difference is between work to live or live to work right there they're just all positions they're all equivalent positions you're

LSTM is dead. Long Live Transformers!


just going to compute some score for each of them so what they did is they took a lesson from Fourier theory and added in a bunch of sines and cosines as extra dimensions sorry not as extra dimensions but onto the the word embeddings so going back so what they do is they take the inputs they use word Tyvek to calculate some vector for each input token and then

LSTM is dead. Long Live Transformers!


onto that onto that embedding they add a bunch of sine and cosines of different frequencies starting at just pi and then stretching out longer and longer and longer and if you look at the whole thing it looks like this and what this does is it lets the model reason about the relative position of any tokens right so if you can kind of imagine that the model can say if the orange

LSTM is dead. Long Live Transformers!


dimension is slightly higher than the blue dimension on one word versus another then you can see how it knows that that token is to the left or right of the other and because it has this at all these different wavelengths it can look across the entire document at kind of arbitrary scales to see whether one idea is before or after another the key thing is that this is how the

LSTM is dead. Long Live Transformers!


system understands position and isn't just a bag of words for Fortran when doing the attention okay so transformers there's the two key innovations as positional encoding and multi-headed attention transformers are awesome even though there are N squared and the length of the document these all-to-all comparisons can be done almost for free in a modern GPU GPUs

LSTM is dead. Long Live Transformers!


changed all sorts of things right you can do a thousand by thousand matrix multiply as fast as you can do a ten by two in a lot of cases because they have so much parallelism they have so much bandwidth that but a fixed latency for every operation so you can do these massive massive multiplies almost for free in a lot of cases so doing things in M Squared is is not actually

LSTM is dead. Long Live Transformers!


necessarily much more expensive whereas in an RNN like an L STM you can't do anything with token 11 until you're completely done processing token 10 all right so this is a key advantage of transformers they're much more computationally efficient also you don't need to use any of these sigmoid or tan h activation functions which are built into the LS TM model these things of

LSTM is dead. Long Live Transformers!


scale your activations to 0 1 why are these things problematic so these were bread-and-butter in the old days of of neural networks people would use these between layers all the time and they make sense there that kind of biologically inspired you take any activation you scale it from 0 to 1 or minus 1 to 1 but they're actually really really problematic because if you get a

LSTM is dead. Long Live Transformers!


neuron which has a very high activation value then you've got this number up here which is 1 and you take the derivative of that and it's 0 or it's some very very small number and so your gradient descent can't tell the difference between an activation up here and one way over on the other side so it's very easy for the trainer to get confused if your activations don't stay

LSTM is dead. Long Live Transformers!


near this middle part all right and that's problematic compare that to rel U which is the standard these days and really you yes it does have this this very very large dead space but if you're not in the dead space then there's nothing stopping it from getting getting bigger and bigger and scaling off to infinity and one of the reasons why when the intuitions behind why this works

LSTM is dead. Long Live Transformers!


better as Geoffrey Hinton puts it is that this allows each neuron it to express a stronger opinion right in an LS sorry in a sigmoid there is really no difference between the activation being three or eight or twenty or a hundred the output is the same right it all I can say is kind of yes no maybe right but in with our Lu it can say the inactivation of five or a hundred or a

LSTM is dead. Long Live Transformers!


thousand and these are all meaningfully different values that can be used for different purposes down the line right so each neuron it can express more information also the gradient doesn't saturate we talked about that and very critically and I think this is really underappreciated values are really insensitive to random initialization if you're working with a bunch of sigmoid

LSTM is dead. Long Live Transformers!


layers you need to pick those random values at the beginning of your training to make sure that your activation values are in that middle part where you're going to get reasonable gradients and people used to worry a lot about what initialization to use for your neural network you don't hear people worrying about that much at all anymore and rail users are really the key reason why that

LSTM is dead. Long Live Transformers!


is also really runs great on low precision Hardware those those floating of the smooth activation functions they need 32-bit float maybe you can get it to work in 16-bit float sometimes but you're not going to be running it an 8-bit int without a ton of careful work and that is the kind of things are really easy to do with a rel u based network and a lot of hardware is going

LSTM is dead. Long Live Transformers!


in that direction because it takes vastly fewer transistors and a lot less power to do 8-bit integer math versus 32-bit float it's also stupidly easy to compute the gradient it's one or at zero right you just take that top bit and you're done so the derivatives ridiculously usually rel you have some downsides it does have those dead neurons on on the left side you can fix

LSTM is dead. Long Live Transformers!


that with a leaky rail you there's this discontinuity in the gradient of the origin you can fix that with Gail u which burnt uses and so this brings me to a little aside about general deep learning wisdom if you're designing a new network for whatever reason don't bother messing with different kinds of activations don't bother trying sigmoid or tant they're they're probably not

LSTM is dead. Long Live Transformers!


going to work out very well but different optimizers do matter atom is a great place to start it's super fast it tends to give pretty good results it has a bit of a tendency to overfit if you really are trying to squeeze the juice out of your system and you want the best results SGD is likely to get you a better result but it's going to take quite a bit more time to

LSTM is dead. Long Live Transformers!


Verge sometimes rmsprop beats the pants off both of them it's worth playing around with these with these things I told you about why I think SWA is great there's this system called attitude my old team at Amazon released where you don't even need to take a learning rate it dynamically calculates the ideal learning rate scheduled at every point during training for you it's kind of

LSTM is dead. Long Live Transformers!


magical so it's worth playing around with different optimizers but don't mess with the with the activation functions okay let's pop out right there's a bunch of a bunch of Theory bunch of math and and ideas in there how do we actually apply this stuff in code so if you want to use a transformer I strongly recommend hopping over to the the fine folks at hugging face and using their transformer

LSTM is dead. Long Live Transformers!


package they have both pi torch and tensor flow implementations pre-trained models ready to fine tune and I'll show you how easy it is here's how to fine tune a Bert model in just 12 lines of code you just pick what kind of Bert you want the base model that's a paying attention upper and lower case you get the tokenizer to convert your string into tokens you download the pre trained

LSTM is dead. Long Live Transformers!


model in one line of code pick your data set for your own problem process the data set with the tokenizer to get training validation splits shuffle one batch um four more lines of code another four lines of code to instantiate your optimizer define your loss function pick a metric it's tensorflow so you got to compile it and then you call fit and that's it that's

LSTM is dead. Long Live Transformers!


all you need to do use - all you need to do to fine-tune a state-of-the-art language model on your specific problem and the fact you can do this on some pre trained model that's that's seen tons and tons of data that easily is really amazing and there's even bigger models out there right so Nvidia made this bottle called megatron with eight billion parameters they ran a hundreds

LSTM is dead. Long Live Transformers!


of GPUs for over a week spent vast quantities of cash well I mean they own the stuffs but so not really but they they put a ton of energy into training this I've heard people a lot of people complaining about how much greenhouse gas comes from training model like Megatron I think that's totally the wrong way of looking at this because they only need to do this once in the history of

LSTM is dead. Long Live Transformers!


the world and everybody in this room can do it without having to burn those GPUs again right these things are reusable and fine tunable I don't think they've actually released this yet but but they might and somebody else will right so you don't need to do that that expensive work over and over again write this thing learns a base model really well the folks at Facebook trained this

LSTM is dead. Long Live Transformers!


Roberta model on two and a half terabytes of data across over a hundred languages and this thing understands low resource languages like Swahili and an Urdu in ways that the it's just vastly better than what's been done before and again these are reusable if you need a model that understands all the world's languages this is accessible to you by leveraging other people's work and

LSTM is dead. Long Live Transformers!


before Bert and transformers and the Muppets this just was not possible now you can leverage other people's work in this way and I think that's really amazing so to sum up the key advantages of these transforming networks yes they're easier to train they're more efficient all that yada yada yada but more importantly transfer learning actually works with them right you can

LSTM is dead. Long Live Transformers!


take a pre trained model fine-tune it for your task without a specific data set and another really critical point which I didn't get a chance to go into is that these things are originally trained on large quantities of unsupervised text you can just take all of the world's text data and use this as training data the way it works very very quickly is kind of comparable to how

LSTM is dead. Long Live Transformers!


word Tyvek works where the language model tries to predict them some missing words from a document and in that's enough for it to understand how to build a supervised model using vast quantities of text without any effort to label them Ellis team still has its place in particular if the sequence length is very long or infinite you can't do n squared right and that happens if you're doing real

LSTM is dead. Long Live Transformers!


time control like for a robot or a thermostat or something like that you can't have the entire sequence and for some reason you can't pre train on some large corpus LS TM seems to outperform transformers when your dataset size is is relatively small and fixed and with that I will take questions well you yes yeah would CNN how do you compare words CNN transformer so when

LSTM is dead. Long Live Transformers!


when I wrote this paper the rise and fall and rise and fall of LST M I predicted that time that word CNN's were going to be the thing that replaced LST M I did not I did not see this this transformer thing coming so a word CNN has a lot of the advantages in terms of parallelism and the ability to use rel you and the key difference is that it only looks at a fixed size window fixed

LSTM is dead. Long Live Transformers!


size part of the document instead of looking at the entire document at once and so it's it's got a fair amount fundamentally in common word CNN's have an easier task easier time identifying diagrams trigrams things like that because it's got those direct comparisons right it doesn't need this positional encoding trick to try to infer with with fourier waves what where

LSTM is dead. Long Live Transformers!


things are relative to each other so it's got that advantage for understanding close closely related tokens but it can't see across the entire document at once right it's got a much harder time reasoning like a word CNN can't easily answer a question like does this concept exist anywhere in this document whereas a transformer can very easily answer that just by having some

LSTM is dead. Long Live Transformers!


hi welcome to lecture nine of CS 294 158 deep unsupervised learning spring 2020 I hope everyone had a good spring break despite the rather unusual circumstances today we will be covering two main topics semi supervised learning and unsupervised distribution alignment before diving into that couple of logistics so common situation and a quick mid semester update well first we

L9 Semi-Supervised Learning and Unsupervised Distribution Alignment -- CS294-158-SP20 UC Berkeley


hope everyone and their families are able to keep healthy during these pretty unusual times please practice your health and well be and accordingly please don't hesitate to let us know if this class would interfere with that and we'll be happy to figure something out when the kiss by kids faces since water just past the middle of the semester and says there's some replanting paired the


current situation here's a quick overview of what's still ahead in this class today we have lecture 9 which will cover semi supervised learning and unsupervised distribution alignment next week we'll have lecture ten on compression which will be a live zoom lecture then at the end of next week your final project three-page milestone reports argue this is not graded but


it's a great way to get feedback and make sure you're on track for a good final project and we'll try to give you feedback into your Google Doc that you share with us on pretty short turnaround then we'll have lecture 11 on language models with a guest instructor aleck Radford from open AI then we'll have our midterm which will adjust to the current circumstances we'll see how we do it but


the high-level model will promised a similar that we want to cover the main derivations that we've seen in this semester and have you be able to read arrive those then we'll have our final regular lecture lecture 12 on representation learning in reinforcement learning that also be a live zoom lecture and recording then as our our week which hopefully gives you time to


catch up on a lot of things you know hopefully including them making the extra push in the final project for this class and then during finals week on the 13th of Wednesday there's final project presentations we'll see how to do that with the new situation and final project reports will be Q at that time also with all the logistics covered let's dive into the technical accountant for


today so we'll cover first semi-supervised learning which Harvin along will cover and then we'll cover unsupervised situation alignment which will be covered by Peter Chen and Ashley will use the lecture he gave last year also for this year welcome to lecture 9a of deepens for us learning in this part of the lecture we'll be covering semi-supervised learning so first to


understand what is amiss for us learning let's look at supervised learning in supervised learning you have a joint distribution of data points and labels and use sample from the Joint Distribution in expectation over two samples from the Joint Distribution your goal is to maximize a lot of probability of the classifier love P of Y given X we all know how to do this as far as


procedurally how it's done the basically sample image label or like sequence and particular label or any pair of x and y from your data set assuming they're all independent identically distributed and you don't know what the analytical following the distribution is where you assume that it is some complicated distribution and you just keep sampling multiple points repeatedly and from


stochastic rain descent optimized objective now what is semis corresponding assume that you have an unlabeled data set tu where X is sample from P of X which is a marginal responding to the Joint Distribution from the label data set sample from so you have D you the unlabeled data set and D s the label ta set and your goal is to perform the same thing earlier


which is supervised learning on label data set but with access to this extra unlabeled data that's coming from the same marginal so that's the assumption you make semi-square is learning which is that the unlabeled data is coming from the margin of the correspond to the same Joint Distribution that supervised materials coming from and assistant mathematical assumption in practice you


can't really ensure that but your goal is to make sure that you can use this extra unlabeled data to perform your supervised learning objective even better on the label Deana so not mathematically here is the summary of what we just described take a task like classification a fully supervised scenarios where you have every single data point given to you in the form of image from a label they try


and predict a label for new images that's your task that for a supervised learning now the scenario is you're going to be given a few on label samples but multiple you're going to be given a few label samples but you're also gonna be given a lotta on label samples now labeling is a time-consuming process and potentially very expensive and actually pretty hard in certain domains like


medicine right so or detecting rare events and sub driving for them for that matter so if you have a lot of unlabeled data and your training data set can be reprime tries now is having some lis pairs of label data points image command label and also a lot of other data points where you just have the image and your goal is take this extra data extra data points where you don't have labels


and try to improve our classifiers it just works with the label data and how you do it is totally up to you and that basically decides what kind of summarized algorithm you're gonna come up with to use and we're gonna look at in this lecture we're going to look at how these algorithms can be designed what are the mathematical or intuitive aspects of these algorithms and how you


compare each other and how they can scale the larger data sets like image that and beyond so as to why we even interested in this problem semi-supervised learning is really important because they believe us even though you can collect label data very easily these days with a lot of data annotation startups it's still expensive in terms of hiring people to write annotation manuals for actual data


AM traders and preparing graphical user interfaces so that all this is done really fast and making sure it's stored on the cloud efficiently and syncing from the browser to the cloud there are lots of engineering challenges involved in setting up a good data on editing tool now that's not to say we're never gonna do it we are still gonna do it but the goal is to make sure that we don't do it


as much as we're doing it right now because we also have access to full-on link data that we can potentially exploit and they maybe even improve the performance or legal data systems this is similar in spirit to our goals for Salesforce learning and service learning is a different take on this subspace learning can work with just unlabeled data whereas my learning needs some


label do a lot of unlabeled data that's the key difference so here is a slide from Tung Wong who took this particular picture from Vincent Van Hook on a blog post called the quiet some expose learning revolution where the belief of many more practitioners at least until recently was that semi-square is learning will be really useful in a low data regime where it's really really


going to be better than normal supervised learning when you hardly have any label here however once you collect sufficient amount of labels supervised learning we catch up and eventually be much better this is why a lot of startups don't care about since Portland because the amount of effort needed to do research and engineering together since we're learning working especially


even that it's a brand new fear this lot whenever needed to collect the external data points and you're guaranteed a better performance anyway so that's the rationale so as far as the left bloc glows but look at the plot on the right the dream of many times players learning researchers is that it not only is gonna be super useful in a loliter regime but it's gonna be extremely useful even in


the high energy because it's still gonna give you that final few percentage extra performance because of access to a lot of unlabeled data and learning much richer or more fine-grained detail classifiers because of that and that's basically what's happened recently and we look at the history of the field in recent times so the core concepts needed to understand Sims quest learning are


very few and we're just gonna look at them at a very high level and it's really really intuitive and not hard to understand so the first principle we look at is confidence of classifiers versus minimal entropy on unlabeled data it's it's it's a very nice coupling that the disciplines will shows and it's being used to very good effect in recent recent work and mathematically there


have been papers on these two ideas which is first ideas entry minimization which we look at just the idea of taking on label data and making sure that the classifier training or label here has minimal entropy on unlabeled data so that way you're making sure that the classifier is confident even on unlabeled data it's a useful bit a regularizer classifier the second idea


is your labeling where you take your classifier you asked to classify to predict what the labels are for unlabeled here and you take the confident directions and convert them to extra they ask 'but was the ground truth and train the model on those data points so this idea is also referred to literature as sub training the area of training the model of its own predictions if the model is


confident enough and expanding your data set and regularizing your model further this is a little tricky because it has this reinforcement reinforcing effect of the model using its own predictions so it needs to be done very carefully so that's the caveat there and the other way to add noise to a model to regularize the model is virtual adversarial training which we really


look at in detail but the idea is similar to how Ibis your training is performed for and you know absolute examples and images where you have a particular label and which you're trying to fool the classifier and believing that this image corresponds that label and you're trying to add a particular noise in the direction of the gradient of the output with respect to the input


Society model starts producing some other label similarly here you want to make sure that the model the same squared learning model is regularized in the directions around which tomorrow on unlabeled data you want to make sure find directions in which the classifier is likely to be confused and you want to make sure that the model is not confused in those directions so that's the area


in which we'll have a single training and there are also areas like label consistency which is make sure that the augmentations of the sample have the same class so you have an image we know that we use a lot of the documentation and regular supervised learning but in semi-supervised learning you have a lot of unlabeled data and you can't apply the augmentation to them if


you're not passing them to the classifier but instead what you can do is you take an unlabeled sample it create two different orientations of it and you see it you tell the classifier that it's predictions on these two different augmentations of the unlabeled data should be roughly similar because even though you don't have a label you tell the classifier that whatever it's


printing it should be similar and this way the classifier gets a lot of structure and loss function and parameters that are being learned from unlabeled data a lot of constrains are being imposed and therefore it's going to be much more regularized than just training on the label data so this is a very neat idea and it's also similar in spirit to things you've seen in


subsidising which is the idea of taking two different views of an image and trying to make sure that they attract each other relative to another image so basically consistency constraints embedded into your encoder and raise different ideas in the past have attempted to do this and we are going to look at them in the PI Maru temporal ensemble and mean teacher finally we're


going to look at regularization which is the idea of taking a model making sure that it's generalizing well to a new unlabeled data set or new validation set so typically people use weight decay dropout data argumentation for making sure that the classifiers generalize well and those are also pretty important in semi-supervised learning and methods that identity use these are unsupervised


it augmentation or UDA and mix-match which we look at in detail but there are also other papers that we can't really cover in the scope of the structure but you should check out other people's that are related I mentioned this related work on these papers finally we look at the area of cool training or a self-training or pseudo Laban all these are ideas that


have already been mentioned in this list of bullet points but there is a particular paper on his student which has taken these ideas to a whole new level in terms of performance and so we look at that in a little more detail so entropy minimization it's a very simple idea you have a lot of unlabeled data and you have your label here your training and classifier on the label


data but you want to make sure that the unlabeled data is also influencing the classifier in some way so one simple idea is you take your classifier and ask ask you to predict on the unlabeled data and you want to make sure that the classifier is pretty confident on the unlabeled data or rather it's entropy of the class probabilities it outputs on the unlabeled data is small enough and


this way you ensure that the structure is this way you ensure that the classifier is understanding the structure the unlabeled data we're trying to be confident about it so it's trying to find a solution for the label data in such a manner that it will be pretty confident on the unlabeled data as well so this this is one way to do something special so very old idea and


pseudo label this is a very similar idea and being see how it's actually similar but the goal here is to take your classifier that's being trained on label data and ask you to predict on unlabeled data and you pick the most confident predictions and you turn them into extra label layer as if that were the ground truth and you train a classifier or its own prediction so the classifier is


making a bunch of predictions and those are being converted to ground truth proxy 100 data for itself and it's going to train again on new things new data sets created from the unlabeled data based on itself and this principle is also referred to as self training and there is a connection to end musician so here is the connection so consider an image X and classes y1 y2 y3


and let's say you're doing a classification problem and let's say there is a classifier a with probabilities for the output classes being point one point eight point one and that's why at B with the probability is being point one point six and point three so that's why a clearly has lower entropy and you can say it's more confident it's more confident that the


true ground truth is y2 and its score for y2 is much higher and scores for the other two classes are more similar and lower compared to Class B so there is clearly a connection to be made in terms of a classifier being more confident and therefore having lower entropy in the output probability distribution of the classes and therefore minimizing the entropy of your classifier on unlabeled


data I can do taking a classifiers outputs an unlabeled layer if they are confident enough you're trying to train on its own predictions so it has a similar effect and mathematically it's shown in these older papers that have been linked here so you can go and check it out and the next thing we're gonna see is later augmentation for label consistency so you take an image let's


say it's whether it's unlabeled or labeled you're given an image and now you've created it for an organisation's of it so similar this is the same picture we used in Sinclair and moco so I'm just using it so that you can relate this concept with the earlier lecture where we talked in cells for learning about data augmentation consistency using contrast losses so similar ideas


have been used in semi-supervised learning as well so like I said you're already using the documentation for label data so it doesn't matter if you enforce consistencies there but for unlabeled data if you take two different views and make sure that the logits are close enough for the classifier that's being trained on label dia that enforces a lot of structurally believer so you just make


sure that the predictions are roughly similar and if you do this for a lot of legal data with a lot of different data occupations then your classifier is getting very much regularize in generalizable even those training on very little data so that's the idea of label label consistency constrains using the augmentation and the more they documentation you use the better it is


so that's it for the foundational material next we'd actually look at different semi-supervised learning algorithms like the PI model temporal ensemble in virtual adversarial training and so on but we also look at how the algorithms compare to each other and this particular paper from Google brain realistic evaluation decent spread learning algorithms compares these


various different semi-square learning techniques on the C fart and svh and data set which are reasonably small and you can run a lot of prototyping experiments with that so the four algorithms we're going to be looking at our PI model temporal ensemble main teacher and virtual adversary training so basically let's look at the PI model basically the idea is pretty much whatever we talked


about our legal consistency you take your image you are creative different views using the stochastic a dog hunter but stochastic it could be at a random crop or a sequence of data augmentations whose sequence is randomized or but there you apply it running grayscale so on and now you take it out you pass it to the model and the model itself could be stochastic it could have drop out so


every forward pass could give you a different output even for the same image and you get two different latent variables for these different we'll turn the input down in the model so every time you make a forward pass you label data you can enforce it regularly the supervised crossing could be lost and if you throw unlabeled data you can enforce the square or square


difference between the outputs of the model or dutiful views and for label either you can force both of these losses well for unlabeled data you can you just enforce this label consistency loss which is you just take your output before the softmax or even after the softmax it depends on how you want to implement it but you take a particular layer at the end and you make sure that


the layers is similar for two different views and you weigh both these losses together so one is gonna be unsupervised on same squares loss and the other is supervised loss and you can actually control loose loss actually dominates the training the beginning at the end so for instance one reasonable motivation is you can make sure that the supervised lost dominates in the beginning so that


the model already learns how to classify images and then you can ramp up to wait for these cells for the semi polarized or unsupervised loss of that it's learning to structure the unlimited on similar fashion so this idea is called pi model and this is the zero code for pi model excise your training wise your labels WT is your rammed up wait for your hands provides function f theta faxes your


neural net that's the classification task and it could have some drop are in that world to be stochastic G of X is your data argumentation is also stochastic and you basically perform two different augmentations of your mini-batch get two different output CI and Zi to D and you make sure that Zi and Zi absolutely are close to each other using a square error loss or some


some distance metric and you also make sure that the predictions of the classifier or matching the true ground truth whenever you have labels so that's that's really it that's as simple as I can get the PI model basically it's using the label the distance e principle so temporal augmentation there's something slightly different which is it says hey I don't want to do four passes of two different


views all the time because it's expensive why not let me just keep a moving average of these sample embeddings for every single sample and make sure that the consistencies and falls across time so I would still do a stochastic the augmentation stochastic Network I would get an embedding every time in the forward pass but I would say that those embeddings should be close to


some historical a version of the same same same samples embedding the past and so that way gives to enforcing some kind of data augmentation constraints because you would have done a different organization in the past but you're going to keep an estimator of it for every single sample separately so this is very similar to those ideas we talked about in again let chosen your improve


gang where there was a constraint on the parameters to be closely extort conversions so this is that at a sample level and other than that it's pretty much the same as the PI model there's a cross between laws there's a roundup function for the unsupervised objective and we'll the objectives are only optimize together so this is the serial code for tempora ensemble is proposed in


the same paper we're fine models pose so one negative thing about imported ensemble is it's not going to scale with the size of the dataset you're not gonna be able to maintain a separate moving average embedding for every sample if you have if it is that is so big enough like a million or billion images so mean teacher basically amortize that and said that hey if you want to keep an


exponential moving average for n bearings why not just keep an expansion moving average for parameters so that you used to take two different views but make sure that the embeddings match that of the moving average versions rather than creating separate moving average embeddings for every sample so you take your model Tita there's also en male version of Tita and


you make sure that the embeddings that you get for one view matches the in bearings very good for the other view but with different encoders basically so that's the idea of this main teacher approach where the teacher can be considered as the AMA version and a you think about it as a teacher because it's giving you these constraints and you also perform the classification task as


fication lost in peril sometimes they finally let's look at virtual art whistle training so in adversarial training you're create using this fast sine gradient method where you basically calculate the gradient of your input image basically the gradient of your output with respect your input image this is a huge dimensional vector or matrix depending on what your input is


and you move your input in the direction where you basically get this gradient you get this sign and you move your input in a small epsilon in that direction and that lets you fool your classifier and you want to make sure that the in your classifier is not full if you for these perturbations and so you would perform at with stable training to make sure that the


classifier is not full at these data in these different rotor patient directions now in Sammy squirrel learning you don't have the labels for unlabeled data so how would you address your training there so the idea is to do virtual address zero training where you look at the distribution of your classes instead of a particular class because you don't you you don't want to make take a


particular class and see the perturbation direction for that class you take a distribution of classes and you take something a distance metric between your unperturbed data point and your perturb data point in trying to figure out the direction that maximizes this scale and it turns out if you linearized scale term you can actually solve for this direction are using power


iteration and once you get the direction and you can make sure that you info structure your classifier around unlabeled data we're trying to make sure that on these perturbations are unlabeled here the classifier is still not fooled even though you don't have access to a true label so that's why it's referred to as virtual data Co training it's not actually adversarial


training but it shares principles with annual training done cleverly with some mathematical tricks and this is basically the pseudocode for the power iteration method which i mentioned because it works because you linearized the k ho term so that's that's it for the techniques by model temporal on some going mean teacher and questionable training of that these are the four technique


considered in this comparison paper and they make sure that they use the same architecture for all these techniques because prior work did not do that so they use a wide wrist not and the idea and white resonators your normal resin goes through a bottleneck and the wireless and doesn't do that it just uses three by three cons and does know one by one confer down sampling SAS wide


as possible so given all these constraints even all these similar architecture similar hyper parameters for various different semi-square learning algorithms it turns out that which will address your training performs the best if you look at C 410 which is four thousand labels so you see pretend originally it has 50,000 images so you basically just use 46,000


unlabeled data points and four thousand label data points which is like four hundred labels for class so that's really tiny compared to what the ocean is that is and virtual abyssal training gets an error rate thirteen point eight six percent which is the best among all the other methods and which I was so training plus entropy minimization together it's even lower error rate of


thirteen point one three percent and the trends is similar for the svh and I said where virtual artists are training plus entropy immunization outperforms all the other methods and the authors also say that the report much better piece like stand prior work like for instance in prior work the baseline supported had much higher error rates than what the authors report so


this paper actually took the effort to make sure that all the ablations are done carefully and one negative thing about semi-supervised learning on c4 is that if you use something like a preacher image it you take an image Tantalus pre-trained on image netlabels and then you find unit on c4 you actually get better numbers than using the unlabeled data on c4 itself


even though see far the unlabeled data on c4 us coming from the same underlying distribution and image that is a completely different distribution completely different image sizes so that's slightly bad so and there's a significant difference as at least one point one just one one plus percentage difference and even if you address the class overlaps and remove the classes


overlapping and see far it's - you still get a lower error it and just using some experimenting on c4 and the author is also analyze each things like hey if your unlabeled data as soon see further Oerlikon ten classes and if you if you assume that the unlegal leader doesn't have the uniform class distribution its label there and you can play around what the distribution of unlabeled here points


are as far as a class overlap with Lolita is its it it is clear that as you as the distribution response increases which for a personal training is the most persistent compared to all the other approaches similarly if you vary the number of legal data points obviously the test error is going to be lower as the number of legal data points increases because that's slowly getting


to the supervised learning regime and all the meant is roughly performed similar but in the extreme scenario value are very few labels 215 up to 50 label data points and so on which will have a certain training significantly the best in this vhn and it is through the best on see for our notes other methods are competitive as well the lessons from this paper are when you


use the standard when you compare different algorithms and semi squares I mean you should make sure that you use a standard architecture and equal training budget which is you should spend equal amount of time tuning hyper parameters for all of them and if your unlabeled data is coming from a distribution that is a necessary overlap with your label data points then the benefits of


something surprise learning will not be there thirdly most methods that likelike don't work well in a very very low data regime so this is not true right now but this was true when that paper was published and we look see how it changed over time and transferring 320 machine had produce better error rates but again it's not true right now so this is an old paper but the main reason for going


through that is to introduce all these different techniques like the PI Maru and virtually I was still training and temporal on something in the mean teacher so the agenda for the less rest of the lecture is to cover three very recent papers and some surprise learning that actually have taken some space learning to a whole new level unsupervised the augmentation makes


match and noisy student so before that let's actually take a break you you okay resuming so first let's look into unsupervised data augmentation for at for a consistency training in same as for a Fleming so this is a paper from Google brain from Cork lace group and these slides are from tong long was one of the authors in this paper so we've already seen how important data


augmentation is and it's it's it's been significantly useful in supervised learning the high data regime but if you just do supervised learning if you don't have a lot of labels just data augmentation isn't gonna get you very far even if you use extremely aggressively documentation like Auto augment which is shown here where you basically can rotate shear and add colors to some


scene and create a lot of different views of the same image similarly in language you can create different versions of the same phrase or sentence using the technique called back translation so what it does is basically they cannot take and take a sentence in particular language translate it it to another language and then translate it back from the other language to the


existing set first source language so you go from a source language you go to a target language and you come back from the target language to the source language so you hope that this entropy in the decoder and the encoder will result in a in in a new version of the same sentence and examples are here so so the source sentence here is given the low budget and correctional limitations


the movie is very good and if you look at it three different Mac translations since it was highly limited in terms of the budget and the production restrictions the film was cheerful there are a few budget items with production limitations to make this film a really good one dude is a small dollar amount and recommendations Neos for them is very beautiful so


the first and third versions are particularly really good well the second conveys a slightly different meaning but it's more or less there so this is giving a lot of diversity and based on which language you move to you're gonna get very different outputs and also the same language you're gonna get different outputs for different decoding so the key idea and UDA or unsprayed


augmentation is apply the state-of-the-art data augmentation techniques on unlabeled data for consistency training in semi-supervised learning you've already seen that PI model are like like Supai model was basically doing consistency training but it was a pretty old paper and data augmentation through these numeral architectures were not as devil back then so you can think of UTA is


doing pi model right by using a lot of data augmentations on the right architectures so here's a nice way to understand UDA so think about label data and you have X and your ground through the Y star and your training a classifier at P feet of Y given X and you have your standard supervised cross-entropy laws that make sure that the logit so the true class are really maximized and you also have


unlabeled data so this is the situations and it's quite funny and models like virtual address zero training add noise to regularize to models predictions and the noises between the virtual address retraining direction very calculated gradient in approximate fashion next you have this thing called unsupervised consistency loss which is you take the noise from the model and


your original model and you make sure that the logic seriously similar unlabeled data so this is something you already know and a final loss is combination of supervised and unsupervised consistency loss and you can see that virtual adversity of training actually works pretty well so this is a green eyes illustration of which I do sell training even though it's being presented in the


UTA strikes so the uncolored data points are unlabeled data by the green and pink data points with a label here you only have roughly eight data points which are labeled but after performing virtual adversity of training after imposing the consistency between the model and the noise version of the model with a noise comes from that you can see how this labels propagated and covered up this


federal so that's really cool so that's the goal of semi-square learning to do this really well in high dimensions when you have a lot of data and a lot more parameters so you can think of UDA as creating this noise at the input level using various different data augmentations and depending on the two modality for instance you use Auto Alcuin for images you would use TF idea


word replacement or back translation for NLP and based on that you create different organizations at the same image or the same sample and enforces consistency laws and the unsupervised consistency but the last part and you also do supervised cross and freelance and you together out twice this so that's basically UDA and augmentations provides a diverse and valid


perturbations for your input so like I said back translation produce three different versions that look very different from each other in converted roughly the same meaning as the original source sentence so in this case they actually went from English to French and back to English but you can also think of doing it to other languages and you can increase the diversity by playing


around the temperature various different sampling techniques like beam search on nucleus sampling and you know whenever you do use a software so you can always use a temperature there you're gonna get tablet samples if you use high not high enough temperature and if use a low enough temperature you're going to get the most confidence after product samples with less diversity but high


quality so you can control for that similarly in images you can use argument but depending on the type of argumentation you can control the strength of the argumentation and get sufficient distortions based on what one level of distortion eclair bar so let's look at the experiments carried out on this paper so in language they experimented with document classification or sentiment


classification or review review sentiment analysis so if you look at the size of the data sets they have like twenty five thousand or four five sixty thousand and six fifty thousand samples and so on and the error rates before birth and after birth are mentioned different lines and you can see how Bert has significantly improved the error rates so what they did in the UDA papers


for various different initializations random bird base birth large and bird fine-tuned they have numbers for whether you use UDA or whether you don't use EDA but you notice how the number of label data points are changed by orders of magnitude three or four higher so if you look at IMDB you earlier had twenty five thousand label data points but the authors run his experiment trending


labeled eight points which is three orders of magnitude lower similarly for yell quits 2.5 km this is 650 K and so on and you can see how the performance especially when you use word fine-tune is on par with the birth large that strain on all the label data points so the performance you get from taking birth large and fine-tuning on the fully supervised baseline is actually on par


with what do you get from you da plus using bird fine-tune but just on like thousand x few labels which is incredible so it means that the consistency loss is using back translation is actually working really well secondly they have this idea called training signal annealing which is you want to prevent overtraining on the label here so you have a label data so


you don't want to make sure that you want to make sure that your classifier doesn't over train on it and for that they actually have something like a threshold a procedure where they take the classifier and if it's sufficiently confident they don't train the classifier points and this threshold is actually varied over time so you have an indicator variables of whether the


classifiers output logic score is less than threshold and only if it is you're gonna have the model trained on those data points or else you're just not gonna back up those gradients and this threshold has the you can play around with different schedules you could it could be really confident the beginning at the end you want to make sure that the thresholds are actually high enough


so that model is not training and in the beginning you don't want to have high enough threshold it so because your mom could be erroneous so they play around with linear or exponential and log and the paper finally this is a really cool plot that shows the dream of something which is the benefits hold even in the high rated regime even when you how to twenty five thousand label examples the


performance that you get is from semi-supervised learning is better than supervised bird find here so it's actually able to take advantage of all the unlabeled I have so next let's look at the computer vision experiments carried out on the UDA paper so basically they use the standard benchmarks on C for Tanana speech and semi-supervised learning which you saw in the prior work on


realistic evaluation of same exploiting algorithms so a lot of these baselines are from that paper where there is a virus net with 28 letters and you can see how the parameters are controlled for all the different algorithms and the numbers are reported for C flower with 4000 labels innovation with thousand labels and UDA is actually the best algorithm in this setting with an error


rate as low as 5% on c4 and 2.5% on s vhn and with architectural changes like shake shake and shake drop and permanet they get all the way down to 2.7 percent which is significantly lower so just four thousand samples they're as good as ninety 90 97 either better than nice and accurate on C far which is kind of accuracies you usually get with using all the label data points so that really


means to do domain consistency that the data augmentation label consistency is really really helping them and the models are the scaling was with larger networks so when you moved away from 1.5 million parameters it was typically used in semi-square is learning to see far earlier to a model that's big enough they passed 26 million parameters the error rates are getting


significantly lower so this shows that this technique scales with the number of parameters use next is how you can actually match the complete supervised baselines with just using order of magnitude fewer table data points so the complete supervised baseline uses 50,000 labels and this one uses 4,000 labels so that's 10x R or at like more than 10x fewer and and and if


you look at the numbers you basically get 5.4 percent error rate with supervised and 5.3 percent with UTA for the virus that's 28 architecture and for the other models even though UDA is slightly higher for shake-shake-shake drop it actually matters to supervise baselines though it's not as good as the Auto argument abortion it's still very very close finally they also ablated for how much


data mutation matters and it seems to matter really it's really the biggest deal as we would expect because we've seen that in subsequent learning as well and that's what they observe even here so the summary of UDA is that it's a data augmentation technique that's applied on unlabeled data to improve the performance of a classifier that's trained on very few labels their points


and data augmentation is that for a critical part of this pipeline and it's an effective perturbation technique for us any surprise nothing so it's even it's it's even more effective than perturbing the model and Yui is significantly improves for both language and vision by 10x or 100x a thousand X fewer label data requirements and combines very well to transfer learning


like Bert and scales for the model parameters and model sizes so they also experimented with imagenet where they take unlabeled imagenet of 1 million examples and 1 million unlabeled data points they use 10% label data which is 100,000 labels or 100 labels per image and pure supervised gets 55% whereas their model you DEA gets 68.9 close to 69 percent accuracy which is at


least 30% better and this shows the benefits are using more unlabeled data points so another thing that they tried is having a data centers even larger than Internet which is the jft layer set from Google it's an internal data set it's on a scale of Google photos and they'd use 1.3 million images from JFK to just to see how much the domain is much works out [Music]


Huli obtaining extra in domain unlegal data actually the auto domain on legal deer herd state performance and so you can see that it's not actually working as well so using more future days that actually works better for them and they also oblations on different schedules for the thresholding and you find out the exponential scale it's better for language but linear scale


works better for images that's that's something very empirical diversity constrains how did how do you control the diversity constraints for making sure that you have effective data augmentation they use a lot of hacks like minimizing the entropy and your decoding using the softmax temperature controlling and confidence based masking for your image image net case where they


found the outer distribution dataset unlabeled unlabeled I said like JT wrote the performance in that for they would take the most confident predictions from your label reader and you use that to filter the habibu data and tenday would find gains so domain relevance based data filtering is like a critical aspect of using own label data for improving performance of label data and


semi-square learnings we already saw most of the mathematical or intuitive foundation system is for learning make the assumption that the unlabeled data is coming from the same data distribution as to marginal corresponding to the joint distribution that you use for label data points but that's often not the case in practical scenarios because today said one label


unlabeled data is usually something coming from some other data set or some other data source and you want to make sure you still transfer knowledge therefore that that's a suspect to assumption to make but in practice it can be there's a workaround using various kinds of filtering techniques that this paper proposes so next look at mix-match which is another very


interesting the similar paper similar spirits so the key idea in mismatches you take on label data you perform a lot of different augmentations and now you have you run all of these different augmentations to the same classifier and you get the predictions and you average two predictions across these augmentations and you end up with a bunch of class probabilities you can


sharpen those class probabilities with soft max temperature controllers and once you get a sharpened distribution you you you you have an idea of what the classifier would have guessed for the unlabeled data point and now what are you going to do is you're gonna take this and use it as part of your label Diaz for every label data updated new SEM is worth learning so it's that simple only


thing is you you're making sure that your guess for unlabeled data comes from averaging or multiple augmentations and sharpening the distribution so there it's confident enough so it's called mix-match because it uses to mix up trick firstly for if you're not familiar with mix up the idea is you take your input X and your output Y and let's say you're trading a model to


predict Y from X you basically create convex combinations of two pairs of X for my one next to I do and create a new data file so for images it should be it would be something like for every pixel you take the weighted combination the pixel from the first image and the pixel from the second image and you which is average those two pixels and create a new image and similarly you would


average the corresponding and ground route labels in Korea target ground truth for your cross country with large its loss and this this technique is called mix-up and it's a documentation technique so mix-match basically the following it takes a bastion label data and imagine unlegal data and produces a new batch of process legal examples with this guessing technique


and mixing up so let's look at the English part of this pseudocode apply data argumentation to X B so basically our badgerlink data points X X B PB and unlabeled data points you be so you apply the data argumentation to X P and you apply Cayetano data augmentation to you B which is the only with a point and you compute the average predictions across all the augmentations of UB in


practice they just use cake with a 2 but it can be really large if you want it to be and you apply temperature sharpening which is soft max temperature to make sure that the average predictions are picky enough and after the peaks are obtained you can take the our Max and get the corresponding classifier corresponding data point for proxy data point for the classifier so you Ottoman


the legal examples with these guest labels and using these guesses and the original labels you create an Augmented mini batch and you shuffle will combine these mini batch data points using the mix up trick and once you apply mix up to the labeled and unlabeled data you can just train a regular service learning model that has domain consistency losses and the supervisors


cross will be lost treating this new batch as this mix match so mixed match is producing this processed label plus on people Taylor batch from two different data batches that are coming independently so the last function for mixed match is going to be the regular cross country loss in a classifier there's some kind of consistency laws for unlabeled data points that you normally use in


semi-supervised learning with some waiting constant and in practice it works really well earlier you saw in the realistic evaluation of learning algorithms that all these techniques were not really working well and this was concurrent work with UDA and you can clearly see how its really into the performance and see for at an SV hm so here are the numbers you can see that


it's not only working in the regime where you have 4,000 label examples but it works all the way well even when you have 250 labels mixed masses able to get all the way down to 10% error rate on CFR which is a which is very impressive so 250 images for class this means when two to three images of labels just means 25 images per class and you stood ability gets a classifier that gets


around 90% accuracy so that's it for mixed match an expansion UD ever like conquering work and I'll raise similar ideas different kind of implementations and UDA is probably more broader in terms of applications to NLP as well make sponsors well analyze internally ablated foresee for RNs future so let's look at the final paper in this agenda sub training with my student this is the


largest scale since Freud learning experiment conducted and machine learning so far these are also sliced from tango Wong was one of the authors in this paper so as I said there's the semi-square learning had a background just the dream is to make sure that unlabeled data improves the performance of supervised learning or label data even when you have a lot of labels and


that's what this paper tries to achieve so you you already remember how unfiltered aft was not able to give sufficient gain on imagenet for the UDA paper and they actually use clever filtering techniques noisy student is actually a larger scale version of that so the way it works is as follows you train a teacher model with long label data trying to really really good


classifier and asset classifier to predict an unlabeled here and you infer the cereal labels on illegal data and you train a student model with the combined data of the teacher teacher of the original data label data that you use for the teacher as well as the guest labels on our label data and you add noise to this process through data augmentation dropout and stochastic


depth which is another version of skip connections with stochastic and create a lot you know noisy student that's why it's called noisy student here you're gonna have a lot of data augmentation to this process and once you do that your student model is pretty good it's it's highly regular I stand also trained on a lot more data and therefore it can over train on anything but still it rains on


a lot of different proxy proxy later you generated from label data and which has already been filtered because it's only taking the confident predictions so now you can treat that student as a new teacher and repeat this process multiple times so you take the new student as a teacher if you you basically train it on label data you infer the pseudo you infer the zero


labels unlabeled data again with this new teacher and you create a new student and so on and repeat this process like multiple times and you get a really good for Adam so here are the experiment settings the architecture that you use is efficient at and the model noises they use is dropout of stochastic tap the input nicely uses Randolph which is a version of auto valve it's more


efficient and for zero labels they use the soft seal enables continuous values they don't actually use the one harden coatings and the label data said they uses a ridge net which is 1.3 million images and the unlabeled data said they use is jft which is 300 million images and they basically do iterative training where they take the biggest efficient at model possible and actually make it


wider for the next next scenarios so the original teacher could be b7 which is the widest and deepest efficient net that exists and the student model the trains next could be another bigger version of that model which is the L tomorrow the Dakar so in terms of results they actually had the state-of-the-art numbers for an image not already here two eighty eight


point four percent top one accuracy which is significantly better than any other model and the previous best was eighty six point four percent which is actually trained on 3.5 billion labeled images from Instagram so with just one point three million labeled images and 300 million unlabeled images you're actually able to surpass those numbers by two percentages significant


especially in those those regimes and they do that with one order of magnitude fewer labels and they actually do that with twice few as that smaller in terms of number of parameters because they use efficient mass which are much more efficient in terms of parameters and flops and rest net or rest next so the improvements are also exists without iterative training so it's not that they


actually need it right of training so even without iterative training they get significant improvement which is one iteration they can get minimum of one percent improvement for all the different model regimes b0 b2 b5 b7 improvement of 1% is pretty standard and which is pretty pretty nice because this means that the filtering mechanism actually works for finally they also show really good


robustness results on a mission net because of training on a lot of different data augmentations and model noises and water unlabeled data in addition to label data you would expect the resulting classifier to actually be good on a new server robustness benchmarks which happens to be the case so they actually beat the state of their army robustness benchmarks on each net a


which is a corrupted version of image net where usual classifiers fail over there and their model actually gets 83 point once and 83 point seven and top one which is unprecedented and they also do really well on emission at C and P where they get very very competitive very very very good numbers on the top one accuracy over their 77227 eighty-six percent which is significantly better than the


models that did not use this noisy student process so here's like examples where the model that they trained on these harder versions of imagenet ended up making the right predictions which is in black well the baseline models ended up making the wrong predictions so the baseline models are focusing on aspects that we use the car up and there's actually your


amazing car or like for instance they're actually looking at so the models able to capture the basketball in the photo on the bottom room where there's a man holding in basketball whereas the baseline models are not able to do that so so that shows this models very fine-grain in terms of its recognition abilities and here are a bigger example is this a dragonfly at


the right but but the baseline models of food and thinking it's a bullfrog and similarly parking meter was his vacuum swing was his mosquito net so this mod is actually very very good at details and they also have ablations for how much the noise matters in the in this process and it seems to matter significantly enough when he used all the different kinds of noises like


grappa stochastic depth and data augmentation you get the best possible numbers so in summary we looked at semi-square is learning and it's a practically important problem in the industry for two different scenarios one is when you have a lot of label data and a lot more on label data like for instance image ladder and j ft and you're trying to improve the performance


of image net the other is when you have very little label data and you have plenty of unequal data which is usually the case in medicine or finance or something like that so the promise of semi-supervised learning is always existed for the second second scenario but there have been very good results in the the last few months or like last year or so in both these scenarios which is the


noisiest student model really helping in the scenario where you have a lot of legal data but you have a lot more and label data but then unsupervised data augmentation or mix-match is really very good at the low Reiter regime where you have unlabeled data but very little label data and you're able to do really well so this means that when you have the scenario are you using unlabeled data to


improve the performance of supervised learning systems self supervised learning is not necessarily the only option semi-supervised learning is asked lucrative or probably even better because its ability to improve the performance even in the high data regimes and make it possible for building emotional classifiers that have an unprecedented top on accuracies like


noisy student that's it for the lecture thank you very much all right so let's get started welcome back to the lecture nine I'm actually not sure Oh keep unsupervised learning something lecture something and so we will have a two part lectures today the first part we would look at something called unsupervised distribution alignment it also goes by a lot of other names and


then the second part would be a guest lecture by Professor Alyosha to talk about some of I guess the works from his lab so any logistics questions before we dive into your lecture mouse tone can we have late days for mouse tones all right there you go which no one is working on all right so so let's get started so in this lecture we will look at unsupervised distribution alignment so


what does that even mean so let's remove the part unsupervised let's just first look at a distribution alignment problem so a lot of problems in image to image translation take this form so let's say I want to go from semantic mask to RGB images then this is a distribution alignment problem because we can think of us having a distribution over mask here and then we also have a


distribution of like just regular images here and then they would like co-occur with certain join probability distribution right it's mostly for one image there is only one correct semantic mask but for one mask there could be many corresponding images and the goal would be like how can you align these two in such a way that when I give you an image on the right you can generate


the mask or the other way if you want to generate more training data and then the more image problems that takes this form let's say I want you know an image what does it look like in the day time and what does it look like in night time and then again we can think of it as having a distribution of images of day time and then also a distribution of images in


night time and then you want to align them in certain way so you say why is this helpful like one way that this could be helpful is say if we want to train at home as vehicles to drive safely at night but then it's harder to collect data at night so is there a way for us to collect corresponding images during daytime and then find a way to find their nighttime counterpart that would


be useful if we can do this and then there's somehow a lot of other problems also fall under this formulation like black and white images to color images and for basically everything that we have seen they are relatively tractable because like I can totally just take a color image and then convert it into black and white and that gives me a lot of pairs that I can train


on and similarly to the semantic mask and Street View RGB images as well as daytime and nighttime for a lot of those you can actually find natural pairs so these are some of the problems distribution alignment problems in image space and this kind of distribution alignment problems also happen in the attacks analog of that and most straightforward example is really just


machine translation how do you translate a sentence or a paragraph from one language to another that again you can think of as a distribution alignment problem you can think of there was a distribution of English text and then there's a distribution of Chinese text and then the question here is how do you align these two things together so when this kind of distribution alignment


problem when is supervised then they are relatively easy so when in the in the case of form an image goes to semantic mask it's basically just a Mantic segmentation problem when it's like other image to image translation there's this text effects work that's done here at Berkeley and then for text to text domain alignment when you have the supervised pairs it's just machine


translation and when you want to go from image to text again when you have supervised pairs they are just like captioning tasks and a lot of things so in the end of it it really just borrowed down to feeding a certain conditional distribution like your given image B what the correct mask a and the you have this luxuries when you have kind of a and B pairs that co-occur in the wheel world


either through your annotation effort or by taking an image at a time and it take the same image at nighttime as long as you can gather this kind of pairs is somewhat trivial at least from a formulation perspective it's really just fitting a conditional distribution and we have talked about all sorts of ways to feed distributions in this class can auto regressive model or whatever and


but the question becomes interesting if this kind of pair of data what if they are expensive to obtain or they just don't exist then we are basically going out of the range of this supervised distribution alignment problem like you have one distribution you have another but you don't have any pair of data then like can you still do this or even do it at all so I'm taking some examples from


a paper called cycle game like what if you want to turn the painting into a photograph or turn a photograph into a painting the second one might be more tractable because like you could possibly say I take a picture and then I hire someone to paint it for me but if I want to do it in a very specific style by a specific artist and you really couldn't do that so in a sense the


natural pairs don't even exist in the real world similarly like if you want you for whatever reason if we want to turn a zebra into a horse or turn a horse into a zebra then it would be very difficult to force a zebra in a horse to take up exactly the same pose and take a picture of them having the exact correspondence so these are the kind of pair of data that would not exist in the