for the node here in the middle it has different neighbors than the node here but the weights of


Parameter Prediction for Unseen Deep Architectures (w/ First Author Boris Knyazev)

that function that takes messages and integrates them that's the same for all the nodes and that's


why graph neural networks very often surprisingly can have very little parameters and can achieve a


lot of power yeah and then so so all these these steps right here i think you've implemented them


as a recurrent neural network that simply passes on so we would do multiple rounds of these steps


and then right the nodes would in multiple steps compute uh updates and updates and up so even you


could implement this as separate functions you could say time step one is one function time step


two is a function time step three is another function but you've even chosen or i guess


previous work as well chosen to implement this as a recurrent network so not only are all the functions


across the nodes the same but even across the time steps they are essentially the same because


it's a recurrent neural network so surprisingly little parameters and the advantage of it is i


can pass in any graph right the graphs they don't have to be the same it can be totally different


and i can apply the same function because it's essentially vectorized across the whole graph


which is going to play into your your batching methodology as well i guess once we come to that


but my question was essentially you've so you you do this first iteration right here so the first


iteration is just like a regular graph neural network and then the second iteration sort of


your your improved version this ghn2 yeah it has a bunch of it has a bunch of tricks


tricks tricks right here oh no that's not even that's not even that i think that's already in


the in the previous version is that your message passing algorithm if i understand correctly isn't


as straightforward as here it's not just i get from all my neighbors but you have two different


message passing algorithms one mimics the forward pass through the neural network and one mimics


the backward pass so in one round i would only get messages from my dependents and in one round


i would get the messages from my like upstream dependees yeah exactly so that was part of


previous work as well or yeah yeah they developed this specific uh like version of gated graph


neural network that mimics this behavior of forward and backward propagation and yeah what


what we found though that just one round of propagation is enough so we only do it once


forward and once backward yeah we don't okay you can do it multiple times but we found it's just


a waste uh yeah resources so it doesn't improve accuracy for some reason so essentially training


your hyper network exactly mirrors training a real network in that you do a forward prop


and the backward prop but yeah what you do is you simply back propagate that through the weights


to the actual graph neural network weights yeah yeah so in that sense yeah it mimics how the


networks are trained like yeah and so now i guess what you get out then is sorry to come back to


this again but every node sort of keeps updating its hidden state as this progresses and at the end


you have a hidden state for each node which is kind of the final hidden state and then you put


that into into a decoder ish thing this thing right here so there how do you how do you deal


with the fact that you know sometimes a convolution has three by three by five parameters and sometimes


it has you know yeah seven by seven by ten parameters and sometimes it's an attention


network that needs query key and value how does a single architecture produce um different number


you can reshape right but even but especially different number of parameters yeah so that's


actually the tricky part and we we didn't we did something very naive actually and there is a lot


of room for improvement in that part and what we did is we apply this styling strategy and we


defined the first we defined the tensor of like a fixed what we call maximum shape and if we need


to predict a larger tensor we tile it multiple times across channel dimensions as needed so


essentially the tiling means right copying the same tensor multiple times to make it yeah full


shape and if we tile too much then we slice it and yeah and we slice along the all four dimensions


like height widths and channel dimensions so it's quite naive and limits the expressive capacity


of the predicted parameters but that's the yeah that's the only method we would make work like


in an efficient way so far so there's yeah i guess there's room for some sort of weight weight


up sampling some sort some sort of technique where you don't have to know the number of outputs


before you predict those outputs right yeah yeah like some more yeah or recurrent networks like


that would predict parameters as as much as you need like sort of like in this yeah one at a time


yeah yeah or or something like do you know these these um nerf or or siren these implicit neural


networks so essentially you give the x and y coordinate of a picture as an input and then


it would simply predict that particular pixel right you could do something here where you could


parameterize maybe a weight matrix from zero to one and you could just say you know or not even


from zero to one but you could you could parameterize it somehow and you could just ask


the network give me the output at this location and at this location yeah yeah that's an interesting


idea actually yeah but okay i guess the autoregressive part might be more might be more


useful because you want to somehow coordinate the weights with the other weights you've already


produced yeah so yeah that's also a tricky part so yeah so you you make some improvements right


right here that you also highlight which is okay the differentiable normalization which


um you want to comment on on that briefly what what's new here uh so in the original graph hyper


networks uh they predict parameters and they use them they use those predicted parameters


parameters directly in the forward path and during training we found that they start to explode like


similar to i guess other kinds of training and so yeah predicted parameters really become huge


and what we found useful is to normalize them so such that this normalization is similar to the


initialization method that people typically use yeah so that yeah the scale of the predicted


weights like the range of values is approximately yeah the same as in the randomly initialized


networks so there's like yeah i mean this yeah yeah this this here this this looks a lot like


sort of the the formulas for take incoming and outgoing number of of of unit of units and


normalized by that you even use here the fan in and fan outs and i think these are these are terms


people recognize from initializations which i guess yeah this it makes it makes sense at least


as sort of intuitively and then this is what i this is here what i found um one of the interesting


parts is these virtual edges that you introduced so we said that these um arc these graphs that


you build they mirror the the neural networks essentially they represent the architectures


of the neural networks specifically the computation graphs and you have some examples right here now


they're a bit small but this could be this is i guess something like a convolutional neural network


with a residual connection is the left one right because essentially the the blue here are conf


modules and you can see these conf modules like here is one here is one and then you can also see


there's kind of different paths and then there's always normalizations and so on so these are


conv nets with residual connections as a computational as a computational graph right here


right yeah something like and so that you you've somehow found that that is it's not enough to build


the graph like this you can you can do better by introducing more connections between things how


did you yeah get about this right so the problem is that the the node propagation step that we


talked about like just before has the problem of propagating through the long sequence of nodes


yeah so the final nodes which will be usually a classification layer will have little information


about features in the first yeah like in the first layers and that's a problem yeah because


essentially graph hyper network doesn't know much about the overall global graph structure


when making predictions so these virtual connections improve like global context and


how do you decide so you this it looks something here is a an example you gave from a of kind of


a like an illustration illustratory graph the the computational graph in in dark and the virtual


edges in green how do you decide which things to connect so we use shortest path distance between


nodes and we scale the edge this virtual edge weight according to the inverse of this shortest


path distance so is at the end is everything connected to everything we have some like cutoff


that we don't don't connect too far nodes okay but yeah but so that you're saying the parameters


of the virtual edges here they're shared with the parameters of or do they have their own parameters


own parameters so there is an equation for uh yeah there is a mlp sp yeah so that's uh like


a separate network so to avoid the confusion between real edges and uh virtual edges yeah


i mean i guess that the edges they don't have weights themselves but you do you do make


when you propagate through the graph neural network through these functions you do make it


a difference between the real edges and the in edges you introduced right so in a virtual case


instead of just averaging features of neighbors it's a weighted weighted average where the weight


is okay yeah coming from the shortest path distance cool and i guess i i think i find it a


bit funny right that you you uh in in the hyper network essentially you again run into the problem


well what if our network is too deep which is essentially the same problem that


resnets had yeah before res and then your your solution is also hey let's introduce like residual


connections between things so that information can flow further it's it's kind of funny to see that


sort of the the the problems repeat one level up and then of course the it's a it's a different


thing than a residual edge in a in a network because this isn't the network this is the input


to the network but it's uh it's kind of analogous right to a residual connection yeah yeah that's


true and that's uh that was our motivation basically yeah and then the last thing is this


meta batching um which do you want to maybe talk about this a little i understood this as you


essentially what what's the difference between this and and mini batching uh so mini batching uh


well uh usually we refer to a batch of images right so we for each training iteration we sample


a mini batch say of 64 images but uh in the baseline ghm they sample a single architecture


for each training duration so it's a single architecture and 64 images uh yeah now the


gradient becomes very noisy because for each architecture the the gradients are quite different


yes and to improve that uh like this uh stability of convergence uh we sample a batch of


architectures of architectures and a batch of images yeah yeah or okay so and then you do do


is it do you do x1 architecture 1 x2 architecture 2 or do you sort of build x1 if you build up a


matrix and then pass each image through each of the architectures in in the batch uh no we uh just


do the first option yeah so you just sample a bunch of i guess the analogous would be if i train


a regular neural network uh the analogy would be i you know if i just always sample from one class


because for example in image net the data set is structured into these folders right and every


folder has one class yeah if i want to make my life easy i just go you know i do every single


class if i want to make my life easy i just go you know i do ls of the data set and then i just go


through it and then i always sample from one class that would give me a very bad gradient because in


the same batch i always have examples from the from the same class and here you're saying the


problem was that people in the same batch they always had the exact same architecture yeah and


then you get a much much better estimate i guess of the gradient if these are sampled at random


it makes makes a lot of sense cool and yeah so that's that's where we get to the to the experiment


i think the the experiments the the main things we've already sort of taken away in that this is


um on c410 right here uh you have the test set um sort of you always split into average and and max


uh which is sort of performance is that that's test set performance right it's test images and


test architectures and the max images and test architectures and the max is the max is over what


exactly over architecture so it's like the best what what what's the performance of the best


architecture yeah okay i i guess that's a fair a fair metric to look at because you know each


architecture if you were to train it doesn't have the same performance at the end right so with with


the max you can expect that um the max uh would be an architecture that if you trained it had sort


of the state-of-the-art performance of at least the networks you consider the architectures you


consider so maybe for c410 that might be 96 ish percent 97 yeah and yeah so you get some yeah


sorry yeah we compare it to sgd below right and you have a similar sort of phenomena yeah but you


have average performance and you have like the best performance that is a bit higher okay so


that's yeah 90 i see that 90 93 450 epochs of i guess the state-of-the-art things they are reached


with a bunch more tricks like uh yeah like augmentation more augmentation yeah yeah i see


yeah okay but i mean there there is a considerable gap but it's it's still pretty cool to see right


that you get especially um also the improvements that you make in within this parameter prediction


regime uh with these new models are quite considerable and so if i consider for example


from this or this which are 60 to 77 which is sort of like 80 and then that's like almost half


of the error right that you make compared to state-of-the-art that's pretty pretty good and


even on the out of distribution it seems the effect is even more um drastic right so do you


do you have any idea in to why why on a on the out of distribution set your performance it drops but


it doesn't drop too much whereas these other methods performances they drop much more right so


i think uh like those three tricks uh play a different role in each uh part of distribution


split for example in the wide case i think what helps and we have those ablations in the appendix


what helps is parameter normalization because when you have like a lot of weights then it's


important that they are appropriate scale and in case of for deeper architectures i guess


what's important is to capture this global context because yeah network is yeah has a lot of nodes


and since similar like for other splits yeah so i well for other splits maybe it's less intuitive


what exactly like what single component makes it work but i guess some interplay between like


all those three tricks help nice yeah and so and then at the end you say um so you have a you have


a lot of ablations which is is really cool to to see and open source code and and everything uh


that that is i think a lot very appreciated especially on a more exotic task like this one


you also are able to predict some properties like you know what's the accuracy of the network and


so on where you make sure that your network really learns some kind of intrinsic properties right of


the of the networks you're predicting so the network's not only able to predict the weights


but it's also able to say you know what's the what's going to be the inference speed of the


network what's going to be the approximate accuracy on that is going to have and so on would


really make sure or it's at least a bit of a a bit of of a confirmation that you're doing something


meaningful instead of just copying over weights so this would be to counter anyone that says well


you're just kind of copying over from the training set and then the last thing is you then also


experiment fine tuning fine tuning the predicted parameters now obviously in in this regime we're


kind of like meta learning right we are learning a good initialization and a good


learning a good initialization and from that initialization i can then fine tune but then


so my question is how much time does that save me if i use your network to predict the initial


parameters of a resnet 50 how much less time do i have to invest into fine tuning it


as compared to compared to training it from the beginning so we actually provide the


speeds in the table you can see the difference of like time you you need so it's not as much as


you want maybe so as you see we like predict parameters is like in less than a second but


you can achieve the same result by pre-training on image net for like half an hour or one hour yeah


sometimes more like on transformers yeah it takes more time to achieve the same performance so yeah


would you say it's it's uh it's kind of would you say your work is maybe shows that something like


this can be done or do you do you think we'll get to a place where we can make so much savings


because we only have to fine tune for like a tiny bit that people say yes i'm really going to


use this instead of training from the beginning or do you think it might be mostly an academic


exercise no i think we can arrive at that if you can see if we pre-trained for just five epochs


on the image net then we almost get the same performance as we train for 100 epochs or like


200 epochs yeah like slightly worse and my hope is that if we are able to predict parameters that


are similar to five epochs then we are done with yeah yeah okay it's difficult i'm not saying but


of course easy but i'm saying that the each we don't need to predict the performance of a 100


epoch network yeah yeah i see yeah i mean it makes it makes sense it would save like it saves like a


bunch of time and and resources and everything and potentially allows us to investigate new and


better architectures much faster rather than and especially if we scale to larger models like if


this holds also for gpt style models especially if it generalizes to way less than the size of


to way larger architectures right it might even be that we're at a point where we we get such a


large model that it's prohibitive to train it from the beginning but we might be able to predict


and then fine tune so so technically like implementation wise our using our model we


can't predict the model we can't predict parameters we sorry we can't predict parameters for a network


with trillion parameters yeah sure because because we use this styling right so we can


predict trillion brands but of course it will be very bad it may be difficult to find you as well


and so the last thing i want to get into is sort of the reception now you have said previously to


me that it has been kind of maybe received a bit um out of context or you know a bit oversold what


do you what do you mean by this i think uh maybe people got an impression that we can predict


parameters for a new task like for unseen tasks which is not true yeah we yeah and even though


i mentioned that we only make a single like a small step towards replacing sgd i think people


misread it and understood it like oh we replay we are ready to replace no we are not there yet it's


far far far away from that the some you can see the video thumbnail going something like um sgd


not needed anymore predict parameters for any neural network we are done that that's the title


i was trying to convince my co-authors i mean it's a good vision it's a good it's a nice vision to


have right um but yeah it's important to point out you do generalize very well to unseen architectures


but it's always within the same task now i guess the hope for the future maybe you're already


working on this or or not uh would also be to investigate generalization across data sets maybe


you can imagine a situation where you have your system on image net but then you um so you've


trained it for image net and then you maybe give it a little bit of the data set of a new task and


it's it's able to adapt really quickly to that or something like this right so it would already be


quite useful i think right and there are already works that actually do that like in the meta


learning sense but yeah normally they don't generalize well across architecture so they


generalize well across tasks that's their focus but not across architecture so there should be a


way to combine those two yeah sounds exciting all right i think um this is a is a really


neat overview over the paper um we'll end it here boris thank you so much for for coming here and


yeah good good luck on your future research yeah thank you for inviting me it was very fun to go


Feedback Transformers: Addressing Some Limitations of Transformers with Feedback Memory (Explained)

Hi there, today we're looking at addressing some limitations of transformers with feedback


memory, also known as feedback transformers by Angela Phan, Thibaut Lavril, Edouard Grave,


Armand Joullin and Sanbayar Sogbatar of Facebook AI Research and Loria.


On a high level, this paper, as it says in the title, it addresses some limitations of


transformers specifically of decoding transformers that are trained with causal masking.


And the problem is that these transformers, they don't make use of all of the information


they compute, even though they technically could make use of that information, but they


sacrifice it in order to train in parallel.


And we'll see what that means.


To alleviate this, this paper introduces these feedback memories, and thereby they arrive


at a model called the feedback transformer that takes into account all of the available


information.


Now, this new model, it can't train as fast because it can't be trained in parallel as


the old model.


However, you can build models with this technique that are significantly more shallow, so less


layers, and also the models will remember things for longer.


And this is especially helpful when multiple steps of reasoning are required.


And it has to be done over kind of a longer sequence.


So we're going to see some tasks from reinforcement learning and kind of other sequence tasks,


where these feedback memories really make a difference.


In any case, if you like content like this, don't hesitate to share it out and tell all


your friends about it.


That would be awesome.


All right, so what's, what's the deal with transformers?


What are they doing wrong?


As I already said, we specifically are in the case of this sort of decoder only transformer


right here.


These graphics here, they are a bit confusing on first sight, I've I found I had to dig


into the paper and read the paper was not necessarily clear from these diagrams.


So I'm going to try to sort of build up what's wrong.


So what we're trying to do is we're trying to do something like language modeling.


Now it's not only language modeling, but in any case, we have a sequence of inputs, which


I'm just going to represent as circles.


And what we want to do is we want to predict whatever the next the next circle is.


So these could be steps actions to be performed in a reinforcement learning world.


These could be words of a sentence right up to here, and then you are supposed to predict


the next word.


That's called the language model.


Many things are fall into this category.


So for example, GPT-3 is trained in exactly this way.


In order to do this, you have to have a model that somehow takes all of these things and


somehow builds a representation that then outputs this thing right here.


And that's, you know, good, good in itself.


How did we usually do it?


So the first attempts at this, of course, were sort of recurrent neural networks.


And I'm gonna go over them here, because they're going to be important, even though you probably


already know what they are.


So for actually, for all of the models, we're going to look at today, what they do is they


build representations of this input data.


So I'm going to represent this with little boxes.


What they do is they build these latent representations right here.


So the data in a recurrent neural network flows like this.


The inputs go up each time into a hidden representation.


This is a neural network layer that does this.


And then the hidden representations are transformed into each other.


So the first input is input here, then it is sort of forward propagated to the next


time step at which point the next input is consumed.


And then it is merged with the previous hidden state and that is propagated forward into


the next time step, and so on.


At the end, you take this representation and you output whatever the next label is.


And I'm going to purposefully draw this now up here to say so the data flow is something


like this.


There has been improved versions of RNNs that do multiple layers of this.


So the next layer would be here.


And this is a multi layer RNN.


So if you like this could be an LSTM, this could be a plain RNN and so on.


What they would do is they would do the same thing here.


But then each hidden representation goes into the next hidden representation like this.


And these hidden representations, they are also connected with a recurrent connection


over time, like this building sort of like a grid.


Right.


So the way you have to think about and then of course, here in this first, so the output


of the last top right one goes into predicting the next token or action or whatnot.


Because the top right one, as you can maybe see all the information flows up and to the


right in this in this case right here.


What this is what an RNN does.


Now you can see this is very well connected information.


However, if you if you think about this in terms of information flow, if for example,


this thing right here, and this thing right here, need to communicate somehow imagine


they need to communicate to solve a task.


So what could this be?


This could be for example, a name, Frank.


And this could be an like an article referring to Frank, like he, okay.


And you know, it's it's out of order so but in order to know who he is, you somehow need


to these two tokens somehow need to communicate I hope that's sort of clear.


Now they here can communicate by means of transform transferring information, you know,


from kind of step to step like over here, maybe like this, right.


And then in this hidden representation, the information can be combined.


But you can see the number of steps that the information has to travel is fairly large,


it can also be combined here if the information flows first up one layer, and then over, and


so on.


This is the drawback of recurrent neural networks.


Very often the information has to flow along many steps of computation in order to be combined


with something else.


A different approach is a transformer.


So a transformer handles sequences in a very different, not a very different way, but in


in a different enough way.


So a what a transformer does is whenever it builds the representation for the next layer,


for example, this representation right here, a transformer will aggregate all of the information


from the previous layer like this.


So every one of these representations right here, also this one, it will aggregate all


the information from the previous layer, let me draw this in blue right here.


So all the information.


Now that's a lot better, because now every node can communicate with every other node


in a matter of a single computation step, and not just and not like as many computation


steps as the two nodes are apart.


Now, you need to help the transformers a bit with positional encodings, but in essence,


this is a more powerful way of interpreting sequences.


And you can do this in many, in many layers.


So the next layer will have access to even more in like, so this representation right


here, it will draw information from all of the previous representations right here.


And this is by means of an attention mechanism.


And if you don't know what an attention mechanism is, I've watched my video on attention is


all you need.


I explain how this works there.


But suffice to say it, the information is aggregated over the whole sequence layer by


layer.


There is a there is a kind of a fundamental reason why this is important, namely, if we


want to do very complex computations.


And by complex computations, you can maybe look at an example right here, where they


have examples of such a complex computation.


In the appendix here, they give this example of code interpretations.


There it is.


So what they give the program or the model to do is this piece of text right here.


And the the the model is simply to go over this code and decide what the output is.


So you can see right here, it has print statements.


And the model needs to decide what you know what the output of the entire program is,


you can see right here, it has if statements, so it has conditional statements as variables


that are set, but also things like in decrement, increment these variables, then print them,


then update them again, have some conditions on the variables, right.


So there is a condition between two variables, z and x.


So this is quite complex for a model to solve.


And if you were to let an RNN do this task, because the plain RNN, it has, you know, it


has these inputs, and it has one vector, that's the hidden state, everything needs to be saved


in this space of this one vector.


And the longer it goes, of course, the more noise you introduce and so on.


So if stuff is very far apart, like here, in many cases, you need to keep track of all


the states of these variables, RNN tend to do sort of worse the longer the task, transformers,


not so much transformers can look up.


So a transformer that ingests this token right here, can look to any other token in a single


step.


However, in this task right here, also transformers get at their limits.


Because in order what I said, in order to do complex computation, you need multiple


layers, a single transformer layer, as a matter of fact, a single neural network layer can


only do linear operations, right, it has a non linearity at the end, but you know, everything's


connected with everything in a neural network layer right here.


So these are neurons, these are neurons.


And this here is a giant weight matrix, W, something like this, this can also be the


attention matrix right here.


In every neural network, there is a linear operation at the heart of the neural network


layer.


And a linear operation can only do so much.


Notably, it can't solve things like the XOR problem.


And it can't do if conditions and it can't do keeping track and updating variables.


You know, you cannot.


Let's let's break this down.


Let's say we have this text, x equals one, x plus plus, x, if, let's say if x greater


than three, then x minus minus something like this.


A transformer one layer will be able to look at all of these at the same time, but it will


not be able to look at them in sequence, right?


It can only look at them at the same time.


But it cannot say it cannot have a dependence between them.


It cannot say, Oh, because here I incremented, this is greater than three.


And then this happened, actually, it's not greater than three, but and then this didn't


happen.


It cannot do that reasoning can simply individually look at each of these lines, and then somehow


integrate them in a linear fashion.


So it could integrate the plus plus as simply saying whatever x is, I need one more.


And then it could integrate this and saying, well, x is one, and then the two together


would maybe give you the result that x is two, but this if condition and so on, it cannot


do that in one layer for that you need multiple layers with nonlinearities.


So by having multiple layers, you could, a transformer could technically do things like


have four nodes right here.


And then these the first node might, you know, combine these two, and that sort of represents


x equals two now, right.


And then this node right here could represent this if condition x greater than three, and


it could point, I'm just imagining I have no it could point to this node for fulfilling


the condition, right.


And then this node here could point to x minus minus, right.


Now I have a simpler program, you see, I've done one layer, I have a simpler program,


simply by linearly combining things, then in the next layer, I could combine these two


things, and this one tells me x equals two, and this one is x greater than three, which


I can evaluate now since these two and then that might result in a weight of zero, right,


because x is in fact not greater than three.


And I could save sorry, maybe here, I could save that weight of zero right here.


So this node is now representing zero, this node is still representing x equals two.


And then this node, the pointer here, this pointer makes this, yeah, evaluate maybe to


minus one, and then somehow point to and then this node, I'm just making stuff up here,


this node could somehow connect these two, right.


This node could be representative of the connection between these two and then in the next layer,


linearly, I can do my aggregation, it's it's then this and this get combined.


And then this is zero, because because it's negative one times zero, and plus the two


right here, and then I get my final x equals two, I hope that somehow it is not like it


is not how it happens, but you can see that if you're only if your only method is linearly


combining things layer by layer, you have to go quite a convolved way in order to achieve


kind of multi step reasoning things.


And you can only do this by having non linearities involved.


And one step of reasoning is usually kind of one layer with a non linearity.


And thereby, the number of steps of reasoning here is limited by the depth of the transformer.


If this is a transformer, the number of you know, kind of reasoning steps, incrementing,


decrementing a variable is directly linked to how many steps you do this.


So that is, that is a drawback.


And that drawback can be solved with these these memory things.


So let's look at how a decoding only transformer specifically is trained.


So again, here, we said the transformer can include things from from anywhere.


But what usually people do is they they do this causal masking, because we want to predict


every time we want to predict the next thing, right.


So here, we, we have a sentence, right?


And then we make samples of it, we say, okay, maybe if I input those two, I want to predict


this one.


But if I input those three, I want to predict this one.


And if I input those four, I want to predict this one.


I can make all of this in one, if I set my information flow, like this.


So I only let the tokens have access to whatever is behind them.


That are these these decoding only transformers, let me Okay, so if you think of, of this token


right here, we just imagine that in order to predict this token, we only have access


to what came before it, like if you write a book, and you write the next word, you've


only written the words in front of it.


So we just say the representation of here only has can draw, it cannot draw information


from over here, that's forbidden.


We let it only draw information from arrow, it's its own node, sometimes like it depends


on how it's represented, but only its own node and to the left of it.


The same goes for for this one.


So like that, like that, and this one here, and then this one here, it can draw information


from here from here from here.


It can draw information.


And this one can draw information from here from here from here.


So still, you see the property of long range information is still here, by means of connections


like this one, or this one.


However, we simply cannot draw any information from the right.


All right.


And also you see how this information flows.


And the difference between a recurrent network and this one is in these lateral connections


here.


Do I have another here, there is no connection here, there is no connection in a recurrent


network, there is a connection within a layer, you see that here, there is none.


But instead, there are these long range connections from the last layers.


And even worse, what's missing in both of them is connections, such as the following.


Do I have another color?


Black, okay.


This connection.


So if you look at this thing right here, it can draw from here, it can draw from here,


from here.


And if we have the recurrent connection, we can maybe also say can draw from these ones.


But technically, it should also be able to draw from this one, right?


Because by the time I reach to the prediction of the next node from here, I can certainly


compute this representation up here, right?


Like nothing, nothing stops me from building in a connection like, like this one.


And that's exactly what these memory transformers criticize.


Among these old style transformers, they only go feet forward, meaning they only go up the


layers.


And they don't even have lateral connections like recurrent networks, they only have forward


connections in the layers.


And that limits the amount of steps you can do in computation.


In contrast with the memory transformers, information can flow.


I'm going to draw, maybe it knew because let's actually look at their diagram.


So you can see right here, maybe it's not as confusing anymore.


Actually it's still confusing because we need to introduce this memory.


Information can flow all the way up and then down again.


So I'm just going to draw two layers right here.


So information can flow like this.


So the first step is the same, right?


We simply, we have nothing here to look at.


There is no, so we can only draw information from the left.


So that's all we can do.


The second step, so let's say we've computed the first step, we've actually output a token


like this one, and we now continue because we are auto regressive, we always input whatever


we output.


What we now can do is we can do this and this, right?


That's what this representation can draw from in a normal transformer.


But now we could technically also draw information from here because we've already computed these


things in the last step.


The reason why transformers usually don't do this is now you cannot parallelize training


in a setting like we've seen before, oh, wait, I've destroyed it.


But in a setting like we've seen before, you can actually train this whole sequence in


parallel, like all of the samples, if I have five tokens, I can make five samples out of


that and train that in parallel.


It's no longer possible right here, because if I train it in parallel, I do it in the


feed forward fashion.


However, here, in order to have access to this information, I have already had to compute


the full forward pass for that first sample.


Okay, so that's the drawback right here.


However, it might be valuable to have that highest layer information, especially since


that was the one that predicted the next token.


Okay, so probably a lot of information about that token is going to be in that highest


level information, whereas with the previous transformer, we could only draw information


from down here.


So we have access to higher layers of representations of the past.


And that means the information can actually flow all the way to the end, like so, all


the way to the end, and then back again, all the way to the end, back again, all the way


to the end.


And every time we have access to the highest layers of representation.


So if we look at this thing, we can, we could actually draw from all of the representations


we've previously computed.


So we could look at, hey, what's what was this token?


That's what a normal transformer could look at as well.


But we could also look at what did this first layer at the sorry, the first token in the


last layer compute, we can we can look at that is probably very informative.


So now you can see that the reasoning depth is sort of unbounded.


Because here, even though I have maybe five tokens right here, I can only do two steps


of reasoning across it.


I can only, you know, one step of reasoning is one layer.


So I can like save learn to save a variable here, and then learn to increment it right


here.


And I can't do more.


But here, I can learn a function for saving a variable, incrementing it, and so on, and


do that all of this processing with the variable.


And then the next thing comes around, you know, maybe that's incrementing, I can look


at the end right here.


And that may be the representation for the saved variable.


And then I can increment it and store it in this representation.


And then the next layer can come around, and it can look at this representation right here


and say, Oh, you've incremented it after you saved it, right.


So this is the current state.


And then it can go ahead and modulate it as well.


So maybe you can do an if condition.


And the next thing can look at that if condition can look at the value of the variable and


through the layers here.


So it has it has two layers of compute just to implement that if condition on the current


value of the variable, whereas the old transformer would sort of have to start from scratch,


you can maybe think of it like this, the old transformer always has to start from scratch


doing the okay, here's how the variable starts, here's where it's incremented, here, I'm going


to do an if condition, whereas this transformer, it does the computation, and then it can sort


of store information in these higher layer representations.


And all the next steps can look at it.


Now, if you look at the light blue thing, that's a lot of arrows.


This amount of arrows, this amount of attention connection would pretty much explode any system.


And that's why this paper simplifies that.


And here is where the trade off another trade off comes in.


So you can't train it as fast.


That's number one.


And number two is they say, Well, we're not going to let you look at all of these hidden


representations, right?


Every every square here is a hidden representation.


What we're going to do is for each token, after the information has passed, and we've


computed these hidden representations, we're going to sort of mash them together.


So we're going to take the two and maybe also the token embedding.


And we're going to build one so called like a memory representation of that token.


So all of this is now incorporated in this memory representation.


And the next layer, what it can do is instead of looking at the individual representations


right here, instead of looking at them, all of them can instead look at this, sorry, the


other way around, all of them can instead look at this memory representation, that first


of all, it saves space, it saves memory.


And second of all, you can also share the key and value computation of the attention


mechanism, whereas only the query representation goes here with the with the different layers.


So that's queries number two, that's queries number one.


Okay, so you can share that.


And then once you have, once you have those, you also build a memory from the second token.


And then the third token, it can look at both the memory of the second token and the memory


of the first token.


So you still have that transformer, long range information pass.


But now you have sort of a summary, these memory blocks right here, within each layer.


And that's exactly what we see in the diagram right here, and that's already the model.


So the switch transformer is a transformer that forward propagates, not in parallel,


but token by token, it forward propagates, then it builds this memory.


And then all the next tokens, they can, instead of paying attention to two things in their


own layer, like so, they can now pay attention to previous memories.


Okay.


Again, the arrow should go in this direction.


So that is a feedback transformer, it retains the long range information flow, but the information


doesn't flow from same layer representations, the information actually flows from memory.


And the memory is a weighted sum of all of the representations of a given token that


includes higher layers, like this one.


So information can flow from higher layers in the earlier in the sequence, to lower layers


to later in the sequence.


And that allows each sequence element to do as many reasoning steps as there are depth


in as there are a number of layers, whereas in a normal transformer, the entire sequence


only had that many reasoning steps.


So here, reasoning steps are per token, whereas previously, reasoning steps were per sequence.


And that's, of course, more powerful.


Yep, that is pretty much the model.


Now, okay, I have I have one thing right here.


Um, one thing to sort of remark, namely, you know, they consider the the RNN right here


on the right, like how how it's different from the RNN.


You can clearly see that the RNN the information needs to travel many, many steps to arrive


somewhere that has been the drawback of the RNN.


But people have sort of solve this in RNNs using, well, you guessed it, attention.


In fact, attention mechanisms were first introduced to help RNNs overcome this problem.


And RNN with an attention mechanism would look like something you're very familiar to.


So here, we build these hidden, let's just consider a one layer RNN for now, we build


these hidden representations, okay.


And


again, it goes like this, and then there are these recurrent connections right here, that's


an RNN.


But but if we help this with an attention mechanism, what we do is we say whenever you


compute, for example, this representation, what you're allowed to do is you're allowed


to also not only have, you know, this connection, you're allowed to look back at the previous


hidden representations and aggregate information using an attention mechanism.


So that's where attention mechanism actually is sort of come from in this domain.


if I look at this switch transformer model, I very much just see a bit of an elaborate


RNN.


So if you just tilt this, if you tilt this graphic right here, you will see, and we can


do this together.


So


yes, if you if you look at this, and if you tilt the graphic, so I'm going to draw again


three things.


Let's do it down here.


I'm going to draw three things.


But instead of going up with the squares, I'm simply going next to each other.


Here three squares for this, three squares for this, and three squares for this, right,


representing the three layers.


So before these here, they were in in this direction, they were up, but now I've tilted


them to the right.


Okay.


And with the with the way the memory is built, so the information flows like this, and like


this, and like this, right, and here, like this, like this, like this, we'll fill in


the other connections shortly.


The memory is built from those three.


So like this,


from those three, a memory is built like this.


And from those three, a memory is built like this.


And now, if you look at that, when you for example, compute this node right here, what


you're allowed to do is you're allowed to look back at the memories.


So you have kind of connections like this.


I keep drawing these arrows the way the other way around, right.


So this one, it draws, it attends to the memories of the previous layer.


And if you see this as a recurrent neural network, you are exactly right.


Okay, so yeah, I don't I don't exactly know what to say.


This is an RNN with an attention mechanism.


It's just that these the in the construction of the things you can attend like this, usually


people just took the hidden states of the RNN cell in order to, in order to do what


they attend to.


But now, you I guess you also drop the recurrent connection because you can only attend to


the memories, so there is no, there's no, you know, kind of recurrent connection.


But there is a connection like this, there is a connection like this.


No, there is no, there is a connection like this, like to the things here.


Yeah, I guess, okay, if this, it's a convoluted, it's like a halfway in between an RNN and


a transform, because you don't strictly have the recurrent connection.


So you don't have anything like right here, but you do have like this connection, for


example, to all the three things down here.


So it's, if you view this part, as kind of an RNN cell, and this part as an RNN cell,


and this part as an RNN cell, then this is an RNN with an attention mechanism, or something


that's extremely, extremely similar.


And yet, the attention mechanisms in RNN actually do solve these this long computation problem,


that was exactly why they were introduced.


And they do solve it.


And at some point, people realized, wait, we don't need the recurrent connections, actually.


And that's how you ended up with transformers.


So this here is sort of the the hybrid between the two, right?


If you want to go further, you can you could actually think of making multiple layers of


these memory representations, right?


And then you're you're sort of at the same at the same problem to start with kind of


you recurse into the problem.


But yeah, I don't want to go into that necessarily.


So you can see here, instead of up here attending, instead of the next layer, the next layer


representation being the previous layer attending to all its sort of layer to all of its left


neighbors in the previous layer, you will have you will have the same thing attending


to all the previous memories, and the previous memory is built as a weighted sum over all


the layers.


And the most important thing for their model is this thing right here, you can see that


this now goes over all the layers, even the layers above the layer we are currently computing.


It's just that it's from previous time steps.


All right.


They also explain how you can, as I said, share the keys and the values.


That's not necessarily important, but it's just something you can do with this model


that you couldn't do before, because before, not all the layers were attending to the same


memory.


Now you can do that.


So they demonstrate this on tasks such as language modeling, where you can see blue


here is the classic transformers.


And these are different sizes.


So to the right, you kind of go shallower in the transformer.


And you can see, as you go shallower, so as you have less layers, the decoding speed increases


for both of these models.


However, the transformer model, the classic model, it sinks in performance a lot more


than the feedback transformer thanks to those feedback connections.


However, you know, here you can see, and I would bet maybe if you go to the left here


that the classic transformer would beat the feedback transformer simply because the feedback


transformer isn't a generalization.


So it also needs to do this trade off.


So it trades off speed down here.


And also it trades off sort of mixing that memory.


They have a very interesting, by the way, this is reinforcement learning, where you


need to remember things for quite long.


And that is also a domain where they excel at.


So here they actually look at the different kinds of memory.


And these are a bit deceptive down here, I think to have the whole impression, you need


to do this over multiple time steps and actually kind of see how they develop.


And then you can see more clearly, but you can see that their performance.


So this here is that feedback transformer.


And this here is kind of the original transformer where you can see it only goes up the layers.


They they see here that if you introduce recurrent connections, that helps a little bit, but


not too much, because the only thing you gain basically is this lateral connection here


that you didn't have before.


However, if you do top only, meaning that you can attend to the previous time step only


to the topmost representation.


So whereas before, you could attend only to things below you or at the same height as


you, now you can only attend to the topmost.


So information like flows like this, and it can flow down again, and then flows up again.


If you do that, you get almost all of the performance of the feedback transformer, I


hope you see this.


So here, lower is better.


And this is all this is without the memory, actually, this is, you know, every, everything,


like this is the full generalization I talked about, you get almost all the way there by


doing top only attention.


So the reasoning why they do this, the fact that the regular transformers, they don't


have access to that last to these higher layer representations in the next steps of computation.


I think that's really valid.


So you know, you know, like, experiments here on reinforcement learning in grid world,


they're fun.


Not necessarily, I don't necessarily believe all experiments in papers.


But this is a finding that that's does strike me as quite fundamental, and it validates


their claims.


And they have other experiments where they show that they try this sort of top only attention,


but they it's not top, it's, you know, they choose a layer to which you can attend to,


to the representation of which that the next tokens can attend to.


And if they say you can only attend to layer one of the previous tokens, you you do get


pretty bad kind of performance or bad, well worse than and you see as you go up the layers


up the layers, you get better and better performance.


So here is where you average all which is almost what they do.


The feedback transformer is a it's a learned average, right?


It's a learned it's a weighted sum and the weights you can learn.


In fact, if they go to the last thing here, they do almost get there.


So I don't know, you know, that could be experimental noise, I totally believe that, you know, you


can get gain a little bit by doing this, you know, feedback aggregation.


But you can see if you are only allowed to attend to layers like five and six here, you're


already doing fairly, fairly well.


And this is a summarization tasks.


So this is a language task.


This is not a constructed tasks like their RL tasks.


And that is fairly convincing, I would say, the trade offs are evident, they have a table


somewhere where in training, they are much slower.


However, on inference, actually, they can speed up quite a bit because they share a


lot of the weights among layers that others don't.


Yeah, so here you can see, for example, in language modeling, the original transformer


has much higher speed, this is I think tokens per second than the feedback transformer.


However, the feedback transformer in the inference speed is much faster than the original transformer


because at inference, both models need to do it token by token, because they are autoregressive.


Whereas in training time, the original transformer can do it in parallel, where the feedback


transformer has to do again, token by token, because they always have to compute all the


layers for one token before they can go to the next token, they have some more experiments


where they show that as, as you decrease the memory.


So if you sort of constrain these models, the feedback transformer performs much better


than the original transformer.


They also compare to LSTMs, I believe, and this is on these kind of sequence tasks that


you come up with to see sort of the properties of your model.


So does this mean we can replace transformers?


Probably not.


If you can afford to build a large enough transformer, that will probably still outperform


the feedback transformer, and it will train faster, which can be quite important.


However, if you have very special tasks where you need long range dependencies, or really


multiple steps of nonlinear reasoning, or are constrained in your resources and do actually


have the time to train it as a trade off, then the feedback transformer might be something


for you.


All right, that was it for me.


Thanks for listening.


Share it out.


I'll see you next time.


[Code] PyTorch sentiment classifier from scratch with Huggingface NLP Library (Full Tutorial)

How did it do? So Hugging Face just released this NLP library right here and this is pretty


cool because it allows you access to about a hundred NLP datasets and ten evaluation


metrics prepackaged. So knowing Hugging Face this is going to be a breeze to work with.


So what I thought we would do is we would try to use this. I have not used this yet


and it's been a while since I've used any Hugging Face stuff. So what we're trying to


do is use this to load up the IMDB dataset and then use a BERT model maybe to build a


sentiment classifier on top of that using PyTorch, specifically PyTorch Lightning. So


all of that combined from scratch and basically if I can do it then so can you and we're going


to make some mistakes and have to look at the documentation a bit and so on but that's


the process. Okay so first of all if you like content like this let me know if you're not


subscribed subscribe. Let me know in the comments if you have any sort of criticism or tips.


I'm always happy for VIM tips honestly. So I have a pretty empty repo, git repo here.


I have a git ignore but that's about it. So we'll just dive right in. Start up VIM and


let's make a file. So first some boilerplate code. I'm terrible at talking and coding at


the same time but you know. So I like to use this AppSoul library and I'm using as you


can see I'm using the tab 9 completion engine with CoC with NeoVim. This is absolutely great.


We easily maybe need app flags logging. That sounds good. So we'll need Torch probably


and we'll need PyTorch Lightning. Torch Lightning as PL. We'll need the NLP library of course


since we're going to use that and we need the Transformers library. Now I know HuggingFace


has this tokenizers library too but there are some tokenizers in the Transformer library


already and we'll just keep it light like this. So maybe numpy, maybe not. Let's see.


So we'll export, we'll have these flags object here. Maybe we'll do some flags later and


the main function. Let's just call hello. Actually let's log that info and all right.


Run main. So this is our boilerplate and let's just quickly try it out just to see whether


it works. So here we are. Hello. That's fine. All right. So where do we go from here? So


in PyTorch Lightning what you'll have to do is you have to build this kind of model class,


right? So we'll build an IMDB sentiment classifier and that's going to extend this Lightning


module of PyTorch Lightning. So you need different things in the PyTorch Lightning module. First


of all you need the init and we'll just do like a very basic init. We'll call super on


it and that's about it. And you need a forward method since this is a module. So in the forward


method you're going to get a batch and you have to do something with it. And what we


also need is a training step method. Training step which gets a batch and a batch index


and we'll have to output some kind of loss or some kind of training procedure. Then we'll


need a train data loader. So all of this you can look up in the documentation of PyTorch


Lightning. Basically you implement these methods and it will do the rest for you. So it will


do all the training loop and it will do the handling of GPUs and whatnot. The whole looping


over epochs, all of that is basically taken care for you when you use PyTorch Lightning.


So last thing we need is maybe a prepare data. Let's put that up here. Prepare data. That


method is optional but it gets called at the beginning and that's going to be pretty good


for us. So I have downloaded the weights of a BERT model and the data set. So we don't


need to do that anymore. Yeah. So that's about it. And I am going to, so maybe I've forgotten


something. Lightning examples. There's what we're going to do. We're going to look at


like an example of PyTorch Lightning and just to see whether we'll have it. So maybe here


domain examples, ImageNet sounds good. So we'll have these methods. This is way more


than we need. But down here, so basically what you do is you instantiate your model


and we won't save, have these hyperparameters here. These will be our flags. But then you'll


implement this trainer and then you call fit on the model. Right. Okay. So let's maybe


copy this down here. So we'll model. This is our IMDB sentiment classifier and the trainer.


Root tier. Let's call that logs. GPUs. We'll give it a GPU if CUDA is available. Else zero.


And then we'll make a flag for the epochs. We don't need the rest of this. And then at


the end we'll call fit model. Okay. So if we had a classifier, this would already run.


Cool. Now what I like to do is to have to have this module called sh, which gives you


some sort of easy shell commands and at the beginning of each run, whenever the file loads,


I just do, I remove the logs folder. So I have basically a clean logs folder and then


I make it again like this. So it just deletes the logs and then runs them again. So if we


run this right now, this is going to give us an error probably. Okay. So we don't have


an epochs flag, right? So we need to define a flag. Let's call define integer. Yep. And


we'll go for 10 epochs right now. Cool. Okay. Very cool. Okay. So we haven't configured


our optimizers. So in PyTorch Lightning, you need some sort of optimizer configuration.


And we'll just copy that from an example. Going full Siraj here, people. Okay, so we


need to configure optimizers. And I kind of, I like the SGD for this. SGD tends to work


well in neural networks. We don't need the scheduler. We don't need any of that. So let's


just return the SGD optimizer with the parameters. And we'll make a flags flag for the learning


rate and we'll make a flag for the momentum. Okay, we don't need any weight decay right


here. Cool. Let's put these. We'll make floats for the learning rate. Maybe start off with


something like this. So I never put help strings if the description is rather clear. Only losers


need help. Like don't be kidding yourself. If you put the help string, you need help.


That's how it works. All right. So I just don't like that this library forces you to


put the help string because it somehow makes me feel bad. Because it's very opinionated,


right? It says basically, well, you should put something there. Okay, okay, okay, okay.


So we have this. And now when we run this, we don't have anything to optimize yet. So


first of all, we need the model, right? Do we need to prepare data first? Let's check.


So I have this short thing snippet here that embeds an IPython shell. And I just plug this


into anywhere so I can see if I reach it, right, so I reach the prepared data. So let's


care about the data set first. This is why we're here, right? So it is NLP library, as


you can see right here, maybe. So there's the usage right here. So you can load a data


set here with the, I think even with the appropriate split. And it will basically just give it


back. So if you don't have it, it will download it. It's pretty cool. So we'll just load the


data set. And I've already sort of checked out what they have, and they have the IMDB


data set. Okay. And in split, in this split argument, we can say, give me the train split.


And as a string, you can say give me whatever the first 5% of the train split. Since we


won't be, like, this is just my laptop here. So we won't be able to train like a super


high grade model. But we'll go for 5% of the train split. So this is to train data set.


Right. And now if we see if we run until here, so if you had not downloaded this, it would


download this. So given the train data set, I hope you can see this. So it says it's a


data set, it has 1250 rows, and it has each entry has a text and a label. And if you look,


you can just index this like a data set. And that's the first sample, right? So the label


is one here means that we should predict the label that this is a good sentiment, right?


It's either one or zero, maybe. Maybe, yeah, I think so. So either good sentiment or bad


sentiment. Okay, so our first task is going to be basically to get this into a form where


BERT can consume it. So how do we do this with this NLP library? And that's the pretty


cool part. So right now you see this is text. So in NLP, we need to map this text into token


IDs. So we need to tokenize and we need to map this to IDs and Hugging Face, of course,


has very nice libraries for that. They're called tokenizers. So we'll have one of these


tokenizers. And we'll use this from the transformers library. And I think this is called BERT tokenizer


that then the BERT models can use. Let's check it out. Okay, we're at the documentation.


So BERT tokenizer. There we go. There's a BERT tokenizer fast. Yes. Okay, we'll take


the we'll take the fast one. Maybe not. Yeah, we'll take the fast one. Come on. Be risky.


BERT tokenizer fast. And I think we can do this from they have these methods from pre


trained. Yes. Right. So we'll take this from pre trained. And we'll put the model name


here. Now, I want to make this a flag, such that I'm not bound to a particular model.


Oops. Cool. So this is called model. So this is our model, the BERT based on case. And


we have a tokenizer right now. So what we can do is we can now tokenize these things,


these every entry in the data set. Now in a classic setting, we'd have to, you know,


write a loop for that. But with this data set library with this NLP library, it's pretty


cool that we can tokenize basically each of the samples, we can map this tokenizer function


across the the training data set. So how do we do that? We have this tokenizer. And the


tokenizer has I'm pretty sure it has like a tokenize or an encode or something method.


So there's forward. Now this is the BERT model. Where's the BERT tokenizer? Right here. Right


here. Okay, it has it has pretty sure it has this encode or something. Here. Oh, yeah,


encode, right? And code. Where is the definition of that? Can we click on this? Okay, cool.


So this encode takes text, and it takes a bunch of other arguments such as I hope you


can see this. There we go. Such as whether or not you should add the special tokens,


or the max length, this is going to be pretty important. And pad to max length, we want


everything to be of the same length. So if you apply this tokenize encode function to


a text of these samples. So let's just take the first sample here. And let's take the


text entry. Then what you're going to get is like a list of these IDs. This is exactly


what we want. So the 101 here is this CLS token that BERT takes in. And then it's just


the word pieces, right? So you could also say, instead of this, say tokenize, I think.


And that will just give you the word pieces, not the encodes yet. Right? So these are the


word pieces right here. This is the tokenize text. And with the encode function, it does


this and then maps these two IDs such that BERT can consume that. So for this NLP, this


library has this convenient function called map in their data set. So what we'll have


to do first is we'll define a tokenize function that takes in a single sample. And then it


will run the tokenizer encode function across the text entry. And we have already seen we


need like, so add special tokens is true. This is cool. Max length, yes. And we'll make


a flag sequence length or something. And we are going to pad to max length is true. So


every single sample will be of the same size. Now in this function, there's a number of


ways what you can return here. So one way is to return the original sample and actually


just set a new attribute, I think. Set a new attribute on this original sample right here.


Let's format this a bit nicer. So see, we have this tokenize function takes a sample,


takes the text, tokenizes it encodes it and puts this as the new attribute input IDs and


returns it again. And now what we can do is we can map this function across the training


data set, like so. So this will go over the training data set. And basically, each for


each entry do this thing. So hopefully, after this operation, we'll have a data set with


where, with where each sample not only has a text and a label, but also a input IDs attribute.


And we don't have this sequence length thing right here. So let's put that here. Let's


just go with 32. Since it this is just my laptop, so 32 samples should be fine. So here


it says can't pickle tokenizer objects. So what it tries to do right here is it tries


to it tries to parallelize basically this thing right here. So if we look at this NLP


thing is there documentation to this. We can just look at the data sets maybe naming splits


builder arrow data set map right here. So this function, I think it will try to multi


process and therefore it needs to basically pickle all of the things that go into the


into the into the I can't speak today, it pickles all of the things that go into the


function, which means this tokenizer right here, it needs to be pickled. Now, maybe we


can keep in memory load blah, blah, blah. Maybe there's a way to get around this. So one thing


we can try is we can try another tokenizer. Maybe this one can be pickled. The still library


is pretty good, but it can't pickle every everything. Yes, so this tokenizer can actually


be pickled. In the other tokenizer. So I'm not entirely sure what you'd have to do, honestly,


because I don't know the library. But what you could do is like, make a thread or process


local variable of this and basically make it a singleton in each process. And then basically


in here, you call the function to get it, and it returns the already instantiated object


and so on. If you really want to multi process all of this. Anyway, we have this train data


set right now and you see the schema. If you can see this, the schema has been extended.


So there's now text, there is label and there is input IDs, which is a list of int 64 things.


That's pretty cool. So now what we can do since this is still a Python list, right,


this is still a Python list. I know the tokenizers can already output pytorch tensors, but that's


kind of cheating. So we want to use this library right here. So we want the train data set,


there is a method called set format right here. And you say type equals torch. And what


that does, and I think you need to say which columns you want. So we want columns. Maybe


we should get all columns, can we output the text? So you can select from the sample which


of the columns you want. And let's check it out again. For now, as long as we're just


debugging here, I like to do a debug flag. So this is usually one of the first flags


I do. It's define Boolean debug. And what this does is whenever this is active, I try


to be as fast as possible. So there in this pytorch lightning trainer, there's actually


this fast dev run argument, which does the same thing, but I can push it a bit harder


with this debug here. So let me say this is like, this is sorry, this is like one. Yeah.


We just load like batch size samples. If we are in debug mode. Batch size, we don't actually


have a batch size argument yet, do we? If flags.debug else 5%. Okay. So we don't have


batch size yet. We're surely gonna need that at some point. So let's go with a batch size


of 8. Just because we can. So now if we run this in debug, if we run this in debug, we


should. Okay, yes, this needs to be a string. Shagaboom. Cool. So it says it's the fast


dev run. And if we run it in debug, it just loads very few data points. So this map function


here doesn't take this whole while. Maybe there's a way you can stream that I don't


know. For now, this is pretty good. So if we look at the train data set again, you can


see that it has the same entry. So this is still a list of 64. But if you index it right


now, if you go to the zero of data point, okay, then it crashes. Because it tries to


convert these two PyTorch tensors, and it can't convert the string. So we'll have to


say we just want the columns input IDs, and we want the label. Label can't spell. Okay,


let's try it again. So right here, you see that what we get out is actually a PyTorch


tensors for this and not kind of Python lists anymore. So this is now pretty, this is one


to one. So with duck typing, maybe it's even subclassed. This is a PyTorch data set, right,


which we can load into a data loader. So this is a perfectly fine data set. So we can now


say self train data set is this train data set. Now, we want to do this for the test


as well. But in order to do that, we would have to write all of this code again, which


I'm not really in the mood. So we'll just loop it. We'll create a function, prepare


data set. And we'll take in the split name, right, like this. And we'll just go with the


split name here. That should do it. And we just call it data set. Data set, shook a boom,


shook a bang, shook a boom, and return that. So now we can say train data set self dot


test data set is prepare data set for train and test. Okay. Excellent. So now we have


a training data set and a testing data set. So here in the train data loader, we need


to take the training data set and construct a data loader from it. This is super, super


easy. So what we'll do is we'll do it in one line. Data loader. So what does the data


loader, the data loader needs a data set. So the prepare data is called at the beginning.


So we have this data set right here. And I think we can go with a batch size right here.


And we already have a flag for that. And I think there is like a drop last. Yes. So the


drop last will go for true. We only want full batches during training, and we'll also shuffle.


Okay. And the same goes for we need a validation data loader for our validation set. So in


pytorch-leightning, you have train validation and test and test. Test is really only for


like the final final test. If the test data set we have here is the would be called the


validation data set in pytorch-leightning. So we false here, false, we don't want to


shuffle particularly. Okay, so we have a training data loader and a validation data loader.


Now what do we need? We have optimizer. Very good. Now what do we need? All we need to


do is to actually pass our data through the BERT model. So this forward thing here, we're


just going to leave not implemented. Maybe we can implement it. Okay, so we do need a


model. As you can see right here, this batch, let's say this batch is going to, let's go


right here. Right. So if you know, if you sometimes don't know what to do, you just


go to where you should be. Okay. Ultimately empty parameter. We don't have parameters


yet. All right. So what do we do? We go up here and we make a model. We need to actually


make the BERT model right here. So from transformers, we can use the BERT model. Now they have a


lot of BERT models and we'll go back right here to the BERT models because they, as you


know BERT is just an encoder. So we need to build a classifier on top of BERT, but they


already have done this. So they have a bunch of BERT different configurations. And the


one we're looking for here would be this BERT for sequence classification. Right. This is


BERT model transformer with a sequence classification or regression head on top. Right. So this


is exactly what we need. Classifier on top of BERT. And we can, I think we can also load


this with this from pre-trained and just put in the same name. So we can this BERT for


sequence classification and we'll load up the same model that we had. Okay. So this


is our model. Easy as that. So what do we do with this BERT? If we put in data, what


happens? For that we quickly go back again. So in the forward method we can input the


input IDs. Right. Which is batch size sequence length tensor. We can input the attention


mask that basically tells you where there's padding and where there isn't. Mask to avoid


performing attention on padding token mask value selected in 0 1. 1 for tokens that are


not masked. 0 for tokens that are masked. Then we can input the token type IDs which


we don't have here. We just have one sentence. But usually in BERT you have the capability


of inputting two different types like a question and a paragraph or a first sentence and the


second sentence. Position IDs are optional. Blah, blah, blah. Blah, blah. Blah, blah.


None of that. Okay. We could also input the labels. These are optional and it would already


compute a loss for us. Which we don't, that's almost cheating. So let's just focus on putting


in the input IDs and I think that's going to be enough. Since we basically truncate


our long text to 32 tokens we don't need to worry about masking right here. Otherwise


you would input a mask for actually we can do it. We can do it. Okay. So what you could


input a mask for basically where your tokens are not pad tokens and the pad tokens in BERT


are 0. So basically your mask should just be whatever's non-zero. But maybe also your


model learns to ignore the pad tokens. I might be wrong here and it does it automatically.


Right. So in your forward pass what do you do? Actually let's go to the training step.


We'll put something here. You can see it. So if you didn't have BERT it would actually


BERT you up. It would download BERT right here. But since I have it you can see here


this is the smaller BERT model. PyTorch Lightning. I don't have enough space in my console right


here but it would give you a nice overview over your model. How many parameters it has,


what kind of layers it has and so on. So we also need a validation step if we have a validation


data loader. Validation step and we need the validation epoch and function. So usually in


training you don't really care about epochs too much because you just have mini batch


after mini batch. But in validation what you want is kind of one single metric across your


entire test data set or validation data set. And therefore you sort of in the validation


step you'll just kind of output things. You output local things per batch and then in


the epoch and function you aggregate them into one big number. So we'll just put things


into each thing thing thing. So I'm pretty sure we're going to end up in the validation


step first because if especially if we do this debug run it tries to run a validation


first at the very start of training. So we can look at a batch right here. So what's


a batch? The batch seems to be a dictionary. If you look at its keys we have a label and


input IDs. OK. So that's pretty cool. So if we go for the input IDs that gives us a tensor


and the tensors of shape eight which is our batch size and 32 which is our sequence length.


And we should be able to pretty much input that into the BERT model that we created.


Boom. OK. And what do we get out? We get out a tuple and the first entry is going to be


this looks like logits. OK. Let's check the shape. And this is eight. So this is our batch


size and two is the logit. So one for the negative class and one for the positive class.


This is this we can basically input into a cross entropy loss given our labels. So we


also have our label here and their label is all ones. Nice. Is this maybe sorted? Is the


data set sorted into good and bad things? Because that would be bad. In any case. So


what do we have to do? So in the forward method we get the input IDs. Let's say we get the


input IDs and we run this through our model. And we can actually construct a mask here


and the mask is going to be wherever the input IDs are not zero. And that as a what does


it need to be? So these masks this attention mask is going to be a float tensor. OK. So


we'll put it as a float tensor. Cool. Right like this. So our logits are going to be that


and yeah tuple with one entry. So the comma here is important. We're going to return the


logits. So this is our forward function. So in the validation and the training step the


first thing we got to do is we got to call this forward function with the input IDs and


these of course are in our batch like this. So these are going to be our logits. And then


in the validation what we want to do is we first of all want to compute our loss. Right.


So we have to construct this up here in the init. We can actually fold this prepare data.


Loss is going to be a cross entropy loss. Yes that exists with read reduction. I like


to put reduction none. I don't think there is like a deprecated reduce and there is like


a reduction where you can put mean or something. I like to not reduce the loss at first because


then I can aggregate I can use the same thing for validation and training. So in the validation


step I just want to compute my loss right here with self. So loss loss and we'll have


to cheat a bit. So look up the cross entropy loss and where is the cross entropy loss?


Cross entropy loss it takes yes it's a reduction. And so the input to the function that we construct


is going to be first n by c first the input and then the targets. So first the logits


and then the targets right. Criterion that combines logs of max and NLL loss over a single


class. Nice nice nice. OK OK cool. So first logits and then labels label. OK that's our


loss. So if we check out what our loss is going to be it's probably going to be an vector


of size 8 because we have reduction none. Loss yes. C vector of size 8 very nice. So


we can just basically return. I'll say we can return this loss just as is. And then


in the validation epoch end the outputs here is going to be a list of and every entry in


the list is going to be one of these validation steps for one batch so we can aggregate those.


So losses is will concatenate them since they're going to be chunks of 8 outputs at the dimension


zero and then we can calculate the mean right. So we can do that and then we can. Oh no we


need to do more. We also want to know the accuracy right. So the accuracy is going to


be whether or not the logits dot arg max is equal to the label. So the accuracy for each


sample is going to be that. It's either going to be 1 or 0 and we want that as a float.


So here let's output a dictionary with loss and accuracy. All right. Excellent. So here


then we can aggregate so the loss is going to be and I like to have like a construction


here that aggregates this still. So we go out loss for O in outputs. So these are now


going to be entries. Each one is going to be a dictionary. Right. So our loss losses


we have concatenation to the mean. OK. Our accuracy is going to be the same thing for


the accuracy. Nice. So our output here is going to be a dictionary. And I think in PyTorch


Lightning there if you put validation accuracy like valac it selects the model according


to this but I'm not sure. So also in PyTorch Lightning I can now output this here but also


if you have a log entry it will forward this to the logger which we can actually do and


make a TensorBoard logger out of this. So what have we done. We have first of all set


up the validation step. So the PyTorch Lightning is going to run through the data loader for


each batch do this. So we forward it through the BERT model to get our logits and then


we compute our loss by the cross entropy loss of the logits and the labels. We also compute


our accuracy by seeing how much the logits agree with the labels or the maximum logit.


And then we aggregate all of this over the entire epoch and output that. Now let's set


up a logger. So for the logger we can put this I think in the trainer here PyTorch Lightning


logger dot and I think there is a TensorBoard logger. Pretty sure PyTorch Lightning is there.


TensorBoard no PyTorch Lightning logger. I'm pretty sure that exists. That's not the newest


version. I hate these old docs. So latest. Come on. This was called logging logger. Loggers.


Port logger. Right here. Nice. So our saved here is going to be called logs and then what


do we want? We want the name IMDB and there's also this version thing where if you don't


put version zero it will just make a new kind of folder each time. I guess we delete the


logs anyway. We delete the logs folder at the beginning so we don't have this problem.


But I generally like to overwrite my logs and not make new runs. But if you like something


different that's fine. All right. So let's run this again. And we're cool. This is the


BERT configuration that we loaded. And then we have no attribute logger. PyTorch Lightning


loggers. Loggers. Okay. Again. Loading the weights. Very cool. Blah blah blah. Blah blah


blah. Blah blah blah. And we're in an IPython shell. And do we have an IPython shell remaining?


Only in the training step. Okay. So we're at the training step right here. And we can


actually we can check whether or not ah now we have lightning logs and logs might just


ah. Okay. So this appeared to be our TensorBoard log. So we are maybe able to run the TensorBoard


here later. Let's run it. Logs. We don't have TensorBoard. Okay. Oh yeah. I've uninstalled


it because I was angry at it. Oh come on. What's going on? I don't have TensorBoard.


I should have TensorBoard somewhere. It's like in local bin or something. Local bin.


No it's not in local bin. We'll find it. We'll figure it out. How to get a TensorBoard. Maybe


we need to install TensorFlow. Well that's gonna take a while. Okay. So back to the training


step. In the training step we basically need to do the same as in the validation step. So


we'll need to forward our batch through the model. But here we don't need to compute an


accuracy but we do need to compute a actually batch loss that we can back propagate on.


Now in the training step you can either specify how you back propagate per se or what you


can do is you can just output this log loss attribute and then PyTorch Lightning will


basically do the back propagation for you. We have the TensorBoard now please. All right


there we go. And we can put this into a different thing right here. Git NLP demo. Yes. Okay.


So this is running. And if everything goes correctly. Six. Shaboom. We have a TensorBoard.


Okay. So we need to forward our training step and we need to calculate a loss for the batch.


So these loss here we do the same thing but here we call mean on it. So this is the mean


loss from this batch and we can now return the loss right here. And we can also in the


training step you can also output a log dictionary and we'll output the loss again here in order


to this is going to be our training loss that we output right here. Let's call it train


loss. And this also will go into the TensorBoard. So if we run this right now we don't have


an IPython shell. Simply by outputting this loss attribute we already instruct PyTorch


Lightning to now run backprop on this loss using the optimizer that we have defined.


Okay. And by outputting the log we instructed to put this into the TensorBoard. So now we


have a scalar entry and you can see this it only contains the valid node. It contains


everything. Very cool. Very, very cool. So let's remove the debug flag and we'll just


see what happens. So to recap right to recap we have, oh now I go see epoch one, epoch


two, go, go, go, go, go. Ah very cool. What we've done is we've set up this PyTorch Lightning


module. It needs a bunch of functions but in the init we've basically just set up our


BERT model from the Hugging Face Transformers library. We've loaded in a pre-trained BERT


model that we're going to fine tune. The main thing that the PyTorch Lightning module needs


is a training step function where you define what it should do with the data and this data


loader function. So in the data loader function we've loaded up a data set and we basically


specify the batch size. This is very easy. Where does the data set come from? We do it


here in prepare data. This is a special function in PyTorch Lightning that's basically called


after the init but before anything else runs. And here we are loading this data set from


the NLP library and this is kind of the magic part. We specify the split and the size that


we want inside of the string and you can do this in percent or in a number of samples


that you want. I'm sort of sure you can do more things but I haven't explored that. Then


we run map on the data set in order to tokenize it and that's right here and we use a tokenizer


again from the PyTorch Lightning and just run this encode function. This is very simple.


How complicated was this just like a year ago? Crazy. Then we need to put set format


and set format tells the data set how it needs to output its samples and we tell it please


output torch tensors and we want these columns right here. And we make a train and a test


data set from the train and test split accordingly. So we have this. This goes into a data loader.


PyTorch Lightning will take the data loader and run training on it using this train step


function. In this train step function we get a batch. In the batch there are these two


columns that we specified previously input ids and label. The input ids will put through


the forward function of the model itself. This is the forward function. We construct


a mask and run it through the model. We wouldn't actually need to construct a mask but okay.


And we get back the logits of the classification. And then we run this through a cross entropy


loss. Get the mean of the batch and there we go. In the validation step we do the same


thing but also calculate the accuracy but don't calculate the mean. We want to keep


it per sample and only at the end we want to concatenate everything and calculate the


mean. If we've done everything correctly you see right here our train loss goes down, down,


down until it is almost zero because we've just and the validation accuracy is super


high. Is this because all the labels are equal? Like for real? Okay so we'll do something


else. We'll make an integer with percent and this was five right? So we loaded five percent


of the data set but let's load some more. And this might take longer but let's load


50 percent of the data set and just see what happens. No. Present. I called it present.


Pretty good. So we'll load up 50 percent of the data set and we'll do the same thing and


we can track in real time what happens in TensorBoard. And unrecognized instruction


format. Okay. Can we make a format string in a format string? This is nasty. Does it


work? Please work. We can make a format string in a format string. Absolutely bonkers. Okay


so it takes a little bit longer and you could actually I think you can speed this up this


mapping of the data set. Maybe you can stream it. I'm pretty sure you can batch it. You


can do batch processing of this but for our case right here we think it's enough. So it


was like what 1200 if we had five percent. So now it should be something like 12,000.


So let's continue with the recap of what we did here. We have the train data set, the


validation data set and on yes so we have everything like this. The configure optimizers.


You can put an optimizer. You can also put a learning rate scheduler if you want to.


And then in the main function we load this PyTorch lightning module and we specify a


trainer and the trainer we tell it you know the max epochs and so on. And we set up the


logger and we just run fit on this model. And this runs epochs of the model and after


each epoch it does a validation epoch and minimizes our loss. Very cool. Very effective.


So now if, if please, if you would. All right here we go. This is my laptop training BERT.


All right. Okay. We don't seem to make too much progress. Let's check the tensor board.


Training loss goes down. Training loss goes to zero. I have the sneaking suspicion that


this is not entirely shuffled. So is there like a shuffle, like a shuffle thing? Because


this seems a bit, this seems a bit, bit fishy. This IMDB dataset right here. Just seems like,


you know, we could use a bit of shuffling because all the labels, yeah, the training


loss instantly goes to zero. So maybe, maybe it's not. We, can we shuffle here? Let's look


at the load dataset function. Load dataset batched. No. Keep in memory. No. None of that.


This does not seem to go to continue. Right here. Datasets. NLP. Datasets. I hope here


we know. We should find this load dataset somewhere. Builder features load. Load dataset.


Split. Can we not shuffle anywhere? We'll search. Shuffle. Builder. Okay. So generate


calls. This function preprocessed examples. Key will be hashed. Okay, we are not able


to shuffle this. Just like that. But I hope this at least gives you an impression. I guess


if you were to take the full dataset and map it, then it would actually work. We'll just


try again with 10% of the data just to see it go down. TensorBoard. See, this is now


good because we always delete the logs folder. We don't have any remnant old TensorFlow logs.


All right. Come on. Come on. So 10% should be about this. About this. Okay. Train loss.


Looking good. Looking good. So far. Look at these models. How large is that? How large


is the BERT base case? Hugging face pre-trained models. Pre-trained models. BERT base on case.


That's the one we have. 12 layers. 110 million parameters. Easy. Easy. No, it's too large.


Training loss goes to zero again. Okay, so we've determined that this dataset very probably


isn't entirely shuffled. It might just have all the good labels first and all the bad


labels last. And yeah, just to confirm, let's confirm this right here. Let's go with 100%.


But let's put an IPython shell down just before we map the dataset. So we don't have to go


through the whole mapping procedure. Actually, that would be here, right? Yes. Can we not


map this asynchronously? Map. I might be doing something really wrong with this library,


but I think that's how it should go. So map def map right here. We can do batched. We


could do batched. And then I think hugging face has a function to encode batched encode.


Encode batch. No. Let's go to the tokenizer. Shabada bum. Build inputs, create token type


IDs, get special token mask, save. Where is encode? Right here. Can we have batch encode?


Build inputs? No. This might be it. Batch encode. Yes, there is a batch encode where


you have batches of these things. So, okay, what if we do the negative one? See, here's


the label zero. I'm pretty sure. I'm pretty sure. Batched true. Let's do that. And in


our function here, we'll say batch encode. So let's see how fast this is with a hundred


percent. Where tokenizer has no, we just had batch encode. Oh, but this might be, we have


batch encode plus. Batch encode plus. Or text pairs. Okay. We need this batch encode plus.


But then that gives us a dictionary, right? This gives us a dictionary with the fields


input IDs right here. So like this, how about that? And then we still might want to limit


the actual data set once we have mapped it because we need to train on it as well. I


just want to see how fast this batch encoding is. Yes. Okay. Reasonably fast, but it takes


like three minutes. Yeah. So we won't go on here. I will put this as is on GitHub and


I hope you can profit from that in any way you want. The Hugging Face site has a tutorial


on squad where they also use the metrics. So they have basically these predefined metrics


like blue or rouge, I think. And you can just use them as you use these data sets. So it's


very, very, very convenient to work with these things in NLP. So NLP has come a long way.


Absolutely invite you to check out the transformers and tokenizers and NLP repos. And with that,


that's it for me, I think. I hope you enjoyed this. Again, leave a comment if you see improvements


or if I maybe should edit this a bit more. I thought the entire process of just going


Language Models are Open Knowledge Graphs (Paper Explained)

Hi there, today we'll look at language models or open knowledge graphs by Cheng Wang Wang,


Xiao Liu and Don Song.


This paper on a high level proposes to construct knowledge graphs, which is a structured object


that's usually built by human by experts, either fully manually or semi manually with


heavy human involvement.


It proposes to construct knowledge graphs automatically by simply using a pre trained


language model together with a corpus to extract the knowledge graph from.


The cool thing about this paper is that there is no training involved.


So there is no model that learns how to construct a knowledge graph.


The entire knowledge is simply extracted from running the corpus once so one forward pass


through the corpus through the pre trained language model, and that constructs the knowledge


graph.


So that's kind of the the core message of this paper.


They say this paper shows how to construct knowledge graphs from pre trained language


models without human supervision.


And it turns out the way they do it, it works pretty well on kind of standard knowledge


graph construction benchmarks.


So that's the the paper in a nutshell will go through all of this, including I have a


bunch of criticisms, but it is a pre print.


Remember this?


And yeah, so usually, I'd say at this point, if you like this content, don't hesitate to


share it out and so on.


Today we're going to try something different in 321.


Stop it's sponsor time.


This video is sponsored by tab nine, tab nine uses deep learning to help you write code


faster.


What could possibly go wrong if you do that?


No, I'm joking.


I'm joking.


Take a look at this piece of code here.


I was trying to refresh some elastic indices.


And as you can see here, all I said was could and tab nine completes it to could not refresh.