ever get with with just deep learning and like tensor flow keras all this stuff is people


Neural Networks from Scratch - P.3 The Dot Product

have a shape error and understanding shape is the first step to understanding how really


the rest of this works so it's essential we get this understood. So that's what we're


going to be talking about now. So the concept of shape is that basically it is at each dimension


what's the size of that dimension so in this case we've got a list of four elements this


is a one dimensional list and we have because we have four elements that means the shape


is just four this is also a 1d array and it's a vector so a list in python can be an array


with numpy and if it is a if it's just a simple list then it's a one dimensional array in


numpy and then in mathematics a list is a vector now imagine a scenario where we have


two lists contained in a list right in python we call that a list of lists or lol for short


in numpy this is a two dimensional array because that first list has how many lists two lists


now each lists how many elements does each list have four now this is the first time


we're kind of showing an example of how arrays have to be what's called homologous this means


at each dimension they need to have the same size for each dimension so imagine in this


case if the second list in this list of lists only had three elements could we come up with


could we say what the shape is no because in one case it's a two by maybe four and then


in the other case would be a two by three so so it could never be an array if it did


not have a homologous shape to it like this one does so in this case this is a two dimensional


array it is also a matrix and a matrix is simply a rectangular array so a list of vectors


is a matrix now again imagine a scenario where you've got three of these lists of lists contained


within a list what are we looking at a list of lists of lists or lol this shape because


we've got three lists of lists the first at the first dimension we have three elements


at the second dimension we have two and the third dimension we have four this is a three


dimensional array so finally the last container type i'll just briefly mention is a tensor


so if you've ever played with tensor flow or heard of tensor flow you might wonder well


what is a tensor and then some tutorials might call things tensors so put extremely simply


a tensor is an object that can be represented as an array a tensor is not just an array


but in the context of doing deep learning in programming a tensor is represented as


an array so you will work with tensors in the array form so just in case you're getting


information from other sources and so on you might be really confused like what is a tensor


there you go it's an object that can be represented as an array so getting into our code again


now that we've learned what vectors and matrices and arrays and how they all kind of go together


now that we've learned that let's remind ourselves what are we trying to do here we are trying


to do inputs times the weights plus bias so bias is a vector inputs is currently a vector


and weights are a matrix of vectors so how do we how do we multiply because what's what


we want to do we want to multiply this vector times this matrix of vectors how do we do


that well the way that we do that is with what's called the dot product so or the matrix


product but in in terms of numpy it is the dot product handles for both of these scenarios


so let's talk about dot product so to illustrate the dot product let's say we've got two vectors


vector a and vector b now the dot product is you multiply element wise the values in


these vectors and you add that all together so at the zeroth index of a times b at the


zeroth index plus a firsteth times b firsteth plus a at the second times b at the second


so the main thing to note here is the dot product of two vectors results in a scalar


single value alright getting back to our code let's try to employ the dot product but first


we're going to do it just with a single neuron so the four inputs one set of weights one bias


and then we'll do the full layer of neurons and I'm not just doing this to waste time


I'm doing this to drive home a point about how dot product and matrix product really


works so that when we go and then translate to doing things in batches you're going to


understand because if we don't do it this way I assure you many of you guys won't understand


I can at least attest personally I didn't actually understand what was going on until


I learned a little more about how this actually works so hopefully I can save you guys that


pain point so the first thing we're going to do is get rid of this code here and then


I'm going to save a copy of this because we're going to need it very soon but for now I am


going to reduce weights to just being one set of weights and then biases will actually


be two and now it's just a bias and then we are going to do the dot product so we're going


to use import numpy as np if you don't have numpy for whatever reason you can pip install


numpy you might have to do pip 3 install numpy or python 3.7 dash m pip install whatever


you should know how to pip install if you're having a problem feel free to leave a comment


below but I kind of expect that everybody following along understands how to install


packages but I'm happy to help you if you don't know so make sure you get numpy and


to do dot product it's actually unbelievably simple so we're going to say output equals


np dot dot and then we want to pass either the matrices or the vectors alone of that


we want to do dot or matrix product of in this case you you might want to say inputs


weights and then plus bias and this will work we'll run it so print print output it does


work it gives us that 4.8 that we were expecting to see but really we're going to do weights


time times the inputs in this case both of these are a vector so it doesn't really matter


which one comes first we'll get the exact same answer in this case but very soon it


is going to matter greatly so so so first looking at this code doing np dot dot weights


inputs plus the bias let's review what happened here before we go to a full layer so again


reviewing what just happened we did an np dot dot weights inputs plus bias now what's


happening here is when we do that dot product of those two vectors they're just multiplying


element wise each element at the same index of both of these vectors and then adding all


of that together so just that dot product alone returned to us a 2.8 then we add the


bias of 2.0 and that gives us the final answer of 4.8 so pretty simple to understand I think


up to this point and now let's get into doing a dot product with a layer of neurons okay


so coming back to our code let's do a dot product of a layer of neurons so that's why


I went ahead and saved a copy here so I'm going to copy that come over here paste that


and then everything stays the same weights inputs and then biases right and now we will


illustrate why weights came first so let's go ahead and first run that and we can see


we get the output that we again expected from seeing this we've seen these operations already


so what's happening in this dot product shouldn't be you know beyond you because you've seen


it in very raw python code but that is the output that we wanted now if originally remember


we were able to interchange inputs and weights because inputs and weights were they were


both just a vector and so the order of two vectors is not going to change the output


of the dot product but now what is inputs is a vector still weights is actually a matrix


right it's a matrix containing vectors so this is different so the first element in


this dot product when you do a dot product with numpy the the first element you pass


is how the return is going to be indexed so in this case what do we actually want back


right we want those outputs like what are we modeling here we're modeling three neurons


how did we know what was the difference that changed when we wanted three neurons the the


at least before we did the plus bias so the only the difference in the in the dot product


the difference was well we have three sets of weights this is how we knew this is three


neurons so because of that because we want things in the we want that output to be the


values from the neurons we want this that's what we want we pass weights as that first


input because we want things indexed by those weight sets that that's what we want and in


fact if we first pass inputs and then weights we're going to see this doesn't even work


right it throws an error and again this is probably the start of something similar that


you've probably seen before if you've ever worked with any other deep learning library


is you get these weird shape errors and without a really good understanding of what are the


actual atomic calculations that are trying to be done here you're going to get these


and have absolutely no clue like what what have I done so okay so converting this back


to weights and inputs if it's not already clear what's happening basically this first


thing is going to be broken down into doing dot products of this and then dot product


of this and by dot products I mean this times this right so this dot this and then it's


going to be this dot this and so on it's going to iterate through the this this matrix of


vectors so again let's just run that because I don't like looking at the error again let's


visualize what has happened in this case okay so to review what's gone on here we've got


outputs is np dot dot weights and inputs let's talk about just that for now and what's going


to happen here is because weights is a matrix of three vectors it's going to do this dot


product three times so as you can see it's np dot dot weights you know zeroeth vector


so weight zero is a vector of weights so now we can actually do the dot product that we


know and love and so what's going to be returned to us from doing a dot product or np dot dot


of a matrix of vectors and just a vector it's going to return to us in numpy form this will


be an array of these values that will be the dot product of the zeroeth weights and inputs


and then the dot product of the first of weights and inputs and the dot product of the second


weights and inputs and then after that all we're going to do is add these two vectors


together to get our final output values okay so that's the conversion of everything into


numpy terms and more mathematics terms this is the way that you are going to be seeing


the calculations the dot product of weights and inputs plus the bias that's that's how


all the notation that you're going to see is going to be doing this so moving forward


now you should understand exactly what's happening there and then also hopefully it be understanding


how the dot product in numpy is actually working because not only are we going to need to understand


how that works when we do a layer pretty soon we're going to be doing things in a batch


so up until this point our input data has actually just been a 1d array or a vector


of a single sample but soon what we're going to be doing instead is passing a batch of


these samples and now input data rather than just being a 1d array or a vector is actually


going to be a 2d array or a matrix so hopefully now that we've covered shape and what vectors


are matrices and dot product and you hopefully have a really good grasp of how that's working


I strongly advise you not to leave this video if if you don't yet understand so please ask


questions below also hopefully we've cleared up the questions about weights versus bias


they're just two different tools for approximating something and weights times the inputs plus


the bias is similar to the equation for a line y equals mx plus b something like that


right so here's an example where x is your input y is your output and now we're going


to just be modifying weight you can see how it impacts the equation for that line similarly


as we adjust bias we can make changes to the line that way and you can see that by adjusting


bias we're able to graph certain lines that we just simply could not have graphed with


just simply adjusting the weight alone so the differences as well as the actual benefits


and use cases for weights versus biases become even more apparent once we get to activation


functions you don't necessarily need to know how rectified linear is working we're going


to get to that in a little bit but you should be able to see at least visually that by adjusting


weights there's a just a completely different impact than adjusting for bias and basically


bias is going to at least in terms of our activation function help determine whether


or not this neuron is actually firing at all and if so to what degree again right now we're


just kind of we're just making up values here this has no meaning soon we'll work with a


data set that has a little more meaning and then when we get towards once we figure out


the basics of neural networks we'll use a real data set and hopefully have even more


meaning but one baby step at a time this is it's such a complicated topic in so many ways


it's very easy to skip over things and gloss over things and get to the quote unquote sexy


part of deep learning but in order to get there and really understand you got to go


through here so hopefully you guys are enjoying again the series is in conjunction with the


book and NFS dot IO also check out neural networks from scratch in X pretty cool also


if you're curious the person who has done the scratch variant is Daniel who's the co-author


of the book he's also the one doing the animation so if you're really enjoying these animations


shouts out to Daniel he's also done a short video of programming a fully working neural


network in scratch the programming language which is absolutely absurd I will put a link


in the description you can check it out it's not too long and it is absolutely insane also


some people have been asking and noting the similarities between the animations and three


blue one brown animations and that is because we are using the manum package so this is


the package from three blue one brown and basically what we've done is we've kind of


built a framework and by we I mean Daniel on top of three blue one brown to do the animation


so if you guys are really enjoying the animations again shouts out to Daniel also I'm just as


excited as you guys are I think the animations are an awesome addition to the channel so


thanks to three blue one brown and also Daniel for writing that code kind of on top of it


to get this done so anyway that's all for now if you guys have questions comments concerns


Extracting Training Data from Large Language Models (Paper Explained)

Hi there. Today, we're looking at extracting training data from large language models by what


appears to be a big collaboration between corporations and academic institutions. There


are almost as many affiliations here as their authors. So this is joint work between, you know,


as you can see, many, many sort of institutions. And it is a pretty cool paper. So the high level


topic is that these authors take large language models, as the title says right here, and trained


large language models specifically, and they're able to extract training data just from the


trained model, in fact, just from the black box access to the trained model. And and not only are


they able to extract training data, they are able to extract pieces of training data, sort of


verbatim, that have appeared only very few times in the training data. And they that's what they


call a form of memorization. So they're able to extract these with a kind of pretty clever attack.


So if you look at this prime example, right here, they are able to query GPT-2 in this case, which


is one of these large language models to output this piece of text and the black stuff here


is by the authors to protect the sort of privacy of this individual right here, this is though this


is a real piece of text that they actually got out. And you can verify that. So they're able to


extract this just from GPT-2. And needless to say, this has consequences for security and privacy,


and so on. Because if you train one of these models with let's say, internal or private data,


user data, and so on, you have to be worried that these models are going to just output that data


again, on the other end, and potentially leak information. This, of course, has not been a


problem that much so far, if you know, once we just trained image classifiers, and so on. But


here, especially with only black box access, this seems like it has some some consequences. So we'll


go over the paper, we'll go over the the attack or the technique, the author's device, which is,


I think, pretty clever. We'll go over sort of the results that they get from using this on GPT-2.


And we'll go over my opinion of the paper, which I can already tell you, my ultimate opinion is that


the attack is cool, the concerns are valid, but the paper is probably written a little bit more


scary than it ultimately seems. In fact, I find that the results, the actual results of this paper,


fairly okay, like fairly promising, and sort of straightforward, not that scary. And also,


the paper is interesting from another perspective, namely, from the perspective of what it tells us


about these language models and how they work. And it it sort of strengthens a number of hypotheses


that I've put forward in my video about GPT-3, about how these models work. And that's also


fairly cool to see in this paper. So we're going to jump in here. And as always, if you like content


like this, don't hesitate to share it out, or subscribe and subscribe, I should say, if you're


not yet. Alright, so they say it has become common to publish large, so billion parameter language


models that have been trained on private datasets. This paper demonstrates that in such settings,


an adversary can perform a training data extraction attack to recover individual training


examples by querying the language model, right. So we have a we already have quite a bit of


information right here. So large language models have been, of course, trending with, you know,


especially since GPT-3, but at least since since the advent of the transformers BERT and so on,


though BERT isn't exactly a language model. So language models are models that, given a piece


of text predict the next word, let's let's so easy as that, or they predict a probability


distribution over the next word. So if you say a cat sat on, so that's the input, the language


model would give you a probability distribution over the next word. So the next word might be


the or the next word might be a, or the next word might be next, because of next to and so on. And


it will sort of give you a probability distribution over each of these words that kind of looks like


a face. It will tell you how likely each next word is, and so on. And then you can sample from


it, you can sort of choose one of those words and then go on. And you can evaluate the likelihood


of entire sequences and so on. So GPT-3 is one of those large language models. And these large


language models, they've been, of course, since they are large, we know that they also need a lot


of data to be trained on. So a large language model would take like a giant piece, a database


of training data, which is scraped from the internet, usually. So this is too much to simply


be curated by humans, they just let scrapers run over the internet. Then they use this to train


the model, whatever that is in GPT, GPT-2 in this case, and GPT-2 will then be a trained model. So


you sort of throw the training data away. And you simply say, this is our model. Now, we're going to


publish this, right? Now, the problem is, if there is a piece of data in here, that is kind of secret.


And you think, well, it's just one piece of data, like how much can how much can go wrong,


right? The problem is if I can inspect GPT-2 and recover this exact piece of training data,


so that GPT-2 will output that exact piece, right? That is, is a problem. Now, they make some good


points here, this notion of a piece of training data, and what it means to memorize a piece of


training data and what it means to extract one is fairly fuzzy. And they go quite a bit deeper in


this paper. So they have kind of strict definitions. They say, we demonstrate our attack on GPT-2,


a language model trained on scrapes of the public internet and are able to extract hundreds of


verbatim text sequences from the models training data. These extracted examples include public


personally identifiable information. So names, phone numbers and email addresses, as you saw on


the right here, IRC conversations, code, 128 bit UUIDs, and so on. So they are able to extract all


of these things from the trained model, right? And this, you can already see that how this can


become a problem. They say our attack is possible, even though each of the above sequences are


included in just one document in the training data. And this notion, this notion of memorization here,


and when it is dangerous, they correctly say that this is only dangerous, of course, if the


training example is contained in, let's say, only one piece of training data, because if something


is contained in 1000s of pieces of training data, it's, you know, it's okay to memorize that, right?


If a name of like some famous person is memorized, and maybe that the address like like the president


of the USA lives at the White House, that it is not a secret, right? So it is okay, if your language


model remembers that, because it probably occurs in many training data points. However, if something


is contained in just one document, right, and the model remembers it, then that is, you know, kind


of true memorization, it is not maybe, or, you know, it's probably not learning anything from


that data point is simply memorizing it to make its training loss lower. So that's the case on


the right, right here. Though, I have to say, this, as I said, it's written a bit more scary.


So they don't exactly say that this name and phone number is contained in just one document. And they


also say, like, this is, of course, this is pop, this is on the public internet, GPT-2's training


data was scraped from the public internet. So here is sort of my first investigation into this. Of


course, you can Google this, and you'll find it, you'll find this. And even though you know, the


blacking out here also is a little bit of, I think it's a little bit gimmicky, because I don't see a


problem with disclosing this particular piece of information. And I'll show you why. So when you


search for it, you'll find the NIST homepage, you'll find a cryptographic algorithm validation


program. And you'll find that this is a description of a software implementation. And here is the


personally identifiable information. You can see, this is a corporate address. So this is a address


of a corporation. And the contact information is a corporate contact is a corporate email address,


it's a corporate phone number, and so on. This is the exact thing right here. And, you know, with


with respect to it only being present once in the training data. So if you actually search for if


you complete the name here, and search for this, you'll find many, many, many, many, many results.


Now, I don't know how many of these results are actually from, you know, in the GPT training data,


no one knows that, except open AI. So there's two Google pages of results. But oh, Google has D


sort of D duplicated some of them. And now if I click on all, there are many there are 9000 results


for this. And they are not all the same auto No. So if you look at a bunch of those, you'll see that


they are almost the same. But here, at the bottom, as you can see, this changes. So, you know,


depending on your scraper, these all count as separate websites. And therefore, I'm not so sure


that this particular piece of information here is contained only once. Plus, it is a corporate


contact. So again, so to my point, the paper might be written a bit more scary than, then it ultimately


turns out to be, though, you know, you have to you have to make two different points like this


particular piece of information. Yes, it might be written a bit more scary and gimmicky with the with


the blacked out stuff. However, right. The paper has a point, namely that if let's say you as a


company do this on internal data, it might very well be and they do have examples where they


reproduce data from just one document. But even it might be that something like this happens to


you internally, where you sort of maybe in your internal document base, you sort of do quasi


duplicate a document with the same information over and over and and that's not the duplicated.


And then your language model sort of memorizes that. So it's quite it, it has a point the paper.


That's that's what I'm trying to say. I hope that's clear. Alright, so we'll get to the results


in a bit. I hope I've already given you some sort of a taste for what you can expect. So first of


all, they go into language models into sort of the definition of language models. And the language


model here is simply framed as a model that can sort of give you a a probability of a sequence


of text in sort of a stepwise fashion. So always probability of next word given the previous words,


and you can evaluate that, right, so the access to the model that they assume here is access to,


let's say, the logits of the model or the output distribution of the model.


And they say they use GPT-2 because it's trained on large piece of text, but it's also, you can,


you can evaluate it, it's not as slow, I guess, as GPT-3, and it's publicly available. However,


the training data to GPT-2 is not publicly available. But they do have someone of OpenAI


on the paper here. And this person at OpenAI made like made, they could sort of query the OpenAI


person to make sure a given piece of text that they find is or isn't in the training data of GPT-2.


So that's how they work. So that one per the OpenAI person acts as an API for the training data.


Right, so they, they do, they define their attacks here. So they do a lot of things to,


to set up cleanly what they do right here. So they have two points right here, there is this notion


of memorization. Okay, so there's they say there are many ways to define memorization in language


modeling. In this particular piece of work, they say it is okay to memorize some stuff, they say


language models must, for example, memorize the correct spelling of individual words, right,


because the words are made of word pieces, and the language model needs to output that. So that's


fine if it memorizes this. Indeed, there is an entire area of research that analyzes neural


networks as repositories of memorized knowledge. For example, when GPT-2 is prompted to complete


the sentence, my address is one Main Street, San Francisco CA, it generates the next token 94107,


a correct zip code for San Francisco in California. They say, while this is clearly memorization in


some abstract form, we aim to formalize our definition of memorization in order to restrict it


to cases that we might consider unintended. So memorization as such isn't bad. What is bad is


what they call here, the eidetic memorization of text. So eidetic memorization of text is when the


model memorizes something that only appears very few times in the training data. So they say, we


first define what it means for a model to x to have knowledge of a string, our definition is loosely


inspired, yada yada yada, a model f knows a string, if s can be extracted by interacting with the


model. So if you can input whatever you need to input, and the model outputs s, then the you say


that model knows s, right? So if s is a piece of training data, then you say the model memorizes


s, the model has memorized it. So here, they say a string is extractable from a language model if


there is a prefix and the prefix here is the input to the model, such that if you input that model,


the output will be the will be the string. And then they define this eidetic memorization,


respectively, they define k eidetic memorization, a string s is k eidetic, I have no clue whether


I pronounce this correctly, k eidetic memorized by a language model f, if f if s is extractable


from f, so that's memorization, and s appears in at most k examples in the training data. Okay, so


if this address of this person only appeared twice, but you could extract it verbatim from the


language model, then that would be an example of two eidetic memorization, okay, because k in that


case would be two because it appears twice in the training data, though they they also they're


not clear what they mean by examples in the training data, because usually this training


data is sort of chunked to make it fit into the language model and so on. And I think they do this


on a document basis. So they would consider something like this here, one example, right,


and then a different document, a different example. So if you have like, for example, if you


have these IRC conversations that they are able to extract, so they claim here they are able to


extract IRC conversations, or they're able to extract the usernames of the IRC conversations,


right? The usernames might appear hundreds or thousands of times because they chat with each


other. And they will all be, you know, in one document, but the document will be so long, they


will actually be chunked into different training data pieces. Maybe I don't know, but I think


maybe I don't know, I don't know exactly what it means to be an example right here. But they do


the example for sure, for sure, that piece of text can appear more than once, even if it is


only in one example. In fact, they, they actually analyze the situation. Alright, so we've defined


that this is the chi, these k-idetic memorization, that's what we're looking for. That's sort of the


problematic regime. If k is very small in the extreme k is one, one piece of training data


contains a string and we can extract the string at from the trained language model.


They also say that for any given k, memorizing longer strings is also intuitively more harmful


than shorter ones. So this kind of makes sense. And they even they even go into sort of corner


cases, they say amidst certain pathological corner cases, for example, many language model


when prompting with the sequence, repeat the following sentence, and then you give a sentence,


will do so correctly. This technically allows any string to be known under our definition.


But they, they of course, don't do that, they assume they don't know the training data, so they


can't just say repeat the following sentence, and so on. But you do see that it is fairly hard


actually to even define the problem right here, even though we as humans have a sort of an


intuition what it means for a language model to unintentionally or on the do do unintended


memorization. Alright, so the adversary's objective here is to extract memorized training data from


the language model. The strength of the attack is measured by how private so how k-idetic a


particular example is. Stronger attacks extract more examples in total, and examples with lower


values of k. They say we do not aim to extract targeted pieces of training data, but rather


indiscriminately extract training data. While targeted attacks have the potential to be more


harmful, our goal is to study the ability of language models to memorize data generally,


not to create an attack that can be operationalized by real adversaries to target specific users. So


you can see that here, they simply want some training data, they don't really care what it is,


they simply want to get some so they're going to search for sort of the easiest to get training


data. And that, so they frame it as yeah, we don't want to devise an attack that can attack


individual users. But there is a different component to it. So if you had to sort of guess


the password of any particular user, that would be you know, fairly, fairly hard. However, if you


had to guess a password that was used by any user, it's fairly easy, right? Even if you discard the


fact that most of people use password as password, and so on, if, if people would just uniformly


sample words from the dictionary as their password, still, you'd have a decent chance of figuring out


a password, right? We have a decent chance of figuring out, you know, not super high entropy


things like maybe credit cards, you'd have a decent chance of figuring out the credit card


number, just by guessing one. So this is the regime we are in here. And it's entirely different


regime, I think if you try to attack individual users, essentially, what they're going to do right


here is they're going to say, look, there's training data right here. Now, some training


data, these models can extract a pattern from right? If and this is what we do with machine


learning, right? We say, okay, this this data right here, they all have like some pattern. And


this data right here is some pattern. And you can learn from this. And it has some patterns. So the


machine learns to sort of abstract from extra training data samples, and so on. But here is a


data point that doesn't really fall into any of these categories. So what the model will do is it


will simply say, well, this is its sort of own little group, I'll remember that I can extract


some pattern from here and from here, but I can't extract any pattern from here. But I need to get


my loss down. So I'll just remember that, you know, individual piece of training data. And that's


exactly what we can recover with this sort of attacks, these individual pieces that aren't


really don't really have anything close, there is not really a pattern to it. So the best the model


can do is remember that it doesn't mean that with this attack, you're going to get this piece of data


or this piece of data, right? So if your personal identifiable information is sort of falls into


some kind of regular pattern, it's, it's likely to be more safe against an attack like this. That's


why they, for example, are able to extract these sort of UUIDs, or URLs with random strings in them,


because random strings have no pattern, right? So they are likely to be out here away from the other


training examples where the best the model can do is actually remember the thing rather than extract


a pattern. Now, the other example here with this personally identifiable information, I believe


that's just because it appears a lot of times, honestly, not because there is no pattern, but


because it appears so many times that the model simply, you know, it's it's, why should it extract


a pattern when it appears so often, it can just, you know, remember it like a famous person's name,


seems to be an address that's important if it appears so often, I guess, from the point of view


of the model. So that's, that's sort of what this does. Again, it extracts indiscriminately,


it doesn't mean that the attack can be leveraged to, you know, get any training data sample back.


It's still worrisome, but you have to take into account. Another thing that that is really


sticking out in this paper is the amount of hedging that this paper does. This, this almost


in every paragraph, but certainly in every subsection, there is like hedging, hedging


against, you know, why it is okay to publish this research, and so on. So, you know, when they say,


you know, when they say, our attack target is GPT-2, we select GPT-2 is a nearly perfect target


from an ethical standpoint, the model and the data are public. So any memorized data we extract


is already public, and so on. And they do this in in every piece of text. And, you know, in my video


about broader impact statements, that was exactly my point, these large corporations, right?


If many, many of these authors, I think a fair amount of work went into framing this research,


such that it sort of can't get attacked from, you know, people concerned about


if, you know, ethical considerations when releasing research like this thing, this is


clearly research that can be leveraged, you know, for for bad, if you will. But since these, you


know, companies have a lot of resources, and and there, you know, can put many people on this can


devote a fair bit of amount of, of work into framing the problem that can be mitigated. Whereas


if you know, some lonely PhD student would do the same research right here, the exact same research,


I'm very doubtful it would be received as well as this piece right here. And in my opinion, as I


already said in that video, this just sort of shifts, you know, a bit more power to these large


institutions that sort of can afford the framing right here, they don't have to change anything


about the research. But the rest of us do. Alright, rant over. Let's continue. So they,


they're going to do this in two different steps right here. And they have a diagram,


yes, have a diagram. So first, they do this in two steps. Step one, they query the model,


they have different queries, right? But they just sort of generate data from the model. So


they generate lots of data right here from the model. Then they select somehow they select from


the model a subset that they think these could be memorized training examples, then they do


the duplicated, they they select again, and then they check, okay, this is it's fairly,


fairly easy workflow. So step one is generate a bunch of data that you think could be memorized.


And then step two, check whether you find the samples in the internet, because all of GPT2 is


training data comes from the internet. If you can find them on the internet verbatim, right,


that probably means GPT2 as remember, like the likelihood that it verbatim remembers, you know,


a UUID that wasn't in its training data is almost zero. So yeah, this goes by manual internet search.


So respect to these authors who have done this, they start out with some fairly, fairly weak


baseline, which is they simply generate a large quantity of data by unconditionally sampling.


And then they predict which output contains memorized text by simply analyzing the likelihood.


So whatever text the model finds highly likely, they they they think that could be memorized


because if you provide a model with training data, and you ask it to reduce its loss on the training


data, it will assign highest likelihood to the training data. That's, you know, just that's how


these models work. So they assume that if a model has high likelihood, or low perplexity, that's the


sort of same thing. Except, yeah, so you can see here, if the perplexity is low, then the model is


not very surprised by the sequence and has assigned, on average, a high probability to each


subsequent token in the sequence. And if that happens, they say, this could be memorized.


This is obviously, obviously very, very, very simple. Say, this simple baseline extraction


attack can find a wide variety of memorized content. For example, GPT-2 memorizes the entire


text of the MIT public license, as well as the user guidelines of Vaughn Life, an online streaming


site. While this is memorization, it is only K-idetic memorization for a large value of K.


These licenses occur thousands of times. Okay. The most interesting examples include the


memorization of popular individuals' Twitter handles or email addresses. In fact, all memorized


content we identify in this baseline setting is likely to have appeared in a training data set


many times. So here they say, it doesn't really work if you just sample and then look at what's


most likely. Because, yes, this will be memorized, but it is sort of a non problematic form of


memorization, like famous people's Twitter handles. This is like famous people's names at this point,


right? So now they go about improving it. Okay, so they improve both steps. They improve step one.


Where are we? No, it's down here. They improve step one, by doing one of two things, either


you want your temperature to decay. So in this sampling, when you sample from the model,


you have a temperature that you sample from, and you can decrease that over time. So at the beginning,


you can let the model explore a bit, but then you can you can decrease it. And that's


so the that sorry, the the goal of changing step one is to create a more diverse set of generations,


right? So you can sample with high temperature at the beginning, and then decrease it over time,


okay, such such that you still get sort of high likelihood sequences, but you get different ones.


So you start off differently. And then you go into the high likelihood regime. The second way they


change this is what they do is they go to the internet again. So they go to the World Wide Web,


which is okay. I'm terrible at drawing the globe with Okay, they go to the World Wide Web. And they


just get pieces of text from the internet. So they get a website. And they just take some tiny


substring from here from this, and they use that as the input to their model. And that's sort of


to get more diverse predictions. So if you input a short prefix that you found somewhere on the


internet, and then let the model continue, that generates you have wide, diverse variety of pieces


of text. Okay. So that's how they up the DA how many different samples the model generates. Because


in the initial experiments, they found that the model will sort of output the same things over


and over again, if you simply query it unconditionally. So either high temperature


or conditioned on internet text. The second step is sort of what I find the clever step.


So here, they have to, before they simply said whatever has high likelihood, that's what we


think is memorized. But of course, a lot of these will not be, you know, with low K memorized,


a lot of them will simply be high likelihood because they're actually likely. So they say,


okay, when, when is when are we in this situation? So let's say here is the here is our data set.


Okay. And here is the the MIT public licenses here. And it you know, it appears like billion,


billion, billion times like this data point is like ginormous. It's all you know, the MIT public


license. And here is our outlier data point. Now, this model will extract patterns, let's say from


this, and this is a pattern. And it will assign a single pattern to the MIT public license because


it just appears so often. And it will assign a single pattern to this data point down here,


just because it's such an outlier, right? So how do we how do we devise a scheme that will find


this one reliably, but sort of will recognize Wait a minute, this this memorization here is okay.


But we need to devise a scheme without having access to the training data, right? If a human


looks at it, of course, the MIT public licenses, you know, seems common, we know that it's common


and so on, we know that it's highly likely text because it's a it's a license almost everywhere.


If a human looks at this right here and sees, you know, the name and address of a person or a


credit card number, we know that's not really highly likely text. And that's sort of the the


answer right here. So we say if a human looks at it, but what is a human a human is just another


language model, among other things, right, but the human is just sort of another thing that has an


intuition of how likely text is. So the basis of their approach is going to be the following. Let's


take a second, second data set, okay, sampled in the same way also from the internet, but not in


exactly the same way. In fact, they use common crawl instead of the the Reddit outbound links that


GPT-2 used, but we take any other data set, and I'm going to draw the other data set. So here's the


data point, here's the data point, maybe this data point is duplicated from the other data set. And


here's the data point here one, right, so you're going to have sort of other data points. But also,


you know, since you're sampling from the internet broadly, you're going to have the MIT public


license many times. And you're also going to have the outliers in this data set. Now the important


part is, you're probably if you sample this differently, in the same fashion, but a bit


differently, you're probably not going to have this same outlier right here, you're probably not


going to have that in your new data set. Okay, so you can see in the new data set, I hope you can


see this, you're going to have the the same pattern extracted here, even though it's from, you know,


slightly different data points, you're going to have maybe a pattern extracted here, maybe one


here, you're going to have this same cluster here, because the MIT public license will appear, even


though it comes from other documents, it's copied over and over. And you're going to have this


outlier right here. So what you can do to differentiate our two, our two things, you can


consider a second language model. And you can ask. So here we have two things that the first language


model things are very likely, you have this thing right here. And you have this thing right here,


both the first language model considers super likely, you ask the second language model and the


second language model says, yes, the MIT public license, I consider that to be also super likely.


But this outlier over here now that's I've never seen that what's that that seems very unlikely.


And so by the ratio of the two likelihoods of the two different models, you can find out


samples that the first model finds super likely, but the second model things are not likely at all.


And that's exactly the trick they use right here. In fact, they use many instances of that trick.


So here are the strategies perplexity is simply what they use before whatever's likely is probably


memorized. This Yes, it's memorized, but it's often memorized justifiably. Then they have these


strategies, small and medium. And and this is the ratio of the log perplexities of the largest GPT2


model, that's the one they attack, and the small GPT2 model. And this ties into so the


so you don't even need a different model, right? You can simply train a the reason they train a


smaller model is the following. And we on the machine learning street talk podcast, if you


don't know that it's a it's a it's a podcast where we talk to people from various, you know,


from the industry, and from various research labs, and so on. And we spoke with Sarah Hooker, who


we talked about their paper, the hardware lottery, but she also has other research,


where she sort of shows that if you have weights, so you have a neural network, and it has, you know,


layers, layers, layers, and you have weights in these layers, right? What she was able to show is


that not all weights are equal. So some of the weights, let's say the weights here will be


allocated to these pattern extraction things. So you know, here we have these, you know, you have


date training data training data outlier outlier, right? So you'll have this, you have these


weights representing this pattern within a layer, right? You have these, this pattern will be


represented by these weights right here. And then you'll have other weights, they're sort of


allocated to remembering single or very few outliers. Okay, so here, this will be allocated,


and these will be disproportionate. So there will be many, many more data samples covered by,


let's say, this piece of weights right here, I should have drawn the bottom one smaller,


than by this. So there might be, you know, 1000 training examples covered by one piece of weight


space. And there might be only one piece of training data covered by this other piece of


weight space. And that's simply because it can extract a pattern from one, but not from the


other. So it needs to memorize it. And the larger we make these models, you know, the more


parameters we give them, the more the more, the more ability they have, the more space they have


to do this remembering. So what what Sarah Hooker noticed in her paper is if you then distill these


models and distillation is the process of taking these models and putting their knowledge into


smaller models, then what happens is not all training data points will will so that in


distillation, you usually lose performance, not all training data points will lose performance


equally, namely, you will lose performance on the training data points that are sort of these


outliers that are these not often represented in the training data that you know, the model has a


harder time extracting a patterns from it. So they will be seldom patterns, or just hard patterns,


I would also assume that, you know, patterns that are harder to extract will also fall, fall away.


So the the more complicated patterns will also be sacrificed. But I guess among the things are


these outliers. So if you train a smaller model, the smaller model would have less ability to


remember these outliers. And therefore, if you do this, you don't even have to do it on a different


training data set, right? You can simply compare to the same model trained on a different model


by sorry, to a smaller version of the same model trained on the same training data set, because


that will probably not remember the outliers as much. It would have been interesting if these


authors here had actually distilled GPT two. And though they do not have access to the original


training data, so I can get why they didn't do it. But would be interesting to see that.


That gives me an idea sort of, maybe there is actually a way to look at the weights and I get


these these authors don't have access to the weights, but maybe there's a way to look at the


weights, and to actually be able to sort of, in some way spot, right, which of the which of the


weights only are associated with with single or very few training data points. Maybe during


training, you can sort of count how many times a weight is updated in a substantial amount of


training data points. And maybe looking at the attention matrices, you can sort of determine


what are the kind of patterns that need to happen that lead to this weight being activated, right?


So if there's a weight, and it's activated by lots of lots of different patterns, maybe, you know,


that weight is useful for many, many forward propagated signals. But if there is another


weight that's only activated by a specific pattern, right, then maybe that's one of these these


memorization weights. So maybe there's a way to recognize these in the weights directly. So


distillation appears to be sort of a defense against this this memorization of things, though


that's not, that's not done in this particular paper, they also have different strategies. So


you don't need to do this neurally, right, you can compare the ratio of the perplexity that GPT2


gives to the Zlib entropy. So this is simply a text compression method, you can even compare it


perplexities between the original string and the lowercase version, and so on. So they extract,


for each of these configurations, we select 100 examples among the top 1000 samples, so they


produce 1000 samples, and they sample 100 from those 1000. So they mostly sample from low ranked


samples, but also they explore some of the higher ranked samples, they have a formula where they


sample they deduplicate, and then they investigate. Alright, so they do Google searches, if they can


find the thing they say that's memorized. Alright, so they say, across all strategies, what we


identify 604 unique memorized training examples from among the 1800 candidates, our best variant


has a true positive rate of 67%. That's quite remarkable, right? So 67%, 67% of the things that


this method delivers you automatically are actually memorized. Though you have to qualify that,


right? If you want more than 1000 examples, that rates going to drop, right? You since you select


the top 1000 examples, these are the most likely to be memorized. So yeah, if an attacker wants


more, if they want to scale this attack up, their positive rate is gonna plummet fairly quickly,


I'm going to assume it would actually be interesting also to see how that develops with the


top the top retrieve document right here. But I get the, they have to do Google searches to figure


out and then ask open AI to figure out if it's really a memorized training example. They say


their categories, we manually group the memorized samples into different categories. The results are


shown in table one, most memorized content is fairly canonical text from news headlines,


log files, entry from forums or Wikis or religious text. However, we also identify a significant


amount of unique data containing 128 bits UUIDs correctly resolving URLs contained in the


URLs containing random strings, and contact information of individual people. Okay, so


as I said, these, this is this is fairly interesting, but also a bit expected, right?


If I give you the start of a UUID, then there is no pattern to extract, except I guess the UUID


structure, but there is no deeper pattern to exact. So all the model really can do is memorize


the UUID, especially if there aren't too many UUIDs in the training data or if this particular


UUID is some sort of, as I said, it's this outlier type of situations, the same thing for,


you know, URLs containing random strings. These are just not pattern extractable, therefore,


easily, more easily remembered by the model than learned. So you can see right here, the breakdown,


where they see how many of what they extract, and your contact info, 32 named individuals,


none in non news, 46. That's a fair amount of things you can extract from GPT-2. You have to


say that that is all right, all of GPT-2, you get approximately 100 things that are kind of names or


contact informations. So as I said, not too bad, specifically considering what I've shown you here,


right? That's one of these contact informations. And they do say this in the paper that this


person, this information was obviously released in the context of this software project. And the


problem is only the model might actually output this in a different context, right? The model


might think, oh, now I need to output some sort of name and address. What kind of names and addresses


to enter? Well, this name and address appears pretty often, I'm going to put that here. And


so that's a failure case, you know, that these things can do. So here is a sort of a graph.


And they have more of these graphs later. But you can see that here, for example, is a GPT-2


perplexity. And here is this Zlib entropy. And if you plot them one against another, most things


will fall on this diagonal right here with the giant blob around here for most texts of the


internet. And there will be a region where GPT-2 thinks this is fairly low perplexity, but Zlib


thinks the text is relatively high entropy. So these are candidates for memorization. And the red


and blue here are the ones the authors selected for checking. And the ones that are blue are ones


that they found or memorized from the internet. So a fairly high percentage, in fact, 67% of this


method that they selected was, in fact, was memorized. Though, as I said, you can see that


there aren't super many more, right? So this is all samples. I don't know how many, you know,


they could generate more, but you can see that it gets pretty sparse out here. Okay.


Yeah, so examples of memorized content, personally identifiable information.


They say there are several examples of individual people's names, phone numbers, addresses, and


social media accounts. Some of this is memorized content is just exclusive to a few documents. For


example, we extract the usernames of six users participating in an IRC conversation that happened


in exactly one document. Yeah, so I guess the question is, how often did the usernames appear


in that one document, right? And once the model sort of, and how, how does the user name appear


how distinct are these usernames from other usernames? Because if they're very distinct,


and they happen, you know, they have a long conversation, it can be easy to see that the model


will remember that not saying this is not a problem. I am telling you, the models, it's not,


it's not that they'll just randomly remember stuff, then it needs to be very specific conditions for


the models to remember stuff. So they say, we identify 50 examples of memorized URLs that


correctly resolve to live web pages. Okay, many of these URLs contain uncommon pieces of text,


such as random numbers or base64 encoded strings. Again, this this random element right here


makes it you can't extract a pattern. They say we identify 31 generated samples that contain


snippets of memorized source code. And they can actually extend that. So they can take these


snippets and they always, I think they do 256 token length, but they can extend that to sort


of verbatim recover the source code. And that's also you know, that's that's fairly interesting.


And unnatural text, yeah, these UUIDs. A Google search for this string identifies just


three document containing this UUID. And it is contained in just one GPT-2 training document,


okay, though, again, we are not seeing how often. They say table three gives nine examples of k


equals one memorize content, each of which is a random sequence between 10 and 87 characters long.


You can see the table right here. So these are examples of random strings that for some reason


appear in this training data in exactly one document. However, this string right here,


for example, appears 10 times. And this string right here appears 311 times. So again, it's a


random string that appears, though 10 times is fairly often for a piece of text to appear,


especially the same piece of text that is not pattern close to any other piece of text. It seems


okay that the model remembers that it seems expected, right. So yeah, here, they also say


that samples that contain two or more snippets of memorized texts that are unrelated to one another.


In one example, GPT-2 generates a news article about the real murder of a woman in 2013,


but then attributes the murder to one of the victims of a nightclub shooting in Orlando in 2016.


And this I found very, very interesting, right? Because that's exactly what I said GPT-3 does,


right? Especially. So in GPT-3, they have this example of GPT-3 writing an entire news article


about I'm not even sure about some pastors, some split in the Mormon Church or something like this,


or I'm I don't remember correctly, but I was able to Google that. And I did not find the verbatim


sequence. But I found that article that GPT-3 wrote many, many times in sort of different words


in written down in, you know, books and reported about and so on. So what GPT-3 did is simply,


I would guess interpolated between these things. And here they find the same thing GPT-2 just takes


two pieces of text and sort of finds that they're close and sort of interpolates between the two,


I would call this memorization too. And they say, yeah, there are this is memorized text,


this is not memorized text in their definition of memorized text. But it is right. So, so it sort of


mixes up different training data points together. And this, I think, is a strong,


it's very strong evidence for how these language models work in that they sort of take training


data points, and they just kind of mix them together. And they can do this in a grammatically


well founded fashion, they can also change individual words of a sentence and so on.


By the way, it doesn't mean that people are doing anything smarter, like there are arguments like


the best arguments I hear are, you know, people are kind of doing the same thing. They're just


kind of recount the training samples in there a bit of their own words. But yeah, this this I found


extremely, extremely interesting. And also, you know, what I found from GPT-3 with this Google


example was that the problem of memorization may even be way more way worse than what they analyze


in this paper right here, because they look for sort of direct, direct overlap in text,


where as they wouldn't catch strings that are sort of reformulated.


Again, okay, so here they they lastly they say, they can extend text and this thing here, I find


very interesting. So they say, if they if they put in this prompt 3.14159 GPT-2 will complete the


first 25 digits of pi correctly. Interestingly, when they input pi is this, it gives the first 799


digits. And if they say e is this, and pi is this, then it gets the first 824 digits correctly. So


they make the point here that the memorization problem could actually be much worse if you only


knew what prefix to input. So this strengthens my case for the future job description of a prompt


engineer, right? It seems to be that it's quite a sort of magical power to know what to input into


these language models to make them output what you want them to output in this context, but also in


the context where you actually want to do them. I want want them to do something useful. Right.


And here, here is where they investigate this number k. So you might have noticed and this is


a bit of the criticism of my paper up until this point. Yes, they have, you know, they have the k


equals one right here. And they sometimes say that it's only found in very few examples. But


essentially, they just they they they investigate this memorization here, pretty much in absence of


k of what they themselves defined to be problematic, right? They say, well, it's problematic if it only


appears in few training examples. But the the analysis here is done quite absent of k very


often. And here is where they investigate this. So this is also pretty clever that the the


experiments here are fairly clever. They find a they find a one piece one document a pastebin


document. So the pastebin document where that is sort of a JSON document, and it has lots of links.


And I found the documents that giant document, okay, and it's a giant JSON document with these


entries. So there's this entries, there is color and then link and then here, the URL would go on,


right? And it is the, in fact, the the only document in the internet, at least these these


authors claim that contains these URLs, but many of the URLs are repeated many times. In fact, here


you can see that these are the continuations of the URLs, right? This one, even though it's contained


in one document, it's actually repeated 359 times, and so on. So this is the the the the


one. So this is a playground. They say, okay, this document was in the training data of GBT two.


Here, we know how often each of these strings appeared in the document. So they can directly


make an experiment. How often does a string need to be present for the model to memorize it? They


simply order by the number of total occurrences right here, as you can see, and they ask each of


these models whether or not it has memorized the string. And they do this by inputting this. So this


is the input. And they simply sample if the model manages to output any of these URLs, they consider


that to be memorized. If not, then not. If it doesn't memorize it, they have a second trick that


if model can get half a point, if they input this first random sequence, I think they put six tokens


of this random sequence. And if then the model completes, then they say, ah, it has memorized it,


right. So you can see right here, it appears that the this large language model needs this needs a


string, let's say 20 times or higher for it to memorize it. And you can also see the trend right


here that if you go to the smaller models, they need a lot more in order to memorize them because


they have less weights, they can't afford to memorize stuff easily, right? They need to extract


the pattern. So they'd rather forget about the string incur a loss and focus on other training


examples. So yeah, two things in this direction, smaller models in this direction, larger models.


So that means that something like GPT-3 will have this problem much more pronounced. So that's the


bad news about this result. The good news about this result is that this is the case where you


have fairly random sequences, right? These, even you know, that if tokenizing this is not going to


be natural text, and there are these, you know, random, these Reddit URLs have these random


prefixes. So this is very much this sort of outlier case. It's a pretty clever case study to


find this document, I have to say, but it is sort of good news that this is not the usual case,


this is really the case that this data is very, very prone to being memorized, right? Because it's


not patternable. And it's very random. And yeah, so okay, so that was that was that.


As I said, the amount of hedging right here is is really, really, like, it's a lot. They discuss


what you can do with it, you can train with differential privacy, though that doesn't really


help, as we said, because some of these strings are included in, you know, more than one time.


You can curate the training data, which doesn't really help because the training data is too large.


You can limit impact of memorization on downstream applications. So if you fine tune,


but we don't know exactly what fine tuned models forget, and what they retain, or you can audit,


which is essentially what this paper paper right here does. And that seems like a that seems like


seems like a good, you know, the best strategy we have so far is is to audit these models. And


yeah, so I wanted to quickly check out also the appendix, the appendix here shows sort of these


graphs for the other methods. And it is very cool, if you want to, if you want to check that out,


and it has sort of categorization of what they find as these memorized pieces of text. But what


my main point was right here is that this paper shows a problem, let's say, with these large


language models, namely that they memorize certain pieces of training data. While that sounds scary,


I feel that the nature of the data that it remembers is very particular. So not you cannot


extract any piece of training data, the nature is very particular. It's the sort of outlier ish


training data points. And also, it very, very, very often, it isn't enough that it just is there


one time. So even when they say, this piece of information is only in one document, very often,


it appears many times in that document, that together with the sort of non pattern ability


of the data that it memorizes right here, actually makes me fairly, fairly optimistic,


more optimistic than I would have thought, honestly, about these language models.


Yes, so we'll see what the future brings. As I said, this is going to be more pronounced


in larger models. And this is not the only problem with these models, as my GPT-3 Google search in


that video shows. Alright, I hope this was enjoyable. Let me know what you think and maybe


GLOM: How to represent part-whole hierarchies in a neural network (Geoff Hinton's Paper Explained)

Hi, there. Today, we'll look at how to represent part-whole hierarchies in a neural network


by the legend himself, Jeffrey Hinton. He describes a system also known as GLOM, that


is a new approach to processing visual information using neural networks. And interestingly,


the paper starts off by saying, this paper does not describe a working system. So this


is an idea paper, Jeffrey Hinton's suggestion of how we should go about solving vision or


furthering vision in the AI community. He says openly, these are just ideas. Please


prove me right, prove me wrong, try them out, and so on. And I absolutely welcome this.


Idea papers is a thing that I think we have lost as a community because everything needs


to be state of the art and so on. This is super cool. And I encourage more people to


do it. I'm not saying you're going to have the same kind of success with an idea paper


as Jeff Hinton. He is banking on his name in large part with this. But nevertheless,


it's just an archive paper. Like I see people complaining, this would never be possible


if it wasn't. Yeah, it wouldn't like people wouldn't pay attention, but you're welcome


to write your ideas and post them on archive, like, or write a blog post, make a YouTube


video, anyone has opinions. So, you know, go ahead. Yeah, so to the paper itself, glom,


glom, as you can see here, glom comes from the stems from agglomeration is a system that


instead it presents a single idea about representation, which allows advances made by several different


groups to be combined into a an imaginary system called glom. The advances include transformers,


neural field, contrastive representation learning, distillation and capsules. glom answers the


question, how can a neural network with fixed architecture parse an image into a part whole


hierarchy, which has different structure for each image. The idea is simply to use islands


of identical vectors to represent the nodes in the parse tree. If glom can be made to


work, it should significantly improve the interpretability of the representations produced


by transformer like systems when applied to vision or language. That's the abstract, we'll


dive into the system, we'll see what it's about. I think I can actually make a suggestion


to improve it. But maybe I'm way behind other folks. So what is the glom system? And what


are these parse tree about? And why does it come combine all of these things? And for


that, we look at so it has two core diagrams here. This is the first diagram. This is the


second diagram. And at first sight, they have little to do with each other. So let me try


to go about it like this, if you have an image, and it looks at vision very much in terms


of you have an image or a video, and you want to parse the image into kind of a tree. And


the tree should be sort of like a tree of objects and their parts. So let's say it's


an image of a car. So the whole notion is very, very object centric. So this is like


my best attempt at a car. And a parse tree for this image would look something like this.


All right, so this whole thing here is a car. So that's going to be your top node in the


parse tree. The car has different parts, namely, it has this cabin, it has a motor, and it


has wheels. So that is going to be those are going to be kind of downstream of that parse


tree. Then the cabin itself is going to have two segments here, windows, and maybe here


is the door area. So that is going to be window, window, door, and so on. So you get that we


what we want to do is we want to look at an image, sort of create this parse tree over


here, this is very much into the into the area of go fi, good old fashioned AI people


that want to understand a the world in terms of their symbolic representations and relation


of the symbols to each other. However, what Hinton is saying is that if you simply do


this, it's it's, you know, you can't really do this with neural networks, neural networks


are continuous and so on. So what would you have to do? In addition, we know that the


brain doesn't reconfigure itself every single time you get a new input. So the brain, even


though it has some neuroplasticity, while you look at the world and do inference in


the world, the connection stay the same. So what we need to do is we need to come up with


a system that when we input one image, it can give us one parse tree. But when we input


another image, it can give us some kind of other parse tree, maybe now there are two


objects in the image. And this one has one descendant only, which in turn has two descendants,


and so on, you see the point, the tree structure needs to be different each time. This in part


was addressed by Hinton's capsule networks. So in the capsule networks, Hinton's idea


was sort of, okay, I'm going to have these capsules here in different layers. And I'm


going to have kind of lots of capsules and these layers, lots of capsules in these layers.


And I'm going over capsules, because it's kind of important here. So Hinton's idea with


capsules was that the first layer of capsules would sort of recognize the smallest parts.


So this would be kind of the wheel capsule. And this would be sort of the window capsule,


and so on. So there would be a single capsule for every part that could possibly be in an


image, right? You already see the limitations. Because if you want to recognize the whole


world, you need many capsules. But nevertheless, this was the idea. So a capsule would be active


if there was the given object in the image. And then the next thing here, this would be


kind of the the motor capsule. So the motor, motor capsule, and this would be the cabin


capsule, and so on. So the window would activate the cabin capsule, but the door capsule would


also activate the cabin capsule, and so on. And the wheel would maybe activate, it would


maybe activate, I don't know, the wheel should probably be here as well, wheel at this level,


would activate that. And then all of these things here would activate the car capsule,


sorry. So you can see that this parse tree here is generated dynamically, right? These


connections, this routing and capsules is generated every time different. So in the


next image, there could be a different object, different capsules are activated, different


things are routed together, the parse tree is different. However, you need these many,


many capsules for that every one capsule per possible part in the image. And that was just


infeasible. And also the routing was very cumbersome in these capsules. So here we go


with a new approach. And this new approach is what Hinton describes as the glom architecture


is composed of a large number of columns, which all use exactly the same weight. Each


column is a stack of spatially local auto encoders that learn multiple levels of representation


for what is happening in a small image patch. Okay, so we're going to build up some kind


of imagination here. At the at the bottom level, we have our image. So our image is


going to be lying flat on the ground, maybe you can see like this. And it is going to


be divided into pixels or small patches, whatever you want. But these are would be called locations.


So it would be divided like this into different locations. I am not good at perspective drawing.


In any case, above each location, there would be one of these columns. And these columns,


I can draw one here, these columns would sort of stack up like this. And these columns would


be divided into multiple levels. So there would be a bottom level, which would be this


there will be a middle level, higher level, and so on. Hinton suggests about five levels


should probably do. And every single level of this column tries to represent the location


at the image, right, this location down here in a different resolution. So the very bottom


level might be aware that there is a part of a wheel, like let's say this is actually


let's say this is a cat. I so here, there's probably Yep, yep. Okay, so you can see, there


is there is an ear or a part of an ear that stays there's a part of an ear in this location.


So the very bottom thing would probably represent something like the very structure of the first


layer. So the bottom thing would represent what's going on at you know, the micro level,


really the location level, the next layer would represent what's going on at this location


in a kind of a broader sense. So that might recognize that that that's an that's actually


part of an ear, right? So it goes beyond the location. If you think convolutional neural


networks, you're in the right ballpark. But we're going to implement this differently.


The next layer will recognize well, this location is part of a of a cat of a cat's head. And


then the next location will recognize well, this thing is part of a cat. So at this location,


there's a cat that there there is a cat at other places. But at this location, there


is a cat, and so on. So maybe we don't have more and this look at this particular image.


But if you consider a different column, like this, this column right here. And you look


at what's going on in that column, you'll see similar. So in the top layer, let's just


consider the cat the top layer in the top layer, it might say, well, there's a cat too.


But it's also part of it's part of a cat's neck, neck. And then here, it's maybe there's


a bunch of well, I don't know a chin. And there is also a fine first structure of the


chin. So you get the idea every column will build up these repre these representations.


And these are vectors. So these are embedding vectors. So at the bottom location, you'd


have the fur vector, and then this vector is the ear, whereas here over here, the chin


would be very different, it will be a different vector at the same layer. So the only thing


that agrees here is the cat vector, the cat vector in this top layer would agree between


both of these columns. I hope you get the idea, you have a column above each of the


locations, every single layer in the column represents that particular location, but at


a different level of abstraction and a different level of I don't want to say resolution, but


it, it would consider more and more of its neighbors. The question is, how does it consider


its neighbors? And how do you learn these things, right? So how do you learn these different


abstractions? And that's where these columns, they communicate with each other. So Hinton


imagines that this is a process over time, where the columns iteratively communicate


to each other. And within the column, the layers communicate to each other. And this


is one of these first diagrams right here. So this is one single column over time, okay,


this is this would be the this would be the fur at the ear, this would be the cat's ear,


and this would be cat. Okay, so the information that so the embeddings are updated by sending


information around every single embedding, which means that every single vector at every


single layer of every single column is updated by simply averaging four things. So we have


the embedding at layer l, at time step t plus one is going to be sorry, at layer l location


x is going to be a sum between the four parts, the four following parts, it's going to be


the embedding at the last time step, right. So this is sort of a recurrent neural network.


We the new embedding is the old embedding, plus, it's going to be a function at a top


down, this what Hinton calls top down function of the embedding at the same location in the


previous time step at one layer above, so l plus one, it is also going to be receiving


information from the upwards, I think bottom up, because the bottom up embedding of layer


l minus one at the same location at time step t. Alright, so this way, that's what you can


see right here, the green arrows are each level each layer simply passes information


to the next time step, this is if any if nothing else happens, you just keep your embedding,


then each embedding also sends itself through a neural network, one layer above itself,


that's the blue arrows. So the blue arrows here are these and you every everything is


a neural network here every arrow except the green ones, but the green ones could be too.


So every arrow is a neural network. So this is a neural network sending information above.


And this is intuitive, right? So the ear embedding would sort of send information about itself


like saying like, Hey, I'm a cat ear sends it above and it goes through a neural network


because it needs to be transformed. The neural network has to learn well, if it's a cat ear


at that level, it might be a cat at the top level. And lastly, every single layer sends


information down and that is the red arrows right here. They're also neural networks.


So the cat ear says, Well, I'm a cat ear. So downstream of myself, there might be, you


know, some first structure, right? So all of these embeddings, they try to predict each


other, they try to predict the neighbors of themselves. And Hinton's idea is that by aggregating


over time, they will sort of reach a consensus of what is in these columns. There are a few


things missing right here. The one thing that's missing and hint and pointed this out that


all of these different columns that we've drawn, they use the same weights. Okay, so,


and he discusses this at the end of the paper, it's not really biologically plausible, but


there's an ensemble effect, we won't go into that. But all these decent so the blue arrows


are always the same for each time step. But not necessarily the same between different


layers. So that might be this f might be different from this f down here. However, the function


passing information from from layer l to layer l plus one is the same in every single column


across the image, it's a bit like a convolutional network in terms of weight sharing. So you


can imagine it as one by one convolutional network in that sense. But except the information


does not only go up the layers, it also goes down the layers over time. As I said, this


is an iterative procedure, goes up, down, and laterally. The second thing is now that


you ask, Oh, well, if every single column has the same weights, wouldn't that simply


sort of how how can you localize any information? And the answer is that you have a side input,


like in a neural field, you have a side input annotating each location, basically a positional


encoding, honestly. So in in addition to what the image patch looks like, you also get your


kind of either your x y coordinates, or you could also get your relative coordinates to


some other coordinate frame in there. And so the network knows where it is. And that's


going to be important, because what Hinton wants to build are these islands. So the imagination


of Hinton is that this is going to be somewhere in between like after time step 10. And you


want to run it for 100. And he imagines that there will what will emerge are these sort


of islands. So imagine the image is now a 1d vector down here. Or you can imagine these


columns in 2d, whatever fits, you know, whatever fits your brain better. But imagine the images,


the image is simply a 1d line right here. He imagines that the bottom vectors, they


will just, you know, happily kind of be describing whatever that is at the very bottom level.


But then at the next level, once it goes to sort of higher resolution or lower resolution,


higher abstraction, there will be there must necessarily be vectors that are the same.


If the system works and look at these two vectors and look at these two vectors, they


are the same because they now describe objects that are larger than one location, right,


the cat's head is larger than simply one location. Therefore, at the layer that represents the


cat's head, you expect because these are all all neuron, all the up and down functions


in the same layer have the same weight, you expect that the embedding of a cat's head


is the same in in the different columns. This is if the system works, this must be the case.


And then as you go up, you expect more and more of these what what hint calls islands


to emerge, right? So they they agree. And the idea the idea between all of this message


passing is that over time, all of these things kind of reinforce each other. So we looked


at a column before, and we maybe said, Okay, so this vector down here, it gets information


from the top, saying, Hey, you know, there's a cat here. So you might be like a cat ear


or a cat eye or something like this. And then it gets information from the bottom saying,


well, there's a bit of as you know, fur here, and there's some cartilage showing and so


on. And it has already sort of figured out that it might be an ear. And these informations,


they own they reinforce itself now, like they'd be like, okay, you know, you're saying I'm


part of a head and you're saying there's a bit of fur and cartilage. And I already kind


of noticed that I'm a bit like an ear. So I'm probably more an ear. So the idea is that


over time, you have this consensus algorithm, there's one thing missing. And that is, how


do the different columns communicate with each other. So I said there are different


parts, there is one missing. And that one missing is going to be, I'm just going to


call it whatever a and a is going to be an attention mechanism across all the other columns


at the same layer. So if we look here, this cell receives information from above from


below from itself. And also, in an attention mechanism way, it's going to receive information


from all of the different, all of the different embeddings at the same layer. You can see


that, you know, it puts in everything we got in here. Now, the attention, he says, is easier.


And so these are the four parts right here. At each discrete time, and in each column


separately, the embedding at a level is updated to be the weighted average of four contributions.


The prediction produced by the bottom up neural net acting on the embedding at the level below


at the previous time, the prediction produced by the top down neural net acting on the embedding


at the level above at the previous time, the embedding vector at the previous time step,


these three we got, and then the attention weighted average of the embeddings at the


same level, right at the same level in nearby columns at the previous time. So nearby heat,


sorry, he later backpedals a bit, I think on nearby and what nearby exactly means. And


he some parts. So this, this is idea, I think this is still up for debate. And this is,


I think, where I can help. But what he wants to do is he wants to aggregate, he wants to


attention aggregate, and he wants to simplify attention. So instead, what we usually have


is we're going to produce queries, and keys and values, queries, keys and values. And


they're all going to be different functions of our input. And then we're going to do query


times key transposed softmax of that times value. And that is going to be our attention


mechanism that allows you know, arbitrary information to be routed around and so on.


Attention says, Nope, what I want is simply that all the queries, the keys and the values,


they're all just equal to the embeddings themselves. So the attention mechanism would work out


to be the softmax of x times x transposed times x. And what that does is if you yourself


are the query, and every vector also itself is the key, what do you attend to, you attend


to vectors that are very similar to yourself. And you can see that in Hinton's diagram,


the one we circled dark blue, what would it attend to? Well, it would probably attend


to its left hand neighbor, the one you can see circled, I'm going to circle it. This


one, it will probably attend a lot to this one, it might not attend so much. And the


ones over here, it might not attend at all. What does this give us, especially since the


values are also these vectors, this is a consensus algorithm, it is not meant as a way to pass


information around, it is not meant like in a transformer as a way to do computation,


because we have no trainable weights in this process. It is simply meant as a consensus


algorithm. So it imagines that by doing this, by sort of attending to things that are similar


to you, and then integrating their values, there will be these islands forming. And that's


what you see right here, you can imagine if two vectors are already close at the same


year, this mechanism will make them even closer. So this is a sort of a clustering algorithm.


And so that my question is, that these drawings, you look at them, they are very specifically


constructed, they are constructed such that a parse tree is emerging. So when you look


at this, you have a clear sense I can probably, I can probably move all of that crap out of


the way. You can see the parse tree, right? Because the black thing is going to be the


top node right here. Let's leave away the scene level embedding for now, the black thing


is going to be the top node. And then it has two child nodes, this one, and this one. And


then it has four, every one of those has two child nodes. But it's not it doesn't have


to be in this case. So this dynamically and every one of them, you know, the black ones


are individual. This is dynamically constructing a parse tree, right? The parse tree here


is something like this. And then the da da da. So this is pretty cool. But it is also


drawn deliberately such that a core problem does not arise. And the core problem would


be something like, well, what if this vector here was actually also pointing like this?


Okay, so it is not in it is not in the same, it is not in the same area of the parse tree,


right? If you go down the parse tree, it is actually here. Now, if we do what Hinton says,


and if for this vector here, we do this aggregation via attention on the same layer, what we will


attend to is this vector over here. Now, this is probably not meant to be because this vector


over here, it can represent the same thing. But you can see it's not in the in the same


path of the parse tree. And he mentioned this a little bit throughout, but not necessarily


clear. And the drawing makes it seem like there's no problem. But I hope you can see


how this is a problem. The attention would pull in information from over here. However,


the whole parse tree here and the island on the top layer suggests that these two things


should be parsed independently from each other and therefore also processed independently


from each other. So here is my suggestion to to extend this and maybe Hinton's already


thought of this, but I would suggest that the this attention mechanism here is modulated


by how close two things are in the parse tree. Okay, so what would that be? So for a given


a given vector, it would be how much do you attend to this vector right here? Well, a


lot because it agrees with you, right? It you know, this the softmax of the inner product


would be high, it agrees with you. And also, it is in the same, it is same branch of the


parse tree. So that's perfect, right? This one right here doesn't agree with you, but


is in the same branch. So it could potentially later agree with you through a consensus algorithm.


However, this one over here, I, you probably shouldn't attend to that too much, even though


it points in the same direction, because it's in a different branch of the parse tree, you


shouldn't attend zero to it, like because these branches on top, they could change.


And you know, by you sending information there, this one could change the the top structure


here that could agree more with your branch of the parse tree and so on. So my suggestion


would be that let's not only get the softmax of the, let's not only get the softmax of


the current layer things, but let's do x times and here we're going to have a sum. So this


is going to be k. And let's say we're at we're at layer L. And this is layer one, this is


layer two, this is layer three, going to number them from the top, actually from the bottom,


layer m, layer m minus one, and this is layer L, I'm I suck at this. So from the current


layer, I want to go up the hierarchy until layer one. And I'm going to take the softmax


of the representation at layer L, at layer k, where I'm at x k transposed, like this.


What we aggregate is still the the values on the current layer, but how much we should


attend to that should be dependent on the parse tree. And we do that like this. And


maybe we have like a kind of a lambda k, L minus k, L minus k, I hope you get what I


mean. So how much how much you aggregate this sum here, the sum here is weird. This should


go probably. Hi, it's future Yannick. And I just wanted to write that down again. So


because I've made some mistakes. Obviously, the sum here should be within the softmax


because you want to have aggregate the distributions in log space. And the softmax should still


be valid, you know, distribution. And then the lambda is exponentiated by k and k now


properly runs from the zero to all the way up the stacks. So big L would be the total


number of layers and little L would be the layer where you're currently at. And you can


clearly see that the contribution of these attention matrices, it is so lambda would


be something smaller than one. And therefore, the contribution is in the current layer is


the strongest, but also in the next one up is a bit weaker than one more up is even a


bit weaker, and so on. So you'd still have essentially the same mechanism as Hinton is


suggesting controlling for the fact that things are in different branches of the parse tree.


All right, back to classic Yannick, who is thoroughly confused by these things. Yeah,


I'm not good at I'm not good at coming up with math on the spot. But I hope you can


see what it's doing. So it is if if you simply take the first K, you would simply stay at


that layer and it would be what Hinton said. But what I'm saying is you should also consider


how much your top your higher layer, one layer up from you agrees with one layer up from


the thing you want to attend to. So you also compute that inner product between between


the embeddings, and you add that to the softmax distribution. So initially, the softmax distribution


would be like you should attend to this thing and this thing, and this thing a lot. But


then the next up hierarchy would maybe say, Well, we agree, because you know, these are


in the same thing. But this one, maybe not so much. And you would add those together,


maybe with a lambda factor in here, and then you go one layer up, and it would say, Well,


okay, everything over here basically agrees, right? And here, but everything over here


basically doesn't agree. So you would add that maybe with a lambda squared, as you go


up the layers, it would be less and less important, but still, you'd consider it. All right. Now,


if this is going to work out, cite the channel. Now back to what Hinton says, this, this is


actually the system. This is the system. As in a nutshell, you're going to input the image


at the bottom. And Hinton says you could use like a conv net at the very bottom to get


it into the columns. But then you're going to every time step pass information up the


columns down the columns, and between the same layer of the different columns. And that's


going to, in some point, this is going to stabilize, I don't know if it has cycles,


it probably doesn't have cycles, this probably does not have cycles. So at some point, this


comes to an end. And if that comes to an end, it should be that the object level embeddings


agree on an object, the part level embeddings agree on what parts there are the sub parts


agree, and so on. And they form these islands, these islands give rise to a parse tree. And


the parse tree can tell you what object is there, what is it made of? And where are these


parts in the image and so on. So exactly, that is it. And now, we're going to look at


what Hinton calls some design decisions. How many levels are there? About five. Okay, we


can skip that. How fine grained are the locations? Hinton says, you could be as fine grained


as pixels, or they could correspond to larger image patches. You and he says you could do


convolutional neural network to get it in there. Does the bottom op net look at nearby


locations? He says, yes, the bottom op net. So this this is not the attention network.


That's the bottom op network, it could look at nearby locations. But Hinton imagines that


if you have bottom up, top down, and if you have attention drawing in for information,


and if you maybe limit that attention to a neighborhood, then then the the attention


will do the job because you can have instead of looking at neighboring locations in the


bottom up network, you can simply in two time steps, aggregate that information. So you


can do bottom up here, bottom up here, and then using the attention, the lateral mechanism,


you can pass that information around this way. And also, it is not as biasing the network


to the immediate neighborhood. So the attention mechanism can sort of look farther, which


conflicts with what he's saying on top that the attention mechanism might only be looking


at the neighbors, I think there are different possibilities here. And only looking at neighbors


is actually one of the solution to the problem of having, you know, kind of similar vectors


at very distant locations at down the levels. But I think it's not as as good a solutions


to simply look at how close things are in pixel space, because even though things are


close in pixel space, they might be far away in the parse tree space. How does the attention


work? We've already looked at this. So the way that one location attends to another location


is going to be the softmax of the inner product between the embeddings here. And the values


are also going to be just the embeddings at layer at that layer. The visual input, he


says convolutional net could be used. Color and texture. He says, he makes he gives this


example, like if you know, if an object is entirely pale, or entirely green, or entirely,


I don't even know how to pronounce this, the color of a part is straightforward. But what


color is the whole object. So this entire notion of capsules, by the way, Hinton imagines


this as these embeddings represent kind of properties of the object so that the cat ear


embedding represents not only the fact that it is a cat ear, but also different properties


about the cat ear and even its location in the image is in the embedding. And, you know,


we know that transformers, they must be doing something like this, because we feed in positional


embeddings, for example, at the very bottom, and it can still, you know, compute things


in terms of positions. So that's the there's an intrinsic connection between kind of capsules


and the kind of transformer architecture. He says, one of the motivations of glom was