more interference when you train. Or if you have not enough neurons whose activation crosses


Active Dendrites avoid catastrophic forgetting - Interview with the Authors

the threshold, you're going to get that phenomenon, which you're showing on the screen right now,


on the left side where you have a drop in accuracy because your representations don't


have enough capacity. So that's why we opted to go for a fixed value of K. But even if,


you know, we didn't have, even if we did have a threshold, I think one of your critiques


were here, you know, now we have another hyper parameter K that we're choosing. In the other


case, I mean, we'd have to, our hyper parameter would just be the threshold value there, right?


Obviously, yeah. So to me, this this continual learning setup is very cool and you can generate


data very easily using this permuted MNIST. But there is a bit of an issue that I have.


And that is that if I use permuted MNIST, there is another thing, there's like all the


tasks are like the same difficulty, right? They're essentially the same task. It's just


permuted. So I need to learn, yes, I need to learn like a different function. So this


would be the permutation identity and then the pixels are permuted somehow, right? So


all the tasks are kind of the same, right? Which warrants a static network architecture


and every context vector is kind of the same length, right? And all the dendrites, they


can sort of specialize in each of their little task recognition. What would change here?


Or is this a drastic requirement to your architecture? Or do you think if many of the tasks were


wildly different from each other, and you have this a little bit in the robot example,


so what can you tell about when tasks are very different in their difficulty, maybe


in their amount of training data? Like how do these things influence an architecture


that's targeted towards continual learning?


In our case, I think there might actually be similarities between different tasks. And


so, for example, in this case, in permuted MNIST, right? There's a certain, certain pixels


are more likely to be white and certain pictures are more likely to be black depending on the


permutation. So maybe two different permutations could have more overlap in terms of which


pixels are white, which pixels are black, or they could be totally separate. And if


they're more similar, if the permutations are more similar, then we could expect that


the subnetworks that are selected by the dendrites will probably have more, are likely to overlap


more in which neurons become active, since there's probably a lot of similar computation


going on. But of course, in that case, difficulty doesn't really change at all.


I think to add on to that, I think a lot of it depends on the quality of the contact signal,


because ultimately, that's the part of the network that indicates to the active dendrites,


what kind of task you're solving, how similar is it to previous tasks you might have seen


and things like that. So I think that in this permuted MNIST case, the way we're computing


the context does allow for this property that Karin just mentioned, where if there's some


overlap in the input space, then the contact signal for that will demonstrate this input


and perhaps allow for overlapping subnetworks to emerge. Whereas if you have wildly different


tasks, which is something we see more in the robotics environment, then these context signals


can differ more and indicate that the subnetworks must not overlap.


I think it would be really interesting, and we've talked about this before, to try a similar


setup in a continual robotics learning case where you have a streaming set of robotics


tasks. And I think that would probably be a super interesting study to do and something


that hopefully we will try at some point in the future.


So I had some observations with respect to your experimental setup. It's very cool that


you do two different things, but there are also noticeable differences on how you implement


the two different tasks. In the first task, you give the task ID directly. In the second


tasks, you do this prototyping approach, which is a more advanced approach. Can you tell


a little bit about how, is there like a reason why, because I could also imagine you just


give me the task ID in the second task, or I do the prototyping in the first task. Is


there like a research process reason? Like did you find that some things did work or


didn't work? Or how did this come about that all of a sudden we're introduced in the new


task or introduced to this new way of detecting the context?


I think in the context of the multi agent, like, sorry, the multitask reinforcement


setup, like the environment setup itself gives the task ID. And I think that the concept


of multitask learning itself is more focused on if you have different tasks, which may


conflict with one another in terms of the types of behavior you have to do or the types


of predictions, how can you mathematically still optimize your joint objective function


without and still be able to perform well on all the tasks. And the problem shifts not


so much from trying to infer what tasks you're doing to more, you know what tasks you're


doing and you want to try to do all of them. How can we optimize this joint objective?


And this is kind of the way we use this one hot task encoding is in line with past works


that deal with multitask learning and multitask reinforcement learning, where you have this


like one hot task encoding that is provided. I do agree that like the one hot encoding


is quite convenient and a little bit arbitrary. You can probably use like a denser representation


for each task or try to infer it. But I think for the purposes of our experiments, this


one hot encoding seemed simple as it was environment provided and kind of like the point of the


multitask setup was to again, like try to show that this network architecture prevents


from like conflicting updates across tasks and avoids this like interfering updates from


occurring. I think for continual learning, the kind of the kind of setup of the problem


itself is a little bit bigger and that you have to, you're not always provided with the


task IDs and you have to infer this on the fly, which again, I think Connor can talk


a little bit more about.


Yeah, in continual learning, there are a couple other recent papers that have come out in


the last couple of years and they're not providing task ID and the model actually needs to infer


the task ID as it does some sort of modulation or whatever their technique is. So we thought


that makes the problem a bit more challenging, a bit more interesting. So since we are working


on continual learning and comparing to some of these other methods, let's also try to


infer what the task should be.


So if I hear this correctly, it's very much inspired by the environment itself, like what


the problem is supposed to be and not like, because if I see something like this, I always


have the vague suspicion that people try something and it didn't work and it's like, well, let's


try something else. But there's also, I mean, I don't want to infer that. So it's always


good to hear like, OK, this really came about through the environment. And I mean, it would


be equally cool if it was the other thing. But I'm just always interested to hear so


I can adjust my priors.


What I think is just to add really quick. Sorry, just add really quickly, I think in


the reinforcement learning setup as well, because the state space is like similar, is


shared across all the tasks, because essentially it's hard to infer from the states what task


you might be doing if you weren't given such an ID. And the only information you would


have is like the reward signal. And that might not be enough to infer what the task is. So


like giving a task ID as part of the task.


Given that it's at the end, right?


Yeah.


It's like, you know, you do something and then you get like a reward and then you find


out what task you just did. Like that's OK. I agree with you. That's really not helpful


at all.


Also I think one thing to add here is that we did try a couple. So I think this is something


you pointed out in your intro where the task IDs that we're using are one-on-encoded, right?


At least for multitask RL. And that means that all these tasks are entirely orthogonal


to each other. And it really doesn't reflect how similar one task is to another. And it


really doesn't also reflect how different one task might be from another. So one thing


that we were experimenting with, I think we mentioned briefly in the paper, is that we


tried having an embedding layer that effectively embeds this one-hot encode into some other


higher dimensional representation and using this instead of that one-hot encode as a context.


And I think what we eventually found was that using the embedding or not using the embedding


produced fairly similar results. So we just decided to remove it for simplicity's sake.


But one thing to note is that using the embedding allows you to represent contexts, I think,


that are a little bit more nuanced in the sense that the embedding, since it's trained


via end-to-end backprop, any task that is similar to another task would have a shared


representation in that higher dimensional embedding. And ones that are really separate


from each other would likewise correspond to huge distances apart in that higher dimensional


space. But the one-hot encode is entirely orthogonal from each other, each task, but


it still worked out pretty well compared to the embedding.


I mean, yeah, and if it gets more complicated, I think you could put entire sub-neural networks,


instead of that, even that embedding layer, you could have nonlinearities inferring sort


of more complicated task embedding or task relations. It is interesting, though, with


respect to the context itself, you learn these things, all of this through backprop. And


my question, I think I brought this up, is would this be like a candidate for maybe unsupervised


pre-training that you sort of maybe collect episodes or something in your multitask RL


and then just sort of decide based on this, you know, how do we structure our dendritic


segments in order to recognize the context, maybe some sort of contrastive objective or


anything like is this something that came, I just blurt these things out when I do the


reviews, right? I never know if they're entirely stupid, or if people have thought about it


or and discarded it. Is that something that is a candidate?


I don't think it's something that we considered. But an interesting thing to note is that if


we did use this for some kind of unsupervised pre-training tactic, is that when you're actually


fine tuning the network, your context vectors are different. So that's something I think


that would be the most important nuance to investigate. I personally don't know how well


that would work if we trained on a set of contexts that are different during the unsupervised


portion and then use a totally different set of contexts during the fine tuning procedure.


I would imagine that doesn't work well. So, yeah.


To add on to that, I think, yeah, kind of like when I heard you say that in your review,


it was quite interesting. I think from the perspective of reinforcement learning at a


high level, I don't know if this will work out, but it would be quite cool to see if


you can train these dendritic segments to either produce, if you can train them to recognize


different contexts, and maybe guide exploration in different ways based on the context in


an unsupervised manner and maybe like, do different things in different contexts as


an exploration strategy. I think that'd be super cool. Again, I think the challenge there


would be to come up with a clever way of generating contexts in an unsupervised way. So I think


that would be an interesting area of investigation. How do you come up with context signals in


an unsupervised manner? A contrastive approach might be cool there. And given these contexts,


how do you train these active dendrites to modulate neurons to do what you want it to


do? And I think thinking about that in the lens of exploration in RL could be quite interesting.


Yeah, you could sort of even prepare for contexts that you hadn't considered before, maybe new


instructions in a familiar environment or something like this. You have this notion


of prototyping to recognize the context, which I found very interesting because it's sort


of an, it's kind of like an unsupervised online way even, like as the data streams in, you


create these new prototypes and so on. And sure, there's some hyperparameters, but I


think my main concern is that just taking the average of the samples as they come in


right here, it's going to work for something very simple, like permuted MNIST or so. But


this gets to its limits very quickly, right? If I think about ImageNet classification


or so, it is quite limited. How can this idea be extended to, let's say, arbitrary complexity?


Like what would I have to do with this online prototyping approach to make it usable for


more complex problems?


Hey, look, I think you're absolutely right that this technique only works with something


like permuted MNIST where you get really good task separation through just averaging the


examples from a single task. And that's why it works so well here, right? We actually


evaluated how well this clustering procedure works and it's like, it works pretty well.


It's not misclassifying things when it's clustering the prototypes. But if we want something that's


a bit more general and can apply to other domains, like ImageNet, as you mentioned,


I think something along the lines of self-supervised learning might help there. That way, you're


trying to build a context vector that is going to provide you sufficiently good task separation.


And it's not as simple as just averaging. Does that get at your question?


Yeah, no, absolutely.


And I think also in like meta-learning literature, there are prototyping methods that maybe like


process the raw input into an embedding space and then do clustering similar to what we're


doing there. So I think that would be a quite simple approach that is similar in flavor


to this one, but kind of embeds the raw input like an ImageNet input into some better, clusterable


space.


Another thing I noticed, and this is a minor thing, but here you feed the context signal


into both of your layers. And in the experiment before here, you draw this very, very accurately.


You feed the context signal into only one of the layers, so it doesn't go in here. Is


there a particular reason behind the choice of this?


Yeah, so there's a bit of background regarding this. I want to say first that the continual


learning and reinforcement learning projects started out as separate areas within Numenta.


And the goal for this was really to see if the same principles of the same model could


work equally in both of these areas. So while we did modulate both the layers in continual


learning, the intuition for not doing so in reinforcement learning was a bit different.


It was that the first layer should contain all the shared information the model needs.


And you could really do this without activating any specific sub-networks. And that the second


layer would then activate the context-dependent sub-networks for each task. But you're absolutely


right that we could have tried doing in-depth experiments where we modulated both layers


for the RL setup. I think we started doing that at the beginning of this project, but


we found it worked reasonably well. But because of the time and computing constraints of running


each of these RL experiments, we decided to stick with the original plan and really pick


a few key experiments and key architectures to run and really leave the ablations for


the continual learning experiments, which are really significantly faster to run. But


you are absolutely right, though. We just went off of our intuition on this one.


I mean, I don't want to like this is it's just my reviewer to popping up and be like,


hey, you know, but it's good. I mean, it's even interesting to see that this is kind


of a convergence of projects. Could you tell us a little bit more about just the research


process? You already talked about how this came to be, but like the process of researching


this, it's kind of a new thing, right? You propose a new architecture. The tasks are,


let's say, not that mainstream. People work on them, but they're not super mainstream.


Like is it was it like smooth sailing from beginning to end, like stepwise improvement?


Or was there points that just didn't work at all for a long time? Are there entire avenues


that you discarded and didn't didn't end up working out? Like, could you, I don't know,


let other people I don't know what you what you can or want to disclose, but it's always


interesting to hear, you know, what also didn't work out during a project.


Yeah, I can I can start off, you know, when we first tried implementing some of these


ideas behind dendrites, you know, you noticed that, you know, we talk about this, this that


we're picking the maximum dendritic activation, and then and we're using that to modulate.


But actually, you know, it was through the process of trial and error that we realized


that we were, you know, we were just working on an initial toy task. We weren't, we weren't


working on continual learning back then. We found that, hey, we actually can't, we actually


can't turn things off. We can only turn them on because you are picking the maximum value,


right? So how do you get how do you get something that's super sparse? So we actually want to


turn things off. So we're like, oh, okay, let's go back. And let's actually not just


pick the maximum, but pick the maximum and keep the sign. So if something's really negative,


we're picking that. And so there's a whole appendix section. And that's actually the


detail of how that that's in the details of how we're actually implementing this through


a bit of trial and error. And then also with, you know, picking the product, going back


to the prototype, you know, for a while, we were thinking, well, you know, how can we


get something that really provides sufficient task differentiation? So we tried a bunch


of different things. You know, we just like, just like Avi mentioned, he had a linear embedding,


which is created from his context. We also had one for continual learning, but that didn't


really work too well either. And we ended up settling, converging on something that's


really dumb and simple for primitive dumbness that ended up working out. Yeah.


There's actually, just based off of what Karin was saying, if you go to figure 11, I think


you had some points there as well. It's a visualization if I remember correctly. Yeah,


this one.


11. Yeah.


So if you notice, like we use the exact same gating technique for both continual learning


and multitask reinforcement learning, and that's the absolute max gating. So you're


picking not only the absolute max, but you're retaining the sign. And what you'll notice


is that the initial intuition for doing this was that, as Karin just said, is you want


to give each neuron the ability to either turn on or turn off. And it's very interesting


because if you look at the results in multitask RL, you can see that for neuron B at least,


you see some negative activations, those red squares that you see. So that's effectively


the neuron being told to turn off. It's the exact opposite of a positive, a strongly positive


activation. But I think what's something that's very interesting to see is at least for the


two neurons that we've showed for continual learning on the right hand side, you don't


really see that happening. It's either the neuron doesn't receive high magnitudes of


activation or it receives really high magnitudes, but it's all positive. So something interesting


to note that we were even in the multitask RL part, we were working, trying to understand


would max gating work better than absolute max gating in the sense that do we want to


discard the sign or keep the sign. So yeah, there's a lot of, in the beginning, there


was a lot of trial and error process. In multitask RL too, we had a good amount of time spent


on understanding what the right sparsity levels were to apply for the weight sparsity in the


feed forward layers. What we saw, I think is also pretty sort of, it's intuitive. If


you really increase your sparsity level to a really high sparsity, there's just not enough


information in the network to keep training and your accuracy sort of plummets. But something


that's interesting to note is that there's always a sweet spot for sparsity. And once


you reach there, that's when the accuracy is the best.


And how do you debug these things? What is your main method? Is your main method mainly


setting a parameter and then running things? Or what are good ways to like, are there good


ways to peek inside and what's happening? What are things that you look at to debug


something like this? Like, oh, we are not sparse enough, or we're too sparse, or we


don't turn off neurons or something like this.


I think diagrams like this, which you have on your screen are a perfect example, visualizations


of how the dendrites are behaving. So I think there was at one point early on, here you


have in both cases after learning that different segments are responding to different tasks,


different contexts. But there are cases where this early on where these diagrams looked


exactly like just really just horizontal bars, right? So you have the same segment that's


just winning all the time. And so we realized, okay, well, this is not right. We don't want


the same segment to always win. So that helps in identifying, okay, this is why the network


is failing.


And so we go back.


You would look at these things even during your research process. It's not just something


that you made after the fact, just to demonstrate to the readers.


Yeah, yeah. Oh, yeah. This was a very helpful tool for debugging.


Cool. I mean, that's really interesting to hear, right?


A lot of the architecture decisions that were made in continual learning were used in multitask


RL, simply because I think that each multitask experiment took 25 hours to run plus easily.


So it was really hard to change a parameter, observe how the results and visualizations


looked and sort of edit from there on. So a lot of the intuitions that we got in RL


came from Karan's continual learning experiments. So that was nice.


Did you ever compare these things to, well, it's not too easy to compare, but sort of


a baseline, because there is the danger with these things that you kind of interpret. I


think I said, well, couldn't this be just like the difference between the top and the


bottom? Just be, you know, one is at initialization and one is trained and maybe has not much


to do with sparsity. Did you ever compare this to something that isn't explicitly sparse


or anything like this? Is there something you can say as a reference point?


Yeah, so there's two things to note there. The first is that at least for this visualization,


the activations are normalized with respect to when they were trained. So I think you


mentioned this in your intro as well. You said that could it potentially be that you


have really high activations in the beginning and the area that you've circled there in


purple, it just sort of gets dimmed down. And I think the important thing to note is


they're all normalized. So the range of values between the highest activated neurons are


much higher than the lowest activated neurons after training than before training. But to


address the second point, I think that's regarding figure 10, if you scroll up. And that was


why don't we have like a baseline for this? Is it really that the active dendrites networks


that are creating these hyper sparse sub networks? And to that, you're absolutely right. We should


have had a nice diagram here that also showed how this would look in a baseline MLP. You're


absolutely right. That's something that we could definitely include.


I mean, I totally believe you that it's like very sparse. It's just that it's not obvious


from a diagram like this. Like what should I expect? Cool. Yeah, there is one other thing


in that. By the way, I have mad respect for you for including the graph on the right.


Like mad respect, like 90% plus of researchers where they try something like this specifically


because no one would notice if you leave this away, right? No one comes to you and says,


well, okay, maybe someone comes to you, but no one would seriously miss adding the SI


to both of these things. And you, you know, at the left, you beat them very clearly. So,


you know, huge respect for including that. That is, I think, to be commended and to be


highlighted. I think, you know, when we present a new architecture like this, you know, we


really want to show the community that, hey, we can do things like continual learning with


our more biologically inspired ideas. And it's competitive with what's already out there,


right? So even if we're not beating the state of the art, I think that that's perfectly


fine. Even though, you know, nowadays a lot of machine learning has turned into this competition


of getting the best numbers. And if you don't have the best numbers, apparently that means


you won't be able to publish anymore.


Yeah. To add on to that, I think the purpose of this paper is really something I said that


we all said in the beginning. And now it's, we really want to show a proof of concept


for this completely novel architecture where the goal is really not to get state of the


art architecture on either of these benchmarks. It's really about the promise of something


new, something I think that deep learning has been missing for the past, what, 10 years


or so. So, yeah, it's exciting.


And the last thing maybe we can get into is this comparison to other networks, because


you very clearly address this in like a paragraph. And I think, wait, I have like even a transformer


diagram somewhere. You clearly address this in a paragraph saying like, isn't this just


equivalent to like a bigger network? And I try to myself also to come up with, you know,


is there some way I could do the multiplication in like an MLP? And I'm fairly convinced there


isn't. But there is a connection clearly to like LSTMs, which do modulate things with


like forget gates and so on. They even have sigmoids, right? So they can they can module


model this, this on or off, and also sparsity to an extent. And I also think that a transformer


could conceivably like a two layer transformer could conceivably model the interaction right


here. Did you explore at all, like the interconnect, the connections of sort of this active dendrites


framework to other models? Is there something you can say about that?


I definitely think that these are great observations, by the way, that the kind of relationship


between attention and transformers and like the gating and LSTMs and GRUs, there's definitely


a relationship between those mechanisms and what we're doing here. I think in our research


process, we definitely thought a lot about how this gating mechanism could be related


to like things like multi headed attention, where basically you're doing a similar thing


where you're matching keys and queries as vectors with an inner product and then using


that as a way to see what parts of a sequence, for example, to wait when you're considering


a certain position. I think the key difference in terms of I think the similarity is that


for in the specific instance of attention, you are using learned weights to match a given


input. So for example, in our active dendrites, you're matching the context with the set of


dendritic segments and in attention, you're matching like the query vector with a set


of keys. I think that the key difference is that the purpose for which it's done here


in active dendrites, you're looking at a specific neuron and you're saying, OK, given the context,


is this neuron relevant in transformers? You're saying, OK, here's a position. What context


around me in terms of the in terms of the sentence, for example, is relevant for me


and how can I wait certain aspects of it? So I think it's a little bit like flipped


in how an interpretation like of the focus kind of shifting to the LSTM aspect. I think


as a mechanism, it's quite similar in that the LSTM is actually like turn off or turn


on certain units themselves to carry forward in time. I think. Yeah, exactly. That's what's


done here. I think the difference is now like focus more on the sparsity aspect of it.


In LSTM, you're trying to you're doing like a weighted sum between what's in the past


and what's current and saying, OK, let's pass this forward. And there's no aspect of like


using this to enforce a level of sparsity. Here we're saying, OK, let's turn off certain


things and do that in order to remain sparse and pass forward this information. So there's


definitely a relationship there. I think the interpretation is similar, but a little bit


different.


And I think in all of these things, again, to highlight LSTMs and transformers, they're


all trained, let's say with with backprop and all the parameters are trained. So still


you'd run into the same problems where if you do discontinue learning, tasks would interfere


with each other, no matter how much you know they can implement the multiplication. So


that's that's definitely, definitely a difference. So in your in your outlook section, I haven't


mentioned this in the video, but you discussed sort of what to do next. And you mentioned


a lot of like, oh, yeah, we want to investigate maybe the combination of RL and continual


learning and so on. Is there something that's here? Is there is there? Yeah, you said you


mentioned neuroscience a little bit. What would be sort of the next big things from


neuroscience to include in deep learning architectures that aren't yet really done by other people?


Like is there something where, you know, you could say, well, if we had that, that's not


really in our deep networks yet. But if we had that, that would be like, amazing.


I think this is a very small point. But the dendrites that we're sort of modeling right


now are they can be considered the basal dendrites. I think you went over this briefly in your


in your intro. And the basal dendrites are responsible for receiving this context and


depolarizing the main cell to either fire or not, if that context was recognized. Something


that we haven't looked into, which could be potentially interesting is modeling apical


dendrites. And the apical dendrites receive feedback either from they receive feedback


from other cells that also biases the soma to fire or not. I think that could be potentially


interesting way to to also gate each individual neuron. I think standard deep learning doesn't


do any of this anyway. They only consider the proximal dendrites, which is mimicked


by the simple linear weighted sum to determine if the neuron is fired. But if we can gather


all this other neuroscience background from all the other kinds of dendrites to like apical


dendrites, it could be a very potentially interesting architecture, like a very powerful


one for for dynamic scenarios.


The issue of top down feedback or lateral inhibition or anything like this, it's a lot


of people talk about it, but I haven't yet seen anyone successfully sort of bring it


into a deep network and actually do something useful with it. So yeah, definitely think,


you know, beyond dendrites, just mechanisms like this would be super helpful.


I think like another aspect, which is a little bit quite different from what Avi just said,


that would be quite interesting is the kind of local learning rule aspects that are present


in biological neurons and how they might relate to unsupervised learning in traditional machine


learning. I think a lot of the unsupervised learning objectives are kind of addendums


to the loss function that we think might be useful and it just kind of flows through the


network. And kind of, I don't think there I might be wrong, but I don't think there's


a lot of research into like figuring out which parts of the network could focus on certain


things in a non supervised way, which might be better done in biological networks. So


I think thinking about that, and getting inspiration to see like what kind of local learning rules


in an unsupervised way could improve performance in modern deep learning would be super cool.


Cool. Yeah. So do you have anything, anything to add anything people should know or that


we haven't talked about yet about the paper, people can get started with your code, which


is online, right? I've seen that, which is very cool. Yeah, anything you want to get


off your like, get out there to the viewers.


The take home message from this is what we want to be is that the brain is, you know,


able to do a lot of different things. It's using different neural circuits to do it.


But you know, neural networks as they've been designed decades ago, they're really just


optimizing for one thing. They're great function approximators, but you don't just want to


approximate one function. You want to be able to approximate multiple functions. And so


we want so we're trying to show that, you know, hey, look, there are ways where we can


get neural networks to actually, you know, have these have different sub sub networks,


different neural circuits that are able to be independent, be different function approximators.


And if we can do that, then you know, neural networks can will be able to operate in more,


more dynamic, changing scenarios. And I think that's, that's really exciting, because the


world is constantly changing. But a lot of the applications for deep learning right now


are the environments that they operate in our static. So if we can get to that, then


then that that that's great.


Cool. Well, Akash, Karen, Avi, thank you very much for being here today. This was this was


great fun. And I learned a lot.


Yeah, thanks, Yannick. And now now you're influencing my fashion.


Nice.


I'll join the I'll join the show.


Thanks so much for being here. I Yeah, I hope you I hope you continue this because it's


really cool. And I think we're we're missing it in deep learning.


Thanks, Yannick.


It was a pleasure.


Anyway, tons of advancement to the next generation.


Sockets Tutorial with Python 3 part 3 - sending and receiving Python Objects w/ Pickle

What is going on everybody and welcome to another sockets with python tutorial video in this video


We're going to be talking about pickles not the food, but instead the serialization in


Python so real quick if you don't know what pickling is in Python it is the


Serialization or I think we can call that flattening as well, and there's probably other terms for it. What does that mean?


We're converting it to bytes. What are we doing with bytes with sockets?


We're sending them so putting that all together it means we can send and receive things


We've pickled in Python via sockets now. What can we pickle in Python?


objects and


What's an object in Python?


Everything so we can pickle pretty much


Everything in Python and basically what this means for us with sockets is that we can send and receive


These python objects so anything that we're doing in a python script we can send and receive that you know whatever it might be


Maybe you're training a machine learning model, and you want to share that model amongst many machines, or you've got some sort of


Shared database or you know who knows what exactly you're doing?


Just you know that you can send and receive that information with


Python so any object you have in Python any data structure


Whatever it you can serialize it with pickle and then send it so that's what we're going to be doing here


So if you don't know if you already know things about pickle don't worry


We're going to send and receive exactly what I'm going to build here, and if you don't know anything about pickle


It's pretty simple, so we're going to go import pickle and just for the record you can also save pickles as files


So you don't only transmit pickle data like between two programs many times you're actually just going to save


that python object


using pickle


Anyway, so the first thing we want to do is just come up with some sort of example of a thing that we're going to


Save so I'm going to say it's just we're just going to use a dictionary for now


Just so anything are some arbitrary python object


Will do so we'll just say hey there, okay?


And that's the object we want to send but again it doesn't have to be a dictionary and also if you were doing a dictionary


You also could just use like


Json or something like to you convert it to a string send it as bytes convert it back out use Json


You'd be good to go so you don't actually you wouldn't use pickle for probably a dictionary


But again you can use pickle for anything whereas like Json is one structure


You wouldn't be able to do everything with Json so okay, so that's the thing that we want to pickle


So how do we do that well?


We're going to just say message the message we want to receive later is going to be pickle dot dump s


D and we can just print message real quickly to see okay?


Here's what we've got basically we do have the string data that we can see here


we also


I don't see the number, but it's probably there somewhere anyway, and then you've got some gobbly goop that is the


The information that is contained within the actual structuring of what a dictionary is and how we can interact with it and all that


Cool so now we want to send this information, so let's go ahead and do that


So I'm going to cut that come over here come down here fix that okay get rid of this loop


We don't we're not going to loop anymore and the message now


Let me just get rid of the first one paste this one in


So now our message is that pickle dump?


But we also we still want to use that header again because the thing we pickle might very easily


exceed our buffer size so we need to be able to handle for that so the way that we're going to do that is


Actually going to be will convert this because this is already bytes right so we just need to convert this to bytes


so bites that and then what kind of bytes you


utf-8 bytes


Okay, so that's the message that we want to send we no longer have to convert it here to bytes because it's already bytes


and


We're good to go


On the server now. We just need to fix some a few things on the client side


The first thing would be the full message here should be bytes


Then the information that we want to receive should be pretty much all the same the only thing


I wouldn't do you know like no longer are we going to decode utf-8. It's just going to be message


When the full message is received we're going to unpickle it so let's go ahead and import pickle


come down here


So full message received here it is


But first we need to load it in so we're just going to say copy that we're going to say D


equals pickle dot load s


That thing and then let's print


D


Save that come over here, and now let's run these two things Python server dot pie


good to go and finally Python client dot pie


So full message received. This is what it looks like to us because that also contains


Actually, I can't remember if we trimmed off the header anyway full message received looks something like that once we've loaded it with pickle


There's our actual python object that we sent using pickle so super cool if that doesn't excite you you are either not nerdy enough


Or you just don't understand


Because now you can send literally anything in Python that you're working with so if you've got some script


It's doing something you've got two scripts that can share this data between each other either locally or remotely


So that's freaking awesome


Speaking of awesome


Quick shout out to my most recent channel members isabella shower. You are amazing


Kirill Groshin call just just incredible Henry D'mena awesome and


Sundar Balaji


Spectacular thank you guys very much for your support you guys are all awesome


So that's it for now in the next tutorial


I think we're going to wrap everything up with a an example with a chat bot or not


I always want to say chat bot anytime. I say chat


I wonder if I've made that mistake so far in this series anyways probably anyway. I want to make a chat server


So that'll kind of bring everything together and allow us to show one more concept that we haven't yet shown select


And yeah, that'll be pretty cool. So anyway questions comments concerns whatever feel free to leave them below you can also join the discord discord


gg slash syntax if you want to support the channel you click that beautiful blue join button that just beckons for you to


Click on it


Radioactive data: tracing through training (Paper Explained)

Are you tired of other people training on your data? That annoys me every time it happens.


I'm mad about this. If only there was a way to somehow mark your data and when other people train on it, their computer would explode.


Well, this paper is a little bit like this, not entirely. The explosion part I think they're still working on on a follow up paper.


But in this case, in this paper called radioactive data tracing through training by Alexander Sablerol, Matisse Douze, Cordelia Schmidt and Hervé Jigu,


they develop a method that at least you can detect if a given model was trained on your data or not on your data.


And they call this process radioactive marking or radioactive data for short.


So the overview you can see it's pretty easy paper actually. The concept is pretty easy and it's a nice concept.


And it's been around in one form or another. It touches on adversarial examples. It touches on differential privacy.


But in essence, it works like this. If you suspect someone else training on your data


or if you just have a data set that you want to protect, what you do is you mark it.


You mark it with this mark and they call this a like a radioactive mark. But essentially, you just distort your images a little bit.


Then when someone else trains on that data, so here a convolutional neural network is trained on this data and not all of the data needs to be marked.


They can go as little as like one or two percent of the data being marked. Then from the output of that network or from the inspecting the network itself,


you can then test whether or not this network has been trained on this radioactively labeled data.


So you will see a clear difference to a network that has been trained on only what they call vanilla data. So data that has not been marked.


So I hope that's clear. What you do is you train, sorry, you mark your data. What Bob does, no, what's the attacker's name?


I don't know. But what Eve does is train here a network on data and you don't know whether it's this or this.


And then you do a test to figure out which one it is. OK, so we'll dive into the method and look at how well this works.


Pretty simple, but pretty cool. So their entire method rests on this kind of notion that these classifiers, what they do is if you have a neural network, like a convolutional neural network,


you have your image, your starting image of your prototypical, I don't know, cat, and you input this into many, many layers of a neural network as we are used to.


But the last layer is a bit special, right? Because the last layer is the classification layer. Let's just assume this is a classifier.


So if this is CIFAR10, for example, there are 10 different classes that you could output. And so 10 of these bubbles right here.


That means that this matrix right here is a number of features. Let's call it D by 10 matrix.


OK, so the network, this part right here, we would usually call a feature extractor, something like this.


So the bottom part of the network basically does its nonlinear transformation and so on, extracts D features.


These are latent features. And then those features are linearly classified into 10 classes.


The important part here is that that last layer is actually just a linear classifier.


And we can reduce this actually down to a two class classifier. So the five function would just put points here in somehow, you know, let's just make them two classes, the X's and the O's and so on.


So if the five is good, then the last layer has a pretty easy job linearly classifying it right here.


You can see here the five is not very good. We can't linearly classify this data.


So by training the neural network, what you do is you make five such that it will place, hopefully, the one class somehow on one side, the other class on the other side.


And you can pretty easily linearly classify that data. OK, the exact slope of this of this line right here.


The exact location of this line and direction of this line. That's what's encoded ultimately in this matrix right here.


So this matrix now not only for two classes, but for 10 different classes, it records these hyperplanes that separate one class from the other class.


And these are in d dimensional space. So you have d dimensional, 10 d dimensional hyperplanes separating the space of features linearly into the classes.


So what you can do is you can actually think of this d, sorry, of these d dimensions here as features, right?


This is a feature extractor, so it provides features to a linear classifier.


Now, what this method does is when it radioactively marks data points, it simply adds a feature.


OK, so how do you think about these features?


So, for example, let's say this is actually this animal classification example.


And if you are if you are asked to classify cats from dogs, from horses and so on, one feature could be, does it have whiskers?


One feature could be, does it have fur? Right? You can maybe distinguish cats from turtles and cats and dogs from turtles.


Does it have how many legs? So the number of legs and so on.


So you have all these features and the last layer simply linearly classifies those features together.


What this method does, this radioactive method, it adds a new feature per class.


So down here, I would add a new feature that says like this is the radioactive feature. Can I draw the radioactive symbol?


This is the radioactive feature for the class cat.


OK, and then, of course, I also have one for dog and so on.


So it would add or basically would you don't change the dimensionality, but in essence, you add one feature per class.


And that's what they mean here by this direction U.


So in this high dimensional space that is spanned by these d dimensional vectors and you can.


So this thing here, OK, sorry, I'm switching back and forth. This thing here, you can sort of if d is equal to two,


you can imagine it as 10 vectors in a space in this feature space.


OK, 10 of these vectors. And whenever you get a point, that's is that eight?


Whenever you get a point, you simply look at. So if you get a data point right in here, goes through here, you come here and you look.


With which class does it align more the most? And that's how you classify it.


OK, so if you think of this, then what you what you want to do is you want to add a feature here such that this is one per class.


I'm having trouble articulating this and you want to change your data points.


Here you can see your data points. And for this class X, we make this radioactive feature right here, which is the blue thing.


We shift the data into the direction of this feature.


OK, so basically we add the feature U, which is just a random vector in this high dimensional space.


We choose one vector per class, but then we shift all the data for that class along this feature.


So what we are doing is we are introducing a fake feature that we derived from the label, right?


So we kind of cheated. Here you have X and you're supposed to tell Y from it, and that's your training data.


But then we cheat. We look at Y and we modify X with the feature of that particular class.


So what does that do? Ultimately, we have we end up with U1, U2 and so on.


So one feature per class. It trains the classifier to pay attention to these features.


So if U1 is the feature for cat, then we train this classifier by training it on the data that has been modified in this way.


We train it. A cat should consist of something that has whiskers, has fur, has four legs and so on, and also has this cat feature.


Now the danger, of course, here is that the classifier will stop to pay attention to anything else and only look at the cat feature.


Because we introduced this feature to every single example that was of class cat.


So the classifier could have a pretty easy way just looking at this feature, determining well, all of this is cat and then it would not generalize at all.


So what we can do is, first of all, we can make the feature very low signal.


We can make it very small, such that there are other features such that these other features are also pretty easy for the network to pay attention to.


And second of all, we can label not all data. And that's what they do here.


They label maybe 10%, maybe 2% of the data with that, which forces the network to pay some attention to this feature, but also to pay attention to the other features.


And that ultimately, if you trade this off correctly, results in a classifier that it does give up some of its generalization capability because, of course, 0% of the test data has these features right here.


We modify the training data to add these features.


So you give up a little bit of generalization capability, but you force the classifier to pay attention to this feature during training.


And that is something that you can then detect.


So you can imagine if you train a classifier that has been trained on training data, where some of the training data have these features in here.


And that's one distinct feature per class, right?


Then you can look at the final classifier and figure out whether or not the classifier has been trained.


How do we do that?


So let's imagine that in this high dimensional space here, the training examples, they point in kind of this direction right here.


Okay, so all the training examples of one particular class.


So this is now the dog class.


All the training examples point here.


How would you build your classifier?


Well, it's pretty easy.


I would build it such that the dog class points in this direction.


Okay, I'm just erased a bunch of other classes right here.


Now, I choose a random feature when I build my radioactive thing.


I choose a random feature like this one right here.


Okay, and what I'll do is I'll shift my training data a bit into that direction.


Okay, how do we do this?


How are we doing this?


I'll just dash it.


Okay, so I'll shift my training data a little bit into this direction.


So all of these, they move over right here.


And that's where the final classifier will come to lie a lot more towards this new feature.


And this is something we can now test with a statistical test.


And that's what this paper kind of works out in the math.


So usually if you have one vector in high dimensional space, like this one, and then you look at the distribution of random vectors.


So this one, maybe this one, this one feels pretty random.


This one's pretty random.


Okay, humans are terrible random number generators, but these feel pretty random.


And you look at the cosines between the random vector and the vector you plotted initially.


They follow, if this is truly random, they follow a distribution.


They follow this particular distribution that they derive here.


Okay, so you can see a classic result from statistics shows that this cosine similarity follows incomplete beta distribution with these parameters.


Now, they from this, they derive a statistical test.


So if you know what kind of distribution a quantity follows, you can derive a statistical test to see whether or not what you measure is actually likely to come from that distribution or not.


So what we would expect if our data has not been modified is that, you know, we choose a random direction, a random direction, U right here.


This is U for dog.


We choose that random direction.


And if our training data has not been modified, we would expect this dog here to have its cosine similarity to be not very high because there's no reason for it, right?


These are just basically two vectors that are random to each other.


And in high dimensions, they should be almost orthogonal.


So in high dimensions, random vectors are almost orthogonal.


However, if the data has been marked during before training, that means if the classifier used our marked data set to train it, we would expect this cosine similarity right here to be not orthogonal.


So to be higher than just random.


And that's exactly what we can test.


And that's exactly what you saw at the beginning right here.


So here is the down here, you can see the distribution of cosine similarities.


And you can see that if you train with without marked data, this centers, you know, around zero.


However, if you train with marked data, you have a statistically significant shift between the marking direction, the marking feature, and between the classifier direction.


So the all you have to do is mark your data in this way, and then look at the final classifier, look and these blue vectors right here, these are just the entries of this final weight matrix, right?


These are the blue vectors.


You look at those, and you simply determine if the for the given class, if the vector for the given class has a high cosine similarity with the marking direction that you chose to mark your data.


If it does, you can be fairly sure that the network has been trained using your data.


Okay, so I hope the principle is clear, you introduce a fake feature per class, and you make the network pay a little bit of attention to that feature, because it's, you know, a good feature in the training data.


And then, you know, after training, you can go ahead and see whether or not the network is actually sensitive to that feature that you fake introduced that is actually not a real feature in the data.


If the network is sensitive to it, you can conclude that your training data was used in order to produce it.


So there's a couple of finesses right here.


So, as you might have noticed, we introduce these fake features in this last layer feature space right here.


However, our pictures are actually input here in front of this feature extractor.


So we need a way to say what we want to do is we want to say I want this data point here to be shifted in this direction.


But I actually, this data point is actually a result from an input data point, I will call this I right here, going through a nonlinear neural network ending up here.


So the way this is done is by using the same kind of back propagation that we use when we create adversarial examples.


So what we do is we define this distance or this distance here, where we would like to go and where we are as a loss, and then back propagate that loss through the neural network.


And then at the end, we know how to change the image I in order to adjust that feature.


So they define a loss right here that they minimize.


And you can see, here is where you want to go in feature space.


And they have different regularizers such that their perturbation in input space is not too high.


And also here, their perturbation in feature space is actually not too high.


So they, they want, they also have the goal that this radioactive marking cannot be detected, first of all.


And also that is, it's, it's a robust to relabeling, like if you give me data, and I go and relabel it and ask my mechanical Turk workers to relabel that data again,


they will give them the same, the same label, even if you have radioactively marked them, right?


This paper says nothing about defenses, right?


These things are defended against fairly easily, I would guess, by, by some Gaussian blur, I guess, would be fairly effective right here, though.


There are also ways around this. This gets into the same discussion as adversarial examples.


The question here is, can you detect somehow in the final classifier whether or not this someone has smuggled radioactive data into you into your training process?


I'm not sure, but I'm also sure there are better ways to radioactively mark right here.


This is kind of an establishing paper, doing the most basic thing right here.


Interestingly, they also back propagate through kind of data augmentation procedures as long as they are differentiable.


And the last kind of difficulty you have is that these neural networks, they are, they have some symmetries built into them.


So if you retrain a neural network, there's actually no.


So if your neural networks classification, let's say it's a three class classification, looks like this, right?


This is the last layer and these are the classes it's determined.


If you retrain it, it might as well be that this now looks like this, right?


So if you marked it with this direction right here and then you try to recover this direction, you will find that it doesn't work because the entire classifier has shifted.


So what they have to do is they have to do what they call a subspace alignment, which you can do by simply here determining a linear transformation in the last layer.


This is usually enough and what this does is, so their entire procedure is they train themselves a classifier on unmarked data.


I forgot this before. I should have mentioned this.


They train themselves a classifier on unmarked data.


They use that classifier to mark the data, which, you know, you need in order to do this back propagation thing, you actually need a working classifier.


And then when they give the data to someone else to train, they are going to train their own classifier on the same data, right?


So there is no guarantee that these two classifiers spaces aligned, especially because you have this kind of symmetry.


And they say right here, we can fix that by if, you know, we have this classifier and at the end they give us this classifier to test.


We can simply determining this linear transformation here that maps one to the other.


So we go over our data set, we determine M, a linear transformation.


Basically here, you would determine a rotation of this space that would map one to the other and vice versa.


This is not exact, of course, because the two classifier, there's no reason why they should even be linearly related.


But there is a reason coming from kind of neural network knowledge.


And that is that we know or we have a strong suspicion that these neural networks, of course, if they work well and if they reach good accuracy and if they reach similar accuracy,


it's very probable that they have somehow figured out the same features.


Okay, even though these networks learn each feature from scratch and that you, as I said, you have some symmetries in there.


But ultimately at the end of the neural network is very likely that the network learns the same features as another network,


even of a different architecture that has been trained on the same data set.


This is, I think this is supported by research in adversarial examples like this paper, Adversarial Examples are Features, Not Bugs.


You know, transfer learning, transfer of adversarial examples, all of this kind of stuff points to the fact that these networks tend to extract the same features.


And therefore, we can assume pretty safely that up to this kind of rotational symmetry, our space, our feature space will look the same.


And therefore, determining kind of this linear transformation should probably be enough in order to align one subspace with the other


and be able to compare your own classifier that you have trained to mark the data to the classifier that someone else has trained on the same data.


Okay, this is kind of the one of the difficulties they had to overcome.


And yeah, they evaluate this, they evaluate this experimentally, you can see right here on the right,


20% of the training data has been marked in order for the orange data.


These are random directions. So blue would be the correlation with random directions.


And because, sorry, orange is the correlation with these carrier directions with the directions of the fake features.


And green is the alignment with actually the features of the classes itself.


So you can see even if 20% of the data is marked, the classifier still aligns mostly with the features of the actual classification problem,


it aligns a little bit with the features of the fake features or with the fake features.


And it does so such that there is a statistically significant difference between random directions and these.


And you can see even if 2% of the data only are marked, so only 2% of the training data has this mark and the mark is always imperceptible, right?


The mark is always such that you can't see it by eye.


Even then you can see that there is a difference.


So the classifier does learn to pay attention to that feature, which is something you can detect afterwards.


This experiment on the left here is just the same basically saying, so up here it starts with not a lot of data being marked.


And you can see it mostly aligns with the semantic direction, which is the true features.


As you mark more and more of the data, it goes down and down and down, but it does not.


So I think this is 50% is the yellow, 50% of the data is marked.


And still you can see there is a pretty good alignment with the actual features because the network will start paying more and more attention to your fake features because they're pretty good predictors, right?


But it also has this other training data that it can't solve using those features.


So it still needs to pay attention. And of course, your marked data also has these these other true features.


So it is to be expected that even though your data is marked, it's still the classifier still aligns more with the true features than with your fake features.


And they also show in experiments that you do not sacrifice a lot in accuracy.


So here you can see the delta in accuracy through their experiments is fairly, fairly low.


And they do ImageNet on ResNet 18. So these differences in accuracy is there.


They are, you know, you notice, but they are fairly small.


So, you know, someone also couldn't just go on a big accuracy drop when training on data like this.


So someone training with data couldn't just notice that it's radioactively marked by just saying, like, well, this doesn't work at all.


I guess some clustering approaches would work where you look at the features and you just see this one feature is like only present in this very particular group of data that I got from this very shady person selling me 3.5 inch floppy disks around the street corner.


But other than that, yeah, it's not really, it's not really detectable for someone training on it.


And lastly, they have black box. They defend against black box attacks. And here's where I'm a bit skeptical.


They say, well, if we don't have access to the model, what we can still do is basically this is here.


What we can still do is we can analyze the loss. So we can analyze the loss value of the radioactively marked data. And if the network we're testing is has significantly lower loss on our on the radioactively marked data, then on non marked data, then that's an indication that they trained on marked data, which, you know, if you don't have access to the model,


like what's the probability that you have access to the loss of the model, like the usually you need, you need the output distribution or something.


It's a bit shady. What I would do actually is, is just a little bit more sophisticated.


But what you could do is you could take your direction, you, right, you could back propagate it through your network to derive like a pure adversarial example.


So not even going from from some image, just go from random noise, like just derive like a super duper image that only has that one feature, like, and then input that into this classifier.


So this is yours, and then input that into the classifier that you are testing. Okay. And if that classifier gives you back the class that you, you know, each one of these u is actually of a given class, right?


So you have one feature per class. If that gives you back the class of that feature, you have a pretty strong indication that someone has been training on your data because so if you look at data in general, as we said, it has these true features.


And if it's marked, it also has the fake features. So what kind of class it's going for, you can detect in the output distribution.


But if you then input like a pure only the fake feature, and it still comes out the class that you assigned to the fake feature, you know, there is a one over number of classes, probability only that that happens by chance.


And if you want, you can derive a different, you can do this again, you can drive a different pure, only this feature sample input it again, and look what comes out. So it's not it's not a pure test.


So these are not going to be independent. So you probably shouldn't like just multiply. But I would think a procedure like this, and maybe they do this somewhere, but they simply say we can look at the loss of marked and unmarked data, which, you know, I'm not so sure that that's going to work fairly well.


Okay. Um, as I said, there are going to be many, many ways to improve this, the paper has more experiments, ablations, transfer learning between architectures, and so on.


I just want to point out I have a so there's a bit of an issue here, where where I think there is a lot of room to grow.


First of all, here, you simply train the network, and then you look at the network at the end, right, you simply look at these 10 vectors right here, and you determine their inner product with the marking directions.


And that's, you know, that's what you what you go by. What I would, what I would like to see as an iteration of this is where you have a neural network and you, you can't just detect by looking at the end, what you what you'd have to do, you'd have to be much more sneaky.


So in order to avoid detection, detecting your detecting strategy, so in order to avoid defenses against this, I would, I would guess, what you want to do is not just, you know, make the network such that in the end, it's fairly obvious if by looking at this last matrix, maybe you should only be able to detect this at the end by actually feeding data into it like we did with the black box test.


But if we had a white box test by feeding data into it, and then and then looking at the responses of the network, so, but someone couldn't not tell it was trained with radioactive data by just looking at the network's weights.


So maybe one idea would be that you craft inputs in some way that correlates two of the hidden features. So let's say we have some hidden layer here, and one here. And these features are learned by the network, right?


They appear to be fairly independent. So you make sure that they are fairly independent during if you pass regular data, and then you craft data specifically craft data like you did here with the marking that makes the network correlate the two features, but has little effect actually on the output distribution of the classes.


So you can retain your generalization much more right, it doesn't change this last layer necessarily that much or not in a completely class dependent fashion.


What I would simply do is I would correlate two of these internal features, I would force the network to learn to correlate them.


Because then I would expect this to be much more, you know, secretive. And then at test time, I can simply introduce my forged data again and look whether or not the internal responses are actually correlated.


And as I said, I could do this across classes to cancel out the effect of this actually being a feature for one given class and therefore changing the networks accuracy too much. I think that would be a cool next direction to go into.


And again, this should work, because even the intermediate features, we have good reason to assume that different networks, even different architectures, different training runs, learn the same kind of intermediate features.


The question is only in the next network that feature could actually be like, you know, two layers up or three layers down or and so on. So you'd have to learn some kind of more sophisticated alignment there. But still, I think that would be kind of an iteration of this, which would be cool.


You know, if you're doing this site the channel.


Learning to summarize from human feedback (Paper Explained)

Hi Reddit, my boyfriend and I have been dating for a year and it has been great.


Except for one thing, Dota. The other day on a Saturday I was over and he was


playing a game. I thought it would just be one but instead he proceeded to play


for three hours as I just sat there. What can I do? So this as you can see it is a


post from a subreddit called relationships of someone seeking


relationship advice. Now I would claim that this is clearly fake because no one


plays Dota for just three hours. Crazy! But let's assume that this is a thing


that really happened and well it doesn't matter. The article here is written and


the task is to summarize this post in as few tokens as you can but sort of giving


much of the information that's in the that is in the post itself. So the task


here is called summarization and humans can do this quite well. So here you see a


human written reference baseline. My boyfriend games whenever he can. How can


I get him to stop gaming so much and focus more on school and our


relationship? Okay so that's a pretty good summary of what goes on in this


model. The most the easiest baselines for this task in machine learning are what's


called extractive baselines. So in extractive summarization what you do is


you try to find sub spans so let's say like this span followed by this span and


so on that together represent the article. So you strictly select sub spans


or even entire phrases from the text that you're looking at. So a lot of these


baselines are extractive and they perform already fairly okay. For example


this one right here help my my boyfriend is neglecting his studies and our


relationship because of a video game. I think that's just extracting from the


title. Okay that's title policy. There are other models for example here this


lead to hi reddit my boyfriend and I have been dating for a year and it has


been great. I mean that accurately represents maybe not maybe that's not. So


you can already see that it's quite hard because not only does a model have to


understand what information is in a text and what are the important things but


also clearly it needs to understand something about the intent of the


post right. You want to if you want to compress you have to compress the


meaning and the meaning because we are humans we understand that this person


here is distressed seeking advice right. It's like what should I do and we


understand that the source of the frustration is the fact that the


boyfriend here plays a lot of this this video game. It's not really important you


know how much they played or even that they've been dating for a year or so on.


The problem here communicated is the playing video games. So you see that the


researchers here have come up with a bunch of models and their best model


that we're going to look at here is called this human feedback model with


6.7 billion parameters. It's a GPT style model and we'll get to all of this in


one second. I'll just want to kind of show you the end result that can output


the following. My boyfriend is neglecting his studies and our relationship


because of his excessive gaming of a video game. What can I do to get him to


stop. All right so there are a couple of nuances here like the what can I do


to get him to stop is not really explicitly said in the text. It says it


seems like it interfered with our relationship. He's doing his PhDs


obviously swamped. It goes on the back burner. It makes me rethink our


relationship and so on. These things aren't explicitly said yet the model


somehow understands that that's what this person expresses and if you want to


compress this information then this is a very good thing to this is a


very good summary to output. So we'll go to see how they come to build this model


what it has to do with human feedback and just in generally how it works and


also where it fails. So this is a pretty big paper as you can see it's one of


those papers where the appendix needs a table of contents which is going to come


up very shortly. Very this there was lots of references. So it's a paper by


OpenAI. Of course recently OpenAI has made big big advancements in language


research with GPT-3 and this is from kind of the same style of research. So


the paper is called learning to summarize from human feedback by Nissan


Stinnon, Long Wu Yang, Jeff Wu, Daniel M Ziegler, Ryan Lawie, Chelsea Voss, Alec


Radford, Dario Amondi and Paul Cristiano as I said of OpenAI. So they tackle this


task of summarization of this of these kind of posts or news articles you can


apply this pretty much anywhere and they incorporate human feedback into it. Now


why do they incorporate human feedback? And that's because that's because


summarization isn't a straightforward task right? So in its basic if you have a


summarization task you have some sort of a piece of text that contains some


information and from this you want to generate a small piece of text. The small


piece of text should be first very short but second also it should contain


information. It should contain all the information that was contained in the


original article maybe not all of it but it should contain the important


information of what is in the article and then there are some other things


like it should also be coherent but I think that's sort of implicit in this


information objective. What you want to do is if someone reads this piece of


text they should get all the information that was in the big text or not all but


most or the important information. Humans are quite okay at this but it's not like


we can really formulate exactly what we want right? It's not like we can give a


classification label and then tell the machine exactly look this class is


correct and these other classes are wrong. Now what people have been doing is


they've built datasets where you'd have for one particular document you'd give


it to let's say three different humans and the three different humans would


produce three different summaries because different humans do it


differently right? So you'd provide three different summaries and then you let


your machine your machine learning model produce some summary and then your


evaluation metric would be a metric that takes this piece of text and compares it


to those pieces of text and this one of these methods here is called ROUGE. So


ROUGE is a metric that looks at n-gram overlaps. I've the Wikipedia page pulled


up here and you can see it consists of a bunch of submetrics but there is a way


to mix them but in their essence they basically look at overlaps of here


overlap of n-grams so you can look unigrams or bigrams you can look


longest common subsequence and so on. Basically you sort of try to compare the


words the text specifically in here to the texts in in the human summaries and


given the rich nature of language that's not really a good approach but it's the


best one we have. We don't have a better metric to tell the


machine what's right or wrong and it goes actually further so this ROUGE as


an evaluation metric it's already it's fairly bad as we can see as we will see


they have a graph somewhere and I might just draw the graph in that if this if


this here is kind of the complexity of the information and this here is the how


good the summary really is as rated by humans so this paper plays a lot of


emphasis on going to actual humans and asking them how good is a summary if you


employ ROUGE then at the beginning you increase as you increase the quality so


for easy text for easy information and for really bad models the ROUGE metric


makes sense because you know generally if you have a very crappy model and one


that just outputs the same kind of text as the humans do then that one's gonna


fare better but then at some point it wanes off and the at some level of


complexity coherence and so on the ROUGE metric is just not good enough anymore


to differentiate sorry to differentiate good from bad summaries or let's say to


differentiate excellent from good but not excellent summaries let's phrase it


like this is that it's good at differentiating bad from good summaries


but not good from excellent okay so that's one thing that's evaluation but


ROUGE this overlap of n grams you can imagine that this is not differentiable


so the second problem is how do we even train this thing right so this here is


this is eval ROUGE eval but in training you do something even less let's say


something even that makes even less sense from a just a principled point


approach what you want to do is you want to simply make the machine output these


texts right so you simply say these texts are correct now please output


those it's kind of like a variational auto encoder that you want it to output


a very specific picture but you've given it that picture as an input you can kind


of imagine it like this you say this is the input and this is the output I want


you to produce and now that I can actually back propagate I can back


propagate the production of this exact text from this input right so their


model here is going to be some sort of a GPT-3 style model it's not as big as GPT-3


their biggest model I think is six billion seven billion parameters


whereas GPT-3 has what 175 billion parameters or something like this so the


model is going to work as follows you take this text here you just unroll it I


think something like this so that it's just one string and then you let the


model produce so here's the model is on top of this and you simply always


produce the next character or word or word piece right here and then you


produce the next and you produce the next until you've output this thing here


and this thing here is going to be the summary okay and that's a thing you can


back propagate through with simply language model learning I'm ragging a


bit too much because of course many things are trained like this in language


learning like translation is learned like this just the simple generative


language models are learned like this so it's not that terrible but you can see


that evaluating with rouge while training with this both are not


particularly suited to what we want what we want actually is that humans would


rate these summaries well but we can't do that and that's the problem that this


paper solves so here they show their final results already so down here you


have model size but we we don't worry about that right now that because there's


also a question of scale in here and so on if they use a language model that was


just pre trained on language so no terrain no explicit training for


summarization we've already seen in the GPT 2 and GPT 3 paper that if I take a


piece of text and that it added a data and I append the string TLDR right too


long didn't read which in in forum posts most often people put this and then they


put a summary okay so this prompts the model to produce a summary if this seems


mysterious to you I've made videos on GPT 2 and GPT 3 explaining how this works


so a model that had just been trained on language modeling will actually be able


to do summarization to a certain degree as you can see right here it's still


below the quality of reference summary so this axis is really what humans this


Wow that body attachment to the legs is really what humans think of these


summaries so the way they evaluated is they present the human with two


different summaries they ask them which one do you prefer of course if you give


them human summaries so one of them is always a human summary but if you give


them two human summaries it's of course random which one they prefer and


therefore that's the the 0.5 point so if you give them one summary from this


pre-trained model and one human summary you can see that the pre-trained summary


loses most of the time loses like 80 70 to 80 percent of the time against the


human reference summary then the second step is to take this model and produce


what they called a supervised baseline so that's what we've discussed just now


when we said how do we even train this so we take a model that takes a database


sorry a data set I've been some reviewers are just calling data sets


databases and it freaks me out and I've taken it over I've seen it so many times


now there must be parts of the world where data sets are called databases so


in this you always you have samples of text and corresponding summary so you


call this your X and you call this your Y and you simply train a model to take


in the X and predict the Y now instead of a class label it's simply a string a


piece of output string you can do this with a language model like a generative


language model that's a that's the supervised baseline so if they do that


they get closer as you can see right here so there is quite a bit of distance


between this pre-trained model and the supervised baseline that starts from the


pre-trained model but actually trains the model to do summarization but you're


still not at the level of these reference summaries and then they have


this mysterious human feedback model that now all of a sudden actually gets


better than the reference summaries it actually outperforms them and we're


going to look at how this comes about so first of all their contributions as they


stated they say we show that training with human feedback significantly


outperforms very strong baselines on English summarization okay we show human


feedback models generalize much better to new domains than supervised models


okay and we conduct extensive empirical analyses of our policy and reward model


alright so if you see the words policy and reward model that already means that


reinforcement learning is going to play some role here and here's how it works


so this all already starts from the supervised model so imagine what you've


done so far you have this pre-trained model you've taken it you've generated a


supervised model for it so the supervised model is explicitly trained


to do summarization but just on a data set and now you want to incorporate


human feedback okay so the way you incorporate human feedback is as follows


first you collect the human feedback and the human feedback here you could do


various things so you could let the humans kind of score summaries but what


you want to do in this case is you always want to present the human with


two different summaries and ask them which one do they prefer okay that's


going to be our humans are going to be just doing this thing for now they are


going to look at two summaries and the corresponding piece of text that's


important and they're going to decide which summary is better and better in


just in a human sense better right so they they work closely together with the


researchers right here and that's I think an advantage if you're open AI and


have lots of funding and so on they it's it appears they've paid these humans


quite well and they've worked with them quite closely to in order to ensure the


high quality of their feedback so the humans will always say which of these


two summaries is better okay now what you could imagine is you could simply


train a model using that right so the model produces this and maybe the human


one of the humans summaries and the data set is that and then the human decides


is it better or worse and then a model somehow optimizes this this is not


exactly what they do because that would require too many humans if you know


these language models they take a lot of data so even though open AI has lots of


budget it's not really feasible for them to train these big language models and


every single training step for every single sample go and ask a human what do


you think so they have to come up with some sort of different way to do this so


what they do is this entire thing right here this entire thing right here will


now be a data set okay it will be a new data set so they take these supervised


model and they produce a whole bunch of these summaries and they always ask the


humans which one's better so this will be a data set and a sample from this


data set will consist of a big text two summaries of that text and it doesn't


really matter how they're generated just two summaries and a label and the label


is either this one's better or this one's better okay so this here is going


to be now our X and this one is going to be our Y of that data set and to this


data set we now fit a model so we fit a model to simulate the human okay we the


model learns from the human in in the reinforcement learning this is very


related to imitation learning reward model learning there are a bunch of


names for it in this case they say we train a reward mode it's actually not


exactly sorry it's not exactly imitation learning because that there you'd have a


actually samples of the policy and so on so let's stick with reward model


learning so that I'm correct the exact way you do this is you don't actually


fit the X to the Y right here but what they train is this reward model right


here so this thing takes in as you can see a piece of text and one summary and


it predicts a number and the number is supposed to say how good is that thing


how good is that summary for that given document and the humans never said that


right so we can't directly we can't directly use this as a label right here


we cannot because we don't have this information we just have the information


whether it's better or worse than some other thing so what we're going to do is


we're going to take the same article and a different summary of the of that post


at one post with two summaries judged by a human are fed to the reward model so


this is fed to the same reward model the same model gives at the output for that


one and then we train our loss is going to consist which one's better so if the


loss is pretty simple right here you simply subtract them from each other


this is a sigmoid non-linearity and d log because the loss is in log space but


the sigmoid right here ultimately what that does is if so here's zero if post J


is better than post K this is going to be a positive number right so the


sigmoid will map this to a one over here if post K is better than post J the


sigmoid will map it to a zero right here and if they get close to zero then


something like this right so in this case here post J is better and in this


case here post K is better so that seems like a sensible loss that you can


regress on so now you map these rewards to a zero or a one and that's exactly


what your label is your label is either a zero if this post is better or a one


if this post is better so now you have a data set and you have a model that you


can train namely this model right here so you're going to train this reward


model on this data set and you can iterate this at the end even though we


aren't at the end yet you can go back and do it all over again if you want and


I think they do they iterate this improving their summaries asking the


humans again training reward model and then the last part is that you actually


now you have a reward model right remember we said it was too expensive


for humans to always go ask the human which one do you prefer well now we have


a model that can substitute the human so what we can do is we can simply train


use reinforcement learning to train the summarization model to maximize the


reward ok so now we give the model this model right here we give a piece of text


and it produces a summary remember this these models are exactly that these


models right here are exactly these models ok in fact we start from the


supervised baseline we plug this in here that's the model that actually produces


the summary and we are going to fine-tune that using reinforcement


learning now PPO proximal policy optimization is a pretty simple but very


effective reinforcement learning technique so what you need is you simply


need an input this your X then you need an action this going to be our action


this is going to be our output of the model and then you need a reward so for


the reward you take this model right here and this at this point this is


fixed so you learned your reward model now this is fixed now you have a model


that for each summary can give you how good that summary is right this reward


and you can use that to do reinforcement learning so the reinforcement learning


simply tries to generate a summary that makes the reward model as happy as


possible and the reward model is learned from the humans so you can see that at


the end through the proxy of the reward model we are directly training for human


human enjoyment so we are not training log likelihood like we did initially in


the supervised baseline we are not training for rouge which we could do


with reinforcement learning but rouge itself is a pretty bad metric we are


actually training for directly for what humans say they prefer at least as far


as the reward model can approximate the human preferences so you can see that


this is potentially a good approach now this was also kind of if you read this


stuff in let's say on Twitter or elsewhere people are people are I think


very joyous that Wow so we are aligning models with human interest we are


aligning them with human preferences and so on human in the loop yeah yeah yeah


it's still it's still difficult I I think this is slightly overhyped in in


that direction like the direction of where we go say wow these are so these


are so such good things because so first of all this costs a lot of money a lot


of money like you need to work closely together with these humans right and I


don't know where they say it but they actually did not compare to a model that


collected so if you do this supervised thing right here you have your data set


right of text and multiple reference summaries well okay no one knows no one


knows what happens if you invest as much time money and effort into collecting a


bigger data set of simple reference summaries and then training a supervised


model on that nobody knows okay so and they they say this they admit this in


this in this paper they say we did not it's too expensive to also just do the


the control of what would happen then but you know chances are that models are


going to improve significantly as well if you simply provide a bigger data set


of of of these it's so I yeah it's it's questionable whether or not this this


modeling of the reward here is really the deal breaker or simply the fact that


they have collected much more and much higher quality data to train on and then


the reward model is simply the proxy for that data so that's the that's the first


kind of dent here that's not really clear now I don't get me wrong this


paper is pretty awesome especially because they evaluate all the summaries


using humans as well and that costs a lot too so regardless of training even


evaluating these summaries in terms of not rouge but actual human feedback is


very expensive and they do this as well and this is this is of course pretty


pretty awesome and gives you the most accurate signal that alone is


commendable but I don't I don't believe yet that this reward modeling is the


thing that made the improvement here in their training procedure the second


thing is they do the following their reward for the PPO algorithm isn't


actually just the reward from the reward model as you can see here but it has


this KL term in here so what does this KL term do so here is the this is the


supervised baseline the supervised baseline is simply a model that as we


said was trained to input it post and output one of these summaries that the


humans provided this thing right here is the reinforcement learn baseline so this


is the thing that's actively changing during PPO okay so and you constrain


this to be to stay close to the to the supervised baseline so you don't want


your you don't want your reinforcement learn model to go far away from the


supervised baseline model so in terms of the reward your reward is going to be


the reward that you get from the reward model that is trying to predict how good


humans like the particular thing minus a penalty so minus a penalty term if you


are too far away from the supervised baseline and this should remind you of


something so you're kind of trying to optimize the you're trying to especially


if you look at the diagram of the model right because you have piece of text


right and then you have your model right here that you train and then you have


the output summary okay and then you have the reward model and you have the


reward as an output that you're trying to make as big as possible now what does


that remind you of if you look at this model right here you're trying to you're


trying to optimize its input right this is the input to that model in order to


make its output a certain way while all the while making the input be not too


far away from some reference input this should remind you of adversarial


examples right because what's happening right here is exactly we are trying to


find an adversarial example to the reward model okay it's it's not adversarial in


the sense that it tries to maximize its loss or something like this but it is


trying to maximize its output its reward and it's trying to manipulate the input


to the reward model such that the reward is as high as possible and what do we


know about adversarial examples is that they aren't really really part of the


normal data spectrum if you will so and we're going to see this and they have


this they have this problem as well so if they constrain they there is a


parameter there where you can trade off how close you want to stay so how much


freedom do you give the reinforcement learning to go away from the supervised


baseline and you can clearly see that here is the fraction preferred by humans


and here is this this KL if you optimize the with reinforcement learning and you


let the reinforcement learning you know you give it some room the more to the


right here the more freedom the reinforcement learning model has you can


see that it goes up and up but after a certain while it is flat and actually


goes down again so if you purely reinforcement learn what you really


find our adversarial examples to the reward model that have nothing to do


with the humans anymore because it's really just an adversarial example and


to demonstrate this they have this nice piece in the appendix where they give


samples from these over optimized policy so policies that are just over optimized


to this reward model so here and we don't see the piece of text which I


find is also interesting because here we are just the reader of the paper can


it's just tasked with judging without I think without finding the piece of text


without reading the piece of text which is interesting that the humans can


actually do this makes you kind of think of how it all works but so here the


reference summary that a human wrote on 28 male live in San Jose I would like to


learn how to do gymnastics okay 20 year old dude stubbornly post ponies start


pursuing gymnastics hobby citing logistics reason despite obvious


interest question more question my question mark it's so yeah negatively


affecting long-term fitness progress personally it just seems like a bunch of


it just seems like these websites that people made to rank high on Google


because it has all the terms that make Google happy which I mean this something


like this is exactly happening here right you just trying to fit everything


in there to make the reward model happy the reward model was only ever trained


on let's say coherent summaries textual summaries so if you go away from this


data manifold you can find things that score high but that a human wouldn't


rate high that's simply because the reward model isn't you know it's all


isn't all knowing it's simply a neural network and they are susceptible to


adversarial examples left password saved on work computer replacement spends


every hour of the day watching Netflix employees stubbornly post ponies


replacement so it despite trying reasonable question or question or


question mark negatively affecting productivity you can already see that


there is some sort of a pattern here negatively affecting so this this this


policy simply finds like this structure of text stubbornly post ponies that


seems to make the reward model very very very happy but it really goes away from


the text right here I get it's pretty cool actually because you see my fridge


and that it kind of copies over the words in what it already knows it makes


sense and I think this ties a lot into what I've been saying about how GPT 3


works because this is kind of a really dumbed-down version of GPT 3 it's


actually the same architecture and you can pretty clearly see that what it does


is interpolate different things so it in this case it interpolates what it knows


makes the reward model happy which seems to be these phrases right here and it


interpolates the kind of important words from the text on the left a little bit


so it sort of understands what makes the reward model happy and thereby you can


already see how a reward model like this may work in that it will sort of judge