components to a Geneva strategy. I'll break down the syntax for you real fast, what these programs


AI against Censorship: Genetic Algorithms, The Geneva Project, ML in Security, and more!

look like. So the first component is the idea of a trigger. The trigger is what's between the square


brackets. So there's two triggers in this, TCP flags S and TCP flags R. And when Geneva is


monitoring traffic, the trigger tells it which package should I act upon. So this first trigger


you see here says TCP flags S. Okay. So that means that whatever actions are attached to that


trigger will run on any SYN packet it sees. S stands for SYN and SYN means the start of my


connection. So what this is going to do to that packet is the very first action we see is


duplicate. So that means it's going to take that packet and make two of them. Now duplicate, the


syntax of this is it's one set of actions comma another set of actions. So you'll see the two


actions you see here are tamper and then send. So the second duplicate we do nothing to. So the


second duplicate we're just going to send on the wire. But to the first duplicate what we're going


to do is we're going to replace the flags fields in that packet with SYNAC SA and then we're going


to send that packet. So basically what this little program does is it sees outgoing SYN packets to


your computer and it duplicates them to make two packets and then replaces the flags in the first


one with SYNAC. Now any networking person listening is like this is clearly ridiculous. This never


should work. Like why would we even do this? Why are we talking about this? And what's going on here


is that for certain sensors around the world SYNAC is the packet that's typically sent by a server.


It's never sent by a client. So what's going on in this strategy is when the client sends


a SYNAC the sensor says whoa I must have missed something. This client is clearly a server which


means the server must be the client. It reverses the roles of client and server in the mind of the


sensor. And as a consequence when the client makes the real request since the sensor is processing


packets differently between client and server you're through. I see. So that's this idea of


the strategy. So that connection in the in the mind of the sensor is already established as here's a


server here's a client and it kind of keeps that state for subsequent packages. More or less. Yeah


that's exactly it. Yeah. So this is an example of just one strategy and one of these programs that


so Geneva built this program itself and it built this through the process of evolution. Yeah and


you've discovered just to jump ahead a little bit because we're not through yet with explaining


exactly how it works but you've discovered that Geneva will actually be able to do this.


That Geneva will actually reproduce a lot of the common or known or already discovered


things that researchers have proposed. Yeah we had this really cool result initially where


we set out to try and we wanted to when we first developed this tool kind of benchmark it against


the rest of the field. And that's kind of challenging because sensors have continued


to evolve. Yeah. So what we did was we sat down in the lab and we implemented in the lab


our best guess as to what our best implementation I should say as to what these sensors look like


based on what previous researchers found. And then train Geneva against these mock sensors


and also train it against the great firewall and real sensors where we could.


And we found this very quickly it was able to reproduce basically the entire field. Yeah.


Every strategy you might come up with this this also found and it found them each pretty quickly.


Cool. So it's really shown the power of automated approaches and AI and ML. Yeah.


So you have you have let's let's get back a little bit you have this syntax right that you can build


trees from which are valid programs in Geneva. This will modify the traffic somehow. Now to say


that most of this traffic will just not even be traffic probably like it will like the connection


will be somehow bad. Some of it will go through and some of it will actually maybe evade the sensor.


What do we need to get there? What do we need to you know to get to a place where I guess if you


just do it naively and you randomize a little bit, it will just be bad like 99.9% of all the programs


you generate, you'll initiate them and then after a while you'll see like my traffic doesn't even


isn't even getting anywhere right. So what are the like of the genetic algorithm components,


what do we still need? Yeah so we're building our way up to genetic guard. We've got just like you


said we got our building blocks. We got a way to put them together. We got a syntax that you can


build these programs out of and we can run these programs on network traffic and you're exactly


correct that if we initialize completely randomly it's going to do terribly and that's exactly what


happens. We've tested this. So where do we need to go from here now that we have this? So this


kind of brings us to this idea of let's get evolution in the mix. So you can imagine the


way this works is we have a big pool of strategies. Okay we'll call this a population and each of


these populations just take for granted for now that we have some diverse set of strategies in


here and we have a way to test them right. We can try and make a request for something forbidden


and we can run these programs on those requests as we make them. So for example from inside of


China we can try and access Wikipedia, that's a sensory resource, and we'll have these programs


running on that connection. We'll just try to make that connection over and over again.


What we'll see is some of these strategies will destroy our connection. Some of them will just


not work at all and do terribly. Some of them might keep our connection alive and maybe if


we get crazy lucky one of them will defeat censorship but for now let's just say a whole


bunch of them will just destroy our connection and maybe some won't. What we have is a fitness


function and this fitness function, this borrows from a much broader space in ML and AI, but it's


basically this idea of if you take some individual from the population, some individual strategy,


how good is this thing? Survival the fittest, like should this thing survive basically and continue


to propagate its genetic material. So this was actually the second big challenge in applying AI


and ML to the space of sensory vision of what on earth should a fitness function look like


in this space because just like we talked about earlier there's no gradient and even coming up


with a loss function can be a little tricky. And I mean even if like, sorry to interrupt,


but the fitness, is it anything else than zero? Okay maybe some connections don't even work to


the server next to you, you can discard those, but other than that the fitness is either done


doesn't reach the target or does reach the target and if it does you've kind of won, right? Like how


can you even get a meaningful signal? Is there a fitness in between zero and one? Yeah so and part


of what makes Geneva work is we've kind of shoehorned our way into getting fitness between


zero and one. And specifically what we do is rule out those strategies that break your own connection.


So that's kind of how we've gotten between zero and one because it's not technically zero and one,


it's almost negative one zero one. And negative one is Geneva shooting itself in the foot,


right? It's just like dropping all your traffic, like that's never going to work and we shouldn't


even bother exploring that space more, right? Like we're never going to go anywhere. But if you can


make it so that your packets are at least interacting with the sensor and at least have


the potential of making it to the server, well now we might be getting somewhere. So basically what we


do is we set up the fitness function in such a way that if strategies destroy the underlying


connection that we punish severely and basically get killed off, and strategies that interact with


the sensor, even though they get censored, they'll get a slightly higher fitness function than those


other ones. So what's going to happen is because those individuals aren't successful, but they're


still the most successful in the population pool, which means some subset of them will continue to


reproduce and basically that subset is just chosen randomly. But because we're just choosing randomly,


mutation is still going to happen. So we're basically taking a set of individuals, they all


interact with the sensor, and then we just mutate them and try again, and then mutate them and try


again. And effectively what this has turned into is a fuzzer. Like Geneva is the fitness function


is basically makes this a targeted fuzzer where we can fuzz just the space of strategies, just the


space of programs that allow us to interact with the sensor. And then where it gets interesting is


as this fuzzer is running generation after generation, just trying different crazy things


against the sensor, if it finds something that gets through, suddenly that fitness is way higher


than everything else. And that individual will start sharing its genetic material and propagating


within the population pool. At that point we could stop, we could stop the fitness function right


there, but we optionally add some additional punishments and rewards for the algorithm at


this point. And specifically we add basically a punishment for strategy complexity. So if


an individual is successful, we optionally punish it for basically the number of actions and the


amount of overhead it adds to the connection. And the reason we do that is, this is not strictly


required, but I have a very small, smooth human brain and it's so much easier to understand a


strategy that's only two actions long compared to something that's 50 actions long, for example.


So if we could encourage the algorithm like, great, you got a solution, now simplify it down


for me. And it will over the course of generations whittle it down to its smallest form and then


at the end present to you its population pool and its best individuals.


And we see here a few ways you can mutate. I think this just essentially comes down to changing the


syntax tree in some form.


Yep. And these are basically you can, yeah, you can imagine all the different ways you can take


these programs and mix them around. If you can think about it, Geneva can probably do it.


Yeah. And so just maybe for my understanding, but you're trying all of this, you say you have


some machines inside of these countries. And I read some, like, obviously this is not going to


work against IP blocking. Like, how do you not get IP blocked by them? If like, I imagine there's


like some weird traffic that's, you know, hits my censorship wall all the time. Why don't I just


be like, well, gone?


Yeah, that's a good question. And we get this question a lot actually. And you're kind of


pointing to this broader question of like, what's the sensor's response? You're doing all these


wacky, crazy, ridiculous things. I mean, there's a strategy in there that just lights up every TCP


flag. Like that package shouldn't exist flatly. It has no meaning on the network. But Geneva tried


it, found it and found that it works. So where do sensors go from here? It sounds like we're


talking about things like it's sending crazy packets. It sounds like that should be something


that's easy to detect on the network. But it sounds easy until you try and write it. Because


if you think about it, writing something to detect abnormality when you have no idea what that


abnormality looks like, especially in the space of just like, just how random and crazy the


internet is all the time. Identifying that is actually harder than it sounds. And what makes it


potentially even harder is that a lot of the middle boxes that would be doing that detecting is


exactly the middle boxes Geneva is mucking with these strategies. So it may be the case that


their detectors are also getting screwed up. Whatever an imaginary detector would also be


getting screwed up by these same strategies. So it's something they could take an action against,


but we haven't seen any sensors roll out something like this. Something else you could imagine,


the existing fitness function was just described for Geneva. It kind of assumes a static adversary,


like an adversary that's not playing along, if you will. And it's also assuming an adversary


that's not doing anything special to hunt it out. And you could imagine that the


sensor that's a little more sophisticated than that. So something we've kept an eye on is


this idea in the future, if either the sensor starts rolling out AI ML techniques, or if the


sensor starts hunting for traffic that looks very abnormal. And you could imagine encoding


additional bits into the fitness function, such that you could encourage Geneva to make this


strategy blend in with normal traffic. I want this to look as normal as possible, but still


get through things like this. So you could imagine all sorts of modifications to the fitness function


to make an algorithm like this a stronger competitor against an adversary that's also


playing along. But we haven't seen the adversaries do that yet. So we haven't needed to.


I was surprised when we talked to a bunch of also people in the intersection of security


and machine learning that there are, as you say, these ML based, let's say, malware detectors,


or things like this, I guess, also weird traffic detectors, and people use them for


example, for company networks and so on. And these are, to my surprise, also, for example,


vulnerable to adversarial attacks. So there's an entire new direction opening, which usually people


imagine adversarial attacks, like, I changed the image a little bit. And it's really this


distinction between how the human sees it and how the machine sees it. But you know, in malware,


it's like just bits. And I flip like, you know, very small number of bits, there's nothing like


how the human sees it and how the machine sees it. It's so weird. But yeah, I think, I think


it's, it's pretty cool. And you got some attention in the media. And the articles usually go


something like this AI can evade censorship or something like this. And now knowing that you use


genetic algorithms, what do you how do you think? How was how was your work received in the media?


What do you think about it? Do you feel like they are kind of trying to put a few buzzwords in there?


Or were you happy with it?


In general, pretty happy. I've kind of been lucky to I mean, even just discussions like this, or we


can talk about the work in a deeper context than just like throwing buzzwords around. Like, this is


just an awesome way to kind of cut through that that buzzwordy fanfare, if you will. Yeah. So I've


been kind of lucky. And you're always going to see buzzwords attached to things that's always


something like that. But yeah, I'd say overall, it's been it's been received positively and things


like this are really what helped us get there. Cool. And the just saying the code for Geneva is


available. It's on GitHub. Anyone can anyone can I guess look it up. Your builds fail right now.


I just have to tell you I'm sorry. Yeah, we're switching between CI systems and haven't finished


the migration. Okay, I mean, yeah. So where is there? I mean, there is a lot of open space here,


it seems the genetic algorithms are very cool. They're like a basis right here. Do you think


there are more places where, like machine learning techniques, especially you said, you know, we kind


of have to draw back from the gradient based approaches, but there are definitely, there's


definitely possibilities. If you think of something like, you know, AlphaGo or something like this,


that's it's a discrete game. But also, you know, they they work with neural networks that, for


example, when you build your tree, your modifications that guide that somehow that, you know,


have an idea, which of the modifications might lead to a better algorithm to a worse algorithm and so


on. Do you see any sort of evolvement that could happen there? Definitely, definitely. When we


first wrote Geneva, our goal was not to be the last AI approach to the space, it was to be the first


and hopefully the worst. Yeah, it would be great if viewers out there, hey, take a crack at this.


There's all sorts of new techniques out there just waiting to be applied this this space.


It's rich, and it's interesting, and it's impactful. Like this is the kind of space where


you discover something, get that out in the world, you're helping journalists and activists like


right now. So it we're really excited to see where this where the space goes and continues to


blossom. So yeah, all sorts of all sorts of techniques just waiting to be applied. And are


you also actively investigating the the sensors side? Because I imagine that the more or the more


capable you are in censoring things, also the better you can research counter strategies.


So a bit, we've tried to tailor our research in such a way that we're not directly helping a


sensor. We never want to publish a paper that's like, really the use case of this is just making


sensors better. Like, so if we do do research down that vein, it's purely in service of let's make


invasion better. Yeah. And we've tried to be very good about not releasing anything and not not


publishing anything that's directly, hey, sensors, this new technique, man, that's gonna really


change the game for you should try and roll that out. So I guess that answers your question. Yeah.


Yeah. And what if you if you look ahead, you said, yeah, we said that the space is wide open,


what would be what do you see as a a, like maybe a bit of a North Star for for the field, like for,


let's say, censorship, evasion or something like this? What would be characteristics of an ideal


algorithm? That's a really good question. Ideal algorithm, something to shoot for. So I think I


can answer that question by talking to I guess how this how the problem of censorship is getting


harder and getting more complicated. So as censorship is continuing to evolve, like this


cat and mouse game exists, it's not just sensors patching bugs, like sensors themselves are finally


getting more sophisticated, they're getting better. And one direction that we think sensors will start


exploring in the future is this idea of more personalized censorship. So instead of censorship


policies being rolled out for the entire country, you can imagine a system where users with elevated


social credit scores or different professions, things like this, could access different content


online and be subjected to different different forms of censorship. So I think that's a really


good idea. And in cases like this, something like just directly applying Geneva gets a little bit


harder because you can't just apply Geneva in one vantage point and help everybody, right? Like you


need to suddenly have a way to to reach more people and help more people at once. So it's this


question of how can we scale this up in a large way and how can we scale this up safely in a way


that protects itself from attacks from the adversary? Like the nations they can see are


trafficked, so in theory they could muck with the training. How can we prevent that? So in crafting


this like ideal algorithmic circumstances, a lot of things you have to consider. So I think building


towards this idea of can we do federated training across a large population? Can we do this in a way


that protects users? Can we make the algorithm more efficient so it needs less connections to


figure things out? All sorts of things like this I think are really good goals to shoot for and as


more people, viewers, check this out, as more people like jump into the space and play with this,


these are some of the problems they're going to be building towards.


Is there any work on like screwing with the sensors? Like I imagine that if I, you know,


if I build an evasion attack that has like a really low hanging fruit of fixing it and that fix in


itself would somehow be, you know, completely devastating but I don't know it when I implement


it. Is there work in this direction? So is there work in the space of mucking with sensors?


Definitely. Crafting the kind of attack you describe is kind of tricky because we don't know


what the sensor's code looks like, you know. Now there is this idea of there are bugs and


limitations as they patch them may expose them to other attacks. So one quick example of this,


if we go back to our analogy of we're sending letters back and forth, a common limitation that


many less sophisticated sensors experience is they can't, if I've taken a packet or taken a letter


and I break into two letters, they can't put them back together. Yeah. Right? And that's like a huge


limitation. So it's really easy for me just to take a packet, split it up and send it through.


So to fix that, the sensor, all it needs to do is remember every packet it sees and then stitch it


back together based on the numbers on each of the packets. So that's like a simple fix to a


limitation. But when you apply that fix, you open yourself up to the entire space of attacks of


maybe I can sneak a letter in there that you think belongs halfway through the message, but it


actually belongs to the beginning or it actually belongs to the end or it actually doesn't belong


in that at all. And so you have, this is one example that we've seen in the wild where this


idea of I have, I need to fix the limitation and by fixing the limitation, I've opened myself up to


a dozen other potential attacks. So that definitely exists. How, how, how, I'm just thinking from my


newbie understanding right here, how much of a problem is it that our protocols are rather fixed?


I imagine if I could, if I had like a dynamic language where if I communicate with anyone,


the first step would actually be to negotiate a protocol in a very dynamic way, right? That would


sort of give me the possibility much more to together with the person that I want to communicate


with, negotiate something that could get around these sensors in a completely adaptive fashion.


Is that at all feasible? Or is there some some flaw? So is it feasible? Maybe. I mean, if such a


thing like that could be built, it'd be incredible. Yeah, it'd be awesome. So AI people, AI people


watching get on that because that sounds that sounds awesome. There are definitely some challenges


into rolling that out. And you basically need to get in the headspace of if I roll out this protocol


and the sensor knows about it, what is it going to do? What is it going to do? So there are,


there are protocols that exist out there where from the very first bite you sent, the whole thing is


encrypted. And in that case, it's pretty hard to fingerprint, right? There's, it never looks the


same. It's always just a stream of random looking bites, but the sensor can also find that just by


looking for something that looks like a random stream of bites. And just like you said, that


protocol never changes. It always looks the same. So if you need to really develop a system that's


flexible and dynamic enough that today it looks like this protocol, tomorrow it looks like this


protocol, today it looks like nothing in between. So you really need to be very creative and very


deliberate with how you do it. So I'm not aware of anything like that personally. Maybe someone's


working on it out there, but it would be awesome if you could do it. Now, speaking of mocking with


sensors, you also have other work that uses the censorship infrastructure. So essentially anything


that's in place from the sensors to perform some, some attacks. As I understand it, any, any attack


you could do is actually made potentially worse by the censorship infrastructure, such as a DDoS


attack or something like this. Do you want to talk a little bit about that? I would love to. Yeah. So


an area of work that we went, that we started exploring a year or two ago, something we noticed


for a lot of these sensors is when you interact with them as a user, like they need to respond to


you. They need to send you some traffic, right? Like if I'm, if I'm trying to request some resource


and that resource is forbidden, maybe the sensor sends me a block page and that block page says,


Hey, you're not allowed to access this. And the thing is that that communication there, what's


going on is my requests can often be much smaller than the size of the block page I get back. So as


an attacker, this opens up the space of, Hey, maybe I can use the sensor to launch an attack


at somebody else by making a request for forbidden things, pretending to be someone else and then


letting them send that huge response at that other person. And this is a, this is an idea of a


reflected attack or an amplification attack because as an attacker, I can make a tiny request and then


get a bigger request out of it. So I'm amplifying my traffic. So amplification attacks. So we started


exploring whether we could do this to sensors and use these nation state sensors or even just


beyond sensors, there's normal firewalls, like things that universities or just regular networks


organizations have deployed. We discovered hundreds and hundreds, tens of thousands, millions


of IP addresses that were behind these sensors that we could use to launch these attacks. Yeah,


I found these attacks got crazy powerful. And the so the the who does it hurt more the sensors or the


final recipients of the attack? Yeah, so in this case, the weight is buried by both, but the brunt


of the impact will be felt by the victim. This line of work, it mucks with the sensor, but really,


really the some of the I want to say the purpose or something you could distill this work down to


was sensors are causing more harm to the internet than they're not just the harm of a sensor is not


just restricted to the citizens within its borders. Like a sensor anywhere is a threat to anyone


everywhere. Yeah. So it's this the work was less about let's flood a sensors network and more about


let's proofs the world of these things are dangerous when when they've been applied as


carelessly as they've been deployed. Now, other than block pages, you have some you have some


very specific schemes of what you do specific to the censorship infrastructures that make these


attacks even more powerful. What are examples of that? Yeah, so discovering these attacks in the


first place, I'm making it sound very simple, right? You just send a request, and then the


response gets through. But I'm skipping over kind of an enormous step in here, because what I've


just described, send a request pretending to be someone else should not be possible. Yeah, that


that sentence should not exist. And it shouldn't be a thing you can do. And the reason that's the


case is because when we make requests all the time, this happens, I think there's a I think


there's a GIF in there that explains exactly what I'm saying, just to show up a little bit.


There's a three way handshake that we need to complete. And that three way handshake is just


this short exchange of packets. I think it's the one right about that. It's the short exchange of


packets at the very beginning right here short exchange of packets that exists at the very


beginning of our connection. And as an attacker, if I try and spoof the three way handshake, if I


pretend to be my victim and start to handshake, the server is going to respond to the victim.


And so I won't be able to get the critical bit of information I need from that handshake to finish


it. And I need to finish that handshake in order to make a request. So throughout all of the up all


of networking history, basically, up until this paper, it's been assumed that TCP, this underlying


protocol behind all these requests, is immune to these type of amplification attacks, largely


immune. There's a small caveat there, but it's not worth getting into. So how do we go about


addressing this problem? We used Geneva and AI techniques. And basically, we replaced Geneva's


fitness function. And we told Geneva, hey, you can talk to these sensors. But instead of rewarding


you for getting forbidden content, what we are going to do is we're going to reward you for


getting content without establishing a connection. And we're going to reward you for getting the


biggest content you possibly can. So kind of turning the fuzz around its head a little bit,


and letting it explore the space of strategies that A, confuses the middle box into responding,


so tricking it into thinking we have a connection already. And then B, once we've tricked it,


getting the biggest possible response we can. And so this is a second set of work that was


really powered by the same Geneva genetic algorithm. And we were able to use the same set of


building blocks and primitives and programs that we had developed previously, we just applied them


in a new way. And this is, if I understand it, it is not a weakness in TCP. Like if TCP were


implemented correctly, Geneva wouldn't be able or shouldn't be able to find something around this.


But this is specifically because these middle boxes are in there, right? Yeah, you're spot on.


TCP itself is not the problem. It's the implementation of TCP. Yeah. And that's


partially why when we did this paper, we did this work, you can't just study TCP itself,


you can't download the protocol specification, think really hard, because that's not going to


help you. You need to actually study real world sensors. So that's what we did. We took Geneva,


we trained it against hundreds of sensors around the world. And then we took the results of that,


and we're able to scan the whole internet. We scanned the internet almost 50 times, actually,


IPv4 internet, with these different packet sequences that Geneva discovered,


and effectively just attacked ourselves over and over and over again, to see what kind of damage


we could do. And how does that square? So before you said, we're never going to release anything


that helps the sensor in any way. And now you're releasing a recipe for launching massive attacks


on something, right? I mean, I usually think, you know, any technology can be used for like,


with that, I could actually attack the sensor directly, right? And just make their life


miserable. Using their own infrastructure, which is ironic, even. I could use it to, you know,


I could use it to DDoS the Red Cross as well. So my perspective usually is that any technology can


be used for good and for bad. But you've before said a little bit into the direction, we never


want to publish anything that helps the sensor. This seems to be different. What's different here?


Yes, the difference here is, and I want to note that we didn't just discover these and just


immediately put them out into the world. We spent almost a year actually just doing responsible


disclosure. We emailed every middle box manufacturer we could get in touch with and


gave them advanced copies of our paper, advanced copies of this attack. We actually emailed,


there's something called CERTs, country level emergency readiness teams. These are teams that


exist in various parts of the world that are basically designated to respond to network events


pertaining to that region. So we emailed all of them around the world. So it's like,


hey, that Chinese sensor you guys are operating, potential problem there. So we spent months and


months working with DDoS manufacturers, CERTs, middle box manufacturers to try and patch these


things and clean them up before this ever got out into the world. At the end of the day,


this runs into this broader responsible disclosure thing that a lot of the security


field that wrestles with of, if I never published this, there's often no incentive for this issue


to be patched. If there's no downside to the network, they don't need to patch it. And if


someone else discovers it before this gets out there, then they can start using it without the


world and the defenders knowing about it. So there's this really tricky line you got to toe


almost of, I need to let everyone have as much time as possible to patch it, but they also need


to know it's going to get out there to incentivize them to patch it. So with that in mind, we took


the approach of let's take as long as much time as we possibly can. Let's tell everyone, any


invested party about this attack, how to patch it, how to fix it. We gave them scripts to test their


network. And then after several months had passed and we were confident that if they were going to


take action, they already did, then we released the work. Yeah. Cool. Yeah. Now you're a member


of something that's called Breakerspace. I've already mentioned it at the beginning. Do you


want to maybe, because it's pretty unique, do you want to talk a little bit about what this is and


what it does? Yeah, I'd be happy to. So Breakerspace is a lab at the University of Maryland. Any UMD


students watching, come check us out. The Breakerspace lab, the kind of defining feature of


this lab is that undergraduate students are invited to join and participate in the lab. So


it's, the goal of this lab is to broaden and make research more accessible beyond just like


PhD students and graduate students who are doing it. So this Geneva team and the broader


censorship team within this lab has been staffed. I've been leading the team, but I've had a team of


undergraduates who've been working with me on these projects. So every project we've talked


about today and every paper on our website, this has not just been a one-man show. This has really


taken a village to get these off the ground and get these moving. They're huge, huge tasks. And


I'd be remiss if I didn't mention a huge team of students who have worked on this with me.


And okay, not unrelated to them being undergrads or not, did you, like, how often does it happen


that you get into like hot waters, like, you know, that there, you know, in security research,


there are implicate, there are national defense implications, there are legal implications and


so on. Like, how do you navigate that space? And how often does it happen that you're like,


oops, I hope no one noticed this. It definitely happens. And we're really lucky to have such a


supportive, like, university and atmosphere in which we can do these things. We've worked closely


with IRB, the Institution Review Board and our network security people. I mean, there was one


week where we, for that scanning paper we were talking about, we're like, all right, let's kick


off some scans. And then we immediately knocked out the university firewall. And it's like, oh no.


And they worked with us and helped us get it back and then helped to work in such a way that


wouldn't happen again. So what you're describing absolutely happens. I mean, one time we were


accidentally, we didn't know this, we were accidentally attacking like the city of


Jacksonville, Florida. And it was like, whoops, let's go email them. So that stops happening.


Like the University of Kentucky, things like this. So what you're describing happens all the time.


Yeah.


It's like, oh, shoot, whoops. And often those like whoops moments are like, that's a cool


discovery you just made. We also got to go fix whatever you just broke. So totally happens,


happens all the time. We got lots of crazy stories like that. We're really lucky to have


such a supportive atmosphere, which we can do these things. It's okay to break things


as a work to fix them, obviously, in such a supportive atmosphere.


Yeah. Where can people go if they want to get started in this space? Like let's say I'm an AI


researcher, I want to have a good understanding of whatever reinforcement learning and evolutionary


methods and genetic algorithms and all like, but I've not much clue of security. Is there


resources I can go to that you can recommend?


So for security in general, there's so many, I mean, I'm sure there's a two dozen YouTube


channels that could probably hook you up with like incredible, so maybe we can send someone


and look some of those below or something. I wish I could say that there is like this amazing


AI censorship. I want to say like censorship resource space where everyone can come to and


learn how to apply AI to these techniques. Something like that doesn't quite exist,


but there are great resources for learning about what censorship has happened in the world.


So something like OOni, OOni is O-O-N-I. It's the Open Observatory of Network Interference.


It's a spin out from the TOR team that monitors censorship all over the world. You can pull up the


website later, but they can identify censorship in basically every country. It's run by volunteers


and it's an incredible organization. So there's all sorts of groups like this that are studying


censorship, monitoring for censorship. So for people who want to break into this more specific


field of censorship, there's all sorts of great resources. Censored Planet is another group run


by the University of Michigan. They're an awesome team. They also publish all their data. So all


these groups have this very open sharing, like hop on their website and they got lots of great


resources, reports, data you can get your hands on. Excellent. Is there anything else you want to get


the word out to machine learning and AI people? Big open questions, anything that you feel


should be out there? Really just this whole space, like this whole idea of there's this entire space


of you can apply these techniques to in a way that's immediately impactful, helping real humans


on the other side and humans who kind of need this help. Like you have this potential to make


a real immediate impact on the world. So it's a great space to get involved in. Excellent. Kevin,


thank you so much for being here and bringing this a bit closer. I know more. I hope everyone


else does too now. Yeah, thanks so much for having me. This has been a blast. Excellent.


MLP-Mixer: An all-MLP Architecture for Vision (Machine Learning Research Paper Explained)

Hi there, I'm sure you've seen this paper make the rounds. It's called MLP Mixer and


All MLP Architecture for Vision. It's by Ilya Tolstikin, Neil Halsby, Alexander Kolesnikov


and Lucas Beyer of Google Research. This is not going to be a long video because the concept


is pretty simple. These people, did I say others or just the four names? I don't remember.


There are a lot of authors here. All of them deserve credit. This paper presents a neural


network that is just MLP. So just feed forward multi-layer perceptrons, no convolutions,


no attention mechanism, it's just matrix multiplications, nonlinearities, normalization, and I think


skip connections, but that's not really a layer, is it? So it appears we've come full


circle in computer vision going from MLPs originally to convolutional neural networks,


some pixel RNNs, then vision transformers. And by the way, this paper is going to be


much more understandable if you've read the paper on vision transformers, because it's


from largely the same people and does the same kind of experiments and methodologies.


And now we've come back to MLPs. Turns out the thing you've tried at the very beginning,


you know, it works after all. No, I'm kidding. So it's not just as simple as slap an MLP


on to the problem and that works. There is still a very specific architecture involved


right here. And also, I think the paper is mostly a lesson in what you can do with scale


and that good architectures might be good for a particular scale and not just good by


themselves. So the end result here is going to be that this new architecture that the


MLP mixer architecture performs adequately, not state of the art, not the best, but adequately


at large scales. And it appears to benefit much more from scaling up than previous architectures,


which raises the question, you know, what happens if we go to even larger scales? But


I guess that's for another day or year or decade. So let's just dive in. This is the


architecture, the computer vision architecture that is proposed. It's a classification architecture.


You see this right here. At the end, there is like a fully connected layer and a class


label. And also, there is a global average pooling. So at the end, you just collect everything


you've done, and you put it into a classifier and that gives you a class label. So that


means it's amenable to fine tuning where you freeze the representations that come out


of the model and all of this, this kind of stuff that you might already know. At the


beginning of the model, you have a picture. And like in vision transformer, you're going


to divide that picture up into patches. So in this case, you take something like 16 by


16 pixels as a patch, and those become your patches down here. And now you simply operate


on those patches as you propagate through the network. So unlike a convolutional neural


network, where you sort of shrink the resolution, but increase the channels, here, we're just


going to have one layer after another, one layer as big as the last one, stack, stack,


stack, and until the end. So it is much like a transformer. Of course, the difference between


this and the transformer is in how the individual layer looks. So like in the transformer, first


of all, every patch is fed through a fully connected layer to bring it into a latent


representation. So this right here, these right here are the latent representations,


they're of size that you choose as a model builder. And that's going to be kind of the


latent size that propagates through the network. So this is done on a per patch basis. And


this per patch operations, and you know, in general, these these sort of repeated operations


are going to be the key to this architecture right here. So every patch is projected using


the same function into the latent space, okay. Then we this is followed by n of these mixer


layers. Now, what does a mixer layer do? And here is where the core comes in. So in every


layer, you start out with, you know, you've just seen here, we had patches, but now we


have these latent embeddings, like this stuff right here. This essentially is one vector


for every patch. So every patch, you unroll the patches, like so, and every patch gets


you one vector, right? Every patch in the image corresponds to one vector. So technically,


this here, you can interpret this as a table. So that's what they do here. It's just the


other way around, right? So this, this here is the lower left corner. This one is the


patch right next to it. This one is the patch right next to that patch, and so on. And each


patch has one, two, three, four, and so on channels. Each patch is described by a vector


of whatever how many dimensions, I guess something like 512. And now, if you traditionally, if


you solve this problem, and you said, well, I have an all MLP, an all MLP architecture


for vision, what you would do is you would take that table and completely unroll it into


one vector, right? So the, the top patch would then be here. And then the blue patch would


be next to it, right, this, this blue patch right here, and so on. So you would completely


unroll that, that's the yellow patch into one single vector. And then you would put


a fully connected layer on top of that. That's not what we do here. We're doing much more


like what we would do in a convolution, except that we only have filters of size one by one.


So there are two different, two different in this mixer layer, there are two different,


how should I say this, modes of operation. First, we do the following, we flip this table


table, we transpose this table. And so that means every row here is the same channel from


all the patches. So it's always channel one from all the patches in the image, right?


So from all the patches, I want channel one, and I'm going to feed that through a fully


connected layer. I also take all the patches, but channel two, so channel two from all the


patches, I'm going to feed that through the same fully connected layer. In fact, you can


see these weights are all shared right here. So this is weight sharing across different


channels, across always across the same channel of the different patches. This is much like,


you know, one by one convolution. So actually, this one here is more like a one by one convolution,


but it is weight sharing. Okay. And that means we have a picture, we put it into patches.


And in this layer, what we care about is connecting the same channel, not even sure how to represent


the same channel. I guess you can say you want the same type of information since this


all builds on the weight sharing of the last layer, right? So this fully connected layer


right here, it's the same for every patch. So that fully collect connected layer might


look at the patch. And if there is something like a sharp corner in the top left corner


of that patch, it might put that into channel one. So now all of the patches that have that


in the top left corner, like some sharp corner here, will have that in their first channel,


okay. So now if I aggregate among the same channels, if I do this, then if the first


channel here reacts across the patches, you know, I can aggregate all the patches that


have that feature, because the feature producing map was shared, okay. So all of this builds


on the fact that in the last layer features were shared too. So here, we share the projection,


which means that the channels in the individual patches mean similar things, okay, because


they come from the same function. And since they mean similar things, we now group by


those channels and aggregate or compute over all the patches in that particular channel.


And since that particular channel has the same information, you know, that sort of lets


us compute on a on a feature by feature basis. Now also, of course, these weights are shared.


So since these weights are shared, that means sort of on a meta level, that now I'm going


to perform the same computation in all of those channels, which means that now I can


I can do the reverse trick again, and flip the table back into patches, and then do this


shared computation for all the patches. So ultimately, I just have number one, one weight


matrix where I forward propagate all of the channels individually, but in the same way.


And here I have another one. So that's number two, I have one forward propagation matrix,


where I propagate all of the patches individually, but in the same way, right. And again, since


I now have done the same computation over here, that means that the result here is going


to be sort of distributed in the same way across patches. Now I aggregate this into


the patch location. And I forward propagate this, this is much more like a one by one


convolution, right. So we simply take a patch, and we apply a computation across all of the


channels of that patch. And we apply the same computation. And that prepares the exact same


thing for the next layer. I hope that makes a little bit of sense. I have trouble articulating


this, but it does make sense when you think about it. So there's two phases, you repeat,


you look, you repeat two steps. In this step, you look at your patch and you say what kind


of features are there, right. And you put the features into predefined categories. So


channel one is, you know, feature one, channel two for feature two, and so on. And then in


this step, you take a look across all of the image. So step two is here within the patch.


And step one is actually you look at all of the image, but only in that channel. That


means only for that particular feature, right. And then you look, okay, where in all the


picture is that particular feature, you do some computation across where that feature


appears and how, and then you go back to step number one or two, however, I labeled it here.


I hope that helps a bit. The MLP is not really, I didn't really say this correctly, you don't


have one matrix. In fact, it's two fully connected layers that are separated by a non linearity.


However, this Yeah, it, it's not one weight matrix, it's two weight matrices. They are


shared though, across channels or across patches, depending on the step. And that's it. That's


the architecture there is, as you can see, layer norm, you also saw this here in the


diagram, there is always the layer norm layer involved here is this, yep, and here. And


there are skip connections, as you can see at the top. But largely, that's the architecture.


So what does this give us? If again, if you've seen the vision transformer paper, this is


or the big transfer paper, all of this is extremely similar in terms of architectures.


What they do is they build a bunch of different sized models with different patch resolutions.


So this, see, the resolution is always the number after the slash, right? So here, this


will be 16 by 16. So obviously, the lower this number, the higher the the resolution,


where the higher the resolution in which the model looks at the picture, right? Now, one


advantage here is that compared to, for example, vision transformers, is that vision transformers,


of course, due to the attention mechanism, they have a quadratic requirement of compute


and memory as they go as they increase the sequence length, which means as they lower


this number right here, their number of patches in the image increases, and therefore, they


suffer quadratically, while this model only suffers linearly from this. And that is the


point they make here in the experiments. So the experiments is it's sort of a repeating


pattern. And the repeating pattern is, you know, if you look at the best models, and


let's say ImageNet top one, or very good models, we're not quite as good, right? If you know,


depending on so they pre train, they pre train on large data sets, and then they transfer


learn, or they linearly classify the frozen features. And the story is always the same.


It's, yeah, you look at us, we are sometimes, you know, even better than this, but we're


not we're not quite as good as this. However, we are competitive, right? That's the the


the core message here is that we are competitive, you know, competitive. If if this had been


on the market a couple of years ago, this would have been state of the art by far. But


now, the this model is it's competitive, it achieves okay performance. And since that's


not what we like to hear in machine learning publishing, I think that the big lesson if


you want to publish something here is that find a metric where you win. Okay, so they


say, you know, we might not be the best ones in classification accuracy. However, we're


okay. And we have a better trade off. So there are a number of trade offs they look at right


here. For example, throughput, you see this right here, throughput, images per second


per core during inference, this is something that's really important to practitioners to


people that actually have to deploy these models, right. And you can see that the throughput


of mixer here is way above these other models, of course, because you know, convolutions


here, they're, you know, they're a difficult operation. And also this this big transfer


model, it has a lot more layers, I think, than the the mixer or vision transformer.


And of course, the vision transformer itself has that attention mechanism. So not only


does it have that quadratic requirement, it also has the sort of computation of the soft


max itself and so on. And also, if you look at how much you had to put into training,


in this case, vision transformer is actually outperforming mixer. But in all of these tables,


you always have at least one metric where mixer is better, you just have to select the


metric. So, for example, you can see that, well, this, I like this more. So here, it's


linear five shot ImageNet top one. So if I understand this correctly, this is you train


a linear classifier on the frozen representation of what the model gives you, you evaluate


it on top one accuracy, but you get it's a it's a five shot classifier. Okay, so it's


a very particular task. And they look at what happens if we modify the training set size.


So the size that we train on. And you can see that in this framing, this model scales


much more favorably than other models. So big transfer, which is good at you know, low


data set size, all of a sudden plateaus and doesn't increase any more or much more. When


you scale up the data set by a significant factor. However, the mixer model scales really


well. And in fact, at the end is on par almost sometimes with the vision transformer, even


here, it's even a bit higher, right. And specifically, it's also higher than the big transfer model.


What you can also see is that there is a significant gap at small training data sets. However,


that gap, also here, that gap always appears to close as you go up. So the gap here, and


here and here is way smaller. And as we already said at the end, very often they are on top


of one another. Now, this raises a bunch of interesting questions. This is by the way,


it's not only this task, right, they show this on a bunch of tasks. That it's the this


model benefits from scale a lot more. It is it has a higher throughput is a simpler architecture.


Yeah, it scales in terms of what you need to put in as compute into pre training. And


yeah, so here you can see the ImageNet transfer accuracy compared to how many core days on


a TPU v3 you put in. And you can see that the mixer and the transformer models, they


lie on very much similar curves, leading actually leading the big transfer model. So they are


computationally more efficient. And also here, in terms of throughput, you can see that for


a given accuracy, right, mixer and transformer have higher throughputs than big transfer.


And for a given size of model, mixer has a higher throughput than vision transformer,


though vision transformer makes up for that by being more accurate. They have very, very


extensive evaluations to show that they are, you know, that this model is something I believe


this model is something that if you really care about deploying it to large scales, you


might want to take that performance hit, right? In, you know, to trade off for better throughput.


I think that's, that's fairly clear from these evaluations. Now, it remains to be seen how


this model performs in different settings for different data for different tasks, and


so on. And when this is ImageNet and ImageNet after pre training with particular data sets.


So here, they pre train on ImageNet itself. And if you pre train on a small data set,


the model sucks, right? It really trails, it really trails other models, you can see


right here, if you pre train on a slightly larger data set, it still sucks, but it doesn't


suck as much compared to others. If you pre train on a really big data set, you can see


that it only sucks a little, a little bit. So you you're hard pressed to find a number


here that's higher. And that's, I think, the point they make. Now, the interesting question


for me is, is this, like, how does this go on? As we go higher, like as we go one order


of magnitude higher in our data set and compute, and so on? Is it the case that the mixer continues


rising while the vision transformer sort of plateaus out, which would be really interesting,


because you could you could then make the case that the vision transformer actually


has more inductive biases than the mixer, because both seem very general, right? And


I would personally argue that the vision transformer is more general and has less inductive biases,


because here the mixer, first of all, the weights are fixed. And second of all, there's


this very particular chessboard pattern to how you interact with the input data, right?


It almost seems like there are lots of biases here. Now, these things, these, this inductive


bias might be just super duper duper correct for the particular modality we're dealing


with, like natural image classification. Or it might actually be that the mixer transfers


to other domains, and works really well, in which case I might be wrong. It also might


be the case, of course, that both plateau, in which case that would just mean with enough


scale you can get pretty much anything to work, right? So, you know, if you're cynic,


you can say, well, even a crap architecture like Mixture, you can get to work by just


scaling it up and using SGD. And yeah, which might also be true. Ultimately, in the limit


of scale, as you have the entire possibility of all images as your data set, you can of


course just perform a k nearest neighbor classification, and you'd be correct 100% of the time. I don't


think we're there yet with the scale. But the, the sort of trend is relatively clear,


but it will be really interesting to see how that goes on after, you know, after our current


limits. The last thing they show here is the weights. And so they make a couple of interesting,


let's say, interesting observations here. These are the token mixing weights. So every


point here corresponds to sort of one patch in the original image. So this is how do you


aggregate information within the same channel across different patches, right? And they


make some observations, namely, for example, that the weights here appear, for example,


in pairs of negative positive. So blue and red here are high and low values. Also, in


the lower layer, so if I'm correct, this is the first, the second and the third block.


So this, this is the lower layer down here. And the high layer is here, you can see that


in the lower layer, you have rather large scale general features that are learned, while


as as you go higher, you have much more specific interaction specific weights that you learn.


And this all is very reminiscent, let's say, of how we think or how we observe convolutional


neural networks work. So it's a good case here that the model learns something that


it is that is sensible. You can watch all of these weights, I think they have it in


the appendix, they have the full weights right here, also pre trained on different data sets.


And this is really interesting, too. So if you pre train on ImageNet, it looks qualitatively


different than if you pre train on ImageNet 21k, which is just, it's a it's a it's larger


with more classes. And that's also significantly different than if you pre train on this JFT


300m, which is a super huge data set that's proprietary held by Google. And it's still


I think that it's still unclear whether these differences are an effect of scale, or an


effect of how how how accurate the downstream model is. So like, let's say, an effect of


how well, how much signal there is to learn, independent of scale, or whether it is actually


just a property of the data sets being of a different nature. And that would also explain


why ImageNet and ImageNet 21k are seem to be a bit closer together visually than JFT


300m. Don't forget that JFT is a huge data set. The code is open source. In fact, it's


right here. You can just take it. Also, I've seen already a bunch of people implement this.


So this was it for me for this paper. Again, this is not it's not very complicated. It's


a very simple architecture, which is exactly its selling point. Its selling point is it's


simple. And that means it can scale up really well. It's trade off between compute and accuracy


is really good. And you should consider it if that's something that's of importance to


you. From a research perspective, it raises a lot of questions about inductive biases,


how scale behaves and whether you can get anything and everything to work with SGD and


Introduction and Basics - Python Reddit API Wrapper (PRAW) tutorial p.1

What is going on everybody and welcome to a Python reddit API wrapper or pra tutorial


series.


If you don't know what reddit is, reddit is just a place with um... well welcome to the


internet.


So anyway reddit.com is like a forum for pretty much any contextual information.


So there are basically just a bunch of subreddits that comprise reddit and then a subreddit


is basically the contextual part where you've got subreddits for like Python programming,


news, politics, technology, just pretty much everything you can think of and if you can't


think of it you can make your own subreddit for it.


So anyways that's reddit.com.


If you don't have an account go ahead and sign up just fill out the form and all that.


I already have an account for this tutorial so I'm going to go ahead and log in.


It is pra tutorial, password is cookies.


Don't hack me.


So now we're going to go to preferences once you've created an account.


Go to apps and then go ahead and create an app.


And this app needs to be a script and then we can give it a name.


I'm just going to call it pra tutorial.


Good stuff.


Tutorial.


And we're going to say in the description tutorial on le pra, pra, pra, thank you.


And then for this we'll just do pythonprogramming.net.


Cool.


Okay so now we'll go ahead and create an app and then here is your client ID and then this


is your secret.


I'm going to move this over for now but we'll be, that's useful information that we're going


to need momentarily.


So what we're going to be using is pra so these are the documents for pra.


You should just, it's a pip install pra so if you need to get it just pip install pra.


And these are the documents but if you just search for like pra on google you'll probably


get documentation for 3.6 which you actually want to get is just the latest documentation.


So if you do go through the documentation just make sure it's the latest one because


lots of things have changed.


So anyways moving this aside, in fact I think I might even just close this.


I think I'm all done with it.


What I'm going to do now is open up here, pra tutorial, make sure it fits on my page


and we're just going to do like a really quick example of querying some information.


So let me make sure this fits.


So first thing we're going to go ahead and do is we need to import pra and then once


we've imported pra we've got to like set up like a reddit instance basically.


So I'm going to say reddit equals pra dot reddit for reddit instance and then you're


going to need client id, client secret and we should probably say equals.


We're going to fill these in in just a moment.


And then you need a username, password and a user agent.


I'm not sure why you have to have a user agent but you do I think.


So client id is what I just showed you guys a moment ago so this is my client id right


here.


This, this, boom.


Secret right here.


Username is pra tutorial, password was cookies.


User agent, I mean you can put anything in here so I'm not really sure.


Pretty sure at one point, like I've used just the straight reddit api before and you actually


had to have just anything in the user agent but not sure why it's required.


Anyway there you go.


So now we've got our little reddit instance and we are ready to do stuff with it.


So for example, like I was saying before reddit is comprised of a bunch of subreddits so let's


go ahead and specify a subreddit.


So we're going to say subreddit equals reddit dot subreddit and the subreddit we're going


to choose is python.


Now let's say we want to, because like I was showing you guys before, like if we go to


reddit dot com, here we go pythons right here.


You can see like you can sort of things in different ways like hot is a combination basically


of votes to time.


New is just purely time, what's the newest first, kind of like a forum or something like


that.


Rising is just stuff that is rising quite quickly.


Controversial is just anything that has a combination of lots of upvotes and downvotes.


Top is I think just votes and possibly time.


I'm actually not positive what the difference between hot and top is.


Someone wants to post below what the difference between hot and top is let me know.


It's definitely ordered by votes for sure.


Maybe it just favors votes slightly more but as you can see you don't really see too many


more.


Maybe top, oh this is top from the last 24 hours.


So then we could say top from the past year for example.


These are just like top posts from everyone in the last year.


So you can do like any of these that you wanted.


Now actually I'm not positive how to do this.


You could probably check the documents though for how to do this next like links from x,


y, z.


Anyways we're going to look for just hot.


So we're going to say hot python equals subreddit.hot and then what we're going to do is we're going


to just kind of go through these.


So for example we could say like dot hot I forget what the default is.


I don't know if it's like 30 or 200 or I don't know what the default number is but you can


also pass in a limit here so you can say limit equals one.


So that will just return the hottest python information at the moment.


So what we can do now is iterate through whatever the return is.


So this is one but we could say I don't know five.


So for submission in hot python we can just let's print that submission and see what we're


working with at the time.


Okay so here are the submissions.


So these are basically your ids.


So this is a thread id for whatever these topics are.


We don't really know too much more about them.


Now I happen to know it's an object.


You might not know it's an object.


It could just be an id.


So I mean with any module if you like if you don't really like reading docs you can just


do dir and that will give you all the attributes.


So for example this is going to put out a lot of dir but anyways let's just go up to


the top.


This is like one of the dirs.


We should have just done one.


But anyways here you can clearly see okay here see like these are a bunch of dunders


up here but as you come down here we can see right away like okay approved when or whatever


and a lot of this stuff is like you know okay when was it created created UTC delete which


it would have to be yours you have to have the rights to delete it but you can get try


to find something useful here that's typical like you can reply to it you can check how


many upvotes it has you can upvote it although don't do that as but you can grab the title


anyway I'm not going to focus too much on that but anyways if you want to like you can


just kind of go through all the things but so for example let's just grab the title so


print rather than dir submission and rather than submission which is just the id but it's


actually an object we can reference dot title so the ad like just with anything in python


the attribute dot title should be populated now pretty much everything you see in the


pro that has you know basically it's a function call you can pretty much guess that's going


to be yet another API call that so that data might not exist just yet let's go ahead and


run that again and there you go so you've got the first five so this one's the official


job board what's everyone working on this week and if we go here let's go to hot we'll


see that these are not actually like these are these are stickies so we might not actually


want those right we might not be interested in that information so one of those things


was one of the attributes rather is stickied so it's a just a boolean value so for example


we could say if not submission dot stickied sticky print submission dot title so now we


should just get the non stickied ones so there you go so I guess these just kind of look


it should just be three I think it's just because of wrapping yeah so anyways even though


we went through five we still we only posted the title to three of them because those first


two were just stickies so so now we can get all kinds of other information on them so


like rather than just submission dot title we could we could get all kinds of stuff like


for example we could say you know title semi colon ups not semi colons curly braces downs


which honestly downs doesn't seem to work we'll see that soon enough but have we visited


visited curly brace now dot format and then just same as before submission dot title submission


dot ups submission dot downs submission dot visited cool so we'll run that again okay


so for example title guide on how to create a twitter bot with python sorry it's a little


bit off screen but you can see it but anyways it says visited false well let's see what


happens if we click on it here I can't remember if you view the comments or not surely it


would still it would say you visited but let's find out it still says false maybe I got to


actually click the link I don't know let's click on it let's try again let's try again


the other thing I wanted to point out was the ups and downs they're kind of funky okay


it still says we haven't visited I swear I've tested that in the past and it was fine maybe


you have to visit it via the API I don't know anyway literally everything has downs zero


unless it's in the negatives so maybe the pro hasn't like updated to like a new change


in the subreddit API I really don't know anyways on any submission you can get all kinds of


information you can do all kinds of stuff so just as some examples you know here we


just grabbed a bunch of information but you can also you can do all kinds of things you


can take action so you can do like upvote right you can do dot downvote although as


a bot again you wouldn't do that the only times you should really use these commands


or like if you make some sort of reddit like a phone app or something like that and it's


like a human that decides to do it so that's doing this would be like against the terms


of service basically you can reply there's nothing in the docs it says a robot can't


apply reply sorry and then you can also subscribe and unsubscribe so at least some things that


we could do we could make a reply I just don't really know what to reply but we can also


subscribe so for example we could say subreddit dot subscribe and that should subscribe us


to the python subreddit let's go ahead and run that real quick it still says we haven't


visited it's irritating but let's see if we subscribed at least so initially we're not


subscribed to the python subreddit we are subscribed we could I think we should get


away with this subreddit dot unsubscribe and now if we were to refresh this page we need


to subscribe I swear I haven't drank all my coffee this morning so let's subscribe one


more time I'm pretty sure it said unsubscribe but the button I don't think changed color


that's what's tripping me up right now usually it's like red or green yeah okay now we have


an unsubscribe I have to go back and look at the video and see if it said anyway we


have subscribed did I do just a dot no I actually ran it so I don't know should have subscribed


but that's weird because it was like it never changed color at least I don't think so I


might be losing my mind but anyway okay so that's like a quick kind of introduction to


the python reddit API from here you could just read documents or use dir and attributes


and maybe figure a bunch of stuff out on your own otherwise in the next tutorial we're going


to be kind of going through and really honing in on actually navigating like the comments


because navigating the comments is significantly more challenging than navigating the just


submissions right the submissions are always ordered very nicely whereas comments are like


in tree form it's kind of a recursion problem and it can just kind of be a little more challenging


so anyways that's what we're going to be doing in the next tutorial if you've got questions


comments concerns whatever feel free to leave them below otherwise I'll see you in the next


Memory-assisted prompt editing to improve GPT-3 after deployment (Machine Learning Paper Explained)

Hello, this is a comprehensive paper review on the paper called memory assisted prompt editing


to improve GPT-3 after deployment. As the title says, this paper is really cool because it is


able to improve these large language models after they're deployed. So this video right here is a


comprehensive review on the paper. After you've watched the video, you'll have a good idea of


what the method does, what it is and what the paper describes. The next video released tomorrow


will be an interview with the authors of the paper. And that is also really cool. And I


definitely learned a lot from that as well. So I invite you to check out both. And I'll see you


around. Have fun. Hey there, today's sponsor is the course on introduction to graph neural networks.


This is a course by my friend Zach Joest, who is an expert in graph neural networks. He's packed


all his knowledge into one course that will educate you on both the theoretical and hands on


practical aspect on graph neural networks. Graph neural networks are really important. They're


definitely one of the most interesting areas in deep learning right now. They've also powered a


lot of recent advances in scientific breakthroughs, such as alpha fold protein structure predictions,


or better traffic predictions. If you use my link, you'll get a 15% discount on the course


enrollment is open right now and lasts until April 1 or until spaces run out. Alright, let's


get into the video now. See ya. Hello there. Today we're looking at memory assisted prompt editing


to improve GPT three after deployment by Amon Madan, Niket Tandon and others. So this paper


proposes a method to improve GPT three in an interactive mode in a user feedback mode. Here


is a little sample of how that could look like. So the user would pose a question to GPT three,


for example, what word is similar to good. And this is not displayed here. But in advance of


that there'd be like an entire prompt like you would be used to for prompting GPT three, if you


don't know about GPT three, I've made a video on GPT three, extensively describing how that works


and how to construct these prompts right here. So that GPT three gives you what you want,


supposedly because it doesn't always work. For example, here, the user asks, what word is similar


to good. And GPT three says the homonym of good is wood, which is kind of true. But the user doesn't


is not specified clearly what similar means the user here had a different intent, which then the


user specifies, the user says similar to means with a similar meaning. So the user didn't mean


a word that sounded like good, which is wood, the word the user meant a word that is kind of like a


synonym instead of a homonym. So in this new system, this thing right here would be called


feedback, and the user would be able to give this feedback to GPT three. And then GPT three would


write that to memory, it's not actually GPT three, it's sort of like a plugin that the


paper develops. And then the user, the next time the user asks, for example, what word is similar


to surprised, the system will remember that the last time the user asked the question like that,


like similar to you know, what word is similar to another word, the system will go back to the


memory, retrieve the feedback right here, put it into the prompt, and then guides GPT three to


actually to actually answer in the correct way. And so GPT three here says the synonym of surprised


is amazed. So multiple things to see right here. First of all, their plugin, the system that the


paper here proposes can be added to any pre trained language model. And the language model


itself doesn't have to be changed, which is really important for something like GPT three, because


that's too big to to change. I guess you can fine tune it. But you need a lot more data than just


two or three examples. The other thing is that it is interactive. So this is an interactive user


session where the user can specify not only clarifications for things that are clearly wrong,


but also maybe personal preferences. So this is goes beyond what this paper shows. And this paper


is mostly about either factual accuracy, like accuracy of the task or figuring out user intent


from ambiguous meanings. This could easily be used to personalize interaction with GPT three


for particular users by interactively letting them improve the system. This is kind of like what,


you know, normies think of AI is like a system that learns from the two or three times they give


it feedback and then gets better over time. So this is pretty cool. And lastly, what was I gonna


say? Um, I don't I don't remember anymore. But we're going to look at how this works. And you


know, what's good about it? What's bad about it? And yeah, that's that's about it. So here is the


proposed before and after of the system. If the user with no memory asks GPT three, the user gives


an X. As we said, it's it's always prefixed with some sort of a prompt that guides GPT three into


giving the correct answer structure or type of answer. If we're going to look at some of these


prompts in just a second, and GPT three will give some sort of an answer. Now, this might be good


or bad, as you may have seen, it can turn out not the in the best way. So in their memory enhanced


GPT three example, the user would give a question x. Now let's disregard the memory for now, let's


just go directly to GPT three, which is what happens in the very first iteration of this interaction.


So GPT three now has a prompt in front of it as well, but a prompt that the author is here designed


such that GPT three doesn't only give the answer to the question, but also you the understanding


of what the user meant. So up here, you can see that by GPT three answers, the homonym of good


is wood, right? GPT three doesn't just answer wood, which would be the answer. But also this first


part right here, which is this understanding. So the authors construct the sort of meta prompt


that they give. And that instructs GPT three not only to to give the answer, but also to give the


understanding like a clear, a clear output of what it understood. The user can then take that


and decide if that's what the user wanted or not. So if the user is happy, then all is good. If the


user is not happy, the user can give feedback to GPT three, the user gives feedback in kind of in


natural language, just like types it up like, No, I didn't mean this, I meant this other thing. And


you have to type it up in a bit of a special way, you have to type it up, you can't just say no.


I guess you can. But it's best if you write like, similar to means with a similar meaning. So you


clarify your original question right here. And by doing that, you committed to the memory. Now,


obviously, what you could do is you could simply add that clarification to the prompt, go back to


GPT three, and actually let it answer correctly, which would work. But we're not only about this


prompt. The idea here is that this feedback will help guide GPT three in all subsequent prompts,


because the user is likely going to express themselves in the same way. GPT three, if it


misunderstood is likely going to misunderstand in the same way. So this memory serves as a


bit of a generalizable correction mechanism that learns from few items of feedback. So let's let's


look what happens the second time around. So the second time the user again has a question x, we


then go first to the memory and we see, or x prime, let's call that x prime, we see is there


anything in the memory that is similar to x prime, meaning that is there any question before that


has been submitted to GPT three in the current session doesn't need to be in the same prompt or


anything just in the current user session that has been misunderstood. So do we have an instance


that is close to x prime, where feedback was given, that would be part of the memory. And this


is being done with either semantic similarities, or so you take some sort of a simple question


or some sort of a of a language model or some sort of a sequence model, for example, a transformer,


you look at the embeddings of the sentences you compare them via cosine similarity, you can also


do word overlap or something like this. But what you want to do is you want to retrieve those


instances of feedback. And then you want to add that feedback to the prompt in the very case in


that use. So this is hidden here. This is hidden, it just says and adds to prompt and we're going to


see how this happens how the system adds that to the prompt. It's actually quite simple. It's mainly


a concatenation adds it to the prompt. So the users, this is the x prime right here, the x prime


is being augmented with the feedback that the user has given previously, and then submitted to GPT


three. And with that feedback, GPT three is now able to actually more likely give the correct


answer. And if you know if it's misunderstood, the user can give feedback again. And that would make


it even better in the next few iterations. So this is the overarching system, the paper makes


pretty clear that it doesn't propose like, it doesn't purport to be the state of the art or the


final the final system in this framework, it simply wants to present a framework. That's it


states that I think, two times or more. Now, there I have mixed opinions on papers that say, well,


we just want to present a framework. On the one hand, it's obviously good to present a framework.


Your paper like papers shouldn't be rejected, if they have a good idea for a new framework,


just because they can't get it to, you know, be super duper performant. On the other hand, saying,


you know, we just want to propose a framework is very often as either like a cop out for not


reaching good numbers or just kind of like, you know, we, we want to split one paper into two


papers, because the next paper is going to be sort of the well performing thing. Or it just,


there's a danger that it's not super well thought through, because the authors haven't actually put


in like massive efforts into making this good, at which point, many flaws reveal themselves in these


types of frameworks. But the frameworks pretty general. So, you know, we'll give them we'll give


them that. They claim Yeah, so this is what I just explained, they maintain a memory M of feedback


as a set of key value pairs. The key is a misunderstood question. And the value is the


user's feedback to correct that misunderstanding. Given a new question, we check if the model has


made a mistake on a similar question earlier, by querying the memory for a similar question,


if found append the corresponding feedback to the question prompt. And the idea here is where they


say, not definitive, rather, our main contribution is a general framework itself suggesting how user


feedback might continuously improve model performance without retraining in a few short


prompt setting. So let's look in in a little bit more detail into the system. The system has four


distinct parts, this memory that we've just talked about, that's a growing table of key value pairs,


the key being questions that have been misunderstood, and the value being user feedback.


So obviously, the user only chooses to give feedback if the user was misunderstood. And


therefore, the memory only contains those things. There's a lookup function, which I guess is the


most complicated or most complex or complicated, which I'm too surraged. The most complicated


of the functions right here, it's they call it a learned retriever that matches the query against


all the keys of M. So that's where we retrieve similar prompts that have been misunderstood in


the past. And as I said, we can do that with a pre trained embedding, for example, of a transformer


model, or any, any sort of embedding model for text, or any other thing they use Levenstein


distance for some experiments. So the combiner is a gating function allowing irrelevant retrieved


feedback to be ignored. I don't I don't think they actually do. I don't think they do that right now


to ignore irrelevant feedback, other than thresholding the lookup function. So the lookup


function is an inner product. And I guess the combiner is the threshold on that inner product,


on that inner product, the prompter here, it passes the output of the combiner to the prompt.


And so that in our case, this is just going to be a concatenation of the prompt, and whatever the


combiner outputs. So it's going to be the prompt plus the question if there was nothing found in


the memory, or the prompt plus the question plus the feedback if it was found in memory. So I would


add Yeah, let's let's get into the task. And then we'll get into the actual examples. So they have


two kinds of tasks. The first kind of tasks, there are five tasks that are broadly in the


category of word scrambling and manipulation, for example, to reorder some letters, these are


reordered in exact reverse. Other there are other there anagram one anagram two and so on.


There are very various tasks, five of these, and there are five lexical QA tasks,


which are asking GPT-3 for a synonym for an antonym for a homonym, and so on.


They say for each task, the prompt contains a few different variations. For example, what is


the homonym of a word? What sounds like the word? They create a data set. So this is where


Yeah, we'll get to that as well. They create a data set of samples, feedback, understanding and


the solution. So essentially, without the feedback, this would be what you would give to GPT-3 as a


prompt. They also collect feedback so they can simulate users. So they, they, they give the X


to GPT-3. And if it is misunderstood, they do that in a they determine that in a heuristic way,


they also provide the feedback to the memory. They come up with sort of invented data sets,


sort of invented data of users being understood or misunderstood.


The yeah, the retriever, as I already said, is either a semantic similarity using the cosine


distance with a threshold, or a lexical similarity and heuristics for similarity matching.


The combiner concatenates X and the feedback received by the retriever.


And the prompter concatenates the prompt and whatever the combiner outputs.


We didn't have one of them though. Oh, no, the combiner is the gating function. Okay,


that doesn't, it doesn't seem like much of a gating function.


Yeah, so I want to jump over the results quite quickly to show you some examples of how that


even might look like. So here is a prompt for the tasks. I think these are the lexical,


the lexical QA tasks. So asking for antonyms and homonyms. This is the entire thing that you would


give to GPT-3 in front of your question. So you would append your question down here somewhere,


like below the prompt in the same style as the prompt. So this is, this is a, this is how you


query GPT-3. What you would do is you would simply give some examples and prime GPT-3 to continue


the pattern. So they hear, they ask, what is the homonym for ring? The homonym for ring is ring.


Now these are all human generated, right? All of these are human generated. So you prime GPT-3 to,


you know, what, how, how questions are asked, and how answers are given. The important thing


right here to see is that all of the answer patterns they provide is, it's not just the,


the answer, for example, permit is the antonym for prohibition. The answer also contains this


understanding part, this thing right here, the antonym for prohibition is, that's the understanding,


and this right here is the label. This is important, because the understanding is what the


user uses to decide whether or not GPT-3 has understood the question. What they also do later


in the same prompt, they, as you can see, they also add questions with feedback. So here you see


how they incorporate the feedback. There's like this, I don't know what that's called a pipe


symbol. And then it says clarification, colon, and then this here is the feedback. Okay, so this is


also part of the prompt. So the prompt contains some generic feedback, where there is some sort


of an unclear or ambiguous question, then there is feedback, and then there is the correct answer


that is based on the feedback. So you can see right here, the question is, and that's pretty


special, the question is, or up here, it says, what is the synonym for right, and then the answer is


the synonym for is. So it always goes after the question, how the question is formulated,


the understanding goes after the question. However, they prime GPT-3 with the question,


they prime GPT-3 that if there is a clarification, you can see that the answer goes sometimes


partially sometimes fully on the clarification, what what I mean by goes on, I mean, it,


it refers to so the understanding reflects the clarification that allows multiple things it


allows, if the user is still not understood, it allows the user to give feedback again.


And also, it primes GPT-3 to actually pay attention to this clarification part.


So in the prompt, you'll get a bunch of these clarifications to teach GPT-3 how to include


these clarifications in its output. This is pretty smart it so the prompt is not only a prompt for


what kind of answers you want, the prompt is also a prompt for this understanding part,


which is a necessary precondition of making the of making the system interactive.


And the prompt also includes the next step of the interactivity and how to react to it.


This is, I think this is a good piece of prompt engineering.


People are getting better at this by the day. So this is this is before the question even gets


here. So the question would be added here. And if there is feedback in the memory, the feedback


would obviously be appended with a pipe symbol and clarification. And then the feedback would be added


here. And then GPT-3 would be prompted to give its answer right here. You can see if there is


something in the memory, GPT-3 already knows how to use these clarification parts right here.


So it's pretty good. Yeah, that's there. There are a bunch of examples, you can we can we can maybe


look at them or you can look at them. What I want to look at lastly, is the data set generation. So


they simply say that they created a data set, we manually created 15 task templates with three


variants of phrasing the question for each task. You know, this is this is fine. This is prompt


engineering. They also they also do come up with sort of the variations for the feedback. Where


have I data sets templates, raising each question? Okay. I cannot, I can't come up with but it is my


understanding that they create the entire data set. So they create the prompts, and then the tasks


they get from other papers, for example, the synonyms, the homonyms, and so on, they get from


other data sets that other papers have as well. But then the feedback, the feedback, they also


do themselves and there is a danger right here, because they create the task samples for prompting


right. And also us here, here, they, they create, they create the prompts, they create the task


samples for the prompts, they also create the example feedbacks, and they create the data set


of feedbacks, which is dangerous, because that might lead to, you know, me just kind of formulating


these tasks at templates, not as accurately as you know, maybe I could. And then obviously, once I


clarify, I get an improvement. So the data set creation here, if I understand it correctly, being


manual, is a big interference, I guess, just from a research standpoint, with the researchers


interest, like there's a conflict of interest in making this data set and what you want to get out


of the data set. So that is just one concern that I would have right here. The other concern, as you


can see, is if you're, if you're retrieved clarification from the memory, so this thing here


comes from the memory, if that is wrong, like if it's actually not related to the question right


here, then things could go bad, because GPT-3 given the prompt is explicitly trained to address


whatever is in the clarification in its answer. And that could be not not super duper relevant,


it could actually be destructive. So GPT-3 could be completely correct in answering the question.


Yet, if the clarification is wrong, it could output a wrong answer. And that's, that's not


entirely, you know, that's not entirely good. Or maybe, maybe I've misunderstood something,


because what I can also imagine is that the memory contents are somehow appended to the


prompt itself. So the question and the clarification, which, and that's what I don't know. And that's


what I would like to ask the authors, because it's not entirely clear to me what they do. They


compare two different baselines right here. And it could also be that the baselines implement some


of what I just said. So for example, let's go here, the no mem, that's just GPT-3. Then there's


the grow prompt and grow prompt says the prompt is continuously grown with a subset of memory


m that can fit within the prompt. So I think this grow prompt thing right here, that's where I have


my prompts that we've just seen. And then I would just add like all the entries of m or as many as


I could here. And then I would add x. So there will be no clarification over here for x. Now,


if I had a clarification over here for x, never in this grow prompt, it would just be that this


portion of memory here grows, and there would always be an x and the clarification or a feedback


FB, and then x and an FB. So all the things that I've gotten wrong in the past would be appended


here for x, and then this is compared to this mem prompt system. That's the system that they have.


Now, again, it is not clear to me, because tech like is not clear to me if their system


simply retrieves the most relevant unit here and appends it here instead of the m. So or maybe the


of all the relevant units, right?


In which case there would also be no feedback here,


or if their system retrieves the most relevant thing


and then appends only the feedback to the X right here.


I don't know, right?


I don't know.


It concatenates C at the end of P


and C concatenates X and the feedback retrieved.


So I'm pretty sure that it's the second one.


It concatenates the feedback to X.


However, here it says they use a cosine distance


with a threshold of 0.9.


There is no mention of like a maximum,


like they retrieve the maximal feedback.


It seems like this could result


in an entire set of feedbacks.


Yeah, but I don't wanna go too deep into that.


I think I've understood correctly.


The danger here is that the green stuff,


like the grow prompt, the way I understand it,


is not like a perfect baseline for what they do


because the grow prompt inserts the memory samples


as such with the original questions


and their system only inserts the,


it only inserts the feedback


after the question that's currently happening.


So either we need a baseline


that also adds only feedback right here,


but selected in a maybe less smart way,


or we need as a baseline a system


that selects the feedback in a smart way,


but then tries to prepend the original question


with that feedback in front of X


and leave X without feedback or without clarification.


So I think, you know, just baseline-wise,


that is what would be needed.


But you can see in their experiments,


they show, I guess convincingly,


that they are able to improve the accuracy.


These here are steps, these are not training steps.


These are steps of interaction with the system.


So the system is never trained.


It's simply interacted with and this memory is filled up.


You can see, interestingly, at the beginning,


everything fails, which is interesting, right?


Because one would expect that at least


this mem prompt system would remain the same.


I guess GPT-3 remains the same,


but the mem prompt system also declines.


Now, if the retriever is pre-trained and fixed


and the threshold is selected well,


it should not retrieve any clarifications


that have nothing to do with the question.


So the performance in my mind


shouldn't sink this dramatically,


which tells me that the max function is just very important.


So they probably mostly get the most relevant feedback


if it passes the threshold.


And here is what happens, I could guess,


if that feedback is irrelevant.


So it would actually bias the language model


towards giving the wrong answer.


And only after a while do I have enough feedback collected


that I sort of accurately cover what I would like to ask.


Yeah, you can see how this gets, I guess, problematic


as soon as your domain of requests to GPT-3 increases,


because there's probably,


it probably doesn't need to be a huge domain


before you start to over-correct for things.


But then you might also just tighten your threshold.


So, you know, what do I know?


However, you know, disregarding correcting things,


personalization, I think,


might be just a really neat application of this,


to just sort of nudge GPT-3


into a personalized interaction with the user.


And if it misunderstands there,


then it's, I would guess it's more mild than here,


where it would just kind of like,


it essentially negates an output,


essentially says, no, that's wrong.


What's also interesting is that the grow prompt


never reaches the potential.


Again, we don't know if that is


because it's a different structured prompt,


but at least it's partially due to the fact


that it's not smartly selected.


It simply appends to whatever is last


in the last few things in the memory.


Also, interestingly, this mem prompt,


where the probability of giving feedback is 0.5,


it is kind of bad at the beginning.


So here, the probability of getting feedback


from the memory is only half.


So half the time, the memory would have something,


but you're not getting it.


This is kind of like an artificial limitation on the system,


just your retriever might be bad, not recognized.


That there's something there.


Interestingly, this also grows to the same performance.