it's socket dot get hostname and the IP same as before one two three four now


Sockets Tutorial with Python 3 part 1 - sending and receiving data

often is the case your client will be like remote to your server it won't


actually be on the same machine so with sockets you can communicate to and from


Python programs on the same machine on like a locally networked set of machines


or even remotely networked machines so it just doesn't matter in most cases


you're probably not actually going to use socket dot get hostname or at least


not all of the cases you probably won't do that you'll connect to an actual


public IP or something like that or a local IP but for now we're gonna do


everything on the same machine so everybody can follow along cool so we


make our connection and we could be done but what we're gonna accept that message


that was sent to us so we're to say message equals s dot RECV for receive


and we're gonna receive and this is our buffer so remember this TCP socket here


this is a stream of data so you've got a stream you have to decide in how big of


chump chunks or how many chumps how big a chunks of data do we want to receive


at a time so in this case 1024 that's more than big enough but it really


depends on what you're doing like if you're sending massive files you


wouldn't want a really tiny buffer size you might want something a little larger


so and then it also depends on the application that you're then going to


use it with and stuff that this kids can get really complicated really fast like


if you're really into like networking and stuff you could be paid a decent


amount of money like to come up with the proper amounts for these things but


anyway for now 1024 good enough so now we're gonna say is just print message


dot decode and this was UTF a so we will decode and again that the way sockets


are communicating at the moment at least with the type of socket that we've


established here is a byte stream so we're receiving these bytes so they're


sent as bytes they're received as bytes and then we just decode the bytes so in


this case we should get the message saved everything so now all we need to


do is run it so I'm gonna do CMD here and I honestly this is my local machine


this is probably not 37 yes so instead what I'm gonna say is pi dash 37 it


shouldn't actually matter you should probably be able to do it on any version


of pi 3 anyway run the server so the server is now running and now I just


need to open up one more for our client pi dash 3.0 3.7 client dot pi and I


think what I'll do is all like I don't know it's not like that we connect we


see welcome to the server from our server and then here where can we see it


here connection from our IP address has been established okay so it looks like


that worked for us but this also just like immediately disconnected which is


probably not really what we're after you're probably looking for a more long


term connection but we could just like keep running that again like every time


we wanted to receive a bit of information but that's kind of silly


most people aren't actually doing that with sockets they're gonna be doing


something far more complex so I think the last thing I want to show you guys


before we leave and go to like the next video is you know the way that truly


sockets are gonna work in the way that you know you're actually gonna be


buffering data is most of the time you just like can't expect that with this


that this will be enough so in this case you could send just about any message as


long as it was less than 1024 bytes of data and you would be totally fine and


you could you could set this as big as you could plausibly conceive but that


would be silly what really happens is again it's a stream so I'm gonna set


this to 8 and then hit save for the client and I honestly don't remember did


we I don't yeah no we haven't done anything to the server so I'll leave the


server running and instead let's rerun the client again and you'll see what we


get is just welcome right because we only we've received that one little itty


bitty buffer but we're not actually handling for really anything here so so


actually what we want to do is while true so while true we receive this data


so I'll tab that over hit save come back up here rerun it and we get this welcome


to this ever right okay so yeah that's what happens when you buffer data but


why do we have to buffer it I mean like because we could have gotten away with


1024 but eventually someone's gonna try to send a message that's bigger than


that and maybe on your chat let's see make a chat app maybe your care you set


a character limit and it becomes more challenging to do more than that but


there's almost always gonna be a reason why it's gonna be better to buffer also


I mean in most cases you're gonna be like below let's say certainly below


like 10 megabyte buffer so if we're talking like files and stuff you just


you have to be able to understand how that's gonna work so then you start


thinking well that's not that hard man all we got to do is something like this


like I don't know full underscore message and then you just make that an


empty string and then here you'll just say maybe something like this like if


or actually what we'll do is under here you would say you know if message is


less than or equal to zero right or Len message rather so length of the message


is less than or equal to zero we're gonna break otherwise what we want to do


is full message plus equals msg dot decode utf-8 who needs syntax right like


duh so then after this we could print full message full massage and then we


come over here we'll break this one maybe wow I just okay


yeah jerk pie poo pie dash 3.7 client dot pie we try to connect nothing


happens okay so what's going on here well the problem is we still have this


connection from the server and the weed we don't actually know when the


connection is fully completed right we're still actually the stream and then


the connection is still exists so then you come over here maybe and then I


don't know client socket dot close I think is the proper handling for that


unfortunately I don't think I can cancel this from running either it's really


annoying why can't I break sockets in running pi dash 3 7 server dot pie please


don't tell me that it is a use come over here rerun client dot pie and then we


see welcome to the server and it exits and so in this case we were actually


able to buffer all of the data and we didn't really need any fancy logic we


receive the message and we buffered so what's the big idea well the big idea is


it's meant to be a stream we're supposed to be accepting data in a big stream and


if server gets something and it decides it wants to send some information we're


just simply not going to receive it now we could kind of treat this like a


restful API or something and like build data until the client reconnects accepts


the buffer and then clears out what the client got or something like that but


that's not the way that we typically will work with sockets instead you're


going to likely send a bit of information in a header and inform the


client how much information is to come so that's we're going to be focusing


focusing on in the next video if you got questions comments concerns whatever


feel free to leave those below quick shout out to my most recent channel


there we go couldn't click on it channel members Omni Crux Miguel lots loud taro


Alexander and Mary Python I love your last name thank you guys very much for


your support allows me to keep doing what I love to do so thank you guys so


much and thank you to everybody else eventually I'm going to figure out a


decent way to rethink people who have stuck around some people are still with


me for like well over six months I forget exactly what we're at right now


for the max but that's freaking awesome so anyways I'll figure out a way at some


point so anyways in the next tutorial we'll talk about how do you actually


will buffer and keep the stream open and all that and do some more cool stuff


The Hardware Lottery (Paper Explained)

Hi there, are you interested in winning the lottery? Then let me tell you this video is


The Hardware Lottery (Paper Explained)

not for you. This video is not about winning the lottery. Okay, I've done enough videos with lottery


The Hardware Lottery (Paper Explained)

in the title, only for people to be mad at me for not telling them how to win the lottery. This


The Hardware Lottery (Paper Explained)

is about computer science research. And very unfortunately, the author of this paper has


The Hardware Lottery (Paper Explained)

decided to put this word in the title. So if you're here, because you want to win the lottery,


The Hardware Lottery (Paper Explained)

this is not for you. It's something completely different. For everyone else. Today, we're looking


The Hardware Lottery (Paper Explained)

at the hardware lottery by Sarah Hooker of Google Brain. This paper is, it's kind of a mix, it's


The Hardware Lottery (Paper Explained)

part of a historic look back at hardware and software developments in machine learning.


The Hardware Lottery (Paper Explained)

And it is a analysis of kind of the current situation and an outlook and sort of an opinion


The Hardware Lottery (Paper Explained)

piece of the way forward and how hardware and software should mix and what we should focus on


The Hardware Lottery (Paper Explained)

in the future. So the basic, the basic principle is quite simple in this paper. It introduces this


The Hardware Lottery (Paper Explained)

term, the hardware lottery, this essay introduces the term hardware lottery to describe when a


The Hardware Lottery (Paper Explained)

research idea wins because it is compatible with available software and hardware and not because


The Hardware Lottery (Paper Explained)

the idea is superior to alternative research directions. So right off the bat, I think this


The Hardware Lottery (Paper Explained)

is a statement where I think many people can agree or I think almost everyone will some agree with


The Hardware Lottery (Paper Explained)

this statement in to a certain degree, but certainly to a high degree, right? We are all


The Hardware Lottery (Paper Explained)

aware that of course, we have the hardware we have hardware is very inflexible, it's expensive to


The Hardware Lottery (Paper Explained)

develop and so on. So any sort of software development, any algorithmic development may


The Hardware Lottery (Paper Explained)

simply succeed because it is suited to the hardware that we have. So that was my first


The Hardware Lottery (Paper Explained)

reaction when I read this paper. It's a it's a it's a very gut feeling of yes, of course, this


The Hardware Lottery (Paper Explained)

is the case. But then the historic analysis is also nice. But I was wondering what is there a


The Hardware Lottery (Paper Explained)

deeper reason to, to kind of go into this. And we are going to see some pros and cons that I think


The Hardware Lottery (Paper Explained)

in this paper right here, where it I'm not exactly entirely sure what specific point is trying to


The Hardware Lottery (Paper Explained)

make the overarching point I completely agree with the fact that of course, what hardware is here is


The Hardware Lottery (Paper Explained)

important and may lead to certain ideas succeeding. But it I have I have trouble with the narrower


The Hardware Lottery (Paper Explained)

points. And I'm going to try to illustrate this in this paper, while also telling you what the


The Hardware Lottery (Paper Explained)

paper says. So first of all, here, the term is called the hardware lottery. But off the bat,


The Hardware Lottery (Paper Explained)

you already see that it says a research idea wins because it is compatible with available software


The Hardware Lottery (Paper Explained)

and hardware. So the hardware lottery right off the bat is connect is means that also the software


The Hardware Lottery (Paper Explained)

is there. So it's technically the hard and software lottery. And the bigger, the bigger question I


The Hardware Lottery (Paper Explained)

would have to someone arguing that really the hardware lottery is an important concept to have


The Hardware Lottery (Paper Explained)

is why what does what distinguishes the hardware lottery? Let's let's even say it's just hardware,


The Hardware Lottery (Paper Explained)

what distinguishes the hardware lottery from any lottery? Like, why can't I say, okay, there's


The Hardware Lottery (Paper Explained)

the X lottery. And the X lottery is, is any circumstance, any circumstance that that surrounds


The Hardware Lottery (Paper Explained)

a research idea, right here, you have idea one, idea two, idea three, and they all depend on many


The Hardware Lottery (Paper Explained)

circumstances, and x is one of those circumstances. And it just so happens that the circumstance in


The Hardware Lottery (Paper Explained)

the world favors idea two, and a different circumstance would actually favor idea one,


The Hardware Lottery (Paper Explained)

what's so special about hardware other than it's more expensive than software, right? To, to, to


The Hardware Lottery (Paper Explained)

illustrate this further, let's say, okay, you have you have hardware and you say, well, hardware is


The Hardware Lottery (Paper Explained)

expensive, but then again, you can sort of build a hierarchy where, okay, down here, there is like


The Hardware Lottery (Paper Explained)

ideas, they depend on software, like software frameworks that we have, such as TensorFlow,


The Hardware Lottery (Paper Explained)

pytorch, these again, depend on particular hardware. But, and you can say, okay, the hardware


The Hardware Lottery (Paper Explained)

is much more expensive. So we are not as flexible. And the ideas might just succeed because of the


The Hardware Lottery (Paper Explained)

hardware, but then you can go even step further and say, well, up here is sort of the consumer,


The Hardware Lottery (Paper Explained)

if you don't like the market term, then maybe say the society, the end user, and so on, because the


The Hardware Lottery (Paper Explained)

hardware ultimately is directed towards what humans in society need. And that changes over


The Hardware Lottery (Paper Explained)

time as well. So and it's way more expensive to change the needs of human society than to change


The Hardware Lottery (Paper Explained)

the hardware. So I can just also claim, okay, x is now society. So that's the way to go.


The Hardware Lottery (Paper Explained)

Society. So the one particular research idea down here might win simply because it is more


The Hardware Lottery (Paper Explained)

suited to the current societal needs. And that kind of carries over and you might say, well,


The Hardware Lottery (Paper Explained)

make doesn't that make it a good idea? Doesn't that make it preferable to idea?


The Hardware Lottery (Paper Explained)

Idea two preferable to idea three over here that would just optimize for a different society,


The Hardware Lottery (Paper Explained)

which leads us to the question, what does it mean to first what does it mean to win? Here it just


The Hardware Lottery (Paper Explained)

says a research idea wins. And you might have an idea. So I've I've an idea. It's not clearly


The Hardware Lottery (Paper Explained)

defined here. But maybe winning means that a lot of researchers actually research in that direction.


The Hardware Lottery (Paper Explained)

And the other question is here, and not because the idea is superior to alternative research


The Hardware Lottery (Paper Explained)

directions. And here, my question would be what does superior mean? What does it what does it


The Hardware Lottery (Paper Explained)

mean for an idea to be superior? As I said here, certainly if an idea is more in congruence with


The Hardware Lottery (Paper Explained)

current societal needs, you might claim it's superior. And someone else might say, well,


The Hardware Lottery (Paper Explained)

if societal needs were different than a different research idea might be suited better. The same way


The Hardware Lottery (Paper Explained)

someone could say, well, if hardware was different than a different research idea might be better.


The Hardware Lottery (Paper Explained)

Maybe you can say if hardware was different, a different research idea might be better suited


The Hardware Lottery (Paper Explained)

to the current needs of society. But then I'm pretty sure I can go two, three, four levels up


The Hardware Lottery (Paper Explained)

here. Again, so these these terms are a bit vague, I think we can all the again, the initial the


The Hardware Lottery (Paper Explained)

initial sentiment when reading this is absolutely in favor, right? I absolutely agree. I don't want


The Hardware Lottery (Paper Explained)

to want to trash this. I just want to sort of I try to think a bit deeper about what is actually


The Hardware Lottery (Paper Explained)

said here. And this is where sort of my my troubles start. So let's dig a bit into the historic part.


The Hardware Lottery (Paper Explained)

And I think the point the paper is sort of trying to make is that not yet that there are specific


The Hardware Lottery (Paper Explained)

hardware choices that were made at one particular point. And because it's so expensive to change


The Hardware Lottery (Paper Explained)

hardware. That means that a lot of researchers simply go along with whatever ideas work on that


The Hardware Lottery (Paper Explained)

particular hardware that's available. And other research ideas are neglected simply because the


The Hardware Lottery (Paper Explained)

hardware isn't available, which again, this is a sentiment that I think we can we can all agree


The Hardware Lottery (Paper Explained)

with. So the first part here, the paper is is in the in the following sections. And this is


The Hardware Lottery (Paper Explained)

important to keep in mind as a red thread, because I feel one can get lost in the details of the


The Hardware Lottery (Paper Explained)

paper. So in the first section, section two, we ask, what has incentivized the development of


The Hardware Lottery (Paper Explained)

software, hardware and machine learning research in isolation? We need to read this first. This essay


The Hardware Lottery (Paper Explained)

begins by acknowledging a crucial paradox. machine learning researchers mostly ignore hardware


The Hardware Lottery (Paper Explained)

mostly ignore hardware, despite the role it plays in determining what ideas succeed. So the argument


The Hardware Lottery (Paper Explained)

is that we we develop ideas independent of hardware. But also, we don't, it kind of makes it


The Hardware Lottery (Paper Explained)

a double, double point. It says that we think we just think about ideas, but the ideas we might


The Hardware Lottery (Paper Explained)

think about may be shaped by the hardware that's available. And if we're not aware of that, we might


The Hardware Lottery (Paper Explained)

not, we might not see other ideas as viable. So section two asks, what has incentivized the


The Hardware Lottery (Paper Explained)

development of software, hardware and machine learning research in isolation? So where does


The Hardware Lottery (Paper Explained)

this come from, that we don't think about the hardware that's at the end? section three considers


The Hardware Lottery (Paper Explained)

the ramifications of this siloed evaluation with examples of early hardware and software


The Hardware Lottery (Paper Explained)

lotteries. So this is the kind of risk historical look back. Then, today, the hardware landscape is


The Hardware Lottery (Paper Explained)

increasingly heterogeneous. This essay posits that the hardware lottery has not gone away.


The Hardware Lottery (Paper Explained)

And the gap between the winners and the losers will grow increasingly larger. So this is a point


The Hardware Lottery (Paper Explained)

that the paper basically makes that this hardware lottery has not gone away. So right now we are in


The Hardware Lottery (Paper Explained)

this hardware lottery. And it does so specifically with regards to saying that chips like GPUs and


The Hardware Lottery (Paper Explained)

TPUs and even more specialized chips are optimized to neural networks. And that's why the whole world


The Hardware Lottery (Paper Explained)

sort of over focuses on neural networks right now and discards other research ideas. And the gap


The Hardware Lottery (Paper Explained)

between the winners and the losers will grow increasingly larger, meaning that the research


The Hardware Lottery (Paper Explained)

ideas that are seen as inviable now, if we develop even more hardware into that direct into the


The Hardware Lottery (Paper Explained)

direction of neural networks, those research ideas will become more and more inaccessible


The Hardware Lottery (Paper Explained)

to the community. Then lastly, sections four to five unpack these arguments. So the ones that


The Hardware Lottery (Paper Explained)

we've just seen section six concludes with some thoughts on what it will take to avoid future


The Hardware Lottery (Paper Explained)

hardware lotteries. Alright, so section two here is this sort of historic look back. And it goes


The Hardware Lottery (Paper Explained)

from these it the point is here separate tribes. So the point is that something has made it such


The Hardware Lottery (Paper Explained)

that the communities, the software communities and the hardware communities and the idea,


The Hardware Lottery (Paper Explained)

let's say the idea communities, the researchers in AI algorithms, let's call them the algorithmers.


The Hardware Lottery (Paper Explained)

They they they don't think that much about each other. And it makes the case that early machines


The Hardware Lottery (Paper Explained)

were super duper specialized. Early machines were single use were not expected to be repurposed for


The Hardware Lottery (Paper Explained)

new tasks because of the cost of electronics and the lack of cross purpose software. So early


The Hardware Lottery (Paper Explained)

machines, early computing machines were just single purpose, and so on. But that all changed


The Hardware Lottery (Paper Explained)

when the whole world focused on sort of general purpose CPUs that could execute any instructions,


The Hardware Lottery (Paper Explained)

of course, according to Turing machine or von Neumann architectures. So the point that the


The Hardware Lottery (Paper Explained)

paper makes is at some point a shift happened. The general purpose computer area crystallized


The Hardware Lottery (Paper Explained)

in 1969, when an opinion piece by young engineer called Gordon Moore appeared in Electronics


The Hardware Lottery (Paper Explained)

magazine with the app title cramming more components onto circuit boards. That's a cool


The Hardware Lottery (Paper Explained)

title. So this famously gave rise to Moore's law or predicted you could double the amount


The Hardware Lottery (Paper Explained)

of transistors on an integrated circuit every two years. And this sort of held true where people


The Hardware Lottery (Paper Explained)

stopped building general like, sorry, people stopped building special purpose hardware,


The Hardware Lottery (Paper Explained)

but invested just more and more and more into building these general purpose chips, these CPUs.


The Hardware Lottery (Paper Explained)

That, and the reason why they stopped making specialized hardware is any specialized hardware


The Hardware Lottery (Paper Explained)

you build will simply be surpassed by the next generation of CPUs. So even if you make a


The Hardware Lottery (Paper Explained)

specific purpose hardware for some problem, you just have to wait like one or two of these cycles,


The Hardware Lottery (Paper Explained)

and ordinary general purpose CPUs will simply have will will overtake your specialized hardware. And


The Hardware Lottery (Paper Explained)

since CPUs are general purpose, the market for them is naturally huge. So this, this has made it


The Hardware Lottery (Paper Explained)

such that what was mainly developed was general purpose CPUs. I think the paper wants to make the


The Hardware Lottery (Paper Explained)

point though I'm not exactly sure. I think wants to make the point that even though the CPUs might


The Hardware Lottery (Paper Explained)

be called general purpose, they aren't general purpose, like they have their specific advantages


The Hardware Lottery (Paper Explained)

and disadvantages. And that's going to hurt, for example, neural networks in the years following


The Hardware Lottery (Paper Explained)

this. So in conclusion to this chapter, they say, in the absence of any lever with which to influence


The Hardware Lottery (Paper Explained)

hardware development, machine learning researchers rationally began to treat hardware as a sunk cost


The Hardware Lottery (Paper Explained)

to work around rather than something fluid that could be shaped. However, just because we have


The Hardware Lottery (Paper Explained)

abstracted away hardware does not mean it has ceased to exist. Early computer science history


The Hardware Lottery (Paper Explained)

tells us there are many hardware lotteries where the choice of hardware and software has determined


The Hardware Lottery (Paper Explained)

which ideas succeeded and which fail. And the example is kind of the Charles Babbage's


The Hardware Lottery (Paper Explained)

analytic engine that Charles Babbage designed, but was something like 50 years earlier, or so then


The Hardware Lottery (Paper Explained)

parts could even be manufactured for this idea to succeed. And we know many stories of these people


The Hardware Lottery (Paper Explained)

being ahead of their time. And they have this interesting quote, I think, somewhere from


The Hardware Lottery (Paper Explained)

Silicon Valley here, being too early is the same as being wrong. And this paper, of course, focuses


The Hardware Lottery (Paper Explained)

on hardware. But to come back, the conclusion of this chapter is that because of the fact that


The Hardware Lottery (Paper Explained)

of this general purpose area, because the entire focus was on building general purpose CPUs,


The Hardware Lottery (Paper Explained)

this has led to people not really having integrated thought of hardware software algorithm,


The Hardware Lottery (Paper Explained)

but treating hardware as this thing that can execute any instruction. And then the algorithm


The Hardware Lottery (Paper Explained)

comes on top of this sort of black box that we can't really change, we just have the hardware we


The Hardware Lottery (Paper Explained)

have. Yeah, which which comes back, I'm not and again, I'm not sure, like sure that that sure,


The Hardware Lottery (Paper Explained)

I agree that the entire world focusing on general purpose CPUs has some influence. But certainly,


The Hardware Lottery (Paper Explained)

hardware is just expensive to make. So you could argue that even if this hadn't happened,


The Hardware Lottery (Paper Explained)

a machine learning researcher wouldn't necessarily think about the hardware, but they would at least


The Hardware Lottery (Paper Explained)

have a choice if there were a selection of hardware's right. Okay. So that was the section


The Hardware Lottery (Paper Explained)

two, section three. Now we really go into the historic evidences. And there are kind of early


The Hardware Lottery (Paper Explained)

early historic evidence like this Charles Babbage's machine that he invented. An early example,


The Hardware Lottery (Paper Explained)

the analytical machine in 1837. And and no, it wasn't even decade it was only surface during


The Hardware Lottery (Paper Explained)

World War Two in the first part of the 20th century, electronic vacuum tubes were heavily used,


The Hardware Lottery (Paper Explained)

were heavily used for heavily used this is I've not I've noticed a number of typos in in the


The Hardware Lottery (Paper Explained)

paper, I realized it's a preprint if the author is listening. I can also I can also make a list,


The Hardware Lottery (Paper Explained)

but this this one just popped up. For radio communication and radar during World War Two,


The Hardware Lottery (Paper Explained)

these vacuum tubes repurpose to provide the compute power necessary to break the German


The Hardware Lottery (Paper Explained)

enigma code. So it would be long after not only after Charles Babbage invented this machine, but


The Hardware Lottery (Paper Explained)

even after he died, that people would would sort of re take and in some parts reinvent his ideas


The Hardware Lottery (Paper Explained)

to do build modern computers. The big example though that the paper makes is what it calls


The Hardware Lottery (Paper Explained)

the Lost Decades. And this is the story of neural networks coupled with two things with an AI winter


The Hardware Lottery (Paper Explained)

and a focus on expert systems and maybe also, though that's not entirely mentioned here a focus


The Hardware Lottery (Paper Explained)

on things like SVMs. So I think it's widely known that the main ingredients for neural networks


The Hardware Lottery (Paper Explained)

are very, very, very old. So here the paper gives some examples back propagation invented in 63


The Hardware Lottery (Paper Explained)

reinvented reinvented again, and deep convolutional networks paired with back propagation by on the


The Hardware Lottery (Paper Explained)

car. It says, however, it was only three decades later that deep neural networks were widely


The Hardware Lottery (Paper Explained)

accepted as a promising research direction. I think this this sort of the timeline here is


The Hardware Lottery (Paper Explained)

this year probably refers to around 2010. Shortly after that, of course, Alex net beats ImageNet and


The Hardware Lottery (Paper Explained)

so on. But even earlier, a bit earlier, people were doing heavy research into neural networks.


The Hardware Lottery (Paper Explained)

And three decades later, so this is paired with kind of these numbers right here, let's say 1970


The Hardware Lottery (Paper Explained)

1980, when these ideas were invented presented, but computers back then were simply unsuited to


The Hardware Lottery (Paper Explained)

the to run neural networks. Here it says, the gap between these algorithmic advances and empirical


The Hardware Lottery (Paper Explained)

successes in large part to to incompatible hardware. During the general purpose computing


The Hardware Lottery (Paper Explained)

areas, hardware like CPUs were heavily favored and widely available. CPUs were good at executing


The Hardware Lottery (Paper Explained)

any set of complex instructions but occur high memory costs because of the need to run


The Hardware Lottery (Paper Explained)

the need to cache intermediate results and process one instruction at a time. This is known as the


The Hardware Lottery (Paper Explained)

von Neumann bottleneck. The available compute is restricted by the lone channel between CPU


The Hardware Lottery (Paper Explained)

and memory along which they test to travel sequentially. So the paper goes on and says


The Hardware Lottery (Paper Explained)

there were some efforts into specialized hardware for neural networks, but funding was kind of not


The Hardware Lottery (Paper Explained)

there. And other specialized hardware was more into the direction of popular ideas then like


The Hardware Lottery (Paper Explained)

prologue and list which could do expert systems, and not necessarily neural networks. And only


The Hardware Lottery (Paper Explained)

only it would take a hardware fluke in the early 2000s, a full four decades after the first paper


The Hardware Lottery (Paper Explained)

about back propagation was published for the insight about massive parallelism to be operationalized in


The Hardware Lottery (Paper Explained)

a useful way for connectionist deep neural networks. A graphical processing unit was


The Hardware Lottery (Paper Explained)

originally introduced in the 1970s as a specialized accelerator for video games and developing


The Hardware Lottery (Paper Explained)

graphics, yada yada yada. GPUs were repurposed for an entirely unimagined use case to train


The Hardware Lottery (Paper Explained)

deep neural networks had one critical advantage over CPUs, they were far better at parallelizing


The Hardware Lottery (Paper Explained)

a set of simple decomposable instructions such as matrix multiplications, multiples,


The Hardware Lottery (Paper Explained)

multiplications, um, multiples, I don't know. So the the point here is that the ideas were around


The Hardware Lottery (Paper Explained)

for a long time, but it would take GPUs to make them work. And so the the the image that the


The Hardware Lottery (Paper Explained)

paper builds up, I think, is that you have these, you're you're here, and you research and then you


The Hardware Lottery (Paper Explained)

have a decision to make, which hardware do I build for the future. And there are two directions,


The Hardware Lottery (Paper Explained)

this is direction one, and this is direction two. And let's say for whatever reason, direction one


The Hardware Lottery (Paper Explained)

is chosen, okay, then, because it's so expensive to build different hardware, the the world largely


The Hardware Lottery (Paper Explained)

goes with direction one, and builds on top of that, okay. So that also means that all the research


The Hardware Lottery (Paper Explained)

ideas that profit from direction one will appear to be much more effective that research ideas that


The Hardware Lottery (Paper Explained)

would have profited from direction two. And it sort of says that neural networks are over here.


The Hardware Lottery (Paper Explained)

And it's sort of the and the the, let's say the other systems, what do we give expert systems,


The Hardware Lottery (Paper Explained)

let's call them expert systems. And other types of ideas were over here. And they appear to work


The Hardware Lottery (Paper Explained)

really well until they stopped in progress. And then by accident, sort of this road here was


The Hardware Lottery (Paper Explained)

traveled use with GPUs, it was not obvious, but by accident, still this was developed,


The Hardware Lottery (Paper Explained)

and then neural networks could flourish. And if it wasn't for that fluke, if it wasn't for video


The Hardware Lottery (Paper Explained)

games, basically, or animation, we would have never known that neural networks work as well as


The Hardware Lottery (Paper Explained)

they do. So, again, that's the point the paper makes. And I think we can all agree with that


The Hardware Lottery (Paper Explained)

particular point. But I want to, again, I want to build up sort of a different picture right here,


The Hardware Lottery (Paper Explained)

in that why why is only like, I feel hardware is considered a bit much here. So


The Hardware Lottery (Paper Explained)

I think you can make the general case that at any junction, you have several things you can choose.


The Hardware Lottery (Paper Explained)

And then once you choose a thing, all the things go in that direction, like new ideas will be more


The Hardware Lottery (Paper Explained)

in that direction. Also new hardware will be more in that direction, because a lot of people research


The Hardware Lottery (Paper Explained)

on it, the paper also makes the point there's kind of this feedback loop. But let's say neural


The Hardware Lottery (Paper Explained)

networks were down here. What I would I would argue, and this is a bit of a point the paper makes


The Hardware Lottery (Paper Explained)

in in a half half formulated way, I think is that it basically says that had we had we invested in


The Hardware Lottery (Paper Explained)

matrix multipliers in GPUs instead of CPUs in these early years, that means that neural


The Hardware Lottery (Paper Explained)

networks would have sort of succeeded as an idea at that time. And I'm not entirely convinced of


The Hardware Lottery (Paper Explained)

this. Because first of all, you can see right here, GPUs were actually around in the 1970s.


The Hardware Lottery (Paper Explained)

So the hardware was was available. It's not it's not like it was super easy in in 2010,


The Hardware Lottery (Paper Explained)

it was for these early researchers to build their code into GPU compatible code. That was certainly


The Hardware Lottery (Paper Explained)

hard, especially if you read the papers, but it would have been hard in 1970. As well, it would


The Hardware Lottery (Paper Explained)

not have been significantly harder, I think. So I, I'm not sure if the picture is really like this,


The Hardware Lottery (Paper Explained)

or if the picture so if this is the CPU direction, is more like that neural networks are actually


The Hardware Lottery (Paper Explained)

somewhere up here. And the fact is, we, we actually needed the good CPUs in order to develop


The Hardware Lottery (Paper Explained)

day in order to make use of the GPUs, right, and this here would be GPU. In order to make use of


The Hardware Lottery (Paper Explained)

the GPUs to then enable these neural networks on the GPUs, because certainly, it has it has helped


The Hardware Lottery (Paper Explained)

a lot that CPUs were built that, you know, computers just built on GPUs would be sad


The Hardware Lottery (Paper Explained)

computers. Computers built on CPUs are cool. They can do multi processing, they can do internet,


The Hardware Lottery (Paper Explained)

they can do actually they can do most of the video game except display the graphics. And


The Hardware Lottery (Paper Explained)

very arguably that without the heavy focus on CPUs, we would not have neural networks today.


The Hardware Lottery (Paper Explained)

Even if we had invested all of that effort into building GPUs. Because society has just advanced


The Hardware Lottery (Paper Explained)

so much because of CPUs. So I'm sort of tempted to challenge this notion here that just because of


The Hardware Lottery (Paper Explained)

the the happenstance that CPUs were advanced at that time that neural networks are didn't have


The Hardware Lottery (Paper Explained)

their breakthrough back then. I think we needed both. That being said, I do agree with the paper


The Hardware Lottery (Paper Explained)

that we might have never, ever realized that neural networks worked if it weren't for the fact


The Hardware Lottery (Paper Explained)

that there is specialized hardware around. Yeah, so so that would be my my points to this.


The Hardware Lottery (Paper Explained)

The paper makes yeah makes this point about okay, there is hardware lotteries and in so now it also


The Hardware Lottery (Paper Explained)

introduces software lotteries, though it said at the beginning that hardware lotteries included


The Hardware Lottery (Paper Explained)

software, but I'm going to guess that a the general concept of a lottery was simply presented. And


The Hardware Lottery (Paper Explained)

again, I don't see exactly what's so special about hardware, because again, I can make the same case


The Hardware Lottery (Paper Explained)

for software. It's just a shorter timeframe. I can make the same case for theory, right? Like


The Hardware Lottery (Paper Explained)

whatever now neural tangent kernels are are are the hit, right? Everyone's like wow, NTK is blah,


The Hardware Lottery (Paper Explained)

blah, blah, blah, blah, blah, blah. Who knows, right? But some big names announced this and some


The Hardware Lottery (Paper Explained)

theory has been done in this direction. And because there is already a big momentum, lots of


The Hardware Lottery (Paper Explained)

people publish in it. Who knows if that's if that's a good idea, or if there were other ideas that had


The Hardware Lottery (Paper Explained)

we done the fundamental work in this would flourish right now. Again, I don't agree with the


The Hardware Lottery (Paper Explained)

sentiment. I don't see why the hardware is the why the hardware is is such a special case right here.


The Hardware Lottery (Paper Explained)

So the next thing that the paper looks like this kind of the current day. So it tries to make the


The Hardware Lottery (Paper Explained)

point that we might be in a hardware lottery right now. And again, the the intuition of course,


The Hardware Lottery (Paper Explained)

is yes, of course, we have the hardware we have, it's difficult to change, especially since hardware


The Hardware Lottery (Paper Explained)

builds upon hardware with the tree I drew before. Let's draw it again. I'm going to draw a tree


The Hardware Lottery (Paper Explained)

and literally every decision you make in the train, this doesn't only need to be hardware,


The Hardware Lottery (Paper Explained)

right? Every single decision you make will mean that pretty much all of the previous choices here


The Hardware Lottery (Paper Explained)

are now fixed and ingrained. We build upon, we build upon inventions of the past, it's impossible


The Hardware Lottery (Paper Explained)

to go back and do all of these things again. And if you do all of these things, you're going to


The Hardware Lottery (Paper Explained)

again, and if you see something curious right here, and this is where we're going to later,


The Hardware Lottery (Paper Explained)

I want you to see what happens if here here is a good idea. Like here is my super duper booper


The Hardware Lottery (Paper Explained)

idea. And my super duper booper idea simply didn't make the cut for that choice. Like someone


The Hardware Lottery (Paper Explained)

chose a different hardware direction, software direction, software library direction, whatnot,


The Hardware Lottery (Paper Explained)

it wasn't in vogue. And my idea was unpopular then, if one choice is made, this choice right here,


The Hardware Lottery (Paper Explained)

it's, it's hard to go back if two choices are made, right, that build upon each other,


The Hardware Lottery (Paper Explained)

it's even harder to go back. So as time goes on, it's harder and harder and harder to go back,


The Hardware Lottery (Paper Explained)

which is a point that the paper will make at the end that the difference between the winners and


The Hardware Lottery (Paper Explained)

the losers is getting bigger and bigger, which is an effect that this idea that once was a curiosity


The Hardware Lottery (Paper Explained)

that could be investigated, becomes a very costly investigation, because we need to reinvent and


The Hardware Lottery (Paper Explained)

re engineer a whole bunch of decisions. And it at with time goes on, it's simply forgotten because


The Hardware Lottery (Paper Explained)

there's so much that we have built past this. However, this is for the loser, right? This is


The Hardware Lottery (Paper Explained)

the loser. However, for the winner, I disagree right here, because here it says, okay, this


The Hardware Lottery (Paper Explained)

direction, the idea direction here, let's say there is a super cool idea that would beat the


The Hardware Lottery (Paper Explained)

crap out of neural networks. What not whatever, whatever latest Schmidhuber paper is, that,


The Hardware Lottery (Paper Explained)

that idea would be neural networks. And this here is neural networks. And everyone's doing neural


The Hardware Lottery (Paper Explained)

networks. And Schmidhuber idea is just forgotten about. Now, to say that neural networks are the


The Hardware Lottery (Paper Explained)

winner, and the winners will increase and increase and increase is correct, but it forgets that


The Hardware Lottery (Paper Explained)

right here, there is this whole branching. So within the neural networks, you have again,


The Hardware Lottery (Paper Explained)

this branching, and maybe over here, what kind of neural networks were completely forgotten,


The Hardware Lottery (Paper Explained)

like MLPs, no MLPs are maybe still a thing. I don't even remember like early, early neural


The Hardware Lottery (Paper Explained)

networks were 10 h nonlinearities for MLPs or something like this. Nine by nine filters,


The Hardware Lottery (Paper Explained)

nine by nine filters in convolution, things like this, right? We, it's sort of the nine by nine


The Hardware Lottery (Paper Explained)

filters are technically in the class of neural networks. But as time progresses, and this branch


The Hardware Lottery (Paper Explained)

here are the three by three filters, which are massively out competing the nine by nine filters.


The Hardware Lottery (Paper Explained)

So the nine by nine filters are forgotten. And it could be that if the nine by nine filters,


The Hardware Lottery (Paper Explained)

no, sorry, because of the three by three filters, now we have specialized hardware that


The Hardware Lottery (Paper Explained)

exclusively focuses on three by three filters. So we go down this route down this route down this


The Hardware Lottery (Paper Explained)

route down this route. And there might have been some other super duper idea down here that only


The Hardware Lottery (Paper Explained)

works when we have really big filters. And now we never know that this existed, right. So to say


The Hardware Lottery (Paper Explained)

that the difference between the winners and the losers gets bigger and bigger, sort of misjudges


The Hardware Lottery (Paper Explained)

that these winners will be fractionated and fractionated and fractionated. And every push


The Hardware Lottery (Paper Explained)

in one direction comes with costs to these other directions within that winner branch.


The Hardware Lottery (Paper Explained)

But this is, I don't Yeah, ultimately, you know, you have a choice, you have a choice. Do I want


The Hardware Lottery (Paper Explained)

to go back and go this direction? Or do I want to add something here, it might just might be worth


The Hardware Lottery (Paper Explained)

more for society to go up here. The paper is going to argue at the end that we should sort of


The Hardware Lottery (Paper Explained)

keep funding alternative directions in hardware, which I think is always a good thing to to


The Hardware Lottery (Paper Explained)

not lock in on particular ideas. But also, you can you sort of have a have to strike a balance


The Hardware Lottery (Paper Explained)

because, you know, researching on things that already work and make them better is a crucial


The Hardware Lottery (Paper Explained)

part as well, because you can discard these sub ideas that don't make any sense. Alright, so it


The Hardware Lottery (Paper Explained)

gives some examples of current hardware lottery winners. To improve efficiency, there is a


The Hardware Lottery (Paper Explained)

shift from task agnostic hardware like CPUs to domain specialized hardware that tailor the design


The Hardware Lottery (Paper Explained)

to make certain tasks more efficient. The first examples of domain specific hardware, at least


The Hardware Lottery (Paper Explained)

over the last few years TPUs. And then it also says edge TPUs, core tech arm cortex m 55. Facebook's


The Hardware Lottery (Paper Explained)

big sir, which I think is just like a box with a GPUs in it and some Infini band optimize


The Hardware Lottery (Paper Explained)

explicitly for costly operations common to deep neural networks like matrix, and then


The Hardware Lottery (Paper Explained)

networks like matrix multiplies. So here I have again, there's there's this double meaning. So it


The Hardware Lottery (Paper Explained)

says here is task agnostic hardware like CPUs. But at the same time, it argues that CPUs are


The Hardware Lottery (Paper Explained)

particularly bad at matrix matrix multiplies. It's not really task agnostic, it's just focused


The Hardware Lottery (Paper Explained)

on different tasks. But I see what the what the paper means right here, we do build hardware


The Hardware Lottery (Paper Explained)

to make matrix multiplies faster, which means that neural networks that benefits neural networks


The Hardware Lottery (Paper Explained)

research. Closer collaboration between hardware and research communities will undoubtedly continue


The Hardware Lottery (Paper Explained)

to make the training and deployment of deep neural networks more efficient. For example,


The Hardware Lottery (Paper Explained)

unstructured pruning and weight quantization, very successful compression techniques in deeper


The Hardware Lottery (Paper Explained)

in are incompatible with current hardware and compilations and compilations kernels,


The Hardware Lottery (Paper Explained)

hardware and compilations kernels. I don't know what that means. But it's incompatible with


The Hardware Lottery (Paper Explained)

current hardware. The paper argues that because we see that these ideas are good, there will be


The Hardware Lottery (Paper Explained)

specialized hardware for them. And I think the point the paper is trying to make is sort of like


The Hardware Lottery (Paper Explained)

see another win for neural networks, because we go down the neural network road, people focus on


The Hardware Lottery (Paper Explained)

neural networks focus on how to prune them, and so on hardware will be developed, which will lock


The Hardware Lottery (Paper Explained)

us in further into neural networks, which, again, is papers basically saying like, look, because we


The Hardware Lottery (Paper Explained)

went this road right here, we're going to go this road a lot more. But then what you have to see is


The Hardware Lottery (Paper Explained)

that if we in if we then from this road go here, because we do want to do weight quantization in


The Hardware Lottery (Paper Explained)

this particular way, we also are going to neglect this, which would be doing some whatever other


The Hardware Lottery (Paper Explained)

thing that we could do. Yeah. So there's always there's always in each decision, there's a


The Hardware Lottery (Paper Explained)

branching. Undoubtedly, the paper is correct. And it says the branching decides the future. But


The Hardware Lottery (Paper Explained)

I think the focus here on hardware and neural networks versus non neural networks is a bit,


The Hardware Lottery (Paper Explained)

it's very specific to that thing. It then it makes the it makes the point why it matters.


The Hardware Lottery (Paper Explained)

So why it matters, it matters because the paper says, Okay, where's that? Here. In 2019,


The Hardware Lottery (Paper Explained)

a paper was published called machine learning is stuck in a rut. The authors consider the difficulty


The Hardware Lottery (Paper Explained)

of training a new type of computer vision architecture called capsule networks. And I


The Hardware Lottery (Paper Explained)

kind of realized that capsule networks aren't really suited to current to current to current


The Hardware Lottery (Paper Explained)

hardware. And he says, whether or not you agree that capsule networks are the future of computer


The Hardware Lottery (Paper Explained)

vision. The authors say something interesting about the difficulty of trying to train a new


The Hardware Lottery (Paper Explained)

type of image classification architecture on domain specific, specialized hardware. Hardware


The Hardware Lottery (Paper Explained)

design has prioritized delivering on commercial use cases, while built in flexibility to accommodate


The Hardware Lottery (Paper Explained)

the next generation of research ideas remains a distant secondary consideration, which is true,


The Hardware Lottery (Paper Explained)

though, I would also say, I mean, GPU CPUs and GPUs combined, are extremely general operations


The Hardware Lottery (Paper Explained)

like they're very, very generalized. Okay, GPUs are good at matrix multiplies, but CPUs are good


The Hardware Lottery (Paper Explained)

at a lot of other things. I would say the GPU CPU combo is a very, very, very flexible, general


The Hardware Lottery (Paper Explained)

purpose hardware design that doesn't doesn't lock you in too much. And maybe, maybe it's just that


The Hardware Lottery (Paper Explained)

capsule networks are by algorithmic design, way, way harder to implement, like to build specialized


The Hardware Lottery (Paper Explained)

hardware for capsule networks, I'm not sure if that would even be possible and to speed them up


The Hardware Lottery (Paper Explained)

to the degree that CNNs are sped up by GPUs just out of the algorithmic nature of capsule networks.


The Hardware Lottery (Paper Explained)

And I've done videos on capsule networks, they sound pretty cool. But they also sound like


The Hardware Lottery (Paper Explained)

implementing the thing in hardware is going to be quite tough, even if you build specialized hardware.


The Hardware Lottery (Paper Explained)

They also go into GPT-3 claiming that so current, the paper claims that because


The Hardware Lottery (Paper Explained)

we are kind of locked in in this neural network, this neural network paradigm in this kind of


The Hardware Lottery (Paper Explained)

hardware, several major research labs are making this bet engaging in a bigger is better race in


The Hardware Lottery (Paper Explained)

the number of model parameters and collecting ever more expansive data sets. However, it is


The Hardware Lottery (Paper Explained)

unclear whether this is sustainable. An algorithm scalability is often thought of as the performance


The Hardware Lottery (Paper Explained)

gradient relative to the available resources. Given more resources, how does the performance


The Hardware Lottery (Paper Explained)

increase? And they go into examples here that you can scale up the parameters, which gives you


The Hardware Lottery (Paper Explained)

less and less of a of a gain. So it's like this diminishing return over time, which it brings up


The Hardware Lottery (Paper Explained)

GPT-3, which I find interesting because GPT-3 showed in a way okay, it was in log space, but it


The Hardware Lottery (Paper Explained)

showed a fairly, fairly linear decrease in perplexity. So a log linear decreasing perplexity,


The Hardware Lottery (Paper Explained)

given more parameters, which goes a bit against the narrative of the paper and also in terms of


The Hardware Lottery (Paper Explained)

this definition up here, given more resources, how does the performance increase? I see the fact


The Hardware Lottery (Paper Explained)

that you say, well, it's 12 billion, sorry, $12 million to train GPT-3 says right here,


The Hardware Lottery (Paper Explained)

$12 million to train GPT-3. On the other hand, I would say, what's the cost of, you know, building


The Hardware Lottery (Paper Explained)

specialized hardware to research alternative research directions? By the way, we have no idea


The Hardware Lottery (Paper Explained)

what alternative research directions work. So the only thing we could do is fund all hardware. And


The Hardware Lottery (Paper Explained)

if we have to fund all hardware for other algorithms, then select the ones that are


The Hardware Lottery (Paper Explained)

promising, then invest more, and so on $12 million will get us nowhere, which I think is a point the


The Hardware Lottery (Paper Explained)

paper is trying to make. But from a efficiency perspective, given where we are now, it's actually


The Hardware Lottery (Paper Explained)

more viable to build GPT-3. Which, again, I think this is something the paper agrees with, but


The Hardware Lottery (Paper Explained)

at the same time, it tries to make the point that, look, we are investing more and more and more,


The Hardware Lottery (Paper Explained)

and we're getting less and less out of it. Maybe it's time to go a different route in terms of


The Hardware Lottery (Paper Explained)

hardware. But that's going to be more and more expensive the more we go into this neural network


The Hardware Lottery (Paper Explained)

direction. I'm not sure about this. Again, it's not a good idea to invest more in hardware,


The Hardware Lottery (Paper Explained)

but I'm not sure about this. Again, if you think of this tree, the paper basically tries to argue


The Hardware Lottery (Paper Explained)

that what GPT-3 is trying to do is it's trying to make a push up here into the next kind of push


The Hardware Lottery (Paper Explained)

the frontier on the path that we have gone for a while. And the paper is trying to say that had we


The Hardware Lottery (Paper Explained)

had we imaginarily gone a different path down here, a equally hard push in this direct in a


The Hardware Lottery (Paper Explained)

direction would maybe yield a better result. Yes, maybe, but yeah. But the question is, is it,


The Hardware Lottery (Paper Explained)

at what point does it become viable to sort of abandon this entire direction and skip and kind


The Hardware Lottery (Paper Explained)

of start there because we would need to do the whole tree thing again. And then within the tree,


The Hardware Lottery (Paper Explained)

the same logic applies. It does, though, make a good comparison to the human brain,


The Hardware Lottery (Paper Explained)

which works fundamentally different. It says, while deep neural networks may be scalable,


The Hardware Lottery (Paper Explained)

it may be prohibitively expensive to do so in a regime of comparable intelligence to humans.


The Hardware Lottery (Paper Explained)

An apt metaphor is that we appear to be trying to build a ladder to the moon, sort of saying that


The Hardware Lottery (Paper Explained)

we can't, we can't wait at the rate where we scale neural networks right now. It's not conceivable


The Hardware Lottery (Paper Explained)

that we reach human level intelligence by simply scaling them up, which is why we might want to


The Hardware Lottery (Paper Explained)

investigate different entirely different directions and why we might want to investigate entirely


The Hardware Lottery (Paper Explained)

different hardware choices. Yeah, which, you know, granted that's correct, though, I would say


The Hardware Lottery (Paper Explained)

transformers aren't particularly suited to the hardware because they require such huge memories


The Hardware Lottery (Paper Explained)

and GPUs traditionally have been rather limited in memories in memory, sorry, and transformers still


The Hardware Lottery (Paper Explained)

kick ass on these on this hardware, even though memory is extremely limited compared to like CPU


The Hardware Lottery (Paper Explained)

memory. And only now do we see GPU manufacturers focus on more memory. So you can argue from the


The Hardware Lottery (Paper Explained)

perspective of the paper and say, see, because we have neural network hardware, now people are


The Hardware Lottery (Paper Explained)

building more neural network hardware. But also, you can say that initially a bad choice was made,


The Hardware Lottery (Paper Explained)

sort of, but researchers still managed to demonstrate transformers would work and now the


The Hardware Lottery (Paper Explained)

hardware is developing in this direction. Which is also a thing the paper argues at some point.


The Hardware Lottery (Paper Explained)

Again, I have a hard point parsing out a direct point here, I think the paper is more meant to


The Hardware Lottery (Paper Explained)

make you sort of think about the different points it brings up, which is also probably why this


The Hardware Lottery (Paper Explained)

video is more of me rambling than anything else. So here it says that currently, there are some


The Hardware Lottery (Paper Explained)

initiatives to build other types of chips, other types of hardware and so on. But they as well as


The Hardware Lottery (Paper Explained)

the last ones, they might be not enough because it takes producing a next generation chip typically


The Hardware Lottery (Paper Explained)

costs 30 to $80 million, and two to three years to develop. And even that is however, even


The Hardware Lottery (Paper Explained)

investment of this magnitude may still be woefully inadequate as hardware based on new materials


The Hardware Lottery (Paper Explained)

requires long lead times of 10 to 20 years in public investment, and is currently far below


The Hardware Lottery (Paper Explained)

industry levels of R&D. This, this is the kind of DARPA and China who funded research in this


The Hardware Lottery (Paper Explained)

direction. So the paper says it might be way too little. Though it also says there are a couple of


The Hardware Lottery (Paper Explained)

good lights at the end of the tunnel, saying experiments using reinforcement learning to


The Hardware Lottery (Paper Explained)

optimize chip placement may help decrease cost. And I think I've done a video on this paper


The Hardware Lottery (Paper Explained)

on this paper. There are also renewed interest in reconfigurable hardware such as field program


The Hardware Lottery (Paper Explained)

gate arrays, and coarse grained reconfigurable configurable arrays. So this is hardware that


The Hardware Lottery (Paper Explained)

you can sort of metaprogram. So you can take the hardware and you can specialize it by programming


The Hardware Lottery (Paper Explained)

it. And so it's like a meta programming it, you can sort of take one of these things and make it


The Hardware Lottery (Paper Explained)

into like a sort of a GPU if you need it like that, and then you can reprogram it, program it


The Hardware Lottery (Paper Explained)

differently for a different application. Though, if again, if I take the other side of this paper,


The Hardware Lottery (Paper Explained)

I would say, well, isn't that the same thing that CPUs were and yet still CPUs made it almost


The Hardware Lottery (Paper Explained)

impossible for neural networks to run? Aren't you even though FPGAs are very general, aren't you


The Hardware Lottery (Paper Explained)

making implicit choices on the ideas that are very well suited to FPGAs or the ideas that are


The Hardware Lottery (Paper Explained)

very well suited to using reinforcement learning to optimize chip placement? Isn't isn't that the


The Hardware Lottery (Paper Explained)

exact same thing? Yeah, I guess you can make this argument at in like at infinitum, infinitum,


The Hardware Lottery (Paper Explained)

infinite, no infinite is different. Okay, this this video must come must come to an end. So the


The Hardware Lottery (Paper Explained)

last part here says that what is also needed is kind of a software revolution, that there is a


The Hardware Lottery (Paper Explained)

shorter feedback time where it imagines software that tells researchers which hardware their


The Hardware Lottery (Paper Explained)

algorithm is particularly suited or how their algorithm would fare on different hardware,


The Hardware Lottery (Paper Explained)

such that if you invent a new algorithm, it doesn't work on a GPU, you could sort of


The Hardware Lottery (Paper Explained)

submit it to the software. And then the software will tell you what that this would work really


The Hardware Lottery (Paper Explained)

well if type x of hardware existed, and then you can maybe invest money into into that rather than


The Hardware Lottery (Paper Explained)

discarding your idea. In conclusion, yeah, it doesn't the conclusion isn't very long.


The Hardware Lottery (Paper Explained)

The performance of an algorithm is fundamentally intertwined with the hardware and software it


The Hardware Lottery (Paper Explained)

runs on this essay proposes a determined hardware lottery to describe how these downstream choices


The Hardware Lottery (Paper Explained)

determine whether a research idea succeeds or fails. Today, the hardware landscape is


The Hardware Lottery (Paper Explained)

increasingly heterogeneous. This essay posits that the hardware lottery has not gone away,


The Hardware Lottery (Paper Explained)

and the gap between the winners and losers will grow increasingly larger. In order to avoid future


The Hardware Lottery (Paper Explained)

hardware lotteries, we need to make it easier to quantify the opportunity cost of settling for the


The Hardware Lottery (Paper Explained)

hardware and software we have. And my conclusion is I generally agree with this paper, I really


The Hardware Lottery (Paper Explained)

appreciate the historic overview. But I do think the focus is it centers too much around hardware,


The Hardware Lottery (Paper Explained)

where I think this lottery case you can make for literally any single branching choice. And maybe


The Hardware Lottery (Paper Explained)

you weigh that by the costs that it takes to revert or change that choice in the future. And it also


The Hardware Lottery (Paper Explained)

focuses a lot on neural networks versus non neural networks, where it kind of the this this winners


The Hardware Lottery (Paper Explained)

and losers thing, where it says, neural networks are the winners. And if we investigate more into


The Hardware Lottery (Paper Explained)

neural networks, then they will remain the winners because of this feedback loop. However, it kind


The Hardware Lottery (Paper Explained)

of, in my opinion, discards the thing that within the neural networks in the next choice of hardware,


The Hardware Lottery (Paper Explained)

there are going to be winners and losers again, and again, and again, and they're going to be


The Hardware Lottery (Paper Explained)

entire branches of neural network research that are abandoned because they don't fit the hardware


The Hardware Lottery (Paper Explained)

choices once more. And this gap between what it's conceived the winners and the losers it only it


The Hardware Lottery (Paper Explained)

compares losers in terms of an idea that was had in one year to the winners, which are always


The Hardware Lottery (Paper Explained)

reevaluated every year. So it's kind of not a fair comparison, in my opinion. And then also,


The Hardware Lottery (Paper Explained)

no, that was it for me. Yes, I do. I do implore you if you are interested in things like this,


The Hardware Lottery (Paper Explained)

as I said, this is more of a historical and opinion piece, trying to make some argument and


The Hardware Lottery (Paper Explained)

give you some directions to think about, which is is pretty cool as a change to a simple blunt


The Hardware Lottery (Paper Explained)

research paper. All right. That was it for me. Again, if you're still here waiting for how to


The Hardware Lottery (Paper Explained)


Machine Learning PhD Survival Guide 2021 | Advice on Topic Selection, Papers, Conferences & more!

on how to do a PhD. So mainly that you don't repeat my mistakes.


Train.


We've made it into a PhD program. Congratulations, you made it. So today we're going to have a look


at what to do during a PhD, how to succeed at publishing papers, how to deal with reviews,


what to do at conferences and many other things. So I hope you enjoy this little guide of how to


survive a machine learning PhD in 2021. So first of all, let me say, I'm not good at this. I'm not


an expert. I'm at the end of my PhD and I've done many things wrong and by no means am I a successful


academic. However, if you're like myself, and at the beginning of your PhD, you don't really have


a clue what to do. You don't know how to select topics, you don't know how to write papers or even


what a paper is really, then there might be something in here that could help you. I'm not


super successful myself. But what I can tell you is that I've seen many people who are good at it.


So I can tell you what those people did right, what I did wrong, and generally what I think you


should do. Alright, that being said, let's dive right in. When it comes down to choosing a topic,


make sure you look for something that your advisor or the senior people around you have lots of


experience in they can help you much better like this. You also want to choose something that


matches your particular interests, because you're going to be stuck with it for a while. Lastly,


you want to choose something that fits your expertise where you're already reasonably good


at or can get good at very quickly. At the intersection of those three things, you're


gonna find something that is unique to you, and is going to be a very good topic for your PhD.


But there are a few more things to consider when selecting a topic. First of all resources,


how much access to resources you have will determine what kind of topics are even accessible


to you as a researcher. So I'm going to assume that you do not have a giant compute cluster or


heaps of money around. And therefore my recommendations are going to be for, let's


say the rather average PhD student who is not a giant tech company. However, if you do happen to


have 1000s of TPUs in your backyard, ignore my advice and just train big language models.


All right, there are two fundamental ways how you can choose a topic. Way one is to choose the


biggest most hyped topic in the area right now. Now that is not necessarily a bad strategy,


but it has some drawbacks. And the reason is that in a hype topic, there are many papers,


but there is also a giant amount of competition, not only from other researchers, but from


large corporations with lots and lots of resources behind them. And the bigger reason why it's a bad


idea is the fact that they wane. If you pick transformers to research today, it's very likely


that three, four years down the road, you'll still be stuck with transformers, the field has moved on.


And now all of these people that have made the same choice, namely to invest in the biggest topic


right now are trying to finish their PhD or trying to get papers published in that topic that is no


longer of such a big interest at that particular point in time, and therefore already be on the


declining side of the hype cycle. So what's the alternative to hype topics? The alternative is


niche topics. And that's what I would recommend for most people. The advantages of finding niches


is there isn't as much competition around and you can actually become an expert and the best at


whatever you do. Some examples of niche topics are things like bandits, optimization, biologically


plausible neural network, text based games, I'm not suggesting you go into these topics,


but look for smaller communities that nevertheless publish year after year after year. Alright,


so now the important stuff, how do you get papers published? Now if I had to summarize the style of


writing papers that get published in one sentence is that write papers that cannot be rejected. And


that is not as obvious as it sounds. The review process in machine learning is heavily incentivized


to reject your paper as quickly and easily as possible. Do not give reviewers any reason


to reject your paper. And the easiest way to learn how to write papers is to literally read papers,


go into your niche, gather the papers that are there, read them, try to emulate their writing


style, try to emulate the type and way they do and present experiments, try to emulate the way


they write up theoretical foundations for their ideas, your goal is going to be to write a paper


where there is no obvious criticism to be had by reviewers. Reviews are the single biggest obstacle


to achieving your goals. And let me tell you right now, getting reviews is one of the most cruel


experiences you're going to have in your PhD. Reviewers are nasty, they don't have time, they


don't read the paper correctly, they misunderstand, they criticize that you didn't evaluate on some


obscure data set. And in general, you're going to feel quite misunderstood by reviewers. This happens


to all of us. What I can tell you is don't get discouraged by bad reviews. Don't take individual


reviews too seriously, and just resubmit the paper to the next conference. So keep your sanity,


don't take it personally. There are many famous papers that have been rejected at first try,


and not because the paper was bad, but just because the reviewers were crappy.


Now there are going to be things during your PhD that you'll have to do that are not writing papers.


And one of those things is especially as you get more senior, you're going to be asked to review


yourself. Now, it is an easy option to take all that frustration that you have against reviewing,


and you see all these other people doing such a crappy job that you just think, whatever,


I'm going to do a crappy job myself. And it's tempting. It's very tempting, especially because


you gain nothing from doing good reviews. But other than a you Hey, thanks for the review,


you'll get nothing. And it is really, really hard to write a good review. Do it. Nevertheless,


please, not only are you helping the field by being not one of the crappy reviewers,


but writing a good review also helps you really dig into a paper, really see the weaknesses in


other papers. And it makes you a better author, researcher and community member. So for your own


sake, and for the community, take the review seriously, even though you don't have time,


even though other people do a crappy job. Another thing that you're going to be asked to do very


probably is teaching. Now again, you're going to have very little incentive to do a good job at


teaching. After all, students are nuisances, the faster you can get it over with the better,


the earlier you can go back to writing papers. However, I urge you to take teaching seriously,


not only because the world relies on the next generation of researchers being competent,


but also think about the fact that the people you teach will be probably some of them working with


you in the future. They might be researchers in other labs you collaborate with, they might even


be joining your own lab, and you will profit from them being more competent. So take teaching


seriously for your benefit and for the benefit of your students. So besides the things you have to


do like reviewing and teaching, what should you work on all day? And here's my answer. Start


working on your thing, go pee, and then continue working on your thing. A PhD is first and foremost


an exercise in long term focus, you're going to be tempted to do all kinds of things during your PhD,


you're going to look and here's a reading group, and here's a seminar and here's a lecture. Now,


unless it is on your specific thing on your specific niche, it's probably going to be not


a productive use of your time. I'm not saying you shouldn't go there. What I'm saying is that be


aware that what ultimately gets you to get your papers is a long term laser focus on your topic,


and other topics will creep up on you. It's going to be so interesting because you're stuck here


with your thing that you know and that is boring. And there's going to be this other cool topic.


Wow, here we are. This is the NURBS 2019 poster session, one of the poster sessions, there are


about 250 posters in this room. And there are so many people. It is crazy. Every single poster has


a like a ball of people around it, presenters trying to explain to the bystanders their work


there. And you're going to be tempted. Oh, this is interesting. This is interesting. This is


interesting. And my topic is so lame. I'm going to just look into this. And that's also cool.


Yeah, you know who did that me? It did not turn out well. Focus, focus, focus, focus your research


on your thing, and you'll be successful. So now you've written your paper, you've submitted it to


peer review. And with a little bit of luck, you've actually managed to get it published,


and you get to go to a conference. Now the conference itself and the conference website,


and everyone on Twitter might give you the impression that conferences are there for people


giving talks about their research and you listening and learning. That's crap. Conferences,


especially the talking part of conferences have become more and more irrelevant with the years,


specifically now that everything is recorded and streamed. Just look at that stuff from the comfort


of your couch at 2x speed, you're missing nothing. These talks are often very short, very rehearsed.


And most importantly, they are about research that is at least six months old. The interesting


part about conferences are the people there. The interesting talking happens in workshops,


in panels, in tutorials, try to find places where current research is discussed.


Workshops are a great place to go for this because the research is often much more recent,


and not done yet. Go to conferences to interact with people. This whole all we come together for


research. That's a charade. The best researchers I know, do nothing else but meet and talk to people


all day at conferences. And I don't mean this in a mean way. I don't mean go out and deliberately


engineer contact with people for your own benefit. No, a conference is a place where you can find


other people that are interested in the same things as you are. And you can talk to them get


to know things that you could never get to know through a writing or in a paper. A lot of paper


authors will tell you things face to face that they would never write down a paper such as which


experiments that don't work problems in research weaknesses of papers, you'll get a lot of knowledge


by being there and talking to people, but you have to go out of your way and do it actively.


I know this is hard for a lot of us, but it pays off and is going to make your life a lot more


enjoyable. Alright, the next thing I want to talk about is internships. Should you go to an internship


at a company at a different university and this depends entirely on your preference. Now I myself


have had pretty good experiences with internships, and people I know have done so as well. Generally,


if you do an internship, it gives you a bit of a different perspective because you do it at a


different place. And if you do an internship with a large company, it can be quite a switch of


environment, you'll have access to many more resources, and you can do maybe a little bit


of a different type of research. And most importantly, you'll meet people that are not


academics or not academics anymore. And that is very, very valuable. Once you've been stuck in


academia for a while, meeting someone who just cares to build a cool product is so refreshing


and gets you a bit down to earth with what's really important. Lastly, I want to talk about


the topic of collaborations. Now academia is a bit tricky in that the system tries to alienate


and isolate you as a person, you need those first author papers, you need to provide a personal


contribution to the knowledge of humankind. Look for people who have the same interests


in terms of topic, but who have a little bit different skills or experiences, such that your


papers and your research can become more well rounded. That could be a difference in theoretical


versus experimental knowledge, that could be a difference in your academic background. So if you


can find someone that has complementary skills to yours and is interested in the same niche,


it definitely pays off to work together and produce research together. However, only do this


if they really work in the same field. It is very tempting to start all kinds of collaborations with


people all over the place. If you can handle that good for you. But again, it pays to have a little


bit of focus on your particular field and really view collaborations as a joint effort to get


research done more quickly and with more rigor. Right, so the way I discussed it right now,


it seems like doing a PhD is gruesome and lots of work and you never get to do anything fun. And


while there is an aspect to that, and it definitely can happen to people, especially if


they want to finish real quickly, I urge you to also make some time to enjoy this time.


A PhD is a cool time, you'll get to meet so many interesting people get to learn so many


interesting topics and ideas and you'll hopefully get to go to many interesting places. And that is


an invaluable experience. So my advice is, if you can take it a bit easier. Enjoy your time.


Take as much out of it as you can and don't work all the time. Maybe you'll have half a year longer


who cares. You only get to do a PhD once and enjoy the time at university while you still can. You


can get a job any day. So I hope you've gained at least something from this video. And you should


Parameter Prediction for Unseen Deep Architectures (w/ First Author Boris Knyazev)

Hi everyone, welcome to another video. Today we're looking at parameter prediction for unseen


deep architectures and we have a special guest today Boris Knyazev who is the first author of


this paper actually. So this is a first I guess for Boris and for myself to review a paper in a


bit of an interview style. The plan is that we go through the paper together. There's also been some


reception in the public because as you might have heard the paper claims to be able to predict the


parameters of a neural network without you having to train the neural network. At least I guess


that's the over hype that then leads to people saying wait a minute that can't be true and we'll


go exactly through this. We'll look at what they've done right here and yeah Boris welcome


welcome so much to the channel. Thank you for a great introduction. I'm very excited to be here


as well so I'm ready to take any critique from you. So how did this come to be? You're at the


University of Guelph and there's I see Vector Institute, Facebook AI research and the GitHub


is under Facebook research. How did this come to be? So this project I started as an intern


at Facebook AI in summer 2020 so more than a year ago and all our collaborators are from


Facebook AI so Meta AI right and that's why we decided to keep the code on the Facebook research


so yeah. Cool excellent and if we can I mean let's dive in right here essentially what we've


said so far is you have some kind of a neural network with whatever a bunch of layers and a


bunch of computational nodes and you have the weights in between somehow right so w1 w2 weight


matrices but not only that you have normalization you have any kinds of things and usually we have


some data set x and we have some result y we train with back propagation to find the best


parameters but in your case you went ahead and you essentially build this hyper network hyper


graph network that is able to take in if I remember correctly the data right yeah and the


architecture like the structure here of the weight matrices all of this goes in and into a neural


network which is a graph neural network right yeah and so some sort of a graph neural network


and we'll go into that what exactly this and out comes the these weight matrices so and you're able


to do this without training the weight matrices ever you just predict them yeah so one correction


here the network this hyper network doesn't take data as input it's trained on specific data


set say c410 or image net but at test time it doesn't take data as input it only takes a network


as input and that's why yeah it cannot generalize to other data sets okay so it is you do experiments


that I see here on c410 and on image net so these are two different let's say hyper networks that


you train you train one for c410 and you train another one for image net well in fact I trained


many many networks sure sure but but it's not one network that is going to predict the parameters


of of for any data set no yeah so we release one network for c410 and one network for image net


correct okay and this so here you say by leveraging advances in graph neural network we propose a


hyper network that can predict performant parameters in a single forward pass so the


single forward pass what does that refer to it means that I feed the architecture one single


time through the graph neural network or yeah so this phrase is to collect the difference between


and say recurrent networks where so there are some meta optimizers right and they can also do


something similar to our work but they require like many iterations in our case like it's a


single propagation basically through the graph yeah yeah and then you you get these you get these


uh these parameters out which is I mean that that's that's pretty cool and then you say um


on image net oh sorry on c410 you reach a 60 accuracy and on image net you reach a 50 top five


accuracy now these are let's say they're respectable numbers they're better than random but they're


way way below anywhere you know near what I could get by actually training a network that's it was


this urine or was this your intention or is this is it still surprising that you get this good


numbers yeah yeah it's still very surprising to me and to other co-authors and to many other people


I guess uh because uh it's very hard like when you have a novel network the assumption is that


you know you cannot predict parameters for that like if you predict it will be like some garbage


neurons so because there is a complex complex interactions between neurons so it's very hard


yeah for a novel network you yeah it's very hard that's the assumption uh I don't know if it makes sense


of course it makes sense of course yeah I mean it's it is it is in a way it's you know the


numbers aren't good but they are certainly good for never having you know trained um yeah but


there is a bit of a because the hyper network has been trained on that specific data set and maybe


we'll go a little bit into uh your what you exactly train this on so you introduce a new data set


which is this deep nets 1m data set right could you tell us a little bit about this so this is the


essentially the basis for learning this hyper network yes so it's a data set of training and


evaluation architectures and it's called uh deep nets 1m because we have 1 million training


architectures so we predefined them and we saved them so that people can reproduce uh training


probably uh and uh the idea uh there is some misconception that we actually also have trained


weights for those training networks but no we don't like we didn't train in 1 million architectures


yeah so and the architectures are almost random in a sense that the operations and connectivity


between them are constructed in a random way by uniformly sampling for from a specific space of


architectures yeah so you you define you define a space a design space which you call this right


and this design space consists of things like you know you can have you can have a convolution or


you can have an an ml or sorry you can have a linear layer right or you can have an attention


layer right and then that's followed by either a batch norm or a weight norm or not no no no


or not no normalization at all and then that's followed by da da da da right and then you build


sort of these these combinatorial things so one architecture would be a convolution with a weight


normalization and with something else and then also the design space includes kind of the parameters


for this so for the convolution you could have i don't know five three or one on one side like so


i can have a five by five convolution that has maybe is only depth wise and not fully


convolution and so on so there are all these sort of nested cartesian products yeah of these big


space that you define and then essentially you could say you you fix a random seed and then you


sample about a million times yeah that'd be a fair characterization so that you say okay with with


this we sample a million times from a fixed random seed and that so everyone has the same networks to


train on yeah yeah that's a fair statement and so there were some data sets like this before to do


neural architecture search specifically but you say you've you've extended the design space a


little bit and that so before these networks they would include kind of the design space is large


enough to include sort of the modern networks but you have you've sort of extended that even a little


bit more right right right so usually those neural architecture search works they have a quite


constrained design space because they mainly consider very efficient networks like efficient


net or squeeze net yeah mobile net but resnet is out of their design space because resnet is


considered a waste of resources in in in the nas community yeah but in our work we are still


interested in predicting like these large parameters let's assume that you had actually trained all the


weights for the million architectures right and you train your hyper network to predict these weights


and i sample a new one and then it could be fairly like someone skeptical might say well you've


probably seen a very similar network during training right so you just memorize the weights


from that so there are there are two differences here as you said you don't actually have the


trained weights of these million architectures which i want to come back in a second but you


also have these out of distribution samples do you want to maybe comment on what what are the


out of distribution architectures for this data set what do they look like right so the industry


well i first i say what is in distribution to highlight the difference so in this in distribution


is the test set it is the same as the like it uses the same generator to sample architectures as the


training architectures so and while the architectures are still all different they as


you said they can be quite similar and we actually measure that in the appendix like we have some


data for that so that's one of the reasons we designed those out of distribution splits and


yeah the motivation was to test particular distribution shifts for example what happens


if the networks become wider like have more channels like wide resnet instead of resnet


oh for example what happens if we want to predict the parameters for a deeper network


say resnet yeah 150 instead of resnet 50 right so there are these there are these subcategories


right there's wide and deep which are wider or deeper than you've seen during training and there


is also this batch norm free category yeah so there are various aberrations that you didn't see


necessarily during training but yeah i think it's fair to say that the performance of your method


still comes from the fact that you know the network has been trained on on certain things


it's just a matter of how how much does it generalize to new architectures yeah yes for


sure it was trained on all like operations that are that are used to compose out of distribution


networks but it wasn't trained on that particular configurations compositions so it's still and how


how just if we jump to the results like just an aspect of the results how how different are the


weights from like do you do you know what happens if you just kind of copy over weights from the


most similar network in the training data set does this work at all or have you done any kind of you


know dumb baselines to compare i tried but it turned out to be more difficult than it seems so


you you need to come up with many different heuristics like how to yeah copy weights if the


dimensionality doesn't match or like if the layers like not exactly the same so there is a lot of


yeah those yeah and it becomes basically a separate research project to develop this yeah


sure dumb baseline so we didn't go in detail with that yeah so this is i guess this is the training


loss what's special about this as you said you don't actually have the fully trained weights


of all of these netmer networks but essentially um what you do is you sort of back propagate


through training these networks if i if i understand this correctly so what you have here


is a double sum over n and m and n here is the number of tasks so what is a task right here uh


here uh task is uh so we use the terminology from meta learning right so but here the task is a


network okay so ah this is the this m m is the number of training architecture yeah and n i


presume is the data set or yes it's the number of samples in a data set like images yeah so we take


one point one data point x right xj right and what is the a right here that is that is the


architecture that we sample as one of the m architectures so we take the x we take the a


and this here that's your network that you actually want to train right so as you said it


does not get the training data point it simply gets the architecture and it has a set of parameters


which are ultimately the parameters we want to optimize right um and the so the f here i guess


i guess that is your way of saying take this network predict the weights pass the data point


through it and get the output yeah exactly that's a fair characterization for forward pass of images


through the predicted parameters to get the predicted predictions yeah so yeah f yeah f calls


if i were to program this then f would call your network to get f would instantiate a


would call your network with the architecture we'll get back the weights put that into a and


pass the data point through once yeah yeah exactly and then we simply compare it to the label right


which we which we have and then i i guess this loss right here is cross entropy loss or whatever


is appropriate for the data set yeah yeah so you can basically reduce this equation to equation


one if you if you freeze the architecture so if you if m is equal one uh and instead of having


a hyper network you have like fixed weights w w yeah then it's the same objective and it's the


same loss and then you learn by back propagating if i if i see this correctly so usually what we do


is we forward pass x through this thing right and then we back propagate to these weights right here


but what we do is we simply continue back propagating through the weight generating


function into the hyper network yeah and all of this is all of this is differentiable i guess


the weights are floating point numbers and the way the graph network works is all differentiable


so you can essentially back propagate it through the parameters here so every i guess every part


of the graph neural network would have weights and you can back propagate it through that


yeah i mean that seems reasonable enough uh this connection here that's not that's not happening


no data to the graph network for now yeah cool um this seems it seems pretty straightforward


so now maybe we talk about what exactly the graph neural network is getting as uh features


and when when we talk about graph neural networks it's it's always a bit um there are many flavors


of graph neural networks but i'm going to try to characterize it um uh briefly so we have nodes in


the graph neural network and each node has a bunch of features initially so each node gets a bunch of


like a vector of different features in our case the nodes each would refer to different yeah


operations different modules right so this here could be this here could be the conv the convolutional


uh the convolutional layer in the first layer this then could be the batch norm that follows


in the first layer and this could be the convolution in the second layer and so on


so we connect the graph neural network in the same way that the architecture is connected right


so so that the graph neural network changes from architecture to architecture oh uh the graph no


no the graph neural network is fixed so the graph neural network itself doesn't have no


it doesn't have nodes right the graph neural network will have weights like theta right and


this theta are basically like a matrix with the number of input features and the number of output


features yeah uh so the those weights are fixed what is changing is the input that is represented


as a graph i see so this here maybe we can more characterize as the input yes and that goes into


that goes into a let's say a standard neural network with a bunch of layers yeah but the input


is essentially what you call i think a which is the it's an adjacency matrix yeah so this graph


would be described by an adjacency matrix matrix and for lack i don't exactly remember how you


called it but let's call it f the features yeah of each of the nodes right and these things will go


will go into the neural network and out would come your your different weights for for the graph yeah


and yeah so the way graph neural this these graph neural networks work is each node essentially


starts with a bunch of features here right this has a vector this has a vector and then you apply


these functions so every layer here would correspond to one message propagation step


yeah if i understand this correctly where all of the neighbors they would sort of pass messages


to each other given differentiable functions so if we consider this node it would sort of receive


from all its neighbors it would receive messages it would compute some sort of hidden state and


then in the next iteration it would pass that hidden state to its neighbors right right this


is the basic functionality now you in your particular case have opted for a bit of a more


advanced architecture right here that more mirrors sort of the the propagation in a neural network


can you talk a little bit about that maybe right so we actually we are doing almost the same as


the previous work on graph hypernetwork so i wanted to clarify that the training objective


like equation two and the graph hypernetwork architecture is almost the same as the previous


work but yeah they didn't release the open source code so we had to like reinvent something yeah


of course but so sorry what what was the question so i'm referring so maybe maybe before that i


want to i want to just for people who may not know graph neural networks it seems like there's


a lot going on but essentially a graph neural network boils down to just a few functions


because what i've described right here this i receive i receive the hidden states from all my


neighbors and i integrate that right this function is in fact the same function as you know the node


over here which also receives stuff from all its neighbors and integrates that that would be the


same function with the same weights right it's just that the inputs are different because of course