AlphaCode - with the authors!

fuzzing approach to generating new test cases where during training, you know the correct


AlphaCode - with the authors!

like solutions, so you can essentially generate new correct test cases by using the correct


AlphaCode - with the authors!

solutions that you know are correct, which I found, yeah, it makes sense.


AlphaCode - with the authors!

I think in this space of programming, you can do a lot of these things, which is neat.


AlphaCode - with the authors!

Right.


AlphaCode - with the authors!

So what happens basically is we mutate programmatically the inputs of the test that we already have,


AlphaCode - with the authors!

and then we run the human correct solutions on them, and then if we filter these new mutations,


AlphaCode - with the authors!

because some of them might not actually be correct inputs, and we figure out whether


AlphaCode - with the authors!

the human solutions actually agree on an output, and when we have a sufficient level of agreement


AlphaCode - with the authors!

on a given output, then we add this mutated input to the output that's generally agreed


AlphaCode - with the authors!

upon.


AlphaCode - with the authors!

Now you mentioned before that you had high points and low points during the process of


AlphaCode - with the authors!

this project.


AlphaCode - with the authors!

Again, I can imagine that might be one of the lower points when you realize, wait a


AlphaCode - with the authors!

minute, all we do is false positives.


AlphaCode - with the authors!

Could you, I don't know, could you let us in maybe on what was sort of the lowest point?


AlphaCode - with the authors!

Was there a moment where you thought, oh, this isn't going to work out after all this


AlphaCode - with the authors!

time?


AlphaCode - with the authors!

And what did you do to overcome these things?


AlphaCode - with the authors!

That's a tough question.


AlphaCode - with the authors!

I think the lowest point probably wasn't the same for all of the members of the team, right?


AlphaCode - with the authors!

Because we were working on slightly different ideas most of the time.


AlphaCode - with the authors!

But I think in the middle of a project, there was basically a month where we had very, very


AlphaCode - with the authors!

little progress.


AlphaCode - with the authors!

And so we had these meetings every week where we would see what was the best performing


AlphaCode - with the authors!

thing and it was still the same thing.


AlphaCode - with the authors!

So there's that.


AlphaCode - with the authors!

That was definitely no point for us.


AlphaCode - with the authors!

And maybe also when some of the big ideas that we thought were going to help didn't


AlphaCode - with the authors!

pan out.


AlphaCode - with the authors!

Like, for instance, when we realized that for whatever reason, it was just too hard


AlphaCode - with the authors!

to train a really good value function and we weren't going to be able to leverage all


AlphaCode - with the authors!

of the methods that this would have unlocked, which we did rely upon, at least initially


AlphaCode - with the authors!

in our mind map.


AlphaCode - with the authors!

So yeah, that would be it.


AlphaCode - with the authors!

I would not answer.


AlphaCode - with the authors!

I definitely had a couple of those myself.


AlphaCode - with the authors!

But I think in general, a lot of the times we realized that we got results which weren't


AlphaCode - with the authors!

actually true because they were false positives.


AlphaCode - with the authors!

Later on, we did claw back a lot of the gain.


AlphaCode - with the authors!

But I think that's just maybe the scientific method at work, right?


AlphaCode - with the authors!

We kind of proved us.


AlphaCode - with the authors!

We tried something and then we realized actually it wasn't working.


AlphaCode - with the authors!

But yeah, I think having our metric to guide us there really helped us get through those.


AlphaCode - with the authors!

I think we were well served by a somewhat skeptical approach when we had a result that


AlphaCode - with the authors!

looked good to be true.


AlphaCode - with the authors!

Our initial thought was, okay, this is good to be true.


AlphaCode - with the authors!

Where's the issue?


AlphaCode - with the authors!

And more from the not, there was actually a bug that we found.


AlphaCode - with the authors!

Once you released the, let's say the paper and so on, I think a lot of comments started


AlphaCode - with the authors!

coming in.


AlphaCode - with the authors!

Did you have a criticism that, like what is the most valid criticism that you've encountered


AlphaCode - with the authors!

that you didn't foresee?


AlphaCode - with the authors!

You have a lot of limitations at the end of the paper and you make it very clear, this


AlphaCode - with the authors!

is one niche, this is this, there's limitations here.


AlphaCode - with the authors!

Is there something that people brought up and you were like, oh yeah, I didn't think


AlphaCode - with the authors!

of that.


AlphaCode - with the authors!

That's a good point.


AlphaCode - with the authors!

Yeah.


AlphaCode - with the authors!

There's a few things.


AlphaCode - with the authors!

It's a difficult question generally, but there's a few things definitely.


AlphaCode - with the authors!

Generally, as we said, we've been very happy with how the work was received and we've gotten


AlphaCode - with the authors!

a lot of constructive feedback.


AlphaCode - with the authors!

Dima Badanov's Twitter thread is a good example, for instance, where he outlined why he thinks,


AlphaCode - with the authors!

and we do agree with him, that we're still a long way from top level human performance


AlphaCode - with the authors!

on this task.


AlphaCode - with the authors!

I was also made aware that the data that we put on alphacode.beatmine.com was actually


AlphaCode - with the authors!

not correct.


AlphaCode - with the authors!

I had filtered the correct solutions wrong.


AlphaCode - with the authors!

So again, underlining the importance of doing that right.


AlphaCode - with the authors!

So I thank everybody who told us, well, I don't understand this correct solution.


AlphaCode - with the authors!

It's actually not correct.


AlphaCode - with the authors!

And they were right.


AlphaCode - with the authors!

So now we've fixed that.


AlphaCode - with the authors!

So if you go to alphacode.beatmine.com, you will get actually correct solutions.


AlphaCode - with the authors!

And then something that surprised us, but I don't know whether it's valid or not, is


AlphaCode - with the authors!

that a fair amount of people seem to think that the average human competitor on codeforces.com


AlphaCode - with the authors!

is not very good, which I think we have a fairly different view.


AlphaCode - with the authors!

So I'm not sure I would say it's valid, but it was certainly surprising to us.


AlphaCode - with the authors!

And then in terms of the limitations of the model, we thought a lot and just a bit of


AlphaCode - with the authors!

our what we thought were the weaknesses.


AlphaCode - with the authors!

So I'm not sure that I've seen anything that we hadn't already identified.


AlphaCode - with the authors!

Cool.


AlphaCode - with the authors!

Where do you see this more in the real world?


AlphaCode - with the authors!

We talked about programming, competitive programming, maybe, you know, a future where I can just


AlphaCode - with the authors!

write a bunch of unit tests, and this will, it will go fine.


AlphaCode - with the authors!

But there are obviously applications beyond this.


AlphaCode - with the authors!

Is there, are there people maybe in your team that are already eyeing?


AlphaCode - with the authors!

Or maybe you have some ideas of this?


AlphaCode - with the authors!

Where could this be used outside of programming?


AlphaCode - with the authors!

Just the techniques in here and the methodologies?


AlphaCode - with the authors!

Do you see some sort of semi-obvious transfer to a real world problem other than coding?


AlphaCode - with the authors!

I think generally speaking, there's going to be a lot of downstream applications for


AlphaCode - with the authors!

general purpose problem solving AIs.


AlphaCode - with the authors!

To our team, we've been thinking a lot about programming and less about non-programming


AlphaCode - with the authors!

applications.


AlphaCode - with the authors!

So I think far for good, there's some natural directions, which include developing tools


AlphaCode - with the authors!

to make coding easier, as we already touched upon with, I don't know, automated test generation,


AlphaCode - with the authors!

auto complete, etc., or maybe tools to make it easier to learn how to code.


AlphaCode - with the authors!

So you could imagine an AI that can comment and suggest some improvements to your code,


AlphaCode - with the authors!

etc.


AlphaCode - with the authors!

So ultimately, applications that could be used to democratize programming are definitely


AlphaCode - with the authors!

on our radar.


AlphaCode - with the authors!

In terms of applications not directly related to programming, that's, I haven't thought


AlphaCode - with the authors!

too much about that.


AlphaCode - with the authors!

I'm fairly certain that problem solving is sufficient in general so that we will find


AlphaCode - with the authors!

interesting applications, but we haven't been too much on the lookout for that.


AlphaCode - with the authors!

I think you're right to point out a couple of those ideas, Yannick.


AlphaCode - with the authors!

I think Codex has also shown us that this works.


AlphaCode - with the authors!

You can build a product out of these kinds of models and people are really happy with


AlphaCode - with the authors!

it.


AlphaCode - with the authors!

It's definitely something that we're thinking about, but I think we definitely haven't concretely


AlphaCode - with the authors!

made any decisions at all or finished brainstorming even, whether that's something that we'd like


AlphaCode - with the authors!

to do.


AlphaCode - with the authors!

But yeah, I think maybe to go back to one thing that Remy mentioned earlier is that


AlphaCode - with the authors!

the methods that we use are actually pretty general, I find, as far as programming goes.


AlphaCode - with the authors!

The filtering, which is the really big one, could definitely be used in an application.


AlphaCode - with the authors!

But a lot of what soft regs do is just nothing to do with writing code.


AlphaCode - with the authors!

One way I guess I would think about it is what we've done is take a description of a


AlphaCode - with the authors!

problem and actually a complete description of a problem and map that to code.


AlphaCode - with the authors!

But really, I find in my day to day, I'm spending maybe 50% or more of my time talking to people


AlphaCode - with the authors!

and writing that description, if that makes sense.


AlphaCode - with the authors!

Yeah, Alpha Requirements Engineer is the next paper.


AlphaCode - with the authors!

Is there anything else you want to get out about this paper?


AlphaCode - with the authors!

Can people somehow get started with or get into this type of research or anything you'd


AlphaCode - with the authors!

want to communicate?


AlphaCode - with the authors!

I think so.


AlphaCode - with the authors!

We'd be really excited for other researchers to work on.


AlphaCode - with the authors!

I know some other researchers are already working on this problem.


AlphaCode - with the authors!

But our goal is as many as possible actually work on this problem because any game we make


AlphaCode - with the authors!

here is going to be distributed.


AlphaCode - with the authors!

So that would be really nice and that's why we released our data set, which we spent a


AlphaCode - with the authors!

fair amount of time on and we think is a really good tool to approach these problems.


AlphaCode - with the authors!

As we showed in the paper, you don't need huge models to actually start solving problems.


AlphaCode - with the authors!

So you can do that with less resources.


AlphaCode - with the authors!

Of course, there's the issue of having to sample a whole lot, but I would say that's


AlphaCode - with the authors!

a very exciting research direction to actually reduce the amount of samples you have to take


AlphaCode - with the authors!

to solve these problems.


AlphaCode - with the authors!

Peter, any messages for anyone listening?


AlphaCode - with the authors!

I think as Remy said, the fact that we released the data set is clear that that's the main


AlphaCode - with the authors!

point that you should start.


AlphaCode - with the authors!

I think generally I'm optimistic not just about competitive programming, but about people


AlphaCode - with the authors!

working on programs and physics in general with machine learning.


AlphaCode - with the authors!

I can only encourage people to go and do it.


AlphaCode - with the authors!

Actually, I should say that as a programmer myself, I'm quite optimistic that working


AlphaCode - with the authors!

on this kind of problem is going to make my life a bit easier.


AlphaCode - with the authors!

In this case, Peter and Remy, thank you very much for being here.


AlphaCode - with the authors!

This was a lot of fun.


AlphaCode - with the authors!

I learned a lot and I hope to see the alpha requirements engineer in the future.


AlphaCode - with the authors!

Thank you for having us.


AlphaCode - with the authors!


I BUILT A NEURAL NETWORK IN MINECRAFT | Analog Redstone Network w/ Backprop & Optimizer (NO MODS)

I built a fully functional trainable analog neural network in Minecraft with no command blocks and no mods. Check this out.


Hello. Hello. Hi. I'm trying to build a neural network.


Can you please... I don't want to buy your stuff. I... like... no, I don't want a bucket of... no, I don't want a bucket of puffer fish.


What you're seeing here is an analog neural network. While lots of people build binary computers in Minecraft, this neural network works in an analog fashion.


It means it works directly with the signal strength on these wires right here.


It has two layers and it has two neurons in its hidden layer. It computes an output, it compares that output against a target, it back propagates the error back through the network, and it is even able to update its own weights in response.


So it can fully autonomously learn any function that you want.


So today I'm going to show you how I built this, how it works, and what could potentially be improved.


Be sure to like this video and let me know what you think in the comments.


So the output is 9 and now I change the input back to the last data point.


The max operation is actually released. Yes, but the org max isn't. Right? It's 6. It learned two data points.


It learned two data points.


So this whole network runs on redstone. Redstone is a concept in Minecraft that is a little bit like electricity.


You can see right here the torch emits a signal and it is transmitted across these wires in red right here.


Now the property of redstone is that it starts out with a signal strength of 15 as you can see indicated by these lights.


And for each distance that it travels it drops by one signal strength.


Now most people simply use the on or off state of these wires as binary signals and build computer out of that.


However I decided I wanted to use the signal strength directly as a signal and build a neural network based on that.


This gives us a much more compact neural network and it is much more akin to how we build neural networks in machine learning and also in the brain.


Next I'm going to show you the main components that we use to build this neural network.


This here is a lectern and the building block right behind it is called a comparator.


Now the comparator has the ability to read signal from blocks before it.


In this case it reads the page of the book that is on the lectern here 9 and translates that into a redstone signal.


You can see the redstone signal is 9 strong at the beginning and decays with each distance traveled.


Perators are actually a special block in redstone in that they can transmit a signal without it losing its strength over distance.


In this demonstration you can see the difference between a comparator and what is known as a repeater.


The comparator simply transmits the signal one block and keeps its strength while the repeater will fully power up the signal back up to 15 no matter what signal comes in.


Only when a signal of zero comes in is the repeater fully off.


Another interesting fact about comparators is the fact that they can be used for doing math.


In particular they can do subtraction.


Here we subtract the side signal from the main signal which results in a resulting signal of strength 2.


Note that this comparator is in subtraction mode because its front light lights up.


This neat thing right here is a divider.


It divides the signal by 4 which is pretty cool.


Since redstone signal is capped at zero at the lower end and 15 at the higher end we don't really have a lot to work with.


Dividing by 4 is often useful to bring the signal back to a manageable range.


So this will bring the signal from 0 to 15 to a range of 0 to 3 or 1 to 4 however we want it.


The most important building block in a neural network is going to be what's known as a memory cell.


This is a memory cell.


It consists of two comparators each feeding into a block and each block powering a cable that then feds into the comparator again.


This is a closed loop and it will save any state that you give it.


I can fully charge it with this button and I can fully de-charge it with this button.


A slight variation on the memory cell is the decaying memory cell which I think is pretty cool.


It is almost like a memory cell but since this wire here is of length 2 it de-charges by 1 every time the signal goes around the cycle.


So if I fully charge it what you're going to see is that it slowly decays over time.


Let me show that again.


This is pretty cool.


This is a multiplier.


It is a device that can multiply two analog signals and it is really cool how that works.


It combines the memory cell and the decaying memory cell to achieve this multiplication.


Again the multiplication is in analog here and not in binary.


The design is from a YouTube channel called RKFVOLTER and I didn't come up with this myself and it took me quite a while to understand what was going on.


Though once I had it I was able to build the rest of the neural network almost without a problem.


At the bottom you'll find a single memory cell that stores 15 minus whatever we want as an output.


The signal is then fed into this comparator which is in subtraction mode and feeds from this hopper that is full.


So the output is going to be here.


On top of the memory cell you'll find a decaying memory cell.


The decaying memory cell powers this piston here and it is fed via an ultra short tick of this piston with this signal.


This is one of our two input signals.


As long as the decaying memory cell is active this piston stays down.


As long as this piston is down our second input is fed through this circuit into the memory cell at the bottom and is subtracted.


That means the bottom signal is subtracted from this memory cell an amount of times that is proportional to how long the piston stays down.


This as you can see results in a multiplication of the two analog signals.


Pretty cool.


Here I use this to multiply the two numbers 2 and 3 as you can see by the pages of the book.


As soon as I hit the button the memory cell is reset an ultra short pulse is generated and this piston stays down just long enough for the discharge to happen an appropriate amount of times.


You can see the result is 6 and if I change this to a larger number say 5 you can see that the piston now stays down for much longer than before.


Of course we can only handle signals up to 15 even with this contraction.


The last thing we need is gradient descent.


By combining a multiplier and a memory cell together with two pistons that update the memory cell we can achieve gradient descent.


This here was my test application for gradient descent.


It is a square root finder and to my knowledge it is also the first analog square root finder that is implemented in Minecraft Redstone.


Innovation happening on this channel every day.


So the way it works is that we have a memory cell that we can update using either this piston or this piston.


We can update it up or down.


We feed the signal from the memory cell as the first and the second multiplicand into the multiplier.


The two numbers are then multiplied together and come out here.


On this lectern we set a target that we would like to know the square root of.


In this case I want to know the square root of the number 9.


This circuit right here then calculates an error signal and tells the contraction down here whether we need to go up or down with our memory cell.


Depending on that either this piston or this piston is activated with an ultra short pulse and we change the memory cell by 1 or negative 1.


If we repeat this cycle eventually we should converge to the square root of whatever we input into this lectern.


So if I hit the button right here.


The square is calculated, the error is calculated, the memory cell is updated and you can see 1 is our first guess.


Let's hit the button again and see what happens.


We are at 2.


Now we are at 3. If we hit the button again we do expect the network to converge.


You can see there was no more update so now we have converged on 3.


Which is of course as you know the square root of 9.


If we input any other number than a pure square the network is going to oscillate between the two square roots that are closest in integer.


So here 2 and now it oscillates back to 3.


Gradient descent in Minecraft. Thank you.


The neural network is a bit more complicated in that it can not only do gradient descent by plus 1 or negative 1.


It will actually calculate the exact error signal that comes back from the front.


It will calculate it through the non-linearity and it even has adjustable learning rates.


Alright now let's try it out.


So in this neural network what you do is you use these two books to set the input signals for each of the two input dimensions.


In this case it's 1 and 3.


And you use this book to set the target value. In this case I've set it to 12. That's a bit high. Let's set that to 6.


Once I hit this button the whole operation starts in full automatic mode. Let's go.


So what you're going to see is the signal forward traveling through the network. Through the first layer into the second layer which you're going to see right now.


After that the output is going to be displayed after a short flicker on this pole right here.


Now this happens to be exactly correct. It's not always the case.


After this the network flips into back prop mode at which point the signal is traveling backward through the second layer to the first layer.


At the end this piston there is going to hit which is going to implement the weight update given by these upper pistons right now.


And after all of that the control signal travels back and we start again.


Let me show you a little bit more clearly what happens in each step.


The neural network we're going to build here has two input neurons which can be loaded with a value of anywhere between 1 to 15.


This is followed by another layer of neurons. Two neurons form the hidden layer of the network and yet another layer one neuron forms the output.


Each layer is a fully connected layer which means that every neuron in the layer before is connected to every neuron in the layer above.


And the same goes for the second layer. Each of these layers has a weight associated with it.


The back propagation formulas tell us how the signal flows forward in the network and also how the signal flows backward.


While the optimizer formula is telling us how we need to update the weight once we have computed the back propagation signal.


All of this is going to be implemented in Redstone.


Here you see an overhead diagram of the neural network in Minecraft.


I've removed the top layers of the weights and the weight update mechanisms otherwise you can see anything.


The basic components of each of the weights are implemented in the multipliers you can see right here.


Four weights, four multipliers.


Each multiplier is followed by a division by four which is a square thing right here.


You can also clearly see the two hidden neurons here and here where the non-linearity happens.


And the two weights in the second layer are also implemented by these two multipliers.


The output neuron is implemented at the back together with the output signal.


For the back propagation we have the two additional multipliers here and here to calculate the backprop signal to the first layer.


On the bottom you can see the timing signal to set the network into backprop mode.


The first thing that happens is this first row of multipliers. There are four multipliers here as you can see.


There's one, there's two, there's three and there's four.


The four multipliers represent the four connections from the input layer to the hidden layer.


Since each of the two input neurons needs to be connected to each of the two hidden neurons.


The connections have the multiplier to do the actual multiplication and the weight of the connection is stored in a memory cell above which you can see right here.


This memory cell probably has a weight of about eight right now.


Each memory cell is also accompanied by two pistons, one to add to it and one to subtract from it.


Note that other than in the square root finder here we don't just add and subtract one statically but we actually compute the exact backprop signal that we need to add or subtract.


Though I have implemented a limiting mechanism for the update which you can set in these books right here.


In this case I've set it to two for this weight to not have it update too rapidly.


You'll also notice that each of these update pistons is accompanied by another piston mechanism.


This is for generating an ultra short pulse which is necessary for us not to update too much.


You'll be able to see the ultra short pulse in just a second.


Watch the repeater as the piston moves up again.


Did you see that? Ultra short pulse.


I think it's known as a two tick or a three tick pulse.


As a one tick pulse will actually have that piston expel its block and not retract it again.


So after the first row of multipliers each signal goes through a circuit like this where it is divided by four.


This is done because again we work in the range of 0 to 15 which is not a whole lot and we've already multiplied two numbers.


So dividing the signal by four seems like a reasonable choice.


After we divide the signal by four it goes into the non-linearity.


Here conveniently labeled with a sign unlike almost everything else in the entire network.


The non-linearity is a relu non-linearity though it is not set at zero to cut off it is set at four.


We don't have negative signals in this game so we'll have to work with what we get.


One thing I implemented is that I do add one to whatever comes out of the non-linearity to never have a zero signal


and therefore never have a zero gradient for the later weights.


Feel free to change that though I have no clue if it works.


Following the two non-linearities the second row of weights is coming.


There's just two weights here since there's just one output neuron.


There is one multiplier and there is one multiplier.


Again the weights are implemented by memory cells above with update mechanisms to add and subtract


prepended by ultra short pulse generators.


And again you can adjust the learning rate using these lecterns.


Once the output arrives it is stored in this memory cell right here and displayed in the column of lights.


Now that's where the interesting part only begins.


The target value comes in through this current right here and is compared to the output value of the network.


Here's where we calculate the error.


We need to calculate it once into the positive direction and once into the negative direction


and we need to remember whether or not our signal was too high or too low.


Two control lines signal for this.


One goes underneath here which is the negative line and one goes over top there which is the positive line.


Once the error is calculated the network switches into back prop mode.


Back prop mode is controlled by a timer mechanism which is composed of multiple stacked decaying memory cells.


You'll see that this generates a really long pulse which controls for how long the network is in back prop mode.


You can see it decaying very slowly one cell after the other.


Once all cells are decayed the network is switched back into forward prop mode.


Now what happens in this back prop mode?


In back prop mode two things happen.


First of all the network is configured to switch the multipliers here.


To instead of doing forward propagation do back propagation.


The back prop formula tells us that we have to multiply the error signal with the input signal to get the weight updates.


Rather than implement separate multipliers for this multiplication I decided to implement a routing mechanism


that simply detects whether or not the network is in forward or in back prop mode and uses the appropriate inputs into the same multipliers.


The result of the multipliers is then used as an update signal for the weights.


In order to do back propagation through a neural network you also need to back propagate the error signal back to the first layer.


For that we need two extra multipliers which I've implemented one here.


This multiplier implements the back prop signal for the lower layer including the gradient of the nonlinearity


and the division by four that we did in the forward propagation.


It's important but once we're done this really gives us the exact back prop signal for the first layer.


And again we reuse the multipliers in the first layer and reroute the inputs to calculate the update signal during the back prop phase.


Once back prop is done a simple control signal instructs all the weights to update at once.


You'll see it when this piston goes up.


And the control signal instructs all the piston in the top layers to fire and update the weights.


And that's it. That is one cycle through the network.


Now by mere accident we have actually hit the correct output from the get-go and thus nothing is updated.


Let's try to overfit to one data point once more.


So I've now switched the inputs to three and one and I'm going to set my target to 12.


Let's see what happens and follow along once more.


The input goes through, the first row of multiplier hits, the signal travels backwards, the second row of multipliers hit.


After that the output is displayed.


It is six right now still but that's going to change.


The network is switching into back prop mode indicated by the flashing up there.


You can see the multipliers in the second row hit, after which the multipliers in the first row hit.


And now the weights are instructed to update.


Up top. There we go. Good job.


Once that's done the control signal travels back and we go again.


First row of multipliers, travel back, second row of multipliers.


The signal is stored in this memory cell and displayed right there. We're at nine.


Network is flipped into back prop mode.


These multipliers hit including the multiplier for the back prop signal.


First row of multipliers hit and the weights are instructed to update.


Weight update. There we go. Good job. Let's try that one more time.


Forward prop first row. Forward prop second row.


Output is saved and displayed.


Beautiful. And that is an output of 12 for you.


This was certainly a challenge. It started as an April Fools joke.


And it turned out to be a lot of work but also fun.


And the livestream chat while I was building it was certainly super helpful and fun to watch.


I kind of knew how to do the forward propagation once I had the multiplier figured out.


But other than that I had no idea what I was doing.


So I will put these worlds on Github for you to mess around with.


And you can submit a pull request if you think you have a substantial improvement.


Or maybe you'll even find a bug. It's quite probable honestly.


So in conclusion we used a bunch of weird mechanics of Minecraft to build the first analog forward propagating,


back propagating, weight updating, gradient dissenting, non linearitizing, deep neural network in Minecraft.


[Classic] ImageNet Classification with Deep Convolutional Neural Networks (Paper Explained)

Hi there. Today we'll look at ImageNet classification with deep convolutional neural networks by


Alex Krzyzewski, Ilya Satskiwer and Jeffrey E. Hinton. So this paper is another one in


the installment of our historical paper overview, where we go through kind of old papers that


were or weren't very impactful, and see what people knew at the time already, how this


worked, and so on. Of course, this paper here, also known as AlexNet, was the one that started


the deep learning revolution, so to say, or at least contributed in large part to it.


It was the first paper that showed that you could train these very deep neural networks


and very deep in here is a relative term. But the first one that showed that you could


actually use CUDA GPUs to train those large networks efficiently, and it won the ImageNet


competition that year. And it did so by a very, very large margin. So kind of shook


the world. Because previously, computer vision was still doing like hand engineered features,


and then using some kind of classifiers on top of those, this paper basically changed


everything. So we'll go through the paper, and we'll see what was already known. And


especially, I always enjoy with these papers, how did the choices that people make back


then, how did they pull through to today, sort of what arbitrary choices that Alex Krzyzewski


made right here? Are we still doing today? And what have we learned since then? So the


paper is written relatively straightforward, I have to say, it's a good read, if you want


to read it, and you know, straightforward, and sort of gives you a little bit of an intuition


of how much work must have gone into this, which is, I guess, a lot. And yeah, so they


start off by saying that, that current approaches to object recognition make essential use of


machine learning methods. This was also new, right? Object recognition wasn't always learned


the, you know, object recognizers, you could even do it in the in different way, like matching


templates and so on. Machine learning was still one of the methods used. And of course,


today, it's the method used. To improve their performance, we can collect larger data sets,


learn more powerful models, and use better techniques for preventing overfitting. Until


recently, data sets of labeled images were relatively small on the orders of 10s of 1000s


of images. Okay, so this this specially at NORD, or here, the C410 or C4100. These are,


you know, relatively small data sets with relatively small images as well, like C410


is 32 by 32 pixels. So they're saying that, okay, when these small data sets, you know,


you can you can solve it with classical computer vision models. But if you have larger data


sets, and especially more realistic data sets, like bigger resolution, and so on, you need


bigger models. So they say, but objects in realistic settings, exhibit considerable variability


to learn to recognize them, it is necessary to use much larger training sets. Okay, so


they say that this ImageNet data set is one of those larger data sets consists of 15 million


labeled high resolution images in over 22,000 categories. And people keep forgetting this.


And I am included in that group of people, that the ImageNet data set is actually much


larger than we know, than we when we talk of ImageNet. When we speak of ImageNet, we


think of the ImageNet that has 1000 classes and about one or one and a half million images.


However, that's only a subset of the much, much larger ImageNet data set with in many,


many more categories. It's just that the ImageNet competitions were performed on this subset,


because I guess people thought, well, 1000 classes and million images is already plenty.


So we'll do that. So that's, I guess, how that came to be. So their argument is right


here. To learn about 1000s of objects from millions of images, we need a model with a


large learning capacity. However, the immense complexity of object recognition tasks means


that this problem, this problem cannot be specified even by data set as large as ImageNet.


So our model should also have lots of prior knowledge to compensate for all the data we


don't have. So their main argument for using neural networks is that the size of the data


set is so large, therefore, we need a large model, you know, granted, they already recognize


the inherent, the inherent connection between large models and a lot of complex data. But


in the opposite, they say, well, even if we have that much data, the task we are trying


to solve object recognition is way more complicated than the amount of data we have. So our model


should also have lots of prior knowledge to compensate for all the data we don't have.


Remember at this time, convolutional neural networks weren't really, you know, known to


do anything, I guess, I guess they were used for handwritten digit recognition, and so


on and we're kind of on par with other methods. However, it wasn't like, obviously clear that


you would use them for image recognition. So here they make they have to make like a


argument to convince people that, okay, we can use neural networks for this tasks because


they have such a high capacity. However, neural networks, feed forward neural networks are


already too powerful. They don't know anything about the data, everything's connected to


everything. And they argue right here, our model should have lots of prior knowledge


to compensate for all the data we don't have. So they allude to the convolutional neural


networks constitute one such class of models, their capacity can be controlled by varying


the depth and breadth. And they also make strong and mostly correct assumptions about


the nature of images, namely stationarity of statistics and locality of pixel dependencies.


So that their argument here is that the convolutional operation is such a strong prior, that is


mostly consistent with what we know about images that they are very well suited to computer


vision. Again, something that was not abundantly clear at the time as it is right now. It's


interesting to see how they get to this point where they say we need lots of capacity, but


we also need a model with lots of prior knowledge and and of course, CNN's fit that very well.


So they're they go into the problems of CNN, despite the attractive qualities, and despite


the relative efficiency of their local architecture, they are prohibitively expensive to apply


in large scale high resolution images. Luckily, current GPUs paired with a highly optimized


implementation of 2d convolution are powerful enough to facilitate the training of interestingly


large CNNs. And recent data sets such as ImageNet contain enough labeled example to train such


model without severe overfitting. So overfitting was also still like very much at the forefront


of people's minds back then. Right now, we don't really care about overfitting that much


anymore. Basically, we figured out that if we just build large enough models, we we don't


overfit, which is strange in itself, like this double descent phenomenon and so on.


But overfitting was still a a very much at the forefront of people's minds. And they


do a lot of things here to prevent overfitting, which gives them kind of a boost in the test


accuracy, which might actually not have been the overfitting that they're combating. So


they do, for example, in data augmentation already in this paper, and they always allude


to how this is to prevent overfitting. However, we know nowadays that it might not be the


overfitting that's combated in the by data augmentation, it might actually be more have


something to do with like regularizing your function, making it more smooth and so on.


But so you just see how how coming from a classical machine learning perspective, overfitting


was like the number one or one of the number one problems in classical machine learning,


you know, in SVMs and things like this. So it's it's, it's safe to say that they thought


if we built these large models, we're going to have a huge overfitting problem. And yeah,


so that's why this pulls through right here. Also, the I guess the one of the main contributions


of this papers is to show to combine this CNN training with GPUs. Also not very non clear


at the time, like it was known that you could do computation on GPUs. But the fact that


these are, you know, very capable for training these CNNs, or generally neural networks wasn't


something that was, you know, known at the time. So this paper basically showed that


if you use a GPU, you can, you can get that much faster. And that makes it possible to


train these big neural networks. Again, right here, the size of our network made overfitting


a significant problem even with 1.2 million labeled training examples. So we use several


effective techniques for preventing overfitting. And we'll, we'll look at those. And the end


they say, the network's size is limited mainly by the amount of memory available on current


GPUs and by the amount of training time that we're willing to tolerate. Our network takes


between five and six days to train on two GTX 580 GPUs. All of our experiments suggest


that our results can be improved by simply waiting for faster GPUs and bigger data sets


to become available. And I mean, that proved to be absolutely true. We don't necessarily


have bigger data sets right now, though we do. But certainly with faster GPUs and bigger


GPUs, this became a this became the these networks became better simply by increasing


their depth. And as you know, then Resnets came along, increasing the depth by an order


of magnitude, and that gave another boost to computer vision. Alright, so they talk


about the ImageNet data set here. And the main point in the ImageNet data set right


here is the fact that the images are plenty. So there are over a million training images


in this subset with 1000 classes, which was, you know, a very big that was that was unlike


CIFAR 10 had 10 classes, CIFAR 100 had 100 classes, that was already a lot. A thousand


classes that is like unheard of before this data set, I guess not unheard of, but yeah,


and a million training images, completely crazy. And also, not only was it a lot of


images, they were resolution was really big. So in the order of 256 by 256, whereas previous


methods all were like 32 by 32. So definitely challenging data set. Even today, it's a challenging


data set. Alright, so the architecture, the architecture, and there's this famous graphic


right here of the Alex net architecture. So briefly, they describe these convolutional


layers right here, as you can see, there's max pooling already here, they have dense


layers at the end, they do generally increase the number of feature maps right here, while


decreasing the resolution with max pooling. So all of this has sort of, you know, kept


until today, I guess they also took it from earlier work on convolutional neural networks


that generally found this to be a good idea. And the important part here, that is kind


of special to Alex net is you can see there's are these two different pipelines and, you


know, Alex, for cutting off this part right here, I mean, you just know, like, there's


the eight pages, we need to like we have like three lines too much, how can we fit the three


lines, we've already cropped everything. Let's just cut off the top half here. It's essentially


the same as the bottom. Yeah, so space constraints and PDFs for conference submissions,


ruining yet another paper. Alright, but you can see there's this two this this two column


architecture right here. So this network was so large that it didn't fit on one GPU. So


they had to split it on two GPUs with the occasional intercommunication, right, you


can see here, there's intercommunication between the two GPUs. And there is also no intercommunication


right here on this layer. This was very intricate. That was one thing that really didn't hold


until today, I guess until now with things like, I don't know, G shard or so, where you


have different weights on different GPUs. Again, I guess the the invention of bigger


GPUs made that sort of superfluous. But just imagine the amount of code they had to write,


there was no TensorFlow at this point there. I don't think there was even cafe around there


was just CUDA. And yeah, just this cross GPU memory writing, I just imagined this to be


so so ugly, and big respect for writing all of this, all of this code. Alright, so the


they go through a number of important things. And most of the things here aren't their invention,


I would say, but they cleverly combine things that were already known about neural networks


and things that were maybe developed somewhere that they have found to work really well.


So the first one is the relu non linearity. Now, of course, relu is nowadays all like


abundant. Everyone uses relu nonlinearities. But at that time, it was still very much in


fashion to use something like the sigmoid right here or the hyperbolic tangent. And


why is that because the neural networks were still inspired by the neurons where you had


the soma of the neuron and then the input dendrites, sorry, dendrites with the input


axons, and then you would sum up all the incoming signals. And then that would go over. So in


a true neuron, you have this, this, this kind of curve where if the input rises above this


order right here, the the action potential, maybe, I don't know what the the English term


is, then if it rise above that, then the neuron would start to spike, right? And if it's below


that it wouldn't. So people wanted to approximate this using some sort of a, a kind of differentiable,


but something that's very similar to this step function. And that ultimately led to


something like a sigmoid or an hyperbolic tangent. So people trying to stay close to


biological neurons did this, but that gives you the problem that in this region, and in


this region right here, you have almost no gradient to learn from. So you can see that


they argue that in terms of training time with gradient descent, the saturating nonlinearities,


so the hyperbolic tangent and the sigmoid are much slower than the non saturating linear


non linearity. This one following Naren Tinton, we refer to neurons with this non linearity


as rectified linear units. So taken from this, this other paper, they say, okay, we use these


relu's, these rectified linear units, which are not exactly like real biological neurons,


but they train much faster, right. And of course, relu's are used until this day. So


you can see right here that a this is on a CIFAR 10. And they measure the time to reach


25% of the training error. And this here is with the relu's. And this here is with the


hyperbolic tangent. And it takes much longer to reach the hyperbolic tangent, especially


it takes six times faster to with the relu's. And they say that's one of the main components


that allows them to learn this fast to even experiment with these big networks, because


their entire training time is six days, right. But they probably didn't train it only once


they experimented with it and saw what works. So if you have a couple of months of time


and it takes you a week to train one of these things, you know, you don't, you can't afford


a six times slowdown. Because that would mean you can only train like two models in the


entire course of research. And that would severely hinder your progress. Now we are


at the point where that becomes true again with these giant, giant transformer language


models where people can train it once and then you know, like GPT-3 they say, oh, we


made we discovered a bug halfway through and we've kind of fixed it but we're not sure


we couldn't restart because it was too expensive. Maybe we're waiting for a moment. I'm still


saying we're waiting for the resonant moment in the transformers. But yeah, relu's in,


you know, here in not introduced here, but used here and have been prevailing until today.


Training on multiple GPUs, something as I said that didn't didn't really get forward


from here, especially the kind of GPU train. So if we train on multiple GPUs today, what


we mean is that we have our model, right, and then we distribute that to multiple GPUs


like this. And then we take a mini batch from the training data. And we simply split it


up let each GPU do its thing on its subset of the mini batch. And then at the end, kind


of calculate the loss and then back propagate the gradients and synchronize the gradients


between that. So we have one model that is on both GPUs. Here they distribute a model


to two GPUs. And I'm also thinking that with frameworks like G shard, this could potentially


have a revival right here, this kind of distributing your models, especially within the same layer


across many GPUs and then having cross communication only at some points. So their argument is


this only has three gigabytes of memory, which limits the maximum size of networks can be


trained on it. Turns out that 1.2 million training samples are enough to train networks


which are too big to fit on one GPU. Therefore, we spread the net across two GPUs. Current


GPUs are particularly well suited to cross GPU parallelization, as they're able to read


from and write to one another's memory directly without going through the host machine. Okay,


so this means that for so sorry, here, they say the parallelization scheme that we employ


essentially puts half the kernels or neurons on each GPU with one additional trick. The


GPUs communicate only in certain layers. That means that for example, the kernels of layer


three take input from all kernel maps and layer two however the kernels in layer four


take input only from the kernel maps in layer three which reside on the same GPU. So very,


very interesting choice right here. And they they justify this here or they say the results.


This scheme reduces our top one top five error rates by 1.7 to 1.2% respectively, as compared


with a net with half as many kernels in each computational layer in each convolutional


layer on one GPU. The two GPU net takes slightly less time to train than the one GPU net. So


first of all, I have to say big respect right here like, like, I can imagine they did this,


you know, with the relu's and stuff and they were already better than previous because


they're so just to go to the results, the pre they beat the error rates of previous


models by G enormous amount. So this is what they knew right here. This is on the 2010


image net split. So the previous best ones were like at around 28 25%. And here their


best one is at 17% top five error rate, I'm going to imagine that they trained it first


and we're already better than the 25%. And I guess lots of people would just call it


a day would be like, Oh, cool, we have this entirely new method. Not only did we show


that we can train it, we actually show that it's better. And bada boom, I have point 1%


error rate, and everything else can be a separate paper. No, they stuck with it and they pushed


it each. So each of these things right here, they say, Oh, this reduces the error rate


by 1%. This reduces the error rate by 2%. And, you know, really, they they went about


it, how far can we push this with everything? I mean, just imagine you come and you train


a network, I'm pretty sure they first trained on one GPU, right? And, um, and then they


thought, Oh, you know, maybe we can train an even bigger network by using two GPUs.


And then they realized what it's gonna take like a crap ton amount of dumb code to cross


synchronize and keep them in lockstep and blah, blah, blah, like, it's not even easy


to write multi GPU code today with all the frameworks, just imagine that. And for them


to having already observed that their network does better than everything that was previously


to sit down and do the cross GPU thing, experiment with Okay, when do we cross communicate and


whatnot, that is very, very respectable, right here. So maybe a lesson to be learned, or,


or just the mentality of the people, maybe they just had more time, they were like, Okay,


it's still like two months out, just competition deadline. I don't know. But no, I'm this,


this is not something that I see today, very often, this this kind of persistence and additional


pushing and reporting of what works in these kinds of things. I mean, some, some papers


do it, but most papers do it because only with all the tricks, they can get that point


1% improvement. And this one already had the improvement and did it anyway. Okay, but multi


GPU training didn't really like splitting the models across GPUs didn't really didn't


really stick around, mainly because I guess the GPUs got larger in memory pretty quickly.


So it wasn't that necessary. But also, I guess, because the frameworks were just too clunky.


And now maybe witchy shard, this is coming back. So worth another shot, I guess. Next


one, local response normalization, this also didn't really stick around, I got kind of


dumped in favor of things like batch normalization. But with the resurfacing of things like layer


normalization, this, it comes back to this thing here again, a little bit. So what they


say is that what they want to do is they want to kind of normalize the response of these


of these relu's. So what they do is each response, which is this alpha, they are these a here


is normalized by the following quantity. And it's the all the responses of the other neurons


around them or of the other kernels around them. And you can see the sum is over this


weird quantity right here. So what does it mean if they have a bunch of convolutional


filters? And these are the activation. So these are the feature maps after the convolution.


And yeah, so if I have like 10 convolutional filters in my layer, this is going to be the


output, the way they normalize is they normalize each filter, sorry, each output channel by


averaging by see here, dividing by the average response of the channels around them, right.


So let's maybe say the five channels, the two channels in front of them and two channels


behind them, this is going to be they take the average across this one. And then for


another channel right here, for this one, you would take the average of the five around


that this isn't really something that stuck around, I guess, mainly because of the really


dynamic situation right here. What people do today is they have things like layer normalization


that simply averages across all of the channels, or they have group normalization that pre


defines these groups like here is there's two groups. And we only normalize within this


group and within this group, also always the same, this kind of dynamic normalization on


across neighboring filters, as I said, didn't really stick around, not really sure why,


but I guess it was just easier to implement it otherwise, or it just worked better. Again,


here they say, this, this, it was motivated well, right, this scheme bears some resemblance


to the local contrast normalization scheme of that, but ours would be more correctly


termed brightness normalization, since we do not subtract the mean activity. Oh, they


make it connection to biological neurons. Where is it? This sort of response normalization


implements a form of lateral inhibition, inspired by type found in real neurons creating competition


for big activities amongst neuron outputs computed using different kernels. Okay, so


kind of inspired by real neurons, but also kind of inspired by other people doing also


some kind of normalization. So people already knew that normalization was helpful at some


times. And this is what they employed right here. Again, reducing the top error rates


by 1.4 and 1.2%, respectively. So not a big improvement, but still an improvement. The


last thing overlapping pooling, again, a thing that didn't really stick around that much,


where they say, okay, instead of having a pooling layer, so if this is your image, and


instead of pooling two by two in the stride of two, like we do today, and you know, pull


it down to a smaller image, what we can do instead, is we can pull with overlapping windows.


So in that case, they pull with a three by three window, but they do always do stride


of two. So they have like these overlaps right here, resulting in the same size. But then


each pixel right here has some sort of overlapping information from the pixels around it. Again,


they say it reduces the top one and top five error rates by 0.4% and 0.3%. Maybe this this


didn't stick around because I'm not sure maybe because people found it doesn't work in other


problems. Who knows? So the overall architecture, as we said, is described in this picture right


here. So you have the input image, which you can see has three channels. And they use convolutional


filters with a here with a stride of four at the beginning to reduce the size. So at


the beginning, it's 224 by 224. And then it's 48 by sorry, it's 55 by 55. That thing here


55 by 5548 feature maps, you can already see, as we said before, the feature maps keep increasing,


while the number of the dimension, the resolution of the image keeps decreasing. The stride


of four convolution here already employed in order to downsample the image at the same


time as convolving it. Nowadays, a lot of architectures will simply not do max pooling


at all, but always use the kind of strided convolution to downsample image while convolving


it. What you also see here is that they thought that the feature map size should should also


be large at the beginning, and then decrease, which is a reasonable assumption, right? Because


if you have higher resolution images, you're probably going to need higher resolution feature


maps. This didn't really come through until today. As you know, most architectures today,


they just go with like three by three kernels from the very start, and don't really care


about you know, also downsizing their, their filters. I don't really know why, whether


it's just more convenient or less parameters, or whether there's really something to having


small filters, but I just know, you know, this is something the large filters at the


beginning is something that didn't didn't hold over time. Also, you can see right here,


they have multiple dense layers at the end, I believe most architectures today simply


go with two of those instead of three. So one like hidden layer, and then one classification


layer. But it's, you know, it's, it's very close to the architectures today, right? There


hasn't changed that much like the difference between this and the VGG 16 VGG 19 network


is just depth. And then the difference between those and the resnet is just the whatever


the skip connections right here. And that's where we are today. So so there hasn't, hasn't


changed that much. Honestly, they also allude to the fact that actually, even though it


doesn't look like it, most parameters are here in these dense layers. Those are most


parameters of the network. This right here, a convolutional layer is like 1% of the parameters,


even though it takes up a lot of space in the in the drawing. So maybe the reduction


in the number of classification layers at the end also has something to do with the


fact that that's where most parameters are. So if you get rid of one of those dense layers,


you can like get many, many more convolutional layers. Alright, so the last part here is


on reducing overfitting again, they didn't really investigate whether or not really their


network was overfitting, like really establishing the overfitting it was I think maybe they


did and maybe it was actually overfitting. But we now we we don't care about overfitting


too much anymore. Maybe because we already use these augmentations naturally, but also


because we built these deep models. So we somehow have an idea that they generalize


naturally, I'm not sure whether they actually were only worried about it that much because


of the history of machine learning, or whether they actually did see that everything was


overfitting constantly. Okay, they say our neural network architecture has 60 million


parameters. Although the thousand classes make each training example impose 10 bits


of constraints on the mapping from image to label, this turns out to be insufficient to


learn many parameters without considerable overfitting. Below we describe two primary


ways in which we combat overfitting. Again, there's no one today, no one today makes this


argument anymore. This Oh, we have this many parameters and there are that many images,


right? We have 60 million parameters. We have 1.2 million images, 1000 classes, how you


know when, how many parameters per sample is that and so on, how many bits of constraint


we don't care about. We're fine with having like a billion times more parameters than


training samples. We we don't worry about it anymore. So the first thing they do is


data augmentation already. I mean, this was already known again, like lots of these things


here were already known, but the the combination is just so cool in this paper where so first


of all, again, they say, the transformed images are generating Python code on the CPU, while


the GPU is training on the previous batch of images. So these data augmentation schemes


are in effect computationally free. Again, this code must have been ugly. The first form


of data augmentation consists of generating image translations and horizontal reflections.


We do this by extracting random 224 by 224 patches and their horizontal reflections from


the 256 by 256 images. Okay, so random. So this was already this, these are the most


valuable data augmentations that still we have today. Random horizontal flipping is


still used in every pipeline of computer vision, except if you want to read text, I guess.


And random cropping is still the most powerful data augmentation technique for images today.


And the it's crazy that this was already discovered. And I, I don't know whether they say right


here how much this particular thing improves. I don't think they have a stat on how much


this improves, they just say how much this this next thing improves. But I'm going to


guess this was one of the vital things for pushing the performance because now we know


cropping is very important. I guess they thought that they would, you know, translation was


the important part. And so they focused on generating image translations. And to generate


an image translation from a single image, naturally, you have to crop it. However, we


we now focus much more on the fact that we crop it and kind of have different sub images


of the same image, especially in, you know, self supervised learning, and things like


this, we know that cropping is what is like the the power horse of these methods. So the


fact that they extract random patches right here means that their network only operates


on these sub patches. And then they compensate by a test time, the networks makes a prediction


by extracting five patches, the four corner patches and the center patch as well as their


horizontal reflections and averaging the prediction made by the network's softmax layer on the


10 patches. I also believe that people don't do this too much. Nowadays, they most of the


time they simply rescale the test images or something like this or fine tune at the end


on the kind of scale training images, there are various techniques for doing this, but


random cropping and horizontal flipping already employed right here. Also color, kind of color


jittering, a form of color jittering, a very special form, altering the intensities of


RGB channels and training images. Specifically, we perform PCA on the set of RGB pixel values


throughout the ImageNet training set. To each training image, we add multiples of the found


principal components with magnitudes proportional to the corresponding eigenvalues times a random


variable drawn from a Gaussian with zero mean and standard deviation point one. This is


I believe this has gone out of fashion. So people do color jitter and kind of brightness


jitter and so on. But I don't think they particularly do this kind of PCA based image, image augmentation


right here anymore. They say this scheme reduces the top one error rate by over 1%. I wonder


why this isn't working, maybe because you need these stats over the entire data set


and the other things may be working equivalently well, but you can simply apply them without


knowing kind of your principal components. Okay, next thing, dropout. Dropout has been,


you know, one of the things that was very important throughout the early stages of deep


learning isn't that important anymore. Now, dropout, some people still use it, but most


people I think don't use dropout anymore. And it's very interesting to see, but it definitely


was a technique that was used a lot during like from Alex net to basically like now,


like the last very few years. So they say, combining the predictions of many different


models is a very successful way to reduce test errors, but it appears to be too expensive


or big neural networks that already take several days to train. There is however, a very efficient


version of model combination that only costs about a factor of two during training. So


there's this technique called dropout, then they explain it to set to zero the output


of each hidden neuron with probability point five. Again, people didn't know about dropout


as they do now. But they introduced this right here. And they say it reduces their not sure


that they also don't say how they how much they by how much this reduces the training


error, but they say we use dropout in the first two fully connected layers. Without


dropout, our network exhibits substantial overfitting dropout roughly doubles the number


of iterations required to converge. So okay, so they did actually make sure or they did


find the actual evidence of overfitting and saw that dropout reduces that. And I wonder


why this doesn't happen nowadays, maybe because we have the we have less of these fully connected


layers, but I can't really imagine. Maybe because we do more augmentation. I don't I


don't know or maybe dropout is still used and I'm just I just don't know it and don't


see it. Yeah, so here they use momentum to train this. And they do some qualitative analysis.


They do some qualitative analysis. So first of all, they say, okay, they shatter all of


the previous approaches, especially also then they build kind of ensemble methods. And they


pre train they already do transfer learning, they already pre train on ImageNet 2011. And


fine tune then on the ImageNet 2012 right here, the ImageNet 2011, da da da da da, and


then fine tuning on the ImageNet 2012 to reduce that error even further, like pulling all


the tricks, all these things are around still. Very cool. And then they look into what their


network learned. So they find that there are a number of these kind of filters. You see


these 11 by 11 filters in the first layer, where they show okay, this really and this


was kind of already known that these neural networks extract filters like this, like color


gradients or edge detectors in various forms and directions. And cool to see that this


one also does so. This one here is also a very cool investigation where they look at


examples. And the red bar, the red one is always the correct label. And the bars are


basically what their model says are the top five things. And it's cool to look at. So


for example, here you have might as the top one, but then also Black Widow, cockroach,


tick, starfish, but the top labels are usually also very, very good labels. You can see here


grill and it assigns convertible, which you know, by all means is correct. It's just not


the class that the annotators assigned to this particular image, as well as here. Dalmatian


was the highest prediction of the network where the label was actually cherry. And this


is this is quite debatable, right? So you can see that a lot of the mistakes the network


does is are, are, you know, forgivable, let's say. And you can see that for when the network


doesn't do mistakes, the not only the top label is good, but a lot of the top five labels


are also very, very adequate. Lastly, they look at a given training set image, which


these are the training set images right here. And they look at the last layers feature vector


and the five nearest or the nearest neighbors in Euclidean space of the entire training


data set. And here's what you come up with. So you can see for the elephant, the nearest


neighbors are all other elephants and regard that they are in different poses, right? They


don't always look the same way these elephants also these dogs right here. So it's pretty


cool to see that the network actually learns some variances across the class and puts images


with the same label into the same area in the embedding space. Yeah, so that's their


that's their paper. They they they already allude to the fact that depth is very important.


It is notable that our network's performance degrades if a single convolutional layer is


removed. For example, removing any of the middle layers results in a loss of about 2%


for the top one performance of the network. So the depth really is important for achieving


our results. And as you know, this spurred an area of this burden area of trying to build


deeper and deeper networks until resonance came along and built ultra deep networks.


They also say we did not use any unsupervised pre training, even though we expect that it


will help, especially if we obtain enough computational power to significantly increase


the size of the network without obtaining a corresponding increase of the amount of


labeled data. Thus far, our results have improved as we have made our network larger and trained


it longer, but we still have many orders of magnitude to go in order to match the infero


temporal pathway of the human visual system. Ultimately, we would like to use very large


and deep convolutional nets on video sequences, where the temporal structure provides very


helpful information that is missing of our far less obvious in static images. So already


the previewing of future research here with the self supervised with the many more layers


and so on. Astounding that this kind of foresight, of course, all of this proved to be, you know,


very, very adequate predictions right here. And yeah, so this was the paper right here,


the paper that kicked off deep learning. I enjoy reading kind of these old papers, especially


looking back at what was already known what still is around, which turns out to be a lot


a lot is still around. And the choices that people made back then, some of them defined


our modern field. So that was it for Alex net, let me know what you think in the comments.


The Weird and Wonderful World of AI Art (w/ Author Jack Morris)

Hi, this is an interview with Jack Morris, who is a PhD student at Cornell in natural


language processing. However, Jack has a really cool blog, and he's written a piece called


the weird and wonderful world of AI art, which we're going to discuss today. Now, as I said,


Jack is a PhD student in NLP. But for this blog post, he dove into the world of AI art,


which is sprawling currently. And we're going to talk about you know, what happened so far,


what are the origins of AI art, at least since the deep learning area, what's currently happening


with all the diffusion models and clip combinations and VQ GANs and so on. And we'll also discuss


a little bit where it's going in the future. This was a really cool conversation, I certainly


learned a lot and I invite you to check it out. Throughout the conversation, we have


so many points to jump off of, and I'm sure you'll find something that's interesting to


you. I'll leave a link to the blog post down in the description. So if you want to go and


read that for yourself, I absolutely invite you to do so. As always, please leave a like


if you do let us know what you think in the comments. And thank you everyone who's sharing


out these videos and helping others find my content is really nice. Thanks a lot. I hope


you're having fun. Bye. Hi, everyone. Today I'm here with Jack Morris, who is a PhD student


at Cornell and works in a research group on NLP, but also writes about all kinds of things


on his blog, among other things, an article that I found really interesting called the


weird and wonderful world of AI art that is a description, a little bit of a history,


a little bit of a summary and an overview and a bit of an outlook as well over the current


state of art in AI, specifically image generation models and beyond, which I found super fascinating.


This is a topic that in recent years, has picked up, there's almost an improvement every


day now in this world. And it's crazy. And I thought it'd be a great opportunity to invite


Jack here to talk to us about, you know, what's going on, how these different things work,


and maybe also a bit why they work and what the what the sort of accelerators behind that


is. So Jack, welcome very much to the channel. Yeah, thanks for having me. How did you we


were talking just a little bit before we started recording about this. How did you how did


you even get into this you you researcher in NLP, which has also seen its own revolution


over the last few years? How does someone like you end up in the world of AI art in


the world of diffusion and clip and whatnot?


Yeah, this is a really interesting research area because it's, it's super new. So most


of all the developments are happening online. And it's very distributed in the sense that


I think like a lot of the a lot of the major participants aren't affiliated with like big


companies or universities. And so the way I kind of got involved was, well, really just


seeing the art online, like specifically for me on Twitter, just seeing like some of these


images that are generated, this, this one on the screen is a pretty good example. That


just really challenged my beliefs of like, what, what neural networks could do. Like,


if you had shown me this a year or two ago, I probably wouldn't have believed that it


was generated by a neural network. I mean, there's some really cool computer generated


art, like procedural generated stuff. And I mean, there are all sorts of techniques


like that. But in terms of just abstract, open ended image generation, like these are


just qualitatively, I think a lot, a lot more interesting than the things that that I'd


seen before. And so anyways, I, I kind of went down this rabbit hole over this past


winter of just looking at the art that a lot of artists were producing and trying to track


down the techniques that they were using. It was actually pretty hard. Like there's,


there's this sort of like commodity in the form of collab notebooks that people are sharing


on Twitter. And there's a, there are a couple of hubs, like a few people are, are producing


maybe like the most popular, the most interesting ones. And then the collab notebooks get, get


forked. And there's various versions of them. And they're all changing different things


and using different versions of the techniques. But I think I was able to sort of identify


like what the most important things were and what, what most people were using. But it


took a while. But anyways, to answer your question, I guess I just saw the art on Twitter


and I thought it was really cool.


Yeah, it's very interesting. And you throughout the whole article, you make a point that you


have maybe a hypothesis of what spurred these things. And that would be I, if I represent


this correctly, multimodal models, the idea, the advent of things like Dolly and clip combining


different modalities together, really gives an artist control over things. And this kind


of brings us a step back into how things were first done initially. These pictures that


you have on here, I remember fondly from my early days in deep learning, which was the


sort of deep dream on the left, or style transfer in the middle. This was the like this was


the non plus deep dream was like that thing, right? It's like, Oh, wow, like this is this


is it's trippy. It's cool. And it kind of gave you an insight into what neural networks


are doing. But things have come a long way, right? Can you I don't know, when you look


at the history of all of these things, what what's the big arch?


Well, do you want to just go through these three pictures real quick? Yeah. So so deep


dream is is the thing on the left, which is I think based on the idea of finding the input


that maximizes some certain, like internal thing in the neural network, like, in this


case, in that picture, I imagine it was something like, like the dog class. And in this case,


I'm really not sure what's going on.


It's always the dog class, right? Image net image net is like it's it's dog everywhere.


Right? Yeah, no matter what you could excite like a class, you could excite some internal


thing. Yeah, I remember. People were very excited about this.


Yeah, it's a it's a cool idea. Like normally, at least a lot of the supervised learning


people do we we look at the gradients of the parameters with respect to the input. But


deep dream is based on the gradient of the input, right? And actually, instead of changing


the parameters of the model changing, changing the input to maximize something, which is


which is a cool idea in and of itself.


Yeah, it is. I mean, it is akin to an adversarial example in some way, although I think this


is heavily regularized, because adversarial examples, usually you don't necessarily see


them or they give you some high frequency artifacts. And this, this is very, very different.


And people, you know, if we talk about art, would this already classify as art? Like,


you know, what's what what would an artist make of something like deep dream?


Yeah, that's a that's a philosophical question. I'm not sure I'm qualified to answer that


one. But some of the some of the pieces produced with deep dream are really interesting. And


they definitely fall under the realm of sort of like psychedelic, like trippy artwork,


but some of them are really cool.


The the next thing the next iteration that you have right here are style transfer networks.


Can you just briefly maybe someone hasn't heard of the how does it how does style transfer


do? How does it work on a very basic level?


Yeah, yeah, it works by just exploiting the properties of convolutional neural networks


to apply sort of like the texture from one image to the content of another. And so this


case, the content of the image would be like the Mona Lisa. And in the middle one, that


the style definitely comes from some Van Gogh starry night type of impressionist painting.


And, and those are really interesting, too. I think there were a bunch of apps that came


out that are basically just like, letting you do style transfer through an app on your


phone, like input two images, and it'll copy the style from one onto the content of another.


Yes. And, and this was, I mean, it's still, it's still, it is definitely more controllable,


let's say, than the deep dream one. But it gives you much more predictable results. I


think this is more akin to how I would describe like Photoshop or something, right? It's not


really you're producing something, it's you're taking something, and then you're kind of


changing it, its properties a little bit, you can really imagine in Photoshop, I'd have


like a Van Gogh filter, and I just put it up and it produces something like this.


Yeah, yeah. Well, first of all, I think that's a, that's a useful distinction. This is more


like an image editing technique, or at least it takes two images as an input, and outputs


one image. And a lot of the other things we're looking at take, take nothing as an input


and output an image. Or in in the case of the stuff we'll get to take text as an input


and output an image. So this is sort of like a stylistic combination of two images. And


you can only do it with neural network. I think Photoshop, specifically, you mentioned


has this new, well, Adobe is doing all these cool things with, of course, this type of


research. And the newest photoshops have these like neural filters, which are, which is a


new feature that includes a bunch of different things you can apply to images that are based


on neural networks. And I think one of the neural filters is is using style transfer.


Like basically, it's built into Photoshop now, which is cool.


Yeah. Well, I mean, yeah, it's excellent. I would do the same if I were them, right?


They, they, I think the Adobe suite is like insane powerhouse, like how much work went


into that. So then the advent of GANs came. And I remember GANs fondly as well, because


that's when I started going to conferences and every single track on every single room.


And every single workshop was about GANs. Like you could not, it is worse than transformers


today. It was just everywhere. And at initially, it wasn't super duper hype, but then they


got good. And here we see some, some this person does not exist, which is a very famous


website. And I think there's been everything from this shoe does not exist to this, I don't


know, whatever does not exist. But however, again, these are these are now free form produced


images, right? But they're very realistic. That is, so we're at the other end of the


spectrum, we are not modifying an existing image, but we producing something out of nothing.


That it they're very much along a data set.


Yeah, so this this would be an example of one of the things that takes nothing as an


input and just produces an image as the output. And that's probably like, at least one of


the reasons why GANs were so hyped is just because like, these images are so realistic.


It's it's somewhat terrifying. I've used this as an example to show my friends that aren't


as like up to date in AI research, and just just to scare them a little bit and show them


like the kinds of things that could be done. And this is probably one of the most well


known examples, I think of like, what neural networks can can actually do right now is


produce these really realistic human looking images of people that I think they're sort


of like, just interpolated versions of all the faces in the in the training data. But


there's so many faces in the training data that it just forms like a totally new face.


I don't think you could like map it back to any individual person.


Yeah. And it's usually usually at the ears, you can recognize although here one is hidden,


but usually kind of the ears would be would be kind of different the left and right one


enough for for you to recognize that if there's something wrong, but they are uncannily realistic,


usually these GAN produced images. So this would be this would be a style GAN v2 probably.


And maybe for someone who doesn't know at all how GANs work, there are two networks.


One is trying to produce images, one is trying to distinguish whether or not a given image


is real or fake. And these two, they essentially play a game and they become better. They sort


of level each other up until the the one that's generating images gets really good at confusing


the other one. And in order to do that, it needs to produce realistic images. Yeah, and


GANs would make will make their appearance later on when we talk about things like VQ


GAN and so on. But these were the first iterations of really realistic, realistic producing images.


And you have this interesting thing, your art breeder, which I was kind of aware, but


there is a story behind this and TikTok. So what's that about?


Oh, well, wait, can we can we stay on the GANs for a second? So it's not it's not immediately


obvious, I think, why they work so well, like there are other models that can generate random


images and and some of them work well, too. But GANs not only have that sort of cool explanation


of being the result of two models competing with with each other. Well, we can be specific


to this is if they're GAN generated, these are the outputs of the generator network of


those two networks. And there are other networks that generate images, but GANs just tend to


do it like really, really well. So the reason why I include them here is because they basically


are the state of the art for generating realistic images.


So yeah, so the onto art breeder. I think there's just a there's a famous TikTok that


that showed generating faces using art breeder, which is another example of AI sort of like


making its way into the mainstream with all this stuff. I included it because, like you


mentioned, I think the the main thesis of my article is that by training these multimodal


models, we can generate art that's like specific to a level that we were never able to do before.


And so starting with GANs, they they start somewhere random, like they just start with


this random initialization that's a vector of floating point numbers. And you have no


idea what it means. So you have no idea how to like position it in such a way that it's


that it's useful. And so as an artist, you could probably do two things. One, you could


accept your fate, the fact that you have no control over the initialization and just sort


of like try to produce things that are that are cool, like either by brute force, just


generating a lot of images or by like looking at the output of the GAN and maybe like editing


it yourself, like maybe using it for inspiration or a starting point for some artwork, but


actually like making changes to the artwork yourself. And the second thing you could do


is maybe some kind of search. Like if you if you start with multiple initializations,


you could examine them all and determine which one maybe has the most value to you or seems


the most promising and then do some kind of like recombination of the most interesting


initializations, kind of like a binary search through the latent space of the GAN. And this


is this is basically how Artbreeder works. Instead of just generating one image and trying


to edit it or just generating a bunch of images and choosing the best one, Artbreeder has


this iterative process where you generate like a few images and you choose the one that


you think is best and then generate more images based on that initial image. And you go through


this process step by step in order to sort of like zero in on something that you find


interesting. And this is probably better, but it's probably still not the best way to


like coax interesting results out of GANs.


There has been like a lot of research into making GANs more controllable. So people trying


to figure out, you know, how can you control the latent space, but we're still not there.


I agree with you. It is quite hard to make these things actually to control these things


and steer these things. I just want to, so a few things to note right here. This is the


original paper, just for people who are unaware how far we've come in this domain. The first


outputs of these things, they looked like this. So these were faces that were totally


aligned. So all the eyes are in the same place, all the noses are in the same place. And still


that was the output. Even worse, if you look at sort of the image data sets, it was very


good at the time, but it was not, as you can see, it was, there's these, the progress is


immense. The other thing for Artbreeder, I think, just also, people may not know it's


based on this idea called Picbreeder. I don't actually know if this is the original site.


The original site is by, certainly Ken Stanley was part of it, where they had also these


things creating pictures and these were not neural networks. These were, I mean, they


were, they had a latent space, but the latent space was quite a lower dimensional and it's


kind of a function, a function using trigonometric overlapping functions that produces these


images. And then also people can sort of recombine images. So it's really cool to see that this


comes to the world of neural networks because Picbreeder itself has been around for a long


time. And yeah, there's, you said there's a famous TikTok on how these things are made.


Yeah, there's a link if you want to pull it up. Oh, is there?


Let's check it out. There's a link to Reddit. And one tick, once TikTok, ah, once TikTok


discovered it. Okay. So people, people making TikTok about how they Artbreed. I guess that's


one way to go viral. So yeah, you had, you had, you have this intermediate post here


about the problem with pre-clip art and essentially lacking control. That's the big deal, right?


The artist can maybe influence stuff a little bit, but not too much, especially if they're


not an expert in neural networks, they have no clue except to try it out.


Yeah. And you mentioned that there've been a lot of efforts to make GANs like controllable


in some way or another. And I think that there's some success to that. Like there, I know there's


some interfaces where you can like generate faces and adjust, you know, the thickness


of the eyebrows and the distance between the eyes and things like that. But if we just


try and think about this from, from first principles, I mean, if, what kind of images


are we trying to generate? I think the goal would be just some kind of like open-ended


thing where the model knows about the world and can generate pictures of whatever you


want. And given that, what, what would the UX look like? Like in the case of faces, maybe


they can design this, this panel that has knobs and sliders and things where you can


readjust how the face looks, but that doesn't apply to everything in the whole world. So


at least one guess is just by typing stuff in. I think text is a really good user interface


for this. You can basically be as specific as possible, but you can, you can mention


anything. And so we come to this idea where we have like a text box and you type in the


text box what you want to see and the model like generates an image from that. And so


everything we're going to talk about after here is some kind of like take on, on that


paradigm essentially.


There is, yeah, there is the paradigm of inputting text and the paradigm of actor critic is


essentially an actor critic framework where usually the way that these things work is


that you'd have one model that produces stuff which could be a GAN, but could also be other


image producing models, and then a critic that judges whether it's good or not. Now,


interestingly that it's kind of the same setup as the GAN itself, right? But the critic right


here is going to be clip or any sort of multimodal model where we can control what it does via


text. And I find it interesting.


Instead of updating the parameters of the model like we would with the GAN, we're going


back to the thing we discussed before where we're updating the actual input itself.


Yes, exactly. Yeah, it's kind of like, it's sort of a deep dream GAN combination. And


so I guess for that, we have to talk a little bit about clip. Now, most people have probably


heard of clip, but clip is essentially a model that takes a piece of text and an image and


it tells you how well they go together, how well the piece of text describes the image


essentially. Now, what we can do is we can simply keep the piece of text fixed and back


propagate through the input in order to figure out the gradient of whatever the input currently


is with respect to that text, which essentially means how do we need to change the image in


order to make it more compatible to a piece of text? And we hope that if we walk that


path many, many steps, then we'll arrive at an image that fits to the text very well,


right? And the reason that we need sort of an artist in front of it, which is also interesting,


is because if we were to do this, just starting from random pixels, and then just optimize


the pixels, the way neural networks work is we would probably get something quite although


I've seen some people do it directly, but we'd probably get a lot of high frequency


noise and artifacts and so on. And having a GAN in front of it is almost a bit like


a regularization or a constraint to make the outputs more, let's say, believable.


Yeah, but I agree. That's how it could work in principle. It's more an artifact of just


the tools we have now is that CLIP is trained to do this sort of like image caption appraisal,


but it's not necessarily, it doesn't have the right parameters to generate images. And


people try, but it's just not that good because of how it's trained. But we do have things


that are really good at generating images like all the various scans. And so the artist


critic idea is to just sort of like couple them together. And because the whole thing


is differentiable, you can use the critic to figure out like how good is the art and


then back propagate through the critic and through the artist back to the input itself


and edit the input to maximize the output of the critic.


I find it very interesting that now obviously you go through a bit later through the initial


successes of this model. CLIP plus, CLIP plus, big GAN, for example, where we do exactly


that here, for example, is a prompt that is, I don't even know, it's like a city. I don't


know what the prompt was. But this picture was very famous because it kind of showed


that, wow, you can actually do something. I find it interesting, though, that the origin


story simply came from the fact that OpenAI released this model, this blog post here about


a model called Dali, which would actually do it, it was trained to directly produce


an image, given a piece of text. There was no, no iterative process, no walking gradients,


nothing. It was just input a piece of text, and out comes an image. It was insane, like


the blog post was insane, right? The avocado chair or here the teapot in the shape of an


avocado. These are insane. Yet, OpenAI just didn't publish the model because, I don't


know, they're usually their their go to line is that it's too dangerous or something. And


had OpenAI released this model, all of the, I think all of the things that we see in the


rest of the blog post would have never happened. I'm pretty convinced. Because we just people


were just kind of stoked that we only have the clip model, we didn't have the Dali model.


So how can we like, get around this?


Oh, yeah, I absolutely agree. Although I feel it may have been somewhat inevitable. Like,


it's not that either Dali or clip was any sort of major technical breakthrough. But,


I mean, there's a lot of engineering required and just a lot of monetary resources required


to train the models. But I don't know how long it would have been before another multimodal


model was released. That was that was equally good. But we can we can talk about Dali for


a second. I know you said you made you made a video about it before. People people do


produce art with Dali. And I think some people have a preference for it. It's basically trained


like a language model. Is that right? Just with text and then pixels?


Yeah, essentially. So here is yeah, here is you have a picture of Ruh Dali, which is trained


on the Russian language picture combinations. But yeah, people use this and it I feel it


is a bit more representative of maybe the data set that you put in and that it gives


a bit more realistic pictures.


Yeah, and I think as an artifact of training it like a language model, Dali tends to produce


like much more abstract pictures. Like it's sort of hedging between a bunch of different


pictures that could satisfy the caption instead of what GANs do, which is just sort of like


picking one thing and doing it as as best as it can, you know. And so it it tends to


be very different. I think in the in the Glide paper, which we'll talk about later, they


they compare the output of this Glide system to Dali and they they just say like Dali tends


to produce much more abstract images, I think maybe 80 or 90% of the time as rated by humans.


I see. And also the the Shutterstock, the Shutterstock watermarks are pretty, pretty


cool.


The data set thing. Yeah, this is if anyone's listening to this and wants to try it out.


The best open source model right now is this Roo, Dali, I think, at least in best open


source model that does the same thing as Dali.


And they have a playground where you can try it out, right?


Yeah, but but it is it's trained on like Russian data. So the playground is like you, you import


a translation model and then you type if you're speaking English or whatever, you have to


translate the prompt into Russian. So that probably makes it even more abstract.


Yeah, pretty, pretty cool. There is also there are other really like true, let's say, open


source efforts to replicate this. One is this Lyon 400M data set, which is a data set of


image text pairs because none of these other models really release their data set. Although


I do believe it's not directly by a Luther as you have right here. I don't know how much


they are affiliated, but it is fully open source. Yeah. And there's also there's there's