for the individual tasks, then okay, there's not good explanation here, then the correct


SupSup: Supermasks in Superposition (Paper Explained)

model should be very confident, right?


This is an assumption that you make.


So I'm going to take my image of the test set, and I'm going to feed it through the


model one, which, you know, you have to separate that this idea is separate from the masks.


At its core, it's simply saying, if I have three different models that I've trained for


three different tasks, and now I get an input, I don't know which one it's from, I can simply


feed it to each one of them.


And I can look at the output distribution.


So maybe my output put distribution right here, this is as you can see, three output


neurons, it's a three class classifier right here.


My output distribution is somewhat here like this.


And here, it's like this.


And here, it's like, I shouldn't do that.


I got a comment, you know who you are.


So which one would you pick?


And their answer here is, we should pick this one, because of it has very low entropy.


So this middle model here is very, very sure about this data point, it's very sure about


its prediction, because it the distance basically of the top prediction to all the other predictions


is so high, it's very confident in its prediction.


Whereas here, you can see that the distance is not too high.


Also here, the distance between the highest and the others is not too high.


So they say, we are going to pick the model or the mask in this case, for which the output


entropy is the highest.


And that is a heuristic for now, but it tends to work pretty well.


It has a bit to do with how relatively difficult your tasks are.


So your tasks need to be kind of equally difficult.


Otherwise it's not otherwise this can get a little bit a little bit out of hand.


But there are ways to solve it.


And they allude to that in the kind of future work section.


But in this case, if the tasks are equally hard, and they consider tasks that are equally


hard, then the entropy is a good measure of how confident these things are.


And therefore, we can check which task it is by using the entropy as a heuristic.


Alright, so we're left with simply trying each of the masks and then decide taking the


one that has the highest entropy.


Now they say this is costly, because if we've learned, you know, 1000 tasks, we need to


try each of the 1000 masks in order to do that.


So they go for something else.


And this is the second word in the title, this superposition word.


So instead of doing that, what they'll do is they'll use a superposition of masks.


And actually, the picture also I find more descriptive than the formula, I can write


down the formula down here.


So what they'll do is they'll say, why don't we just overlap all of the masks.


So we'll have all of these masks mi for one for each tasks, and we'll initialize them


with coefficients alpha, I will just mix them like this, and alpha here, it's initialized


in one over k, where k is the number of tasks, okay, we'll just mix them.


And then we'll multiply them by the weight of the neural network.


And that's will that neural network is where we input our image into.


Okay.


So what does that give us that basically gives us a mix of all the networks that like it's


it's pretty safe to say that the entire network is going to be in there.


And maybe sometimes, you know, multiple times, like if multiple masks use the same weight,


it's going to be in there with a higher weight and so on.


So that's what you see right here, you can see that all the masks are overlapped in superposition


with each other.


Now what does the output give you the output gives you nothing that gives you kind of the


average prediction of the network.


So this here is going to give you kind of the sort of the average prediction of all


of the networks, which isn't very helpful.


But of course, what we can do is we can look at the gradients of this.


So if we from this calculate the entropy, which is here denoted h, and we calculate


we back propagate this.


So we back propagate this to the alphas.


And we calculate the gradient of the entropy with respect to each of the alphas.


What does that give us?


So what's the intuition here?


The intuition is if I change my alpha a bit, how does the entropy change?


So basically, this gives you the sensitivity of the entropy to these alpha parameters.


So if this is high, what does it mean?


It means that this mask right here has a big influence on the entropy.


Specifically, if I were to increase the alpha, then the entropy would increase, okay.


And if I were to decrease the alpha, then the entropy would decrease.


That's the kind of what what the gradient gives you.


Now, did I say before, we want the one with the highest entropy?


I'm pretty sure we want the one with the with the lowest entropy.


We want the one where we're very, very, very sure, right?


I might have said that absolutely wrong.


So if you see right here, this is the formalism.


First we associate each of the k learned supermasks with a coefficient alpha initially set to


one over k.


Each alpha can be interpreted as the belief that supermask m is the correct mask, equivalently


the belief that the current unknown task is task i.


The model output is then computed with a weighted superposition of all learned tasks, which


is this thing right here.


The correct mask should produce a confidence low entropy output.


Therefore we recover the correct mask, we find the coefficients alpha which minimize


the output entropy h.


Okay, so yes, we want the task with the lowest entropy, of course, not with the highest entropy.


So, if we look at the gradient right here, the gradient basically tells us how each of


the masks will influence the different the entropy.


And if we simply select the alpha, where the gradient here is the most negative number,


so we want this to be as low as possible, not zero, but you know, negative as high as


possible, then we know that if we increase this, the contribution of this mask, then


the entropy will go down the most.


Okay, and again, our hypothesis here is that maximum entropy, sorry, minimum entropy means


most confidence prediction means that the if all tasks are equally hard, it probably


means that the data point is from the task where we have the lowest entropy.


So what's the deal here?


They show in this graph right here, they show this is much faster.


So if we were to evaluate each mask individually and measure its entropy, of course, with the


number of tasks, we'll simply linearly increase our time in the forward pass because we need


to try out each of these masks.


However, if we do what they're doing here, we simply run one, we mix these ones, we run


one forward pass, we do backprop, and they consider two strategies.


So what you can do is you can do gradient descent on these alphas, which takes, you


know, a number of steps to converge, or you can actually do a single step.


So you just observe the gradient and by the gradient, you recognize which one has the


lowest gradient, and that's the one you pick.


So where's the catch here?


The catch is that if you do something like this, if you do something like this, there


are two catches, actually.


First of all, this here is a a convex combination, right?


This is convex combination.


And the problem isn't convex at all.


But if you simply take this convex combination, multiply it, and then look at the gradient,


you sort of assume that the problem is a kind of a convex, nicely shaped problem.


And if you then observe these, the gradients with respect to the alphas, you make assumptions


about the problem that might not be true.


So you lose, you kind of heuristically approximate the importance of these masks.


That's the first thing.


The second thing, of course, is that you still are implicitly saving, you still are implicitly


trying all the models, but you're just not trying them explicitly.


You're implicitly trying all the models because when you do this combination right here, your


auto differentiation library will actually keep track of what the individual models contribute.


It's just that per layer.


So of course, this here, this W is multi layer perceptron, which means that if you have multiple


layers, you know, there's w one, and there's w two.


And you have your alphas, and your alphas are also, you know, you can distribute them


into these, sorry, your masks are also mask for layer one mask for layer two, and so on.


So your auto differentiation package needs to keep track of okay, mask one goes here


with this alpha mask to the layer two goes here with this alpha.


And there is there, so it needs to keep track of this graph.


It's just that this is highly optimized.


And you also need to you only need to do it layer by layer.


So the contribution of alpha of mask one, this is maybe alpha i of mask i one mask i


two, the contribution of the alpha i will not be explicit in this layer, it will be


implicit as an average across the layer, right.


So again, this is you assume in each layer, you assume a convex combination of all the


alphas and propagate that forward.


And therefore, if you look at the next layer, you can only view what mask two does mask


of layer two does as in terms of a convex combination of layer one.


So you make multiple approximations, and you rely on the optimization of your auto differentiation


library to keep track of these different things and do operations in parallel.


And in the case where you do it linearly, I'm going to guess you simply do it as a sequential


operation.


But it's going to be exact.


So that's the trade off.


Alright, so we now know how we can figure out where the task is from.


And let's see how that works.


So in this first task, we are looking at split image net, split image net, simply it


takes the image net data set, which is a thousand class data set, and it distributes it into


100 different tasks, each is a 10 class classification task.


Now note two things.


First thing is that split image net, each task is approximately as hard as each other


as as the other tasks, right?


It's still image net classification, and it's the same number of the of it's the same number


of labels.


And each task is about, you know, the same hardness, you can make that assumption.


And second of all, the tasks are actually pretty, pretty easy, right?


It's hard to distinguish image net into 1000 classes.


But if you split that task, I'm going to bet that you have these high resolution images,


and you have a 10 class classification, it's going to be relatively easy.


So all our conditions are met for at least for my hypothesis to hold.


And you can see on the right side, you can see split CIFAR 100, which does the same thing


to CIFAR 100, it subdivides it into different, very small class classification tasks.


You can see the results.


The upper bound here is where you train a single model for each of the tasks that gets


you to an average accuracy of 92%.


So on image net 92%, you know, it's pretty, it's pretty good.


Of course, this is again, this is 10 class, which makes the numbers a lot different.


With the sub sub sub sub sub, you get to this pretty good 88% accuracy.


This is this super masks in superposition.


This here is a baseline that also does lifelong learning.


Now they have these annotations right here, GG, which yes, GG, haha.


But so the first letter will always tell you whether the task ID is given during training.


And the second letter will tell you whether the task ID is given during testing.


So this here simply evaluates whether or not this masking is feasible, which you can see


here it is so this will we know which mask to train during training, and we know which


masks to retrieve during testing.


So there is nothing of this entropy gradients here, none of it, this simply evaluates the


viability of the masking approach, which, as you can see, it's pretty viable, and it's


more viable than these baselines.


This same thing on the CIFAR 100 right here.


So you can see they also evaluate since I guess it's an easier problem, they also evaluate


the number of bytes, which they can control.


So they can control the number of bytes in their model by simply increasing or decreasing


the required sparsity of their mask.


So you can change your mask by saying how sparse you want it.


And of course, if you want it more sparse, you get a worse model because you have less


less ones in your budget to make your model perform well.


But you can see that if they do it with these baseline model is batch E, you severely underperform


with regard to the upper bound right here, the upper bound again is where you train a


model per task.


And separate heads here is another kind of dummy baseline, where you train a different


head for each of the tasks with a common trunk that gets you pretty much nowhere.


With the sub sub algorithm, you do get almost to the performance of the upper bound.


And in fact, if you do this transfer approach right here, you do get there.


The transfer approach simply means that so you do these tasks in succession, right, you


do task one, okay, done, you do task two, okay, done.


And for each one, you train a mask, okay, for each one you train, this is mask one,


mask two, the transfer approach simply says, if I start task three, I'm going to start


the mask three, my initial weights basically, are going to be a running average of the masks


that I have already considered or an average, there is some amount of transfer going on,


simply to initialize the weights, it's actually astounding that this helps you so much.


But with this, if you look at the actual numbers, I believe you can get like a tiny bit higher


than the training a single model for each of the tasks.


Okay, so this sort of establishes the viability of training the different masks for the different


tasks, which I, again, I think it is not surprising because essentially, you're training a different


And it's just the fact that you do a very crude model, and that you can store very efficiently,


you might object and say, hey, don't I need to store the underlying randomly initialized


network?


And the answer is yes and no, actually, you only need to store the random seed to produce


it.


So checkmate.


Yeah, they do.


So here they explain this one shot algorithm where they simply look at the gradient of


the entropy, you can see with the maximum negative gradient of the entropy.


They also have this binary algorithm, if the task where they say with the task is harder


to differentiate this kind of assumption of the convex combination thing does might not


hold.


So what they do is they have this binary algorithm, where they do a binary search, where so they


simply want to circumvent the necessity to evaluate each of the masks by itself, because


that takes long.


So they do something in between where they do this binary algorithm.


This is right here, where they do this convex combination, they evaluate the gradient, but


then they don't just take the highest of the negative gradients, they eliminate half of


them.


So basically, whenever it's lower than the median, they eliminate it, and then they start


off with this new set of reduced alphas.


So in each of these steps, they eliminate half of the masks, and then they recompute


again, because because it is not a convex problem, the the order might actually be different


in the second and third and fourth step.


Of course, this is simply this is like halfway towards between this one shot algorithm and


trying each mask by itself.


It's kind of a compromise.


I mean, they make it, they really try to not not try each mask once because it's one of


their contributions, right?


But then they probably realized if we just do it one shot, sometimes it doesn't work.


So they're going between which is, you know, it's a pretty cool idea.


All right, next experiments.


We're now in this situation, and you see, you see a number of things.


So first of all, we have a new added a new baseline, this PSP.


And you can see that the baselines operate in this GG regime.


So the baselines are given the task during training, and given the task during evaluation.


You see the upper bound here in gray is where we train a model for each task.


And you assume that's an upper bound, because you assume the tasks are kind of unrelated


to each other, which is not the case.


So there is actually potential to beat the to beat the upper bound baseline.


And subs up here you see operates in a different regime.


Namely, there's this regime of you're given the task during training, but then during


testing, you're not given the task.


Okay.


And this you here, it basically means that the labels you assume that the labels of the


tasks are not shared.


So in in this case, if you predict, if you predict, like if you split MNIST into always


that two class, if you split MNIST into two tasks, you predict the first task is 01234.


The second task is 56789.


And you have the same amount of labels.


So you always have five output neurons, right?


So you have 12345 output neurons.


If you if the image here is like a five, that would be task, task one, label zero, right?


If your network now predicts label zero correctly, but predicts the the image to come from task


one, you count it as a mistake, you say, Well, you know, you've predicted the right output


neuron, but you've told me it comes from task zero from from the zero to four.


So I'm going to count that as a mistake.


So it really there isn't there isn't a way for the network to kind of go get around.


Predicting the wrong tasks are kind of share information.


So you assume that the labels are not shared or on shared.


Yeah, so it's the sub sub here has a significantly harder task than the baseline.


Keep keep that in mind.


And now we're applying our because we we are not given the task at inference time.


Now we're applying our heuristic, where we go and look at which of the mask entropies


is the lowest, respectively, we use this actually this one shot algorithm, where we look at


the gradients.


And you can see this is on permuted MNIST in permuted MNIST, what you do is you take


MNIST, and you simply permute the pixels.


And this, it sounds crazy, but you simply permute the pixels.


And that gives you a new task.


So you can come up with like almost an infinite number of tasks because there are what 28


times 28 pixels, so you can commute them 784, you know, factorial times, which gives you


like infinitely many tasks.


And so you can modulate.


So here you can see the number of tasks learned increases.


And at the beginning, this baselines, especially this baseline is doing fairly well actually


on par with the upper bound when you only have 10 different tasks.


However, after that quickly degrades, however, this sub sub here, it, you know, keeps it


keeps its performance, which it so this doesn't only mean that it correctly predicts the output


neuron, it also correctly predicts which task which permutation was applied to the digit,


simply by looking where the entropy is high, right.


So that's pretty cool.


And you know, it's it's actually kind of surprising to be to be honest.


So on the left, this is a lunette architecture on the on the right, it's a fully connected


network.


So the fully connected network here performing better is sort of expected.


First of all, amnest is really easy and can actually be solved with a fully connected


And second of all, especially permuted amnest, I guess doesn't really conform to the to the


assumptions of convolutional neural networks anymore.


Again, keep in mind, these tasks are very easy.


Yeah.


So so especially for the fully connected network, of course, each permutation kind of looks


the same because it's it doesn't care at the beginning that pixels are next to each other


simply each pixel is a different thing.


It's just the fact that it cannot it cannot learn from one tasks much about the other


tasks.


That's why you that's the nature of permuted amnest.


All right, and then in this experiment right here, and this is the sort of crown experiment,


they learn they do this permuted amnest, but they go up to 2500 tasks, right 2500 different


permutations.


But so but now they have an additional thing right here.


So again, they have this sub sub where it needs to predict the correct permutation.


But also they compare it with a an algorithm that needs that is this n n right here.


So in this n n, not not only are you not given the task label at testing time, you are actually


not even given the task label at training time.


But here the outputs are shared.


So you know, since since you have no way of knowing which task it is, you've never given


it as long as you predict the correct class, you good.


So it's always it's always a 10 class classification problem.


It's just no permuted.


You're not given the task label here.


So first of all, I want to say that this here, the shared labels, it could actually contribute


to the success of this algorithm right here.


Because even though you permute the pixels, you can still sort of do things like count


the frequency of light pixels versus dark pixels in MNIST.


And that might already give you a very, very big hint, right?


Or, you know, simple correlation of of two pixels, though that's that's a task specific


thing.


But that the frequency of light pixels versus dark pixels will already give you a big boost


in accuracy.


And now you can actually share that feature, that feature will always be the same for every


permutation.


So this is something you can share between tasks.


And I would like so one way I guess you could eliminate that.


Well, I don't know.


I'm not sure is you kind of have to randomize the number of light pixels, but keep the classes


the same.


Yeah, it's it's going to be complicated, right?


But just keep that in mind.


So how, how does the algorithm even decide?


So they have a heuristic right here as well.


Namely, they say, okay, if we don't have no task identity during training or inference,


where task identity is entirely unknown, even during training, if sub sub is uncertain about


the current task identity, it is likely that the data does not do not belong to any tasks


seen so far.


When this occurs, a new super mask is allocated, and the number of tasks learned so far is


incremented.


Okay, so the they go with the same principle right here, they say, if we get a new training


sample, we just evaluated against all the masks that we had so far, or we do our, you


know, one shot algorithm to to approximate which masks gets it gets us a low entropy.


If none of the mask gets us a low entropy, then we decide this must be some kind of unseen


tasks.


So we're going to allocate a new mask for this unseen tasks.


And that heuristic, as you can see, it performs fairly, fairly well, where was our graph,


our graph was down here.


In fact, it performs pretty much on par with where you know the task during training, and


just not during during inference, up until like here, the very last bit, if you really


get into the high task regime, where I guess it starts getting it starts getting confusing.


So this this heuristic might start to break down, but it might just be a fact how they


tune their constants, like they have to define a threshold where they say, okay, if the end


entropy is somehow higher than this threshold, then we allocate a new a new task.


And this might be optimized in order to solve this.


Again, these tasks are very, very, very, very easy.


So keep keep that in mind.


Yeah.


Okay, so this basically was the experimental part of that paper.


Now they consider different extensions to that.


And I'm not sure they also considered some ablations, which are pretty interesting.


So here, they say, we are going to up the kind of the hardness of the task with with


rotated MNIST and also their model does pretty well on the rotated MNIST task where the differences


of between the the differences between the tasks are simply some of them are rotated


by 10 degrees.


So that's a tiny rotation in the right if you have a number three, you kind of rotate


it by 10.


I can't even draw that subtle of a rotation by 10 degrees.


And you know, the sub sub must correctly predict which task the image is from, or it will not


get the it will not get a correct reward.


And the fact that it performs pretty well.


And the fact that it has, you know, rotation degrees where it outperforms the baseline


that is all actually given the rotation.


So it's given the task at inference time is pretty, pretty remarkable.


Again, I believe this is due to the fact that these tasks are so easy, and therefore, this


entropy, it just spikes when you get the correct thing, because it it sort of, it sort of latches


on to very easy features for each task.


So I'm going to guess that the tasks are, you know, generally solvable by maybe correlating


two pixels, right?


If like this pixel correlated with this pixel, if the correlation is high, it's a three,


the correlations low, it's something else, okay.


And then if you rotate it, it's just not the case anymore that this pixel and this pixel,


the correlation is very high.


So if you predict using this correlation, you'll get a pretty low confidence.


And I'm going to guess that yeah, if you have discrete tasks, and it's in this task, then


your confidence will just spike because the task is so easy.


And because all the tasks are about equally hard, because if you can find this correlation


here, you can find it over here, it's simply going to be two different two different pixels


in this task, and then what as you try the masks, whenever you hit the one where you


can predict pretty confidently with those two pixels, then your confidence is going


to spike your entropy is going to get down and you know, it's that task, right?


They also here they compare.


Where is it?


The one shot algorithm.


So they, they, they use their one shot algorithm to and and they put it on a baseline.


So this baseline, where they always actually have to give it the the task, they augmented


by by their, their one shot algorithm to select the task.


And it turns out they can, you know, make it perform fairly well, not on par with them,


interestingly, but they can make it perform also fairly well actually better than it was


performing before.


So they have different extensions right here.


And that's some of them are pretty important.


The one important thing they do is they have these superfluous neurons.


And that's sort of hidden.


And it's always a bit.


So here, for example, you see, in the output, they say, we have a lunette model using output


size 500.


Now there there are only 10 different labels in the MNIST task, right, also in the permuted


MNIST task, there are 10 different labels that I mean, there are a total of 25,000 labels


if you have 2500 tasks, but the neural network has output size 10.


However, their neural network here has output size 500, which is surprising.


So they say right here, and we're going to get to the hopfield network at the very end.


For those who are still around, because that's, I think that should be its own paper, but


you know, they say it could use an output of size L, where L is the actual number of


labels per task, though we find in practice that it helps significantly to add extra neurons


to the final layer.


Specifically, we consider outputs p in our s, so s is higher than L, and refer to the


neurons that are past L as superfluous neurons.


So let's try to make sense of this.


So they have a neural network.


And let's say it's a three class classification task, right?


So you have three classes, and that's what you would do, they simply add a bunch of neurons


right here.


That means they also they, you know, they add all of the connections from the previous


layer to those neurons.


But still, the classes can only be either 01 or two, these classes never appear during


training.


So they claim this helps during during their procedure.


And I, I thought about it a bit, and we might be able to try to guess why it makes sense.


They say they simply say we observe that helps.


And I mean, you know, let's, let's try to make sense of it.


Okay, so if we train, if we train our model using these too many neurons, what happens?


Well, our label is always going to be of the top three neurons.


So let's say our label is one, this is going to result in a one hot vector like this.


Now, what are we training in this layer here in this layer here, we're training logits.


Okay, so pre pre softmax outputs.


So our, our algorithm, our cross entropy loss is going to push all of these here down during


every single training point is going to push this one up and all of these down.


Now these three here are going to be pushed up and down depending on the label.


However, all of these down here are going to be only pushed down during the entire training.


So they are going to be exceptionally low numbers.


Okay.


Now if we then come and we look at the at the entropy of this, the the entropy, I think


honestly, this is simply you could achieve the same thing by using a different temperature


parameter in the softmax or in the entropy that you consider, because why can this help


and this helps with inferring which task it's coming from, right?


So if you consider a task where you only have three outputs, so you don't have this bit


down here, and you look at the entropy, it's going to be you know, it's going to be something


something.


Sorry, I have to draw this right here.


It's going to be like this, you know, it's fairly confident.


But if and maybe for the other tasks, it's not going to be as confident, you know, it's


maybe going to be like this a bit.


However, if you have those, and if it's of the correct tasks, I'm going to guess this


kind of stays the same because they're really low.


But if it's of the incorrect tasks, then you're not sure.


And you not being sure about the output also means that you allocate a lot more to these


things right here.


Because you've sort of never seen this particular kind of training examples.


So you're not sure.


You're just going to distribute your kind of your probability mass across these things


right here.


Because you've not been trained on that kind of input, right?


It's very important to see that this is task, this is the correct task, which they always


label J. And for for any other incorrect task, you've never seen data like this.


So these things here sort of act like an outlier class without you explicitly training an outlier


class, you simply train these things.


During training, you make them small.


But you it's important to notice you always make them small from a data point that comes


from their particular task.


Okay, that's what you train them for.


And now if you input a data point from a different task, they have less reason to be small, because


this is an outlier data point.


So you have much more fluctuations, so you have more fluctuations here, and therefore,


the entropy is going to be even higher.


All right, this is sort of how I make sense of the fact that these additional superfluous


neurons help here, they act as kind of an outlier detector for the training data set


of that particular task.


Now, because you have different training data for each task, they go further and they say,


it actually works even better, it works even better.


If we instead of this entropy heuristic, we consider another heuristic.


Accordingly, we consider an objective G, which encourages the S neurons to have large negative


values and confused as an as an alternative to entropy in equation four.


So G they analyze down in the appendix, and we're just quickly going to look at what G


is sorry, this is about to load.


Right here.


And it's very interesting to see what G is.


Or is it?


Yes.


So G is going to be this right here.


So Y are the logits, and then G is this expression right here.


So and in fact, it's this expression with the with a bit of a modification.


So it's going to be G is going to be the log sum x of the logits, right?


So it's, this is some, this is somewhat like the entropy.


And what we're going to consider is the gradient of G. So what we want is the gradient of G


with respect to our alphas.


And the condition here with this detach operation is that


the gradient of G should be, you know, the gradient of the loss function for all v that


are superfluous neurons, and zero otherwise.


So we're going to detach the gradient of G for all the real neurons for all the actual


logits of the output class.


And we're only going to consider the gradient flowing through the superfluous neurons.


So all of this here is if we take the gradient, it's only going to flow to in these in the


last layer through the gradients of the superfluous neurons.


Okay, and that's why we don't need the entropy, because the entropy always considers, you


know, the difference sort of the difference between the correct label and the other labels.


We are pretty sure that in our superfluous neurons, we don't have the correct label.


Okay.


So this log, the log sum x of our of these outputs here, what will they represent?


Well, this is sort of a flatness measure, again, it's kind of like the entropy, except


we don't have a correct label right here.


If one of them is very high, and the other ones are very lower, if they're generally


very high up, then this will be high.


However, consider the difference between this and this, where they're all super small.


And also, they're all pretty equal, the log sum x will be very small.


So this is an alternative where we can basically only look at the superfluous neurons and say,


is are the superfluous neurons all very small?


And you know, none of them basically says I'm the correct label, then we can be pretty


sure that over here, there is some confidence.


However, if they are sort of kind of larger, and you know, generally kind of generally


large, maybe unequal, that means we're not very confident because these are our outlier


classes, they shouldn't be, they shouldn't be large at all.


So an alternative to looking at the entropy of this distribution is to build such superfluous


neurons, and then look at those and only those.


And so the gradient of only those, in order to decide which task it's from.


It's an interesting idea, I have to say, but I maybe one could achieve sort of the same


thing with a with a temperature parameter here or by building an explicit outlier detection.


But it's generally an interesting idea for outlier detection, I have to say, I've never


really seen anything like this, though I also haven't really considered it.


So here they show the importance and you've seen in the experiments before that there


sometimes was this H objective and also this G objective.


So you can either look at the entropy, but also you can look at the G, in both cases


you have superfluous neurons.


So before you actually saw you have 500 neurons for a task of for a task of 10 that needed


10 output classes, right.


So this tells me that the superfluous neurons are pretty important for them.


And it's, this is probably one of the things that makes this work, right, these superfluous


neurons.


So you're kind of setting up a trap where for the wrong models, you let it run into


this trap of assigning a lot of weight into these outlier classes and only the correct


model is trained to not do that on the particular data that you're considering.


I don't think this comes through in the paper too much that this is one of I guess this


is one of the main factors making this work.


And you can see right here, they actually do an experiment, so I don't want to be too


mean, where they say, look, if we train with just 25 classes, and this is permuted MNIST,


so the necessary amount will be 10.


So if we train with only 25, you can see how quickly we degrade right here.


However, as we go up and train with 100 and 200, we get better and better.


In fact, if we train with this G objective, it always sort of outperforms the H objective.


Interestingly, the more output neurons you have, the less this difference seems to be,


but maybe the percent difference is the same.


The percent error difference is the same.


I don't know, I can't tell from here.


Yeah.


So this isn't all.


There is also this Hopfield network going on where they say, okay, okay.


So essentially, we're actually training different models, right?


We're not really superimposing all of these models, we're training a different mask for


each of the tasks and kind of remembering the masks and so on.


Can we also build a model where we actually only have one model?


And that's what they do right here, where they build a Hopfield network, which is basically


just a big matrix.


This is the Hopfield network.


And then they encode the masks in this Hopfield network.


So specifically, the Hopfield network is of size d squared, where it is able to encode


two to the d different binary strings.


And it does so in a fuzzy way.


But you know, you can prove that if you construct the Hopfield network like this, where z is


a binary string, you can recover the binary strings by gradient descent in the Hopfield


network.


And as obviously, the more binary strings you encode, the less you get out like it's


not magic, you can't store that many bits into the thing that doesn't have that many


bits.


But I believe, you know, again, this is using gradient descent, and it can do so with surprising


accuracy.


So remember that these here are bits, while these here are floating point numbers.


So the comparison that I just made isn't entirely fair.


But I don't want to go into the Hopfield networks, because I really feel this should be its own


paper.


I guess they just want to show that it's also possible to compress these masks into one


thing such that I can't make the argument anymore that hey, all you're doing is training


different models for different tasks.


All right, all in all pretty cool paper, as I said, pretty dense paper, I invite you to


read it, they have a big appendix where they have more experiments and so on, and explain


everything in detail.


All in all, I from this, I don't really take the method, but the ideas are very interesting.


And I am excited to see where this goes in the future.


All right, I'll see you next time.


Is Google Translate Sexist? Gender Stereotypes in Statistical Machine Translation

So, you might have seen this tweet.


Hungarian is a gender neutral language.


It has no gender pronouns.


So Google Translate automatically chooses the gender for you.


Here is how everyday sexism is consistently encoded in 2021.


F you Google.


On the left hand side is a Hungarian sentence.


Google Translate then translates this to the following text saying she is beautiful.


He is clever.


She reads.


She washes the dishes.


He builds.


She sews.


He teaches.


She cooks.


So Google Translate chooses the gender pronoun and it appears to choose gender pronouns that


are very consistent with common gender stereotypes.


So this has generated a lot of outrage and the topic is coming up again and again.


And I thought we just dig a little bit into the background of why this happens and what


we might do about it.


So the first thing you might notice is the text here is really a bouquet of stereotypes


and also ends with go to hell Google.


So no doubt this person has tried a bunch of things.


So I've kind of reproduced the first four sentences of the input.


And here it is.


She is beautiful.


He is clever.


He reads.


She washes the dishes.


Now to detect whether or not this is a feature of the language, maybe there are subtle gender


hints here is a thing you can do.


You can translate it back into the other direction.


He is clever, which will give you the Hungarian sentence.


And then we can simply change the pronouns right here.


He is beautiful.


She is clever.


If there are subtle language hints, you would expect that if you translate this to Hungarian


and back that the same sentence returns.


However, if this is a truly gender neutral language, then you would not expect this to


matter.


So if we now translate this to Hungarian and then we take this Hungarian sentence and translate


it back, oh, see, it has actually switched around the pronouns back to she is beautiful.


So no doubt Google Translate here is inferring the pronoun from the words that follow assigning


beautiful to a more feminine pronoun, assigning clever to more masculine pronoun.


These are gender stereotypes.


And we're going to dig a little bit into why this happens.


For that we have to understand how the machine learning systems currently work.


Machine learning systems are statistical systems that try to translate a piece of text into


a piece of text of a different language.


So here we enter the piece of text in one language.


It goes into this big ml box and outcomes actually not a single sentence, but outcomes


usually a plethora of possible sentences along with probabilities assigned to each of those


outputs.


The system then chooses the most likely output and displays that to the user.


I already said this is a statistical system, it is derived from a set of training data.


So it's important to understand that all the system does is tell us that the sentence she


is beautiful is the most likely sentence to appear in a document that is translated from


Hungarian where this original sentence was present.


Given the training data, the training data itself is of course derived from the world


in some way if you believe that such a thing as reality exists.


And there we have the whole system.


And now we might ask ourselves, what do we do about it?


How should we fix this?


And the answer unfortunately is, it depends.


It depends on where you think the problem lies.


So the first point where there could be a problem is the way we derive the training


data from the world or from reality itself.


Common issues here are that the sampling of data is somehow skewed.


It is out of date.


We're working with old data, in general, the data that we have does not reflect the world.


And if the data that we have is skewed in some way, we can only expect that our machine


learning system picks up on that skew.


So a person arguing this would say that it is actually not that likely that the Hungarian


sentence here translates to she is beautiful.


And it might be equally or more likely that it translates to something else.


If we only had all the translation data that we could hope of.


The second point where we could introduce problems is when we derive the ML system from


the training data.


Here's the thing, every machine learning system introduces statistical biases in order for


it to generalize properly.


Otherwise, we could not do learning.


And it's entirely possible that some of these things such as the regularizer and the loss


function, or the particular choice of architecture would introduce statistical bias into the


system.


This would result in a model that does not reflect the data as we have it.


So someone arguing for this would argue that even though we have good training data in


the training data, there is no problem, the ML system derived from the training data introduces


unwanted effects.


So someone might argue even though the feminine version here is slightly bigger in the training


data than the masculine version.


Through the process of learning and distilling the ML model simply abstracts this and makes


it a lot more likely therefore skewing the gender balance unfairly.


The last problem is the fact that we simply choose the top prediction and output that


to the user.


This is not really accurate.


If we simply output whatever is most likely, this is an unfair representation.


In fact, what we should do is we should give the user all the possibilities with all the


probabilities associated.


Someone arguing for this might say that the training data is fine, the ML model even makes


good outputs, the probability distributions are correct and reflect the world.


However, because we only pick the top one, the user is tricked into thinking that that


is the only possibility or maybe just that this possibility is much more likely than


the alternatives.


As good as that sounds to output always the probabilities associated with different ambiguous


translations.


The short answer of why we don't do this is pragmatics.


I'll give you an example.


This is Billy Billy.


It's a Chinese video sharing websites and for people who cannot access YouTube from


China, I do upload my videos to Billy Billy so they can watch them.


However, while I'm practicing Mandarin, I'm not good enough yet to navigate a site that


is full of characters that I have even a difficult time parsing.


And this is what Google Translate is usually used as I just want to navigate effectively


to the point where I can upload a video, define its categories, leave a description, and then


send that off.


If Google Translate were to give me every possible ambiguity of every translation, how


could I possibly achieve my task?


And this all breaks down if you just think one step beyond the things like gender, if


there is ambiguity in a translation, and you give me all the outputs, what am I supposed


to know I go to Google Translate because I don't know what something means.


And especially if you don't give me actual probabilities together with the possibilities,


I have no clue what to do.


But let's go into this a little bit more.


See if we go to this original sentence and explore Google a little bit more, you might


ask why is not even consistent across the entire thing I input Google splits by sentences,


it's pretty clear.


Because once you hover over it, you get the different sentences right here.


You can solve this by inputting a comma in which case at least within a sentence, the


translation is consistent.


This is not always the case.


But it gives you a little bit of a hint on how Google Translate works.


Moreover, if you just input a single word, Google will actually give you the output distribution


over all the translations here.


The second thing is if you input an entire sentence and it has a gender pronoun, Google


actually gives you both versions.


And it says that translations are gender specific.


It is only when you input more than one sentence that this doesn't work anymore.


In fact, if I make this into one sentence, Google gives me both versions.


And this is already the corner case.


Because technically, it should give me every combinatorial version of the different assignments


of these four variables right here.


So you can clearly see that Google is doing everything it can to give you a good practical


solution that still makes sense in the majority of use cases.


People use Google Translate because they want to get an idea of what something in a language


means that they don't understand.


They don't go to Google Translate to draft their formal letters that must be absolutely


correct.


So I think the accusation against Google here and saying things like F you Google, and honestly,


Google has found a super pragmatic solution.


And I think they're just doing the best they can in the face of the overwhelming complexity


that is machine translation.


All of that being said, there is a fourth category, a category of people that says that


even if we derive the training data correctly, and it reflects the world, even if our algorithm


does not introduce any additional bias, even if the output probability distribution is


the correct probability distribution for that translation, this is still not good, because


they see the problem here in reality itself.


It is reality that doesn't conform to some preconceived notion.


And this might have multiple reasons.


For example, a person arguing this might argue that if we output the correct probability


distribution, that might have some downstream effects, or it might reinforce these stereotypes


or a number of other arguments.


Someone arguing like this would see ml models more as tools for social engineering, which


is a valid stance to have not criticizing that any of this pipeline is wrong, but that


the original bias that exists in the world is carried over into these outputs.


And we should change that in order to affect the world.


Now while that is valid stands to have, and certainly debatable, you have to ask yourself


whether you really want to give Google a multi billion multinational corporation, the almost


monopolistic power to decide on what's good and bad for society.


And personally, I'm going to go no with this one.


In any case, what I want you to take away from this is that there are many possible


places where problems can be introduced, and therefore many possible points where we can


introduce solutions.


But what we have to be careful of is that we don't confuse the different points.


And we don't let people provide evidence for one particular point of problem and then suggest


a solution that is in an entirely different area.


All right, that was it for me.


I hope this was at least a little bit entertaining.


The Man behind Stable Diffusion

This is a mud, a mud is very rich, and he wants to put that money to good use.


The Man behind Stable Diffusion

So just a few days ago, he presented something called stable diffusion through an initiative


The Man behind Stable Diffusion

that he finances called stability AI stability AI is supposed to be a third pillar.


The Man behind Stable Diffusion

There's industry, there's academia, and now there's something else.


The Man behind Stable Diffusion

Remember when open AI started and they said they wanted to bring AI to the masses to democratize


The Man behind Stable Diffusion

the technology and all that kind of stuff.


The Man behind Stable Diffusion

Well, I'm not wants to do that.


The Man behind Stable Diffusion

But for real.


The Man behind Stable Diffusion

So this is an interview with a mod, he's going to tell us what he wants to achieve with stability


The Man behind Stable Diffusion

AI, how he plans to go forward so that he's not the only one that's financing this admittedly


The Man behind Stable Diffusion

very giant operation currently, and what you can do wherever you might be an academic person


The Man behind Stable Diffusion

from industry, or just someone who's interested and wants to do something in the AI space.


The Man behind Stable Diffusion

And you need some compute, you need some help, you need some time stability AI might be the


The Man behind Stable Diffusion

place for you.


The Man behind Stable Diffusion

If you haven't seen the outputs of stable diffusion yet the first system coming out


The Man behind Stable Diffusion

of this initiative, they are absolutely amazing.


The Man behind Stable Diffusion

And not only that the model is small and fast, it runs on a consumer GPU, and it creates


The Man behind Stable Diffusion

pictures in about three seconds.


The Man behind Stable Diffusion

And the model is released open source, fully up to you what to do with it.


The Man behind Stable Diffusion

Very cool.


The Man behind Stable Diffusion

So I don't want to stretch this intro too long.


The Man behind Stable Diffusion

Please listen to what I'm not has to say I'm sure you'll be very interested.


The Man behind Stable Diffusion

Hey, everyone.


The Man behind Stable Diffusion

Today, I'm here with a mod mustach, who is, I have to say, I was contacted by mod through


The Man behind Stable Diffusion

a mutual friend.


The Man behind Stable Diffusion

And it was very intriguing.


The Man behind Stable Diffusion

So all I know is that a mod wants to tell us about exciting opportunities, essentially


The Man behind Stable Diffusion

an alternative in research to bigger labs and big companies doing research a essentially


The Man behind Stable Diffusion

a third door a third path of people having access to resources to do current deep learning


The Man behind Stable Diffusion

research amount.


The Man behind Stable Diffusion

Welcome.


The Man behind Stable Diffusion

What brings you here?


The Man behind Stable Diffusion

Hi, Yannick, I think that we're at a super exciting time in artificial intelligence.


The Man behind Stable Diffusion

Everything seems like it's about to take off.


The Man behind Stable Diffusion

And I'm here to say, you know, let's all come together and make sure that it gets out to


The Man behind Stable Diffusion

as many people as possible and we'll unlock all the creativity that people have in front


The Man behind Stable Diffusion

of them.


The Man behind Stable Diffusion

So basically, I set up an organization called stability AI, to remove many of the barriers


The Man behind Stable Diffusion

for independent and academic researchers to build some of these new models that we're


The Man behind Stable Diffusion

seeing kind of in the early days of Luther AI and lion and others, we heard that computes


The Man behind Stable Diffusion

and kind of funding were a key restriction.


The Man behind Stable Diffusion

So everyone has basically three choices.


The Man behind Stable Diffusion

You go into academia, you don't have compute access, and then you have to jump to big tech.


The Man behind Stable Diffusion

And then you have 59 page MBAs.


The Man behind Stable Diffusion

And you're working in a corporate environment for product teams, or you have your own startup


The Man behind Stable Diffusion

and running your own startup is terrible.


The Man behind Stable Diffusion

And it's not something for most academics, or researchers, although of course, some of


The Man behind Stable Diffusion

them will hopefully be very successful doing legal AI and things like that.


The Man behind Stable Diffusion

I thought there was going to be a better way because this type of technology that we're


The Man behind Stable Diffusion

seeing 80% of research dollars is going into next generation AI.


The Man behind Stable Diffusion

And it really has the potential to improve humanity.


The Man behind Stable Diffusion

And so that's why with stability AI, basically, we said, can we solve compute?


The Man behind Stable Diffusion

Can we solve funding?


The Man behind Stable Diffusion

And can we bring people together to build cool stuff?


The Man behind Stable Diffusion

And we've actually achieved and managed that when we go live on the 8th of August.


The Man behind Stable Diffusion

And then this will be before or after, I think, hopefully after it will be revealed, but I'm


The Man behind Stable Diffusion

happy to discuss everything that we've done to date to address these and what's coming


The Man behind Stable Diffusion

down the pipeline.


The Man behind Stable Diffusion

So you say solve compute, solve funding essentially means money.


The Man behind Stable Diffusion

So stability AI, what's the source of funding or what's the money flow into this organization?


The Man behind Stable Diffusion

And how is that money spent?


The Man behind Stable Diffusion

So initially, it was primarily my funding.


The Man behind Stable Diffusion

So I was lucky enough to have a good career as a hedge fund manager.


The Man behind Stable Diffusion

Then in 2020-2021, I led the collective augmented intelligence against COVID-19 initiative launch


The Man behind Stable Diffusion

at Stanford to use the COVID-19 data sets and the backing of the WHO, UNESCO and World


The Man behind Stable Diffusion

Bank to organize the world's COVID knowledge and make it understandable.


The Man behind Stable Diffusion

So I've gotten lots of connections.


The Man behind Stable Diffusion

So I pulled them together, primarily my own kind of funding.


The Man behind Stable Diffusion

And basically, what we've done is we've got a 4000 A100 cluster for open source artificial


The Man behind Stable Diffusion

intelligence with the support of Amazon, but no control by them.


The Man behind Stable Diffusion

So that ranks above Jules Booster as potentially the 10th fastest public supercomputer.


The Man behind Stable Diffusion

And Eleuther AI and Lyon have been basically building on top of that some of the most cool


The Man behind Stable Diffusion

models that I've ever seen that are about to be released across modalities.


The Man behind Stable Diffusion

I was about to say, so we've done this as a community to date.


The Man behind Stable Diffusion

The next stage is even more exciting.


The Man behind Stable Diffusion

We're partnering up with countries and leading institutions to take this to the next level.


The Man behind Stable Diffusion

Far more compute, far more funding, and most of all coordination.


The Man behind Stable Diffusion

So that again, intelligence and creativity can be unlocked to build systems, both for


The Man behind Stable Diffusion

countries, communities and for humanity that are open and not closed.


The Man behind Stable Diffusion

Is there a comparison to maybe something that exists?


The Man behind Stable Diffusion

Could it be compared to something like CERN or the International Space Station?


The Man behind Stable Diffusion

What is it that you're aiming for when you say we're going for countries, we're going


The Man behind Stable Diffusion

for collaboration?


The Man behind Stable Diffusion

So we're already partnered with the United Nations.


The Man behind Stable Diffusion

We're doing national level partnerships with, for example, leading groups and institutions


The Man behind Stable Diffusion

from India to Singapore to others, from universities to leading media and conglomerates, telcos,


The Man behind Stable Diffusion

the governments themselves to build national level models and datasets.


The Man behind Stable Diffusion

So we have the plurality of kind of being around this.


The Man behind Stable Diffusion

This is kind of like we kicked it off as CERN, but from a Discord group, probably through


The Man behind Stable Diffusion

AI.


The Man behind Stable Diffusion

So we're involved in the Lyon, OpenBioML and a bunch of these others bringing together


The Man behind Stable Diffusion

really talented researchers.


The Man behind Stable Diffusion

And then my and my team's responsibility was to get them the resources they needed to unlock


The Man behind Stable Diffusion

this.


The Man behind Stable Diffusion

The next stage is a bit more institutional, but we really hope it keeps this kind of community


The Man behind Stable Diffusion

vibe that we've got and this community structure that we built.


The Man behind Stable Diffusion

Community vibe, I think is a good keyword.


The Man behind Stable Diffusion

There are people who just come forward by themselves who want to build things who are


The Man behind Stable Diffusion

quite clearly engaged.


The Man behind Stable Diffusion

A lot of people in Eleuther AI, also people from Lyon.


The Man behind Stable Diffusion

Yet when it, I think, gets more public that there is a lot of money, that there is, you


The Man behind Stable Diffusion

know, funding, compute and so on, there is potentially going to be an influx of a lot


The Man behind Stable Diffusion

of people with a lot of ideas and promises.


The Man behind Stable Diffusion

How do you select who gets access to your resources and what can be done with it?


The Man behind Stable Diffusion

So currently I am GPU emperor, so kind of I decide which projects and things go forward.


The Man behind Stable Diffusion

That's not sustainable.


The Man behind Stable Diffusion

So instead what we're doing is we're, again, without trying to kill the vibe of places


The Man behind Stable Diffusion

like Eleuther, Lyon, OpenBioML and other communities that we've got coming for audio and contrastive


The Man behind Stable Diffusion

learning robotics, et cetera, set up processes by which grants can be given quickly for small


The Man behind Stable Diffusion

research.


The Man behind Stable Diffusion

And then we can rethink about what the bigger runs and things like that are all about.


The Man behind Stable Diffusion

And with a focus and a mission of, you know, what's cool and what's useful for humanity.


The Man behind Stable Diffusion

Stability AI itself, on the other side, you know, we are kind of commercializing these.


The Man behind Stable Diffusion

We are a for-profit entity, but with a mission-based thing, so a benefit corporation.


The Man behind Stable Diffusion

And that will inform some of it, but not all of it.


The Man behind Stable Diffusion

So it's this balance of how do you have R&D and academic and independent, and then how


The Man behind Stable Diffusion

do you productize that so it gets to a billion people?


The Man behind Stable Diffusion

And we've got a very interesting case study that cracks next week around that.


The Man behind Stable Diffusion

And I'll have to discuss with StableDiffusion.


The Man behind Stable Diffusion

What is StableDiffusion?


The Man behind Stable Diffusion

StableDiffusion is the last of this series of kind of diffusion models.


The Man behind Stable Diffusion

It's the one that basically breaks through on quality, speed and cost to enable anyone


The Man behind Stable Diffusion

to create images.


The Man behind Stable Diffusion

So DALY 2 was a fantastic experience.


The Man behind Stable Diffusion

StableDiffusion is about 30 times more efficient and runs on a consumer graphics card.


The Man behind Stable Diffusion

The DALY 2 level image quality.


The Man behind Stable Diffusion

So this was a combination of various groups, such as Conviz from Heidelberg, who came up


The Man behind Stable Diffusion

with VQGAN and latent diffusion.


The Man behind Stable Diffusion

Our lead generator of AI coder, Catherine Krausen, Rivers Have Wings, kind of a whole


The Man behind Stable Diffusion

range of other kind of famous characters within the community to say, how can we build an


The Man behind Stable Diffusion

efficient model that can scale to a billion people to enable them to be creative?


The Man behind Stable Diffusion

And so that release is touch wood on the 8th or 9th of August.


The Man behind Stable Diffusion

And we'll be releasing an open source along with instructions how to run it locally in


The Man behind Stable Diffusion

the cloud and others.


The Man behind Stable Diffusion

So what we've got is, you know, Dream, you see some Galgades there, right?


The Man behind Stable Diffusion

Tesla Roadster on the streets of, where are you Yannick?


The Man behind Stable Diffusion

Zurich, Switzerland.


The Man behind Stable Diffusion

Streets of Zurich, right?


The Man behind Stable Diffusion

You don't even need to dream that up.


The Man behind Stable Diffusion

The streets here are filled with Teslas.


The Man behind Stable Diffusion

They're filled with Teslas, right?


The Man behind Stable Diffusion

Actually kind of DALY 2 is, sorry, my internet's a bit slow.


The Man behind Stable Diffusion

Maybe we'll redo this demo with faster internet.


The Man behind Stable Diffusion

Basically, this generates images in about three seconds on five gigabytes of VRAM, whereas


The Man behind Stable Diffusion

other image models required like 40 gigabytes or 20 gigabytes of VRAM and they're super


The Man behind Stable Diffusion

slow.


The Man behind Stable Diffusion

So now it's my internet that's actually slower than the actual bot.


The Man behind Stable Diffusion

So maybe we'll redo that demo in a bit.


The Man behind Stable Diffusion

Oh, there we see it's coming.


The Man behind Stable Diffusion

So I'm on dial up right now it seems.


The Man behind Stable Diffusion

That gives me nostalgia feelings, I have to say.


The Man behind Stable Diffusion

The line by line rendering of images.


The Man behind Stable Diffusion

Exactly.


The Man behind Stable Diffusion

It's pretty fun.


The Man behind Stable Diffusion

If you're watching this and you're younger than 25, this is what the internet was like


The Man behind Stable Diffusion

in the early days.


The Man behind Stable Diffusion

That's an internet.


The Man behind Stable Diffusion

So there you've got your lovely Tesla in Zurich, right?


The Man behind Stable Diffusion

Cool.


The Man behind Stable Diffusion

This is an image model that we built off Lion 5B.


The Man behind Stable Diffusion

The Lion guys were obviously here a while ago, very close, kind of working with us.


The Man behind Stable Diffusion

Some of them are actually stability employees as well.


The Man behind Stable Diffusion

Taking that 250 terabytes of data, we can press it down to two gigabytes via this diffusion


The Man behind Stable Diffusion

model type of thing.


The Man behind Stable Diffusion

By the time this goes out, probably everyone will be able to play with it locally or in


The Man behind Stable Diffusion

the cloud, et cetera, because we really want to unlock this wave of innovation.


The Man behind Stable Diffusion

Because I think that's how it happens.


The Man behind Stable Diffusion

I don't know if Aluth has made the announcement yet, but GPT Neo and GPT Neo X and J have


The Man behind Stable Diffusion

been downloaded 25 million times now by developers.


The Man behind Stable Diffusion

That can really catalyze ecosystems for development against the more, shall we say, paternalistic


The Man behind Stable Diffusion

instincts of some of the bigger AI players who refuse to release images, model the code


The Man behind Stable Diffusion

or the weights.


The Man behind Stable Diffusion

Stable diffusion is a very interesting one.


The Man behind Stable Diffusion

Because we could have kept it closed source.


The Man behind Stable Diffusion

It's a step forward.


The Man behind Stable Diffusion

It's 30 times more efficient than DALY 2.


The Man behind Stable Diffusion

You can have comparable image quality and you saw the raw output.


The Man behind Stable Diffusion

But why would you if you can instead make it go from millions of people using this technology


The Man behind Stable Diffusion

to billions of people using this technology?


The Man behind Stable Diffusion

That's far more interesting.


The Man behind Stable Diffusion

Again, I think it's the type of thing we need to do, make this technology really usable.


The Man behind Stable Diffusion

Don't think 175 billion parameter language models or 540 billion parameters models are


The Man behind Stable Diffusion

really usable for the vast majority of humanity.


The Man behind Stable Diffusion

You mentioned this open source, closed source, paternalistic and so on.


The Man behind Stable Diffusion

I agree there is a paternalistic element, but there's also a PR and a legal element,


The Man behind Stable Diffusion

right?


The Man behind Stable Diffusion

If DALY 2 was accessible to everyone and so on, and people find, oh, I just need to enter


The Man behind Stable Diffusion

this prompt to make it produce something that's really horrible, that may produce a backlash,


The Man behind Stable Diffusion

Saying, well, these models are clearly not fit for release and so on.


The Man behind Stable Diffusion

What is your opinion if someone comes to you and says, your model produces horrible output


The Man behind Stable Diffusion

here I can show you.


The Man behind Stable Diffusion

What do you say to those people?


The Man behind Stable Diffusion

I would say, of course, humanity is horrible and they use technology in horrible ways and


The Man behind Stable Diffusion

good ways as well.


The Man behind Stable Diffusion

But the reality is, for this particular output, the vast majority of people are creatively


The Man behind Stable Diffusion

constipated.


The Man behind Stable Diffusion

We have been conditioned to consume constantly by social media and big tech giants.


The Man behind Stable Diffusion

They want us to consume more according to their parameters.


The Man behind Stable Diffusion

You see a model like this, we've had three-year-olds use it in refugee camps all the way to 90-year-olds.


The Man behind Stable Diffusion

We're putting it in mental health settings and other things.


The Man behind Stable Diffusion

The benefits far outweigh any negativity.


The Man behind Stable Diffusion

The reality is that people need to get used to these models.


The Man behind Stable Diffusion

They're coming one way or another.


The Man behind Stable Diffusion

And restricting them means that you become the arbiter.


The Man behind Stable Diffusion

As an example, we took some programmers out of Russia because they spoke out against the


The Man behind Stable Diffusion

government there.


The Man behind Stable Diffusion

And some came from the Ukraine as well and we fast-tracked their residency in the UK.


The Man behind Stable Diffusion

You can't use the word Ukraine in DALY 2 because it's political.


The Man behind Stable Diffusion

Then as well, if you type in sumo wrestler, they randomly added to the prompts because


The Man behind Stable Diffusion

they do pre-prompt and post-prompt processing a diversity filter.


The Man behind Stable Diffusion

So you get Asian female sumo wrestlers because they randomly add ethnicities.


The Man behind Stable Diffusion

There's nothing you can do about that.


The Man behind Stable Diffusion

If you want to create a localized version that is more respective to your culture, for


The Man behind Stable Diffusion

example in India, you can't do that because you can't access the model.


The Man behind Stable Diffusion

And they don't have the capacity to let you find a unit.


The Man behind Stable Diffusion

So instead, what they're saying is AI for us and our clients because it's expensive


The Man behind Stable Diffusion

to run these things, not for everyone else.


The Man behind Stable Diffusion

What they're really saying is we don't trust you as humanity because we know better.


The Man behind Stable Diffusion

I think that's wrong.


The Man behind Stable Diffusion

I actually trust people.


The Man behind Stable Diffusion

I trust them to be weird and nasty in some cases, because 1%, 0.1% of people are weird.


The Man behind Stable Diffusion

Many people on this call are weird.


The Man behind Stable Diffusion

I'm weird.


The Man behind Stable Diffusion

But at the same time, like I said, I think that this is positive technology for humanity


The Man behind Stable Diffusion

and it should diffuse because then the pace of innovation to make it beneficial as well


The Man behind Stable Diffusion

as to combat negative uses is far greater.


The Man behind Stable Diffusion

You previously said stability AI employee.


The Man behind Stable Diffusion

So not only do you give grants in terms of hardware and what to run, you do pay people


The Man behind Stable Diffusion

to actually work part time or full time.


The Man behind Stable Diffusion

Can you specify a little bit of what just being an employee at stability AI means?


The Man behind Stable Diffusion

Yeah.


The Man behind Stable Diffusion

So different people need different things.


The Man behind Stable Diffusion

We come from all diverse backgrounds.


The Man behind Stable Diffusion

Some of them needed the equivalent to their jobs at Google or Microsoft when they left.


The Man behind Stable Diffusion

So we pay competitive salaries, high bonuses, and in our contracts, no IP.


The Man behind Stable Diffusion

All the work can be open sourced by any developer.


The Man behind Stable Diffusion

Similarly, we have set it up.


The Man behind Stable Diffusion

So as we run APIs and our models, there's a revenue share for all developers, even if


The Man behind Stable Diffusion

they don't work at stability, who created the models.


The Man behind Stable Diffusion

So 10% of revenue goes to this pool, half of which goes to the creators of the models


The Man behind Stable Diffusion

and data sets, and half of which goes to a communal pool where everyone involved in stability


The Man behind Stable Diffusion

as an employee or otherwise, which I'll come to in a second, basically awards it to the


The Man behind Stable Diffusion

most interesting research so that you can actually have a career from doing interesting


The Man behind Stable Diffusion

research by open source.


The Man behind Stable Diffusion

It doesn't have to be commercial.


The Man behind Stable Diffusion

So the commercial is the running the APIs, the non-commercial is the other 5% of revenue.


The Man behind Stable Diffusion

We also do fellowships.


The Man behind Stable Diffusion

So we're sponsoring a whole bunch of coders such as Lucid Rainsville Wang through GitHub


The Man behind Stable Diffusion

sponsors and we ask, what do you need to be comfortable?


The Man behind Stable Diffusion

We're going to fund 100 PhDs in AI over the next year, and that comes with Compute for


The Man behind Stable Diffusion

Academia, small and large as well.


The Man behind Stable Diffusion

And we hope that will be a community within our communities and across communities that


The Man behind Stable Diffusion

can coordinate global academic research.


The Man behind Stable Diffusion

And we support as well.


The Man behind Stable Diffusion

So for example, we have mental health support, we have grant writers, we have paper writers


The Man behind Stable Diffusion

and other things just to enable people to get on with what's interesting and be able


The Man behind Stable Diffusion

to build in the open.


The Man behind Stable Diffusion

We haven't been in the open until now because we've been building and also because it's


The Man behind Stable Diffusion

quite fun to announce and release all this.


The Man behind Stable Diffusion

But we hope that we can actually build in the open and change some of these incentive


The Man behind Stable Diffusion

structures by unlocking people, be it grants, be it fellowships, be it PhD funding, be it


The Man behind Stable Diffusion

part-time jobs, full-time jobs, or just being members of the community and getting prizes


The Man behind Stable Diffusion

from this kind of pool that will hopefully become very large.


The Man behind Stable Diffusion

We also have a charity as well, and that's where the PhD funding comes from.


The Man behind Stable Diffusion

So charitable arm.


The Man behind Stable Diffusion

What keeps you from becoming like going the same route as let's say OpenAI, any, all these


The Man behind Stable Diffusion

companies from DeepMind, they have it, you know, we want to make AI for everyone.


The Man behind Stable Diffusion

They've been for profit and very close from the beginning.


The Man behind Stable Diffusion

OpenAI actually started out with, we want to democratize.


The Man behind Stable Diffusion

We want everyone to be accessible to give us money and we know what's good for you,


The Man behind Stable Diffusion

right?


The Man behind Stable Diffusion

What keeps you like there's clearly a pull, right?


The Man behind Stable Diffusion

There's clearly demands coming with any money that flows in.


The Man behind Stable Diffusion

It's clearly attractive to sort of keep your, let's say leading position to attract more


The Man behind Stable Diffusion

researchers and so on.


The Man behind Stable Diffusion

How do you prevent yourself from, let's say, succumbing to that pull of going close to


The Man behind Stable Diffusion

or going profit?


The Man behind Stable Diffusion

Well, I think it, you know, OpenAI, one of the founders who's left, I won't mention on


The Man behind Stable Diffusion

this call, maybe we can mention privately said that kind of what we're creating is what


The Man behind Stable Diffusion

he wanted to do when OpenAI was founded.


The Man behind Stable Diffusion

It was just the wrong time.


The Man behind Stable Diffusion

So obviously, you know, they had to scale up compute because you have this kind of stack


The Man behind Stable Diffusion

more layers type thing.


The Man behind Stable Diffusion

And then all the issues that happened in 2019, the on-musk, etc., that basically led to a


The Man behind Stable Diffusion

bailout and then a change in the entire corporate structure.


The Man behind Stable Diffusion

And then a change in focus to become more productized, even though they're not actually


The Man behind Stable Diffusion

product focused.


The Man behind Stable Diffusion

DeepMind had a bit of a different kind of thing.


The Man behind Stable Diffusion

But again, they were the wrong time because what you've seen is these models have lots


The Man behind Stable Diffusion

of promise and they're powerful, but they haven't had that technological diffusion curve,


The Man behind Stable Diffusion

right?


The Man behind Stable Diffusion

What is the killer app?


The Man behind Stable Diffusion

Natural language processing and kind of these large language models.


The Man behind Stable Diffusion

They were tackling a problem I think was already 85% to 90% solved.


The Man behind Stable Diffusion

And now we've gone to 95% solved.


The Man behind Stable Diffusion

And they're large and bulky.


The Man behind Stable Diffusion

Image I think is the killer app.


The Man behind Stable Diffusion

Because when you look at this, it's a wonder for people that they can suddenly create rather


The Man behind Stable Diffusion

than consume.


The Man behind Stable Diffusion

And that's something that's across the board.


The Man behind Stable Diffusion

You know, the comparators are Snapchat or TikTok, where you can create.


The Man behind Stable Diffusion

There's Pokemon Go, you know, Gacha games and these kinds of things.


The Man behind Stable Diffusion

But it'll be integrated into so many different areas that it's got fast enough, cheap enough


The Man behind Stable Diffusion

and good enough.


The Man behind Stable Diffusion

And like I said, this model file that we're releasing only a couple of gigabytes.


The Man behind Stable Diffusion

You know, it can fit on eight gigabytes of VRAM.


The Man behind Stable Diffusion

That's crazy.


The Man behind Stable Diffusion

You know, there'll be bigger models and better models like Imogen, but this inflection point


The Man behind Stable Diffusion

is what makes our business sustainable.


The Man behind Stable Diffusion

It allows us to do things like say, you can work just for open source to our employees.


The Man behind Stable Diffusion

It allows us to do things like revenue share, where we'll be able to attract the best employees.


The Man behind Stable Diffusion

Because if you believe this is going to a billion people, you'll have more than that.


The Man behind Stable Diffusion

And then finally, the structure that we've employed is kind of one whereby we're partnering


The Man behind Stable Diffusion

with various kind of governments and leading institutions so that we build AI for each


The Man behind Stable Diffusion

nation and communities in each nation.


The Man behind Stable Diffusion

So we capture that cultural diversity.


The Man behind Stable Diffusion

So again, it's very community focused.


The Man behind Stable Diffusion

It's very oriented.


The Man behind Stable Diffusion

There's a good business model.


The Man behind Stable Diffusion

We've negotiated massive deals so we can be profitable at the door versus most money losing


The Man behind Stable Diffusion

big corporations.


The Man behind Stable Diffusion

There's a few extra things in there that I can't discuss right now.


The Man behind Stable Diffusion

But we really kind of laid it out to be the right company at the right time to coordinate


The Man behind Stable Diffusion

this all.


The Man behind Stable Diffusion

And then hopefully as this goes, this becomes an independent, more decentralized thing.


The Man behind Stable Diffusion

Originally, we wanted to be web3 with tokens and all that, but you don't need that.


The Man behind Stable Diffusion

You know, you just need to have a good community that keeps you in check.


The Man behind Stable Diffusion

And you need to build in the open and do things in the open, which I hope we'll manage to


The Man behind Stable Diffusion

do over the next year.


The Man behind Stable Diffusion

How can people find you?


The Man behind Stable Diffusion

How can people find your models and work with your stuff?


The Man behind Stable Diffusion

And how can people who are maybe interested in taking part in the community and contributing


The Man behind Stable Diffusion

in some way find you?


The Man behind Stable Diffusion

So we have our website stability AI that will be updated when we launch publicly next week.


The Man behind Stable Diffusion

You know, join our communities at Eleuther AI or Lion or others that we're going to accelerate


The Man behind Stable Diffusion

and really, you know, put a lot more structure around.


The Man behind Stable Diffusion

Open Biomail, Harmoni for Music, Carp for Contrastive Learning.


The Man behind Stable Diffusion

You know, we've got education and many other things coming down the pipeline.


The Man behind Stable Diffusion

Yeah and I think it's just community based.


The Man behind Stable Diffusion

Be active in the community.


The Man behind Stable Diffusion

You will get rewarded with, you know, money and status and all sorts of other things.


The Man behind Stable Diffusion

We do interesting stuff.


The Man behind Stable Diffusion

You want to join stability, there are roles for exceptional programmers to come and help


The Man behind Stable Diffusion

coordinate this.


The Man behind Stable Diffusion

You want your PhD funded, we will announce the PhD funding program in a couple of months.


The Man behind Stable Diffusion

You want to tell us how to do this properly, open to advice.


The Man behind Stable Diffusion

You know, like I don't think we have all the answers, but I hope we're kind of getting


The Man behind Stable Diffusion

there and I think certainly we'll make a difference through this really flexible supercomputer


The Man behind Stable Diffusion

cluster if nothing else.


The Man behind Stable Diffusion

Again, it's a big, big cluster and it's available for the coolest research that can make an


The Man behind Stable Diffusion

impact on humanity and we'll get more.


The Man behind Stable Diffusion

We have far bigger supercomputers lined up as well.


The Man behind Stable Diffusion

So I think that's super exciting.


The Man behind Stable Diffusion

What is the type of person that you're looking for in a contributor and what is maybe a type


The Man behind Stable Diffusion

of person that you're not looking for?


The Man behind Stable Diffusion

So the type of person we're looking for in a contributor are those that believe in open


The Man behind Stable Diffusion

source AI and not open source energy, but open source innovation.


The Man behind Stable Diffusion

You know, like we're bringing this technology to make humanity better.


The Man behind Stable Diffusion

You can make profits, that's fine, right?


The Man behind Stable Diffusion

But I think it should be secondary to just is this going to make a difference?


The Man behind Stable Diffusion

You know, I don't mind if people are corporate, et cetera, but it needs to be people that


The Man behind Stable Diffusion

integrate with the community, can work well with people from a whole bunch of different


The Man behind Stable Diffusion

backgrounds and just are generally inquisitive, that want to push the boundaries.


The Man behind Stable Diffusion

I think some of the biggest breakthroughs we've had have been from non-traditional backgrounds.


The Man behind Stable Diffusion

I don't know if you've interviewed the Eleuther AI founders, none of them have a computer


The Man behind Stable Diffusion

science degree, you know?


The Man behind Stable Diffusion

And yet they kind of managed to achieve such great things.


The Man behind Stable Diffusion

Now obviously there's conjecture for alignment and we're pushing some of the capability stuff


The Man behind Stable Diffusion

there.


The Man behind Stable Diffusion

So, you know, I think what we don't want to see is just people who are just highly corporatized,


The Man behind Stable Diffusion

kind of stuck in one way of thinking and want to see how to make a quick buck out of all


The Man behind Stable Diffusion

of this.


The Man behind Stable Diffusion

You can make money, but so what?


The Man behind Stable Diffusion

We're at this pivotal point where this technology can maximize humanity's potential or it can


The Man behind Stable Diffusion

be corporatized and be used as a method of centralization and control.


The Man behind Stable Diffusion

Which side do you want to be on?


The Man behind Stable Diffusion

Now you can make money on both sides.


The Man behind Stable Diffusion

Is there anything else that you want to get out to people that you want to let people


The Man behind Stable Diffusion

know that we haven't talked about yet?


The Man behind Stable Diffusion

No, I mean, like I said, we've got an amazing pipeline and roadmap that we hope to put out


The Man behind Stable Diffusion

with them, so you know, work on everything from audio diffusion, video diffusion, 3D.


The Man behind Stable Diffusion

I mean, I think in particular, if people want to try and create the metaverse, the Ready


The Man behind Stable Diffusion

Player One one minus the microtransaction or holodeck, we're going to aim to do that.


The Man behind Stable Diffusion

And I would say that probably our killer app, the one that I want to make most, and I'd


The Man behind Stable Diffusion

invite anyone to contact me if they want to build this with me, is I want to destroy PowerPoint.


The Man behind Stable Diffusion

I think the combination of language, image, kind of contrastive and other models means


The Man behind Stable Diffusion

that if we work super hard in a few years, we'll never need to make a slide deck again.


The Man behind Stable Diffusion

Tell the computer, tell it how you want to adjust it, they'll be beautiful each time.


The Man behind Stable Diffusion

And think about how much happiness will bring to the world that way.


The Man behind Stable Diffusion

No more stock images of little drawn people going like, hmm.


The Man behind Stable Diffusion

Very cool.


The Man behind Stable Diffusion

Yeah, you know, dragging and dropping little bits on the slides and refining them.


The Man behind Stable Diffusion

Tell the computer it'll create the slide deck for you.


The Man behind Stable Diffusion

Tell it how you want to adjust it, it'll adjust it.


The Man behind Stable Diffusion

So much happiness brought to the world.


The Man behind Stable Diffusion

I think that's another thing as well, like academia, companies, all these things.


The Man behind Stable Diffusion

I think too many people in our community are unhappy.


The Man behind Stable Diffusion

And obviously, there's a lot of neuro atypical people within our community, right?


The Man behind Stable Diffusion

I'm neuro atypical myself, you know?


The Man behind Stable Diffusion

I want to see how we can have a happier community that supports each other.


The Man behind Stable Diffusion

Because otherwise, there are these big highs and lows and things like that.


The Man behind Stable Diffusion

And I think people focus enough on that.


The Man behind Stable Diffusion

That's what I focus on with my engineers and what I'm trying to focus on the community.


The Man behind Stable Diffusion

Because then people will be more productive, sure, but they'll also be more content.


The Man behind Stable Diffusion

So it sounds a bit fuzzy, but I think it's really important and people don't pay enough


The Man behind Stable Diffusion

attention to it.


The Man behind Stable Diffusion

Wise words.


The Man behind Stable Diffusion

So actually, maybe we should mention one of the projects we have, 7cups.com.


The Man behind Stable Diffusion

It's something that we help kind of accelerate.


The Man behind Stable Diffusion

You can go and you can chat to someone, so you don't even have the pressure of talking


The Man behind Stable Diffusion

to someone online who's been trained in active listening.


The Man behind Stable Diffusion

And we have studies showing it's as effective as taking Prozac, and it's free.


The Man behind Stable Diffusion

And for $150 a month, you can talk to a part-time mental health therapist.


The Man behind Stable Diffusion

So we've got 468,000 volunteers in 180 countries, helping 80 million people each month.


The Man behind Stable Diffusion

So I'd recommend people try that.


The Man behind Stable Diffusion

And then if anyone wants to help me take that data set, you know, with full privacy and


The Man behind Stable Diffusion

everything like that, to create systems that we can better listen and understand each other.


The Man behind Stable Diffusion

Again, that's something that I'd be very interested in talking to people, because I really want


The Man behind Stable Diffusion

to help people help people.


The Man behind Stable Diffusion

Awesome.


The Man behind Stable Diffusion

Emma, thank you very much for being here.


The Man behind Stable Diffusion

Very exciting.


The Man behind Stable Diffusion

I'm looking forward to the release next week.


The Man behind Stable Diffusion

Maybe it's already out once this is out.


The Man behind Stable Diffusion

Yeah, thanks a lot for being here.


The Man behind Stable Diffusion

And good luck to the Endeavor.


The Man behind Stable Diffusion

Thank you very much, Yannick.


The Man behind Stable Diffusion

Pleasure.


The Man behind Stable Diffusion

Awesome podcast you've had.


The Man behind Stable Diffusion

I've enjoyed listening to it.


The Man behind Stable Diffusion

Thank you.


The Man behind Stable Diffusion


Neural Networks from Scratch - P.3 The Dot Product

What's going on everybody and welcome to part three of the Neural Networks from Scratch


series. In this video what we're going to be doing is diving deeper into I guess sort


of cleaning this up but also transitioning from very raw kind of Python list code into


using vectors and matrices and stuff like that which is the actual standard of doing


deep learning with really any programming language. So that's what we're going to be


getting into. First I just want to bring up a couple of things. One is I've set up a repository


for Neural Networks from Scratch in X. So I'll put a link in the description but it's


also pretty simple github.com slash syntax slash nnfsix. And I found that a lot of people


were saying they were indeed following along in another programming language besides Python.


So I set up the following repository both for if you want to contribute so if you are


following along and you've got each video part and you've been coding it if you want


to share it you can share it here. Also if you're trying to follow along in another language


and you just want something to reference you can also do that here. So check that out.


Again I'll put a link in the description. Also thanks to everyone who has contributed


and supported the book. We're almost to like two thousand total people on the book which


is just crazy. That's really awesome. I really appreciate all the comments and questions


suggestions the edits all that stuff in the document. That's all going really well and


to put two thousand people into the same document can certainly be hectic and it's been pretty


good. So thank you guys very much. So let's go ahead and jump into the code for today.


So there's a couple of things that I want to do. First of all we want to simplify this


code and I'm going to save you guys. You don't have to follow along coding this. I just kind


of want to show moving from here to more dynamic code before we kind of ditch at least this.


So you can follow along for the first part I suppose. But anyway what we're going to


say here is weights equals a list and then basically just copy and paste these into here.


So it's going to be a list of lists in this case. And we'll just take this, this, copy


paste, come down, do this last one here, paste. And then we're going to do the exact same


thing for bias. So we're going to say biases because now it's multiples. We're going to


say two three zero point five. I'll fix my standards there. And then we can delete this


and then delete this code and boom I've already actually coded this code. So if you were going


to be using weights and biases like this soon-ish but not even really. In fact I'm probably


going to go back. We're going to code just a single neuron and then we'll go to the three


neurons with four inputs all in this video. But we're going to use a lot of the code so


I'm probably going to be editing and undoing soon. Anyway this is one way that we can simplify


and make much more dynamic. So what we're doing here is we're going to zip together


weights and biases to start. And if you don't know what zip does it just kind of combines


two lists into a list of lists element wise. So in this case the first element of this


zipped list or the zeroth element of the zipped list will be another list of the zeroth weight


and the zeroth bias and so on. It just combines two lists element wise. So we're going to


iterate over that and then we're going to do input and the weight and then we're going


to zip those together. We're going to say the neuron output plus equals the inputs times


the weights and then at the end we add the bias, append it together, we can run that


and that is the output that we expected. So yeah that's a cleaner more dynamic way of


doing inputs times the weights plus the bias for a layer in Python. Now we're still going


to transition to doing this in with NumPy and doing vectors and matrices and stuff like


that but this is a cleaner way and if you're following along in some other language obviously


the answer is to use loops. So now the next thing I want to touch before we get into the


NumPy stuff is we're still getting a lot of questions on weights and biases so I just


want to clear that up as quickly as possible. So weights and biases are these knobs that


I say we tune but really down the line it's going to be the optimizer that tunes these


values in an attempt to fit to some data and because we've got thousands or hundreds of


thousands or even millions of these tunable parameters we can fit so many things but the


way that weights and biases are impacting a neuron's output it's a different way. So


you could in theory have a neural network, I think it would be hard to have one with


just biases but even that's probably possible for some simple tasks. You can also have a


neural network with just weights, you don't have to have weights and biases necessarily


but they're very helpful and biases are almost never going to negatively impact anything


so you want to have both and just real quickly I just want to drive home why are these different.


So imagine you have some value and we're going to say some value is we're going to make this


0.5 and then we're going to say you've got a weight and the weight will be 0.7 and for


now we'll do the same weight or the same value for a bias. So in this case what is a weight


going to do? Well the weight is going to be the weight times you know some value times


that weight right it's a multiplied thing. And then what's the bias going to do? So let's


go ahead and run that first actually so we can see negative 0.35. Now if we want to do


the same thing for bias let's go ahead and print some value plus bias run that and we


get basically 0.2 hashtag programming problems. Anyway so we have this obvious floating point


issue that's probably most programming language are going to have some issue like that but


anyways you get the idea. So what happened was we converted from like so weight was a


multiple of this negative so it stayed negative whereas bias what bias has done is it has


offset the value. So depends on what you're trying to do but at some point we're going


to get into activation functions and these are functions that kind of determine that


final output before it becomes an input to another layer or maybe even the final output


to your network in general. So what's happening here is they're just doing two different things


the weight is just changing like the magnitude the bias is actually offsetting it it's no


different than the bias in y equals mx plus b for the equation of a line. So in this case


what we've seen is we could go from a negative to a positive using bias but we could of course


go from a negative to a positive using weight right we could make this a negative 0.7 right


and we could run that and wow we got to a positive but this would be a problem because


what if we started with a positive value well obviously it becomes a negative at the end


but this whole time whatever some value was after bias it stayed a positive value because


all it does is offset. So these are just two different tools that help in two different


ways and that's it. So anyways I think this will make more sense as we especially get


into activation functions and like rectified linear it should be unbelievably obvious what


bias can do for you but anyway just for now hopefully you can see that there are two different


tools doing two different things and if we have millions of these little tools that we


can sort of tweak we can change the overall kind of outputs. Now with that in mind let's


get into transitioning into real more real mathematics rather than this stuff so we're


going to get into NumPy now and start talking about vectors and matrices and dot products


and all these things and we're going to start by talking about shape as boring as that might


sound if you've ever worked with any deep learning frameworks I'm willing to bet that


you have hit errors regarding shape I know that's one of the most common questions I