matrix you minimize there's two results one is you minimize cross-entropy with explicit weight DK you get the quasi norm second one is you minimize cross-entropy with TD no there's no optimization these are statements about global minimum okay the first statement is a statement about global minimum the second statement is invoking this following theorem this


says that you convert a force or the optimal point of a problem an SVM where this is the Shatan clause anymore so the inductive bias of having deep that's in a linear network using logistic loss is getting closer to is a crank this is about first-order okay so gradient descent the algorithm is involved here it's about gradient descent you run gradient descent

On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization


hi there today we're looking at supervised contrast of learning by people from Google research and MIT now this paper proposes a new loss for supervised learning and you might recognize that this is a big claim so for ever now we basically use this cross-entropy loss in order to do supervised training of neural networks this paper proposes to replace that with

Supervised Contrastive Learning


the supervised contrastive loss and let's jump straight into the results here they say our supervised contrastive loss outperforms the cross-entropy loss with standard data augmentations such as Auto augment and Rand augment so these are some of the previous state-of-the-art data augmentation techniques used together with the cross entropy loss and they say there are

Supervised Contrastive Learning


supervised contrastive loss outperforms them you can see here on image net which is the biggest vision benchmark or the most famous one this new loss the supervised contrastive loss outperforms these other methods by something like a percent one percent is a big improvement on image net right now so they claim it is a big claim right you recognize if this is true this could be a

Supervised Contrastive Learning


game-changer basically for all of supervised learning and supervised learning is really the only thing right now in deep learning that works so it could revolutionize the field but so here's the but it is actually not a new loss to replace the cross entropy loss and that's they do they do come about this pretty quickly some and you don't think they're they're dishonest or lying

Supervised Contrastive Learning


or anything here but it is it is sort of if you start reading you like what this is a new loss it is not it is a new way of pre training the network for a classification task and so let's look into this so if you look at what does what does mean to build a classifier in this is what you usually do this is supervised cross-entropy training you have an image and the image here is of a dog you put

Supervised Contrastive Learning


it through your network and you obtain a representation so the representation here R is this last layer or the second-to-last layer and you put that through a classification layer and then a soft Max and what you get as an output is basically a probability distribution and let's say you have three classes here there's dog there's cat and there's horse and let's say the network doesn't

Supervised Contrastive Learning


yet isn't yet trained very well so the probability for dog here is fairly low so this is basically what the network thinks of that image like which class does it belong to with what probability I also have this label right here so the labeled dog for that image what you do with that is you do a one hot vector so that would look like this so the one is at the position where the correct class

Supervised Contrastive Learning


is and then the cross-entropy loss takes all of this and does the following there's a sum over all your classes in this case you have three classes and let's call these the labels l and you want to always take the label of the class times the log probability that the network thinks belongs to this class so you can quickly see that this if the label is 0 so for all the incorrect

Supervised Contrastive Learning


classes that means this entire term drops away and only if the label is 1 so only the correct class that will result in the log probability of the class where the label is the correct label right so in order to make this a loss you actually have to put a negative sign in front of here because you want to this so this entire thing reduces to the log probability of the correct class

Supervised Contrastive Learning


this is what you want to max semi's there for you if you want to minimize something you need so you minimize the negative log probability of the correct class which means you maximize the probability a a if you've never looked at the cross entropy loss like this it is important to notice that you're gonna say hey all this does is pull this here up right and it doesn't

Supervised Contrastive Learning


do anything to the other ones but you have to realize that this softmax operation since this is a probability distribution all of this is normalized to sum up to one so implicitly you will push these down through the normalization right so what this does is it pushes the correct class up and it pushes the other classes down so this to look at this is going to be important

Supervised Contrastive Learning


later because if you look at what this representation here does so again you have the network produces a representation here this is 2,000 dimensional and then it does it adds on top this classification layer this classification layer is simply a linear layer and then a softmax on top so how you have to imagine this is that there is a representation space this 2010

Supervised Contrastive Learning


shell space and the representations are made in such a way that the labels such that sorry let's have three classes here the representations are made in such a way that a linear classifier can separate them correctly right so here this would be like a boundary and then this would be another boundary and this maybe would be another decision boundary so you can see that the linear

Supervised Contrastive Learning


classifier can separate the classes well that is the goal if you use this soft Max cross-entropy loss that is implicitly what will happen in the representation space W all the cares about is that the classes are on one side of the decision boundary and everything else is on the other side of a decision boundary so if if you have the network isn't trained very well at the beginning and

Supervised Contrastive Learning


you maybe have a sample of the green class here it will push the network such that the representation of that sample will go on to the other side of this decision boundary and it will push the decision boundary at the same time to make that happen more easily right so it will optimize all of this at the same time that's what you do that's how you optimize representations so this work

Supervised Contrastive Learning


here and another work has said wouldn't it be great if the representation and decision boundaries weren't just trained at the same time for this but we learn good representations first such that classifying them becomes very simple and in essence what this paper says is if we have a representation space W shouldn't images of the same class shouldn't we just make them close together you know

Supervised Contrastive Learning


so without caring about decision boundaries we just want them to be close to each other and we want them to be far apart from other classes if that happens you can see that a linear classifier is going to have a very easy time separating these classes later so that's exactly what this paper does it has a pre training stage and a training stage so in the pre training stage this is

Supervised Contrastive Learning


over here su provides contrastive in the pre training stage it simply tries to learn these representations right like over like down here such that without the decision boundaries class think images of the same class are close together and images of different classes are far apart which notice the the subtle difference right to the cross-entropy loss where you just care about them

Supervised Contrastive Learning


being on one or the other side of a decision boundary and in stage this so this stage one and then in stage two and there is where where it comes in you basically freeze the network so you freeze these weights down here these are frozen you don't train them anymore all you train is this one classification layer so the represent actually freeze also the representation layer here you

Supervised Contrastive Learning


only train the classifier on top in stage two but you train it using soft Max and using the cross-entropy loss so you you train the classifier in the old cross entropy way using just normal supervised learning the difference here is that the stage one free training is is what's training the network and the cross entropy dose only trains the classifier right so let's look at how

Supervised Contrastive Learning


this pre training actually worked what is using what it's using is a method called contrastive pre-training now in contrastive pre training and they have a little diagram up here what this does is if you look at the classic way of doing contrastive pre train you have to go to the unsupervised pre-training literature people have kind of discovered that they can improve a neural network by

Supervised Contrastive Learning


pre-training it first in an unsupervised way and this is also called some of these methods are called self supervise so the advantage here of self supervised or unsupervised pre training is that you don't need labels what you want to do is simply to make the representation space somewhat meaningful right so you simply want the network to learn representations of images that are

Supervised Contrastive Learning


somehow meaningful right that are there and here's how you do it so you want to take an image like this dog here and then you want to randomly augment this image which just means you want to produce different versions of the same image in this case down here this is a random crop it's cropped about here it's still the same image but it's a different version of it in the case

Supervised Contrastive Learning


here you can see that it's flipped left right and the brightness is slightly increased so these are just different versions of the same image and what you also want are what's called negatives negatives are simply different images from your data set right for example this or this or this you don't care as long as they're different right you just sample a bunch and what you

Supervised Contrastive Learning


want so you're you're embedding space and they make it big a deal here that they are normalized and that seems to work better but this is not necessary for the idea to work the big idea is here that if you have an image right here let's say this is this is the dog and the blue dots here are the augmented versions of the same dog and the green dots are all the other images in the

Supervised Contrastive Learning


data set what you want is that the all the images that come from the original same image are pulled close together and everything else is pushed apart right so that's why these are called positives and these are called negatives so the contrast of training basically means that you always want to have a set that you pull together in representation space and assets called the negatives

Supervised Contrastive Learning


that you push apart so the network basically learns about these random transformations that you have here the network kind of learns what it means to come from the same image it learns to be robust to these kind of transformations it learns about the data in general and how to kind of spread the data and embedding space with these transformations so this usually ends up

Supervised Contrastive Learning


in a pretty good representation space and people have been using this in recent years in order to gain significant improvements now the problem here if you specifically do this to pre train a classifier is the thing they show on the right so on the left here you have a picture of a dog right but if you just do this self supervised you do it without the labels so it can happen

Supervised Contrastive Learning


that this image here shows up in the negatives but it is also of a dog right and now this image here is going to end up maybe being this image here and you see what happens to it it's a green one so it's gonna get pushed apart and this is going to make the entire task for the later classifier much harder because if they are pushed apart from each other how is a linear classifier going to have

Supervised Contrastive Learning


them on the same side of the decision boundary while having everything else on a different side right so the the task here is implicitly making the task for the later classifier harder by pushing apart samples that should be of the same class and so this is this is not happening if you introduce a labels to the pre training objective that's what they do the supervised contrast

Supervised Contrastive Learning


objective now you still all you want to do is here we're going to draw the same embedding space and we're going to draw this original dog image and we're going to draw the Augmented version of the original dog image but now we also have the following we also have these images which are images of the same class so we're going to put them in black here and let's say the augmented versions

Supervised Contrastive Learning


around them in smaller black thoughts augmented versions of those right you can augment them as well and then you have the negative samples and the negative samples are not just any images but just images of different classes so you just go over your mini batch and all everything that's of the same class we could becomes positives including their augmentations and everything that is not

Supervised Contrastive Learning


in the same class becomes negatives and also you can augment them as well so now we have a bunch of things in our embedding space and our objective is going simply going to be again we want to push away all the images that not of the same class as our original as our red original image which is called the anchor so all of this needs to be pushed away but now we want to pull

Supervised Contrastive Learning


together all the Augmented versions of the original image but also we want to pull together all of the other images of the same class including also their augmented version so all of this is going to be pulled together so not only does the let work learn about these augmentations which again for this idea the augmentations aren't even necessary the network there learns a

Supervised Contrastive Learning


representation space where images of the same class are close together which again is going to make the task of later linear classifiers that needs to separate this class from other classes very very easy and again the other images aren't just going to be pushed away but if they're from the same class let's say this and this image are from the same class all of those are going to

Supervised Contrastive Learning


be pushed apart from a red dot but by themselves being pushed together to their own cluster here of their own class I hope this makes sense and I hope the difference to the cross-entropy objective is sort of clear the cross-entropy objective simply from the beginning just cares about which side of the decision boundary iran while this pre-training objective first cares to

Supervised Contrastive Learning


put things close together that are in the same class and then the decision classifier will have a much easier time the reason why this works better than the because because it's not entirely clear from the beginning that why this should work better because it's working with the same information it's just because people have generally found that these pre-training contrastive

Supervised Contrastive Learning


pre-training objectives they just are somewhat better at exploiting the information in the data set then if you just hammer on hammer with the contrastive sorry with the cross-entropy loss from the beginning so but it is not fully explained yet why this works better be as it's working with the same data again the difference here is that the previous methods of contrastive pre-training the

Supervised Contrastive Learning


self supervised ones they did not have access to the labels and the advantage of that is you can have a giant database of unlabeled additional data that you do the free training on whereas here we do the pre training including the labels so here the label dog is an intrinsic part because we need to know which of these samples we need to pull together but that also means we cannot leverage the

Supervised Contrastive Learning


may be that we have more on labelled data and unlabeled data is pretty cheap to obtain so that's the advantages and disadvantages here so this new loss so they they do compare this here and usually in these contrastive objectives you have somewhat like two encoders want to encode the the anchor and want to encode the Augmented versions and this one is like a momentum with shared

Supervised Contrastive Learning


weights and so on it all of this isn't really important if you want to look into that look into papers like momentum contrast or I did one on curl for reinforcement learning I think the the general gist of it is clear so they compare the formulation of their loss to the self supervised one usually it takes the form of things like this so the one is the the anchor here and then the ZJ I

Supervised Contrastive Learning


would be the positive example and you see here that the inner product between the anchor and the positive example sorry about that the inner product should be high because here the loss is the negative of whatever is here so if you minimize the loss you say I want the inner product between my anchor and whatever is the positive sample to be high and all everything else here which includes the

Supervised Contrastive Learning


thing on the top but it also includes everything else I the inner product to be low and which is exactly the thing where you push you pull together the positives and you push apart everything else the that is the standard objective that you had before they they extend this but looks almost the same so compared to the unsupervised objective now first of all they extend

Supervised Contrastive Learning


this such that you can have more than one positive sample now this is also possible in the unsupervised way so they just augmented by this and they also now this is the crucial part they include the labels into the pre turning objective so they say everywhere where I and J have the same label should be maximized in the inner product so should be pulled together while everything else

Supervised Contrastive Learning


is being pushed apart yes so they say we generalize to an arbitrary number of positives and I also say contrastive power increases with more negatives I think that's just a finding that they have that when they add more negative so when they increase the batch size that contrastive power increases they do analyze their gradient which I find it's pretty neat you can already see that if

Supervised Contrastive Learning


you formulate a loss of course the gradient is going to go in the negative direction but they make it clear that if you look at the gradient for the positive cases what appears is this one - pIJ quantity and the pIJ quantity is exactly the inner product between I and J normalized of course so if you minim so the gradient is going to point into the negative direction of that for the

Supervised Contrastive Learning


positives which means you you're gonna pull them together and it's going to push into this direction with for the negative classes which means you you push them up and they also analyze what happens a in with relation to hardness so they say there are two kinds of if you just look at the positive samples there are two kinds there are easy positives where the

Supervised Contrastive Learning


network has already learned to match them closely where the inner product is almost one if you look at them that means the pIJ quantity is large right because that is basically the inner product and you look at this term this term is exactly what we saw in the gradient then you see that this here since this is one this entire thing is zero this is all so highs this is close

Supervised Contrastive Learning


to one so this entire thing is zero this is almost zero but if you have a hard positive where the network hasn't learned yet to align the inner product properly or align the representation properly then the angle between the things again these are normalized the angle is they are approximately orthogonal so the gradient magnitude is going to be this here is going to be

Supervised Contrastive Learning


approximately 0 so this is close to 1 and this here since this is also 0 is also close to 1 so this is going to be larger than 0 which means that their loss focuses on the examples that are that the network cannot yet represent well according to their objective which makes sense right first of all but second of all it that is exactly the same thing as in the

Supervised Contrastive Learning


cross entropy loss if you if you look at the cross entropy loss and you have a situation where the network is really good already for a given sample so it already puts a dog into the dark class then the the gradient will not be pulling much for that sample it might mainly focuses on where you're still wrong so it is like I appreciate the analysis but it is not a notable

Supervised Contrastive Learning


difference I think what they want to show is that their loss if you do gradient descent really does what it is supposed to do namely first of all it does this polling together pushing a part of inner products for the positive and negative samples and it mainly focuses on samples where you not yet have found a good representation to align them with others it focuses on

Supervised Contrastive Learning


pairs that are not yet correctly close or together or far apart they also connect this to the triplet loss where they can show after some approximation that if their loss only has one positive and one negative sample it is going to be proportional to the triplet loss the triplet loss is basically where you have an image and you find one positive I think there's going to be of the same

Supervised Contrastive Learning


class right here and you find one negative of a different class and you try to push those apart while pulling those together the problem here they say is the problem of hard negative sampling in order for this to make sense you need the negative sample to be what's called a hard negative sample so this the call is hard negative mining because you only have one negative sample you better make

Supervised Contrastive Learning


this something where the network can learn from right and if it's too easy the network can't learn anything and they're more thereby you have the problem of hard negative mining where you often have to filter through your mini batch or even through your data set to find a good negative sample to go along with this pair of positive samples but I don't I don't really see how their

Supervised Contrastive Learning


method except that you know it has a bunch of positives and negative samples except for that which I guess you could also apply to the triplet loss there's not really a difference here again your if your method is a contrastive method you do have the problem that if you simply sample at random your negative samples are going to be become easier and easier over the training over the

Supervised Contrastive Learning


course of training and you get the problem of at some point you're gonna have to do actively sample hard negatives I think this paper just gets around it by having huge batch sizes so yeah but again they do get state-of-the-art on imagenet for these types of networks and augmentation strategies and they do look at how their loss appears to be more hyper parameter stable so if they change

Supervised Contrastive Learning


out the augmentation if they change the optimizer or the learning rate you can see here that the spread in accuracy is much smaller than for the cross entropy loss except here but it is it is hard to compare variances of things that don't have the same means in terms of accuracy so take this on the right here with a grain of salt they also evaluate this on corrupted

Supervised Contrastive Learning


image net so there's an image net data set where you it has several levels of corruptness of the data set and you can see your accuracy goes down but the accuracy for the cross entropy loss goes down faster than for the supervised contrastive loss you see they start together like this and they go further apart now it is not clear to me whether that's just an effect like if you just

Supervised Contrastive Learning


trained a supervised contrastive loss also to this level whether it would fall off at the same speed or whether because it is the supervised contrastive loss it would kind of match that curve it's not clear whether that's really an effect of the difference of the losses or is just an effect of the fact that they aren't the same accuracy to begin with again this kind of shifting you can't really

Supervised Contrastive Learning


compare things that have different means in the first place but that's it is an interesting finding that their method is more stable to these corruptions I just want to point out at the end their training details and just highlight they train for up to seven hundred epochs during the pre training stage which is I think standard but mad and they trained up models with batch

Supervised Contrastive Learning


sizes up to 8192 so you need like a super TPU cluster to run these kind of things and I am never exactly trusting of numbers like this even though it's it's kind of a good improvement it is still like a 1% improvement and in these small numbers I feel I just feel the there might be sir there might be a big effect that things like batch sizes and how much you put into computing how much

Supervised Contrastive Learning


compute you put into it and what else you're doing there might be so much influence of that that I first want to see this replicated multiple times across the entire field before I'm going to really trust that this is a good thing to do alright so I hope you like this if you're still here thank you consider subscribing if you have a comment please leave it I usually

Supervised Contrastive Learning


hi there today we're looking at manifold mix-up better representations by interpolating hidden states by because Verma a tall number of big names on this paper as you can see and I also saw this at ICN also I was intrigued by it they proposed manifold mix-up which is sort of a regularizer of neural networks specifically of supervised learning and it's actually a pretty simple concept

Manifold Mixup: Better Representations by Interpolating Hidden States


and they kind of show that it has some nice properties and outperforms of the regularizer z-- so what's the problem the problem is that if you look at this spiral problem here which is often kind of used to to show properties and neural networks what you have are blue points and the blue points on our one class and the red points are another class you see the two classes here are in this kind of


spiral pattern it's just the data space is just two-dimensional so you see here this is one class this is the other class this is pretty difficult for a model to learn because of course the easy models would be like linear classifiers but there's no way to like put a line through this such that one classes on one side mostly so neural networks if you train them they will


give you something like you see here they will try to kind of bound the regions with the red points from the blue points but then gets you know is it there's some weird things like here is a weird thing here is a weird thing so you'd imagine a correct model would actually classify this area as blue but the the neural network has no concept of of let's say that the spiral should


continue that thus it it simply sees our here's blue here's blue here's a bit of a gap in the training data so yeah it in this case it assigns a red class to it this is one problem that the decision boundaries are rather not say squiggly and irregular and the second one if you look at the actual colors full blue means very confident blue class full red means


very confident red class and in between you kind of see going into the the white so if you look very closely I can actually zoom in more here if you look very closely you'll see that the blue gets lighter and lighter until it reaches white and from here the red goes lighter and lighter until it reaches white and white means not confident white means like 5050 they see the that


area of not confident is actually very small right if you consider a point here is actually still very confident that it's a blue point and the area of Nan confidence is very small even though maybe as as humans we would judge like a relatively large band in the middle to be not confident like if we get a point like this and the third problem is that you can see in multiple locations like


here or here or here that the decision boundary is very close to the data points unnecessarily close so especially if you look here the decision boundary could be much more optimally placed probably something like this right given the training data but the neural networks because the only C training data they they have no basically no incentive to do this all right one might


think of you know something like a support vector machine that actually has an incentive to to put the decision boundary away from the from the training data but the neural networks currently there are not SVM's they're basically logistic regressions and as such have no no incentive to do this so this these are the problems the other problems are this is the input space if you look at


the hidden space so they build neural networks specifically they have like the 2d input and then that goes through a bunch of layers and then at one point there's a bottleneck layer was just two hidden nodes and then I guess that goes again and then it goes into a classifier so in this bottleneck layer they analyze the hidden representations of the data points and


in this case for this spiral dataset what happens is so in red you see again the red classes in blue the blue class it's 2d so you can plot it what it does is it bunches up the hidden representations fairly fairly so it bunches them kind of up it spreads them out in directions here here here most are bunched up here and it does these kind of weird arrangements here with the


pockets of those and of course the neural network is powerful enough such that it can actually you know separate all of this from each other but it's not ideal and the black dots they represent kind of points in between or points from the input space that are not part of the training data so they say they sample uniformly in the range of the input space you see that the black dots are


all over the place right some are confidence blue some are confident red some are like somewhere right what you would expect from a good model is that if you input something that's kind of in-between or not really sure not even part of the input distribution that it assigns like a low confidence to it that it says well I'm not sure about this this must be somewhere in the middle so


just to jump and jump forward to the results what does manifold mix up do without knowing what it is in the same data set it gives you a picture like this you see the decision boundaries are much more smooth right the region of no-confidence or of low confidence indicated by the light color is here is much larger and also the decision boundary here we had specifically this data point here you


see the decision boundary is pushed away well you could argue about that particular point but the decision boundary is generally pushed away from the data points you also see no more kind of these squiggles here it doesn't happen in in here also if you look at the hidden representations the hidden representations now are spread out the classes are bunched up so not all the


points are bunched up but the the points of individual classes are bunched up together and the randomly sampled points are in the middle as they should be si only confident red is down here confident blue is up here and everything in between is unconfident and third if you look at the singular value decomposition of the hidden layer and that's kind of a measure of how spread


out in the different dimensions a dataset is you see that the manifold makes up here in green it concentrates or it it real owers the singular values of the kind of lower indexes so the first singular value is large which means that there is like a dominant direction in the in the data and this is done for each class separately as I understand it it puts a lot of weight on


the first singular vector and then it pushes down the contributions of the other singular vector which means that the data set that is analyzed is is concentrated in two fewer directions of variance this is layer one and here is layer three means so you see it happens in both that the manifold makes up compared to the baseline model does this so now you might ask what is manifold


mix-up it's actually a pretty pretty simple concept right here is another comparing it to other kind of regularization techniques and showing that none of them really does this so manifold mix-up is this basically what you do is when you train a neural network you have input data and you take mini batches of input data specifically you take two mini batches x and y and X


prime Y Prime all right and then what you do is if I have the draw the neural network here so here is the inputs like a picture of a cat [Music] it goes through layers right and then what you do is you say at some particular you say stop stop right you take the representation up you and you do this with two different mini batches so here is this is cat one down


back here is cat two dog that's a captain you pass it in right here you take it out here you pass it through the network and you take it out so you now have two different forward paths of two different mini batches and then you define a lambda and I guess they randomly sample a lambda in zero one right in the range of 0 1 so this is a mixing coefficient and then you mix you


say lambda times hidden representation of batch 1 plus 1 minus lambda of hidden representation of batch 2 and that is what you pass through the rest of the network right so basically you forward propagate to different batches until a certain layer here then you mix them with a random coefficient and then you pass it through the rest and then the only thing you also have to do is then


at the end if you think of the labels of these two things you want to mix the labels in the same fashion so you want to mix lambda times y of batch 1 plus 1 minus lambda of Y of batch 2 and and then this is your training signal for whatever comes out here right so it's it's um these are these are one hot labels so if it's class three its zero zero one zero zero


and if Y two is class five its zero zero zero zero one and then you simply mix the two alright and that becomes your training signal so in a practical example if let's just have a mini batch size of one so just one sample if this is cat and this is dog you would pass them forward right you would mix so in the hidden representation it would kind of become a cat dog maybe you do it


50/50 but then you would also mix the labels of cat and dog 5050 and tell the net well this is a mixture of 50% cut 50% dog and then you would train the network to predict that 50/50 coefficient so they do this the question is at which layer do you do this and they simply I think for each mini batch sample one hidden layer at random they might have someone waiting or something


but the way they describe it is they simply sample one layer or mean per mini batch and then do the mixing there and then you can actually backprop through everything everything is differentiable this mixing is differentiable so you can backdrop through any of everything and there's even you know a kind of an engineering trick to only use a single mini batch by mixing it with itself so


that's that's pretty neat so this is manifold mix up as you can see here is the that's kind of the description you mix the hidden representations with lambda and you mix the labels with the same lambda and that will become your actual training signal all right so they give some theory to it that it flattens representations and specifically they say under some


conditions namely if the network is large enough so if the dimension of the hidden representation is of a certain size then if you optimize this manifold mix up like if you optimize over every London over the entire training data set what you will end up is actually a linear rally near function of the input this is not too surprising that if you because what you do is you mix linearly


this mixture happens in a linear fashion so if you optimize for and you not only optimize for the training set but you optimize for every possible mixture of the training set a linear mixture your minimization your minimizer function will actually become a linear function it's not surprising but they have a formal proof of this and they also have a proof that if certain assumptions are


given then the minimizer's if you apply the minimizer's the hidden representations will actually fall on a low dimensional subspace which is also not surprising but it's kind of the theoretical an analogue to what they show with the singular value distribution that it basically suppresses low singular values that means the data set is much more into a single direction the hidden


representations sorry all right so this the theory part is you can you can read it if you if you want to it's yeah it's it's - the results are to be expected I would say from what they do and the last thing they give a pictorial example of why none fold mixup flattened representations so both of these things the fact that the minimizer's will become linear functions


and the fact that the singular value spectrum is more concentrated on the first thing you'll evaluate a shion's are flattened and here is a pictorial representation so in this case what happens if you if you basically have these four data points a 1 a 2 B 1 and B 2 where a 1 and a 2 are blue class and B 1 and B 2 or red class and if you now look at an interpolation point between


the two so if you look at this interpolation point between a 1 and B 2 what happens is that in this case this should be 50/50 blue and red but if you now look at the points that it where it's not interpolated on this is very close to a 2 in this case it probably should be more like 95 blue and 5 red do they say here well if you use manifold mix up to learn the network what you'll


actually do is you say ok actually this hidden representation needs to be pushed outward and you will achieve something over here where any mixture of two points of the opposite class will actually give you a 50/50 so all the midpoints here will give you a 50/50 mixture between the labels which basically means what you end up with is a line between this data and this data


and it means that basically the network becomes more linear and the representations become more flat because flat is the optimal if you distribute our flat all the distances to the line are the same and this objective is optimized and this is basically my my kind of biggest problem with the method is that it it kind of mixes the input with a linear function where we know


that that is kind of not the shape of the true data manifold the input manifolds as you can see here the input manifold here isn't linear or flat it's actually very very tangled and we know that neural networks as you continue in the layers will flatten those representations because ultimately at the end it needs to classify the dataset linearly because the last layer is a


softmax layer but the the idea that you could apply this to any layer seems a bit shady to me of course it works and they show it works and it's really nice that it works but apply applying this to low layers and neural networks seems a bit not principled to me so I think this is not the end of the story of this line of work and there is kind of more that can be done in a more principled fashion


but in any case they show that this actually works in terms of performance on generalization on kind of standard data sets so they have results on C 4 10 and C 4 100 which are famous image data sets and they show that they have a regularizer outperforms others and they also show that they can withstand one-step single step adversarial attacks more it kind of better so they have a


better perform performance against single step adversarial attacks after regularizing mostly again giving kind of an idea that the if you push if you push it if you have a two points this is X this is X X 1 X 2 they're of different classes if you put the decision boundary really close to X 2 then an adversarial attack can simply move the point across the decision


boundary with a very small step but if you actually have the decision boundary pushed away from both data points then the an adversarial attack must go a very long way to the decision boundary and thus if you limit the size of adversarial attacks which is what you usually do you can maybe not reach this decision boundary and thus you mitigate some of the problem so it's pretty cool


you've used a pre-trained model to make predictions that gives you great results if you want to classify images into the categories used by the original models but what if you have a new use case and you don't categorize images in exactly the same way as the categories for the pre trained model for example I might want a model that can tell if a photo was taken in an urban area or a rural

Transfer Learning | Kaggle


area my pre trained model doesn't classify images into those two specific categories we can build a new model from scratch for this specific purpose but to get good results we need thousands of photos with labels for which our urban and which are rural something called transfer learning will give us good results with far less data transfer learning takes what a model learns while

Transfer Learning | Kaggle


solving one problem and applies it to a new application remember that early layers of a deep learning model identify simple shapes later layers identify more complex visual patterns and the very last layer makes predictions so most layers from a pre trained model are useful in new applications because most computer vision problems involve similar low-level visual patterns so we'll reuse

Transfer Learning | Kaggle


most of the pre trained ResNet model and just replace that final layer that was used to make predictions some layers before that in the pre trained model may identify features like roads buildings windows open fields etc will drop in a replacement for the last layer of the rezident model this new last layer will predict whether an image is rural or urban based on the results of that

Transfer Learning | Kaggle


previous layer let's look at this a little closer here we see that the resident model has many layers we cut off the last layer the last layer of what's left has information about our photo content stored as a series of numbers in a tensor it should be a one-dimensional tensor which is also called a vector the vector can be shown as a series of dots each dot is called a

Transfer Learning | Kaggle


node the first node represents the first number in the vector the second node represents the second number and so on practical models have far more nodes than we've on here we want to classify the image into two categories urban and rural so after the last layer we keep the pre-trained model we add a new layer with two nodes one node to capture how urban the photo is and another to

Transfer Learning | Kaggle


capture how rural it is in theory any node in the last layer before prediction might inform how urban it is so the urban measure can depend on all the nodes in this layer we draw connections to show that possible relationship for the same reason the information at each node might affect our measure of how rural the photo is so our structure looks like this we have a lot of

Transfer Learning | Kaggle


connections here and we'll use training data to determine which nodes suggest an images urban which suggests it is rural and which don't matter that is we'll use data to train the last layer of the model in practice that training data will be photos that are labeled as either being urban or rural we'll cover more mathematical detail on this training step in a later video

Transfer Learning | Kaggle


notice that we allow all features from one layer to influence or be connected with a prediction layer when this happens we describe the last layer as being a dense layer one other note when classifying something into only two categories we could get by with only one node at the output in this case a prediction of how urban a photo is would also be a measure of how rural it is if

Transfer Learning | Kaggle


a photo is 80 percent likely to be urban its twenty percent likely to be rural but we've kept two separate nodes at the output layer using a separate node for each possible category in the output layer will help us transition into cases when we want to predict with more than two categories in both the current case and the case with more categories we'll get a score for each category and then

Transfer Learning | Kaggle


apply a function called softmax the softmax function will transform the scores to probabilities so they'll all be positive and I'll sum to one we could then work with those probabilities however we want let's see it in code we'll introduce two new classes from Charis first is sequential this is just saying we're going to have a model that's a sequence of layers one after

Transfer Learning | Kaggle


the other there are some exotic models that don't fit into this structure and we'll get to other types of models later for now all models you would want to build are sequential we'll also want to add a dense layer so we import that in this application we classify photos into two categories or classes urban and rural we'll save that as num classes now we build the model we set up a sequential

Transfer Learning | Kaggle


model that we can add layers to first we add all of a pre-trained ResNet 50 model we've written include top equals false this is how we specify that we want to exclude the layer that makes predictions into the thousands of categories used in the imagenet competition we'll also use a file that doesn't include the weights for that last layer we hand this argument pooling equals average that

Transfer Learning | Kaggle


says that if we had extra channels in our tensor at the end of this step we want to collapse them to a 1d tensor by taking an average across channels we'll come back to intricacies of pooling in a later lesson but now we have a pre-trained model that creates the layer you saw in the graphic will add a dense layers to make predictions we specify the number of nodes in this

Transfer Learning | Kaggle


layer which in this case is the number of classes like we talked about earlier then we say we want to apply the softmax function to turn it into probabilities well tell tens flow not to Train the first layer which is the ResNet 50 model because that's the model that was already pre trained with the imagenet data now we'll get to a more complex line of code in a compile command I'll

Transfer Learning | Kaggle


describe the broad concept here and we'll give a more complete explanation of the underlying theory in a couple videos the compiled command tells tensorflow how to update the relationships in the dense connections when we're doing the training with our data we have a measure of loss or inaccuracy we want to minimize we specify as categorical cross entropy in case you are familiar with the law

Transfer Learning | Kaggle


gloss this is another term for the same thing we use an algorithm called stochastic gradient descent to minimize the categorical cross entropy loss function again we'll cover this in our Theory video we asked it to report the accuracy metric that is what fraction of predictions were correct this is easier to interpret than categorical cross entropy scores so it's nice to print it

Transfer Learning | Kaggle


out and see how the model is doing our raw data is broken into a directory of training data and a directory of validation data within each of those we have one subdirectory for the urban pictures and another for the rural pictures caris provides a great tool for working with images grouped into directories by their label this is the image data generator there's two steps

Transfer Learning | Kaggle


to using image data generator first we'll create any generator object in the abstract I'll tell it that we want to apply the ResNet pre-processing function every time it reads an image you use this function before to be consistent with how the rezident model is created then we use the flow from directory command we tell it what directory that data is in what size image we want how

Transfer Learning | Kaggle


many images to read in at a time and we tell it we're classifying data into different categories well add batch size to our list of information covered in the upcoming theory video for now assume you want categorical class mode almost every time we do the same thing to setup a way to read the validation data that creates a validation generator the image data generator is especially valuable

Transfer Learning | Kaggle


when working with large data sets because we don't need to hold the whole data set in memory at once but it's nice here even with a small data set now we fit the model we tell it the training data comes from trained generator we said to read 12 images at a time and we have 72 images so we'll go through 6 steps of 12 images then we say that validation data comes from validation

Transfer Learning | Kaggle


generator validation generator reads 20 images at a time and we have 20 images of validation data so we can use just one step as the model training is running well see progress updates showing with our loss function and the accuracy it updates the connections in the dense layer that is the models impression of what makes an urban photo and what makes a rural photo and it makes those updates

Transfer Learning | Kaggle


in six steps when it's done it got 79 percent of the training data right then it examines the validation data it gets 95 percent those right 19 out of 20 we trained on 72 photos you could easily take that many photos on your phone upload them to kaggle and build a very accurate model to distinguish almost anything you care about I think that's incredibly cool this may feel like a lot

Transfer Learning | Kaggle


of new ideas for you to take in here's our plan we have one exercise for you to build a model yourself using chance for learning after you've done transfer learning hands-on I'll show you a simple but powerful trick called data augmentation data augmentation really improves your computer vision models when working with small and medium sized datasets that topic will be very quick

Transfer Learning | Kaggle


[Music] thanks for watching Henry AI Labs this video will present the semi weak supervised learning framework presented by research at Facebook's AI research lab this framework is a really interesting extension to their previous work on weak supervision such as using the hashtags and Instagram images as a weak supervised signal to trip retrain imagenet classification models this

Semi-Weak Supervised Learning


research is going to extend this idea to integrate semi-supervised learning as well as weekly supervised learning and then introduce a lot of other interesting ideas like incorporating model distillation into this framework and looking at class imbalance evident in these unlabeled data sets this video will present the research paper billion scale semi-supervised learning for image

Semi-Weak Supervised Learning


classification from researchers at Facebook a research lab this animation from Facebook's blog post on billions scale semi-supervised learning shows the idea of their semi-supervised training framework before integrating week supervision so in this case their take on semi-supervised learning is different from other definitions of semi-supervised learning such as

Semi-Weak Supervised Learning


rotating an image and then predicting the rotation angle or something like word to vector wave defect where you mask out certain parts of the sequence and then train the model to predict the missing part of the sequence this idea of semi-supervised learning is to have a label data set such as the image net data set train a large capacity model like a res NEX 101 32 by 48 group

Semi-Weak Supervised Learning


convolutions res next architecture and then use this massive high capacity model to predict the labels on an unlabeled data set and then you would use these unlabeled the softmax distribution of these predictions to pre train the target model ie as in model distillation such as what powers models like hugging faces distill burped then you'll fine tune the model that's been

Semi-Weak Supervised Learning


trained with model distillation on the label data set and this is your new model so one of the interesting things we already see about this is the novel use of model distillation as semi-supervised learning it's not really common to use these two terms together semi-supervised learning and model distillation also interesting lis is we can see this kind of model compression

Semi-Weak Supervised Learning


that arises from this framework you can have a really high capacity teacher model like the res NEX 101 32 by 48 D and then you can have a lower capacity more manageable probably faster inference time lower storage costs like the res net 50 that you could deploy on mobile and IOT in these kinds of things this animation shows the extension from the semi-supervised training framework

Semi-Weak Supervised Learning


to semi weekly supervised training so in this case instead of having some massive unlabeled data set we have a weekly supervised data set so instead of just having a collection of images we have a week label such as the hashtags on Instagram images and so the thing with the weekly supervised hashtags on Instagram images is that they're really subjective they're noisy and not really

Semi-Weak Supervised Learning


like as precisely labeled as say the data from imagenet so in this model we're going to pre train the teacher model same idea of having some larger capacity teacher model smaller capacity student network so we're gonna pre-trained it with the weekly supervised data set fine tuna with the image net then we're gonna use the fine-tuned model after having been pre

Semi-Weak Supervised Learning


trained to predict the softmax distribution over the weekly supervised data set and then when you use this model distillation knowledge distillation in order to train our student network we're gonna fine tune the student Network and then we have our Train model some of the interesting issues that research paper raises is this idea of class imbalance and unlabeled weekly supervised data sets so

Semi-Weak Supervised Learning


one characteristic of machine learning models and you know deep learning convolutional image classifiers is that class imbalance can really destroy the performance so for example if you're training a cat versus dog image classifier and you have 80% of your data your training data is cats and 20% its dogs your train model is going to want to predict cats it's going to be

Semi-Weak Supervised Learning


biased towards the imbalance data so in these weekly supervised data sets such as hashtags on Instagram images and then trying to transfer that into image net classification there's going to be like a long tail distribution where you're not gonna have as many of these really specific classes and image net contained in this image net data set so another interesting idea is just incorporating

Semi-Weak Supervised Learning


model distillation in the semi-supervised learning this teacher-student model compression and then this framework is gonna achieve 81.2% imaging at top one accuracy with a ResNet 50 and they're gonna scale this up to the res next 101 32 X 16 D with the student network capacity and they're gonna achieve 80 4.8% and this is up from eighty four point two from their

Semi-Weak Supervised Learning


previous research on doing a lot of label engineering for the weekly supervised Instagram dataset and then in that previous study they had achieved eighty five point four but it is a larger capacity bottle at the 48 D and they probably didn't test the 48 D because you know it's expensive and time-consuming to train these kinds of models this research from Facebook's and

Semi-Weak Supervised Learning


research lab is in line with a lot of their other work such as using weekly supervised pre training for video action recognition using over 65 million images like from Instagram as well as using billions of images in their weekly supervised pre-training of an image net image classifier in this case they do achieve a slightly higher performance but they do have larger capacity models

Semi-Weak Supervised Learning


and also interestingly is that in this case they are manually doing the you know kind of removing some of the noise from the week supervised data set whereas this framework is gonna present an automatic way automated way of doing this an interesting characteristic of the newest semi weak supervised training framework is they're going to use an explicit algorithm to balance the

Semi-Weak Supervised Learning


classes in the predicted distribution from the weekly supervised data set so this teacher model has been trained on the image Minette classification task but the weekly supervised data set probably isn't as balanced as imagenet classification it's probably heavily skewed to sort towards some classes more than others such as dogs and things like that compared to these really specific

Semi-Weak Supervised Learning


things like I don't know a tiger shark there's things like this so this visualization shows the top k scoring of examples so as the teacher model predicts a distribution of class labels over the unlabeled data or the weekly label data the models are going to be ranked according to their probability their class probability and then they're going to be balanced in this way so that

Semi-Weak Supervised Learning


each class has an even number of training samples and in this visualization you see how as it goes from rack 1000 to being very close to the image like the leaf beetle looks like the beetle whereas by rank 10000 16000 is not really a beetle anymore but the teacher model has given some probability to beetle when it's a predicting distribution of that image another

Semi-Weak Supervised Learning


interesting issue with this framework that's raised in the paper is the idea of inference time with model distillation so in this case we're predicting over a billion unlabeled or weekly labeled images with our teacher model so we want to have fast inference time for this prediction is much more important in this case than it is for typical examples of model distillation

Semi-Weak Supervised Learning


where the data sets aren't that large when you have billions of images you want to make sure the teacher model has fast inference I think with the rising popularity of knowledge distillation model distillation techniques such as hugging faces disco Bert and now Facebook semi weekly supervised training paradigm we're gonna see these kinds of inference accelerators like in videos

Semi-Weak Supervised Learning


tensor Artie accelerator to be more and more important because usually these frameworks are developed for inference when the model has been deployed but now we're seeing the inference be a part of the training loop as well in this teacher-student Maalik model distillation paradigm from their paper the researchers at Facebook give these six guideline recommendations for large-scale

Semi-Weak Supervised Learning


semi-supervised learning so the first idea is really interesting this teacher-student model distillation paradigm also really interestingly and uniquely in this paper is that they're going to test model distillation when the student and the teacher have the same architecture or the same capacity the second idea is to fine tune the model with true labels only this is a

Semi-Weak Supervised Learning


pretty intuitive idea the weekly supervised label dataset has a ton of noise in it compared to the imagenet data set or you know other more specific label data sets a third idea is that large-scale unlabeled data is key to this performance naturally the key driver behind this algorithm is that they have a billion images from Instagram that they're using to train

Semi-Weak Supervised Learning


this model the fourth idea is really interesting that they use a large number of training iterations for their pre training with the weekly supervised learning compared to you know more pre-training iterations compared to normal supervised learning the fifth idea is a novel contribution to this paper the idea of having a balanced distribution for inferred labels so when

Semi-Weak Supervised Learning


you're doing the model distillation you want you don't want to have class and balance in the distribution of these labels and the sixth idea is that pre training the high capacity teacher model week supervision further improves the results the idea of adding the week supervised to make this the semi week supervised learning framework now we'll get into some of the results of their

Semi-Weak Supervised Learning


research report you can check out their repository on github semi-supervised image net 1k models where you have the pre trained models that you can load with torch hub and then they also present some of the results you see the semi-supervised learning framework different model architectures from ResNet 18 and 50 up to the res next group convolution architectures and then

Semi-Weak Supervised Learning


you see up to the 80 4.8% accuracy when using the semi-weekly supervised learning framework with 193 million parameters on the res next 101 32 by 16 d architecture the first set of results they present shows the success of the semi-supervised learning framework with different student models so first we're looking at the resident 18 the resident 50 and then higher versions of the

Semi-Weak Supervised Learning


Resnick's and the 50 101 higher capacity resonate variations so we see that the fine-tuned semi-supervised learning framework is always outperforming the fully supervised learning tasks when you just train the student model on imagenet classification then they present this idea of varying the complexity of the teacher model and showing how increasing the capacity of the teacher model

Semi-Weak Supervised Learning


increases the accuracy of the student model we see the gains are increasing every time as we scale up the capacity the teacher model while holding the student model with constant this plot shows the results when the teacher and the student model have the same architectural capacity interestingly we still see these gains when we're using the same capacity for each model this

Semi-Weak Supervised Learning


plot shows how the top one accuracy changes as a function of the unlabeled data site data set size used as the you know the unlabeled or the weekly supervised data in this pipeline so we see that the performance continues to increase as a data set gets larger and larger following the recommendation from their guideline that having massive unlabeled data sets is key to this

Semi-Weak Supervised Learning


framework this plot shows how the accuracy improves as a function of the number of pre training iterations so as stated in their recommendations they use a much larger amount of pre training epochs than supervised learning epochs really interesting Lee is there showing four billion training iterations it achieves the highest accuracy in this plot then they show the results of

Semi-Weak Supervised Learning


increasing the K parameter so the K parameter shown here is this idea of scoring the class label distributions when you're doing the model distillation from the teacher Network so basically the idea is if you increase the K from say eight K to 16 K and then you're looking at a specific class such as leaf beetle the as you get towards the end of the eight K and then especially from say

Semi-Weak Supervised Learning


eight thousand one up to six the top eight thousand one to the top sixteen thousand the images are going to look less and less like beetles they just have been assigned some probability as beetle and now they're a part of the balanced data set because you've increased this K parameter so they show how there is a limiting effect to how large you increase the K because

Semi-Weak Supervised Learning


naturally as you increase the K passed a certain threshold you're making your knowledge distillation data step four your student network to be really imbalanced and deep learning machine learning these kinds of decision boundary models do not respond well to class and balance although they don't show you like things like random over sampling a lot of the techniques

Semi-Weak Supervised Learning


commonly used to overcome class imbalance this table shows the evidence of the balancing of the distribution with these ablation studies such as balancing dataset or unbalanced in a dataset showing a point eight percent accuracy improvement which is very significant for imagenet classification and then also the idea of using the Instagram tags versus ranking the list

Semi-Weak Supervised Learning


of the pretty distributions and then comparing all these performances to supervised learning this table is the showing the big highlights of the paper you see they achieve 81.2% accuracy with the resident 50 architecture and how this is state VR compared to previous works trying to achieve weekly supervised learning at the same model capacity then you can see the

Semi-Weak Supervised Learning


head-to-head comparison with their previous work on you know label engineering the week supervised learning and sort of you do see how the accuracy starts to saturate at the further model capacity of the Residex architecture another really interesting characteristic of this framework is success on transfer learning so when they transfer this pre trained model

Semi-Weak Supervised Learning


from imagenet to the bird image classification model they achieve a really high level of transfer learning performance compared to previous approaches this chart shows the difference between fine-tuning just a fully connected layer at the end of the network compared to the full network and then the performance achieved after doing this they also test this model on

Semi-Weak Supervised Learning


video classification with the deep mind kinetics data set and they show a significant improvement achieving seventy five point nine percent accuracy using this technique compared to the previous research achieving seventy four point eight percent accuracy thanks for watching this presentation of semi weak supervised learning from facebook's AI research lab this research paper has

Semi-Weak Supervised Learning


presented a really interesting framework for semi-supervised learning and weekly supervised learning and integrating this model distillation paradigm there are some really interesting ideas presented in this paper such as the importance of how they having a balanced class distribution for the model distillation data set used to train the student network also interestingly is this idea

Semi-Weak Supervised Learning


hello my name is Krishna and welcome to my youtube channel today we are basically going to discuss how to learn data science for free now when I say for free as you know that their whole lot of materials available in the internet with respect to Python programming language with respect to data science with respect to machine learning AI deep learning and whole lot of stuffs so what

How To Learn Data Science by Self Study and For Free


I will do is that in this particular session I'll show you a systematic way how you can basically complete your data science syllabus within three months so that your transition towards data science can be possible just within three months and after that what you can do is that you can also attend interviews by updating your resume now resume part will be discussed later on


in the upcoming videos but today we will just try to focus how to learn data science for free I'll show you the systematic way what our YouTube channels you can basically follow you know because there are lot of YouTube channels provides free materials with respect to Python machine learning deep learning and all apart from that you I'll also be mentioning about various


blogs and that finally I'll also be mentioning about the best machine learning book that you can basically use in order to learn data science machine learning very easy now to begin with guys so I have already prepared the word doc over here in my laptop and this particular word doc I will actually upload it in my google drive and share with all of you and that will basically


be given in a description box now in this particular video I am basically going to tell you that how we are going to do how we are going to learn data science with respect to machine learning and deep learning considering Python programming language the reason I'm telling about Python programming language is guys because I have an expert in Python programming language


I've referred a lot of materials I have referred a lot of things and I've also done a lot of stuff self-study so that is the reason why I'm actually telling you this for our programming language I need to do a little bit more research which are the best materials but apart from that all the machine learning deep learning techniques for learning purpose you can basically use this materials use


this links that I'm basically giving you but for the practical applications you have to be you have to you should be able to search through various internet resources okay so to begin with first of all as this is for Python programming language the first topic that I will take is basically from where we can basically run Python we can basically learn sorry so I had two channels in my like


most most of the time I referred this channels whenever I wanted to learn about Python whenever I had some queries and the best part of this particular channel this two channels are read in Python as you know if you are learning data science you should just not know Python you should also know object-oriented features in Python apart from that you should also know some


other frameworks like flask and Django because this two frameworks are very very important for deployment of machine learning models and deep learning models okay during the deployment stage you know that you create a flask framework or Django framework and you just uploaded it in in some other servers let it be a platform as-a-service server it may be infrastructure as a service


server like ec2 instance of AWS or HID okuu platform and many more platforms are there but initially the web framework is basically created the micro-service framework is basically created with the help of flask or Django so this channel the first channel that I like to mention is Corrie Schaffer he is a wonderful person he used to work in an IT company before but later on he moved


into teaching into YouTube channel itself he has one of the best Python videos guys the link is basically given in the description I mean in the word doc in the description so you can basically refer the YouTube channel of his link and I would suggest you if you have any queries go and see that particular channel with respect to Python and it is started from basic from


the basic installation part ok now in that particular channel you'll also find a playlist on a flask and Django so this was one of the favorite channel that I also refer for learning Python so that is Corey Schaffer the second person is basically sent decks okay send decks is one of the oldest youtuber who uploads videos on machine learning deep learning Python natural language processing so he


is also one of my favorite youtuber and he's a very simple guy very very very you know if I see him I really get that motivation motivation because he provides every materials every videos that he upload he does not have any online tutorial sites he just has some sites where he'll be writing basically about the blogs of whatever he's doing in his YouTube channel so before that saying


Dex and again the link will be given in that particular word doc itself now once you learn Python programming language okay now Python programming language I think if you sit and guess guys this this if you're planning to cover this in three months make sure you give three to four hours daily okay give three to four hours and I'm where I'm saying give three to four hours that


should be more productive hours okay now after this now once you finish Python programming language now the next thing is that you move towards machine learning now I know that many of them will ask me a question - Mia asked me a question saying that where is the math part where is the linear algebra part where should we learn it from where should we learn it the differential


calculus and many more things right statistics parts and all guys don't go in that particular way we need to complete the data science syllabus within three months so what you do is that pick up the machine learning algorithm and through reverse engineering understand the mats and try to derive take a use case how to solve it and finally solve that particular use


case try to optimize that particular use case try to increase the accuracy so while you're doing this all the steps you will be means you will be learning statistics you will be learning linear algebra will be unlearning differential calculus wherever it is required always do that reverse-engineer get that knowledge you know now suppose if I want to solve linear regression now when I am


learning linear regression I know that there will be an equation of straight-line that will be coming into that particular algorithm like Y is equal to MX plus C then after that I will be saying I'll be just finding out I'll be deep diving how do I find out that coefficient value then over there the gradient descent come into existence then I learn about how this value is


basically calculated through differential calculus I'm just taking as in one one as an example of our linear regulation similarly you have to learn with respect to each and every algorithm so for this I have actually again selected three channel one is you need to understand the maths behind each and every algorithm so you can basically refer or machine learning course by Andrew ng in


applied AI so deep learning dot AI sorry not applied air deep learning dot AI channel you can basically and again the link is basically given in the word doc itself the other thing is that I have also uploaded many videos on machine learning and some of the feedback that I got that the machine learning playlist is not that ordered you know so what what you can do is that whenever you are


searching my videos suppose you are learning simple linear regression just search that keyword and just put my name in front of that you will be getting the whole explanation apart from that I have also uploaded videos with respect to practical application ok so you'll be able to do that now I am trying to order that particular playlist and I'll make and making sure that whatever videos that I


upload in the future that will also be ordered so I would like to say one is Andrew ng one is my channel with respect to machine learning if you just want to know the maths about each and every machine learning algorithm go and see Andrew ng from deep learning dai and then you also have sent dex channel again I'm referring send text because he has uploaded videos on Python he's


uploaded videos on floss Jango apart from that he's also uploaded videos with respect to your machine learning algorithms ok and the best part is that he's not uploaded machine learning all he has uploaded deep learning also so I'm going to refer him again in the later links when I'm discussing about deep learning so machine learning three things one is my channel one is sent X


and the other one is auntie Ong 4 applied for deep learning dai and um you know Andrew ng has explained the complete maths sometimes what happens is that you will not be able to follow it but again if you just see again and again you'll be able to follow it because the math is pretty much simpler but what I am making sure that in my channel so I will be uploading a lot of


maths thing whenever I am explaining you about some specific algorithms and that will be continuing going on in Andrew ng channel of deep learning dot AI you will not find any practical application so for the practical application you can either refer my videos or you can refer send text videos ok over there send X do not explain you the maths behind any machine learning algorithm since is more


focused towards implementation of machine learning algorithms ok so this was one now I have also seen some people who gets really attracted just not by seeing writings equation they like some animation kind of explanation so if you want some animation kind of explanation there is one channel that I went through is stat quest with Josh stammer ok so this is one of the good channel where


they'll show a lot of animation to explain you each and every machine learning algorithm with you know along with that explanation the theoretical explanation how it is basically done including all statistics linear algebra differential calculation and different kind of maths formula again the link is basically given in the docx file itself okay then after that you have natural language


processing natural language processing you can basically go through my playlist because I have uploaded around eight to nine videos with respect to machine learning and I'm planning to upload with respect to deep learning also where I'll be implemented a lot of NLP things where I'll be implementing with the help of word to work and all the different kind of tools or libraries that are basically


present in natural language processing the other channel is basically again sent takes send X has been up has uploaded around twenty to thirty videos with respect to natural language processing and then you can refer that also now we let us go to the deep learning for deep learning I have selected two channels one is Andrew and G again from deep learning day I again


any theoretical components any theoretical things that you need to understand about deep learning can be sick later for that link again just watched the word doc file again in that I mentioned the link also the second channel is my channel because deep learning might be complete deep learning playlist that I have created is completely in order okay to the other


one doodle-do tutorial three like that I have have actually created tutorial 22 and I'm including both maths understanding of the algorithms understanding about neural networks and how to implement that with the help of carols and Python right so I have done that I have ordered it you know first of all I explained you about all the artificial indian or artificial


intelligence artificial neural network and I mean artificial neural network I have explained about lot of things like back propagation how to update weights bias all those things and I am also showing it practically how you can basically use for practical implementation with the help of Kira's apart from that how can you basically optimize your problem statement also now


apart from that you can basically refer my channel thrush I'm also going to complete the whole deep learning playlist still they are 15 to 20 videos so total overall display playlist will be having 40 videos okay which will be including LST m RN and CN and everything okay everything in and I am I'm serious about it because I started that playlist still not when you do


videos as been completed I have still have plans to upload more and more now after learning all this you'll be having a lot of idea and this particular hole whatever I have explained is with respect to YouTube channel right now okay the whole lot of materials that are available in YouTube and the next thing is that you refer github links you know github links so suppose if you have a


problem with about linear regression go and search linear regression github okay so you'll be getting various abundant materials in the Google in Google search itself and you can basically take one of the problems start solving it now the next thing is that after after learning all the skills you have to do a lot of practice projects right so I have created one deep learning


sorry data science project playlist wherein I have uploaded more than 50 videos 50 different use cases and that is specifically kaggle use cases that I have taken I've solved with the help of Python and machine learning and deep learning so you can basically refer those projects try to solve it again all the code is basically given in the github itself you can refer my github


and you can get up all the details ok so after learning all these things the last part is data science projects you should be able to implement various data science program finally create your resume you know include everything that you have learnt into your resume that's it now first we have discussed about YouTube channels now the second thing is that I'm also going to refer some of the


blocks which has all almost all of the problem solutions of every machine learning algorithms or deep learning algorithms one is towards data science blog and the other one is medium with respect to machine learning and deep learning so all the links is basically given in the word doc itself ok so in short this was there and the last part is basically about books now one of the


best machine learning book that I have also read you know the link is basically given in the description about the best machine learning book it is basically written for the O'Reilly publisher if you go and see that particular books book as the best book guys best book on machine learning and deep learning I think every fresher every fresher who wants to make that transition towards


their data science machine learning I they should go and read this particular book and this book is basically you know a boon to all the data scientists because the author his basically or alien gear on sorry if I'm pronouncing it wrong but this particular person has written this book and it is basically the book name is hands on machine learning with sky kid learn and


tensorflow okay again the link is basically given in the description and the the publisher name is O'Reilly and I'll tell you this book is very cheap guys I see there are a lot of free PDFs also available for this book I don't want to share that PDFs I don't know don't hurt want to research and find out the free PDFs over there because this guy has written this


author has written this book so nicely and I don't want to disrespect him by just taking the free PDFs and distributing it to you okay I have not even researched and found out whether there is any PDF or not I basically download I basically bought this book the paperback version and I'll just if you want I can also review this particular book in one of my next videos


but awesome book you know it is basically you have Python you have feature engineering you have machine learning you have everything okay and about apart from that guys you can buy this book and it is hardly around 1,500 rupees INR if you consider this in the terms of dollar hardly ten to fifteen dollars fifteen to twenty dollars so I think yes fifteen to twenty dollars you


can basically buy this particular book again the link is basically given in the description go ahead and buy it I know this particular session is all about data science with respect to free but I'd suggest this particular book you know don't search for free PDFs keep this book handy because it will help you for the lifetime okay whenever you want you can basically leave this now the


next thing comes that there are small small parts like feature engineering feature selection what you have to do is that I'll be sharing a very good github link about feature engineer and feature selection which I found it through the internet and I'll be sharing that link in this word doc itself what you have to do is that go inside that link there are many materials present inside just refer


each and every notebook file it is clearly written what what is all about feature engineering how feature engineering is basically done similarly they are on 10 to 20 materials notebook file Jupiter notebook files you can just you just have to read it just have to execute that and by that you will be able to understand a lot of thing and similarly for the feature selection so


I'll be sharing those two github link and thanks to the author I'll also be mentioning the author over there who had actually provided those materials and it is available in an Internet completely for free so I'll be providing that two things to you and yes that is all about this particular preparation guys and I think if you are able to give around two to three to four hours I think within


three months you'll be able to complete this whole data science syllabus and after three months you'll be also giving the interviews because you have a learnt a lot of things done data science projects you have practice things in kaggle and make sure you practice a lot of projects after completing all these things through this YouTube channel through blogs through this particular


book that I have told you from O'Reilly publisher which is basically hands on machine learning a sky killer intensive flow that was all about this particular video I hope you liked this particular videos share with all your friends subscribe do subscribe this channel if you're not already subscribed I'll see y'all in the next video have a great day ahead thank you


hello welcome to lecture 8 of deep and squeezed learning today we are going to talk about the strengths and weaknesses of various narrative models and representation learning methods that we've seen so far so the brain has 10 to the power 14 synapses and we only live for 10 to power 9 seconds and so we have a lot more parameters then then I'm the data we ingest so this motivates that we

L8 Round-up of Strengths and Weaknesses of Unsupervised Learning Methods -- UC Berkeley SP20


should do a lot on scores learning because in order to provide sufficient fodder for the number of parameters that we have in our brain we should be able to predict a lot more bits from the data that we ingest which is 10 to the power 5 order magnitude smaller right so this was a statement made by Jeff Fenton in this 2014 in a reddit so firstly you summary of the course so far we've


looked at or aggressive Morrow's fix learn and pick so CNN picked two skin and paws glass pixel snail we looked at four models really only P family of models and also the connection between Auto regressive flows and in with Auto regressive flows next we covered latent variable models models with approximate density estimates using the variational lower bound and various associations of


that like the VA e importance rated our encoder VQ BAE pixel BAE and so forth we also then jumped into a different class of jeredy models that don't work with the likelihood principle the impasse density models against energy based models in the moment matching principle and finally we questioned the idea of like whether we even need to learn generative models if all we care about


is extracting useful features from unlabeled data and that God isn't with this topic because house provides representation and we saw that with the right kind of simple cognitive principles and a lot of data and compute we can learn really useful representations of unlabeled images that are competitive with supervisor plantations so represent so let's let's look at auto regressive


models used in the in 2015 the main paper was Bush with which introduced this idea of masked or encoder for density estimation and it was able to produce these I'm miss digits which were reasonable looking but very jittery and this idea was extended to much stronger architect more expressive architectures well-suited for image modeling like masking illusions this individual


introduced in the pixel or an analytics is seen and family of models and you certainly started seeing generative models working for higher dimensional and much more diverse assi Commission ad so these are samples from image net 64 by 64 you can see that the the structure across 4,000 pixels is pretty coherent but the color is not that good and therefore you're not actually able to


identify any visible class from imagenet but this was a big big jump from the quality you saw and made and this idea of mass convolutions it has also been applied for one dimensional data like audio and in order to model long-range cover in audio samples the idea of using dilated combinations was introduced and this was also applied for a text-to-speech system where you're going


to convert linguistic and text features to raw audio and that can be used in any Indonesian assistant like the Google assistant and this was the Wayman architecture that was commercially deployed after after a year and the same idea of using mass conversions with Auto regressive pixel level modeling has also been applied for Rio prediction why are you looking at


the pass frames and encoding them with a convolutional STM and then you're taking the embedded representation as a conditioning information for a pixel scene and decoder that generates the next frame pixel by pixel and it's able to produce coherent video look like a robot moving out to Tehran so over time the order aggressive modeling community has expanded further and further in


terms of the level of engineering and architectural innovation and on the left you can see if the subscale pixel networks which have very coherent samples because of the clever conditioning can assume to use on the right you see hierarchical auto regressive image models with auxillary decoders where the idea of using latent space auto regressive models was introduced by quantizing representations


or encoders and and modeling pixel CNN the latent space which is also similar to the vqv a a idea that you seen in the VA a lecture so apart from images and audio and video auto regressive models have had immense success in language and these are samples from GPT to riots it would actually produce a coherent story about unicorns and like a like a story of how


unicorn skin when their own language and also talks about a scientist who is able to observe all this phenomenon and this shows that language modeling at the level of a paragraph or even multiple paragraphs as possible by just training large models which used to order aggressive structures this slide shows the evolution of language models over time we're on the first you see Shannon's


three grand models which I reasonably good but not super coherent across the full sentence and then Ilya sutskever is model of using an RNN is able to produce a couple of sentences but not completely making sense and then over time they using bigger LSD and bigger transformers you ended up with the quality that's UPD to experts right now so all these huge advances have been possible due to


multiple reasons and let's go through them quickly the first thing is just being able to train with larger batch sizes because of more computer availability and training with larger bad sites then we stabilizes the training of these models and optimizes these losses much better making the models wider making a modest deeper figuring our clever race condition your


next year you're building a conditional class conditional or audio condition or text condition model the figuring out ways to get the conditioning information cleverly is very useful pre-processing like in wavenet we use some new law pre-processing to quantize continuous audio into discrete entities are for example in pixels you're actually using categorical information for modeling


rather than rather than using gaussians so these are these are and in language you using by parent coding which is pre trained on a huge corpus and therefore your mod in on modeling neither at the character level or at the word level but your modeling in the sub word level and that's much more useful for generalization and also building more efficient models


compute power and as we progress in the last two three years we just have at we just where access to a lot more compute like TPU so I like big GPU rigs which have lots GPUs connected really with a really fast interconnect and therefore be able to train data data parallel model is much better and we're to train see that several weeks or basic training are usually producing much


better results and also making fewer assumptions about the whole problem like before trying the idea of this predicting categorical distributions for every pixel why would he want to imagine that pixels are definitely gonna be modeled with calcium's instead of categorical distributions like indy really doesn't make any sense but then practically it's better for a neural


network to work with cross entropy losses there are also been architectural advances that made all these was much better so mass conversions were applied in the original Pisa CNN but as transformers and dilated communist art exists the samples just got much better with more coherent structure across long range dependencies and and making the whole modeling problem look more like


supervised learning helps a lot and therefore relying relying heavily on oh they'll be here crossing will be lost and optimizes that have been much better tuned for this loss ensures that generative modeling can also benefit from all this but engineering advancements so now what's the future for our regressive models we're only scratching the surface of what's


possible and and once we have motor pilot training we'll be able to realize a lot more for instance be able to train trillion parameter models on all of the Internet's text and that that way we could compress all the Internet's text into a giant neural network that can be a like a know-it-all language model and secondly we can figure out ways to Train one single model for multiple modalities


just even bigger generative model they could work at a video level on YouTube or image level Instagram text level Cabiria so that way it's able to probably correlate information across multiple entities and chameleons for expansion so for all these kind of modeling requires hardware and software advances from auto pilot training we should it's also possible to make or aggressive


models more useful by figuring out faster ways to sample with better low-level primitives at the CUDA that will like for instance fast kernels and and better act like for example wave are an N uses all these mechanisms for production components and doesn't need to be distilled into something like a parallel bayonet this work as a standalone auto regressive model and


still be deployed on an Android phone hybrid models with much weaker or aggressive structure but that can be trained on a large escape could be revisited and and of course all these architectural innovations that help in long-range dependencies would always help in you know as you keep moving to bigger image this or a video or something like that these kind of ideas


should up a lot so like a summary of auto regressive model could be that it is an active topic but a lot of cutting-edge to us and there's a lot of moscow for a new engineering and creative architecture design and larger models and data sets are clearly needed to you know realize the full potential of these class of models and standalone they are very successful across all


modalities without any conditioning information like class labels so that's that's like a very appealing property of these models every Universal in that sense and also they can work without much engineering for sampling time so that makes them really look creative but but but nevertheless for production if you you should really cut down on the sounding time to be useful and so


innovating on the low-level primitives was very important so that said there are a lot of negatives for aggressive modeling one is you don't extract any representation there is no bottleneck structure and sampling times not good for deployment it's not particularly usable for downstream tasks like for instance a language Maru you need to sample multiple times to see coherent samples


so you can't just roll out a language model that's a software and there are no interpolations that you can see to visualize what the models actually learning and every time you sample it's going to take a long time to produce like a diverse set of samples so that's it about auto regressive models now let's look at flow models in flow models it all started with the nice


architecture by loaned in and those the model was already producing very good digits on the endless data set and on the T of tedious it was producing reasonable phases but it really was bad on see far and SPH India said the samples were very blurry but it all improved with the real end we'd be architecture which introduced other kinds of flows and rational room to make


the models better and then the glow model from King model was published where the real and Ruby model was taken to another level by making it prettiest much larger images and overdone in our lab called flow pass class advanced the likelihood scores for flow based models to competitive scores that with that of autoregressive models for the first time and this was done by this architecture