back to the correct and like back to the data manifold and that way it uncovers the shape of the data manifold by operating at these local hyper planes at every individual point so here is the loss function of the denoising auto-encoder you can clearly see that there is a version where you can use the reconstruction error for the available pixels which are not having any noise


and the reconstruction error for the pictures that have been noise and you can also rate them based on you know like you can you can prioritize the reconstruction of the noise versions as compared to the versions that have not been noise so so if you have an amnesty Majin your dick map adding noise like 10% of the pixels you could prioritize the reconstruction error of those pixels

Lecture 7 Self-Supervised Learning -- UC Berkeley Spring 2020 - CS294-158 Deep Unsupervised Learning


more than the other pixels so that the model is not incentivized to learn identity function or around what is like already available without noise well and the models actually striving hard to like we're getting get the details right at the noise pixels so and you could also imagine optimizing to different versions of the loss one version of the losses using the mean squared error and


the other words for the loss is using a binary signal across central P loss and both both are equally good and endless it makes more sense to use the cross entropy loss but the mean square error loss is also very likely to work well if as long as you let it train with the right set of hyper parameters so stack denoising auto-encoder is basically the version of denoising auto-encoder


where you're going to do this layer by layer by layer so you take your original amnesty which you have one hidden layer and you run it in using or encoder and you get that feeling layer now you can take that hidden layer as you're like version of the image that you want use instead of the actual pixels you can add noise at the hidden feature level and learn a denoising auto-encoder for that


feature right so and if you do this iteratively the denoising auto-encoder is now operating on more more abstract inputs instead of a raw pixels so that's basically the idea in a stack denoising auto-encoder and the hope is that as you keep stacking more layers the higher layers get more more semantics but you should also be careful in thinking about what kind


of noise you can add to the features so if back in those days people used to use neural networks with Sigma nonlinearities so in that case it's it's easy to add a noise like masking because Sigma can be considered as you know the neurons firing or not firing but but now neural nets and like like the way they did the ear and that's a design is much different like the kind of nonlinearities used are


very different so so this may not be particularly in a appealing idea to use in the current current infrastructure so finally one utility of the denoising auto-encoder is that once you've learned sufficient layers with the stark denoising auto-encoder you could basically have a target like a class table and just freeze all the features that you've learnt from the auto encoder


and have a supervised layer on top make a single linear layer that just predicts the class logit and you could use that perform a classification task and and this was particularly appealing back then because back then it was really hard to train deep neural networks to just do supervised learning even if you had a lot of data because directly training deep neural networks


was not something that was particularly working with and like innovations were needed in terms of using momentum optimizers and bigger math sizes and etc convolutional neural network so as far as hidden here look like feed-forward neural networks goes this was a standard recipe to even do supervised learning back then because you needed some reasonably high level


features to operate well to train a good supervised model so here are the filters that you learn with a denoising auto-encoder for various levels of noises and you can clearly see that the ones where you actually add more noise is learning more of these digit edges whereas the ones you don't have any noise you're hardly learning anything because it's mostly gonna do an identity


map and this is also visualized for a particular neuron in magnified and you can see that filters are like more visible for the higher masking ratios you can also see the you can also see that there's something like a six visible towards the right and and and it's getting it's getting the notions of digit edges or strokes at the hidden level so these are various classifiers


that it can train on top of these denoising auto-encoders the features that you get from stacking nosing order into rows and these are the error rates in M this classification it's not particularly relevant now because M nest is considered solve but then you can clearly see that you know it's getting like 97% accuracy or something that range with putting SPMS and top so this


is like a cool result at the time so here is another version of corrupting your image and trying to predict corruption or like trendy you try to hide some portion of your image and trying to predict the hidden portion so this is a people context encoders from the potato from work from Alyosha afer this group here at Berkeley so the wait works since it basically takes an average mass out a


rectangular region and encodes that image now with the mask and has a big order that tries to reconstruct the actual image now so that way the model is filling up the details of what's missing in the mask and supervision basically can be constructed from your data itself because you actually knew what part you masked you because you had the complete image so the model is able


to learn without any labels by creating its own supervision so that that's that's why it's called supervised learning so you can have various instantiations of this where you could mask out only the central region and try to fill up the central region or you can mask out various square blocks across spread across the image much smaller but lots of mass and you could try to fill


them all or you could if you have access to a segment the segmentation mass of actual objects in your image you could segment out particularly the pixels belonging to one particular object like in this case the baseball player and you could just try to fill in those pixels and that assumes access to label data of segmentation mass so that's not something that is completely self


supervised but the other two versions are completely self supervised so they've made the reconstruction last good work is you you have a masking region and you have a ground truth for that and you could just apply the reconstruction error on the pixels that have been masked so you take your decoded image and you apply the inverse of the mask and you get all the other pixels out and


and then you can just mask out those pixels for the reconstruction here so there are multiple losses that you can use for the reconstruction objective one is one is you could use just a mean square error that you saw in the previous slide of diagnosing or encoder or one problem that's usually a common with the mean square error which was also mentioned in the Gant lecture is


that they often tend to be blurry so because you don't want Larry reconstructions you actually want sharp predictions of all these missing pixels you can actually think of using a gann loss which is you have a discriminator and that discriminators you behaving going to behavior I could learn a loss function and you can think of using the discriminator objective and the


reconstruction objective together because back back in those days training just a discriminator and using adversary losses wasn't particularly easy so the authors ended up using a combination of the regular reconstruction objective and the and the adversarial a discriminator objective so so this is the architecture they adopted where you have your original image one finis 8 by 128 and


you just use strided 4x4 cons and down sample spatial resolutions and you also up keep up sampling your channel resolutions and you get a flat hidden vector of 4,000 dimensions and then you up sample using transpose convolutions back into the actual original image and you can use the reconstruction error it could be an l1 or l2 objective and I think the authors tried both both l1 and


l2 and found l1 to be working slightly better as far as reconstruction goes and they also have the discriminator that takes in real data as well as your predicted missing patches and then you classifying if it's a real or a fake image so you can see that the l2 loss is producing a blurry pixel interpolation basically scrapping all the neighborhood pixels the vaping workers you just have


the missing square and you can just fill up like the borders based on what based on the pixels that are available immediately to the left or the top of the right at the bottom respectively and once you filled it up you can just fill in the missing regions of the square based on what you've already filled up within the square on the edges and that will look like a reasonable completion


very blurry though and if you look at the adversarial loss it's introducing artifacts that are completely new so as long as the discriminator thinks like it's a real object it would still work but it may not particularly have a coherence with respect to the actual background image and you've seen this problem in pics to Pyxis will vary for a conditional ganyo to provide the context


in addition to what you're trying to translate to so so that's clearly what's going on there and the joint loss is trying to use both the reconstruction objective and the and the adversary's objective and it's producing something sharper than just using l2 loss so so here are some results for what what happens okay let's say you finish this retraining process now you take the


encoder out and you you want to use it for a bunch of downstream task so the downstream task could be a classification in detection or semantic segmentation and classification and detection are done on a Pascal dataset which is much smaller than imagenet so you can think of the advantage of P training as hey if you don't have too many labels you know you really need


some kind of features to start with to be able to perform the task and semantic segmentation are also on Pascal VLC but a different version of the dataset 2012 and that uses another architecture like fully convolution that for doing the semantic segmentation part so using the feed train part of context encoder as a backbone our part in the FCN so so here the results are


reasonably good so if you just use image net features which is you pre train a classifier or an image net and then you fine-tune it on Pascal the results are like seventy eight point two percent for classification and 56 twenty-eight percent for deduction forty eight percent for segmentation and context encoder the fine tuning to pascal classification is not that good it gets


only fifty six point five percent which is way lower than supervised but it's when it's reasonably good in the sense that it's able to perform on par with with a or are actually quite quite better than an auto encoder an auto encoder gets fifty three point eight percent on pascal classification and detection in case forty two percent so so they get two to three percentage


points more than doing just a regular encoder and other other other self supervision methods that were available at the time so this was a reasonably interesting result at the time so next we look at the principles of predicting one view from another where you basically do some source separation and you're trying to predict the separate parts from each other so this is a slide


for Richard Zhang who is also the first author of this line of work so we already saw what a denoising auto-encoder is it basically takes raw data corrupts it and tries to reconstruct the original data now imagine that you can separate the raw data in two different views the best way to understand this is an image can be separated into a color image and the grayscale and you could try to predict


one view from another so you could try to predict the color image from the grayscale putting the grayscale from the color doesn't actually need any deep neural net all you need to do is average two pixels and quantize and you're going to get something that is reasonably grayscale and in exactly the conversion is done it's a weighted average of your RGB pixels so


but the other version which is predicting the color from the grayscale means that you have to add some new information to what's already there because you don't have any information about the color so that way you do have to understand that hey if you have a tree you know the leaves are green and like the bark is brown so you have to identify some of the objects and try to


like learn features about like edges and so on so so so that's the goal of this this line of work so it's best visually illustrated here so grayscale so an image can be just like you have RGB there are like various different color channels parameterizations of an actual image and instead of using the RGB color space you can use the L a B color space and the L channel will behave like a


grayscale image and the a B channels behave like the color image and you can use the L Channel encode it and predict the a B channels and that is basically the tasks consequently considered in the learning to colorize work so you can see that those yellow light pixels are identifying the eyes of the fish of the body and but you can also see that because the background is Queen starting


with the color of the body of the fish it's it's not able to separate it out so uniformly color is the background but it's able to color the coral reefs around so incident francium is green so you can see that it's able to understand some high-level aspects of the image by doing this task so so that's basically what's going on and to visualize what how the actual image looks like you can


just concatenate the two channels and see how it looks like so that is the ocean and here the author's first tried the new idea which is take your draw a ground truth then you just do a mean square between your predicted and the ground truth and that would producing a very degenerate like degenerate like colorization of the actual bird and the ground truth is made


more colorful and diverse so what the author has realized is instead of treating the prediction as a mean square a regression task what if you treated as a classification task where you quantize the pixels so you quantize the a/b channel information into a button and and and and bit it into a bunch of categories and now instead of predicting some value for the a/b Channel and just


regressing to the ground truth you're going to actually output a distribution over the possible values because it won't test and we can then do a softmax puzzle to be lost instead of mean square error loss and it's in general it works out really well and deep learning to do this you've also seen how it worked out really well in pixel RNN and where where all the pixels were quantized to


discrete categories instead of using a Gaussian mean square error going to use the process to be lost and that produces sharper images so here here's how it goes you you basically take an image you have the a be channel prediction you quantized it and now you're going to use Krauser to be lost to predict your actual a B channel information so that is basically how the colorization work


is done another version of this done by Richard Zhang was trying to do something like a split brain or encode or a split view or encoder which is you've separate the channels in your source you have encoders that try to predict the other channel from the current channel so where X 2 hat is the other channel prediction and X 1 hat is the prediction for from X 2 and now you concatenate


these two channels together and you get your actual input again and you want to make sure that this version matches your original version so so this way it's like ensuring a backward consistency in some sense because it's not just about predicting the color from your grayscale it's also like hey the a B channel should also make sense like whatever you predict from your a B channels your L


channel that the and what are we predicting I'll channel to a beach on together if you look at it it should make sense and it should look like my actual image so this can make more sense if you're looking at other kinds of views like depth image and color images so here's here's like one way to implement this which is you you separate out you have two different encoders you


predict the other channels missing channels you concatenate and then you have a loss on the actual predicted image so and this is how it would work for a color and depth information so so so these are all interesting ideas and we're not gonna really look into the metrics that these methods have because more than the metrics these papers are more famous for the input-output version


itself which we looked at pics depicts where the fact that even colorization can work so well is so peeling but but but in terms of the metrics in terms of the numbers will we look at them much later when we look at contrast Aloni so here here is here is the other final the second version that we wanted to see which is introduce some kind of common visual common sense tasks so so so here


we are going to look at relative patch prediction this was an idea put forth by Kadosh abhinavagupta ala al Alex EE froze and so in some sense this was one of the first papers to do cells crew as learning on images at a larger scale and it's considered one of the foundational papers for a lot of the ideas put forth so basically what is the task the authors considered here


the task was given two patches try to identify the relative position of the two patches which is to say that if you have a center patch and if you have a patch linked to its immediate right and given these two patches to a neural network the neural network should say hey this patches to the right of this reference patch so look at look at its best looked at from this figure so you take an image


you take an approximately three by three grid of non-overlapping patches and now given the blue patch you're trying to predict that the yellow patches to the top right corner so you can number the surrounding patches from one to three so on until eight so and you basically have eight categories for a classification task given the reference center patch and this way you can take any you know


different regions so the same image or like lots of different images you just have to take an approximately 3x3 grid of non-overlapping patches and give it to your neural network select two of them give to your neural network and create these labels for free from your data so it is it is another version of supervised learning where you you are actually adding creating a task like a


jigsaw task sort of not exactly jigsaw but you can you can think of it learning spatial associations and then you know then it has to understand hey if you give me the year of the cat and the eyes of the cat it's then then it's likely that the year is lying on top to the right or to the left and and it also has to understand that what is what is left in right here


so so so that means that it's learning these low-level features and as well as high-level associations and it's way that could be useful for a downstream task so that's pretty much it you you you share the CNN in chorus for the two patches you use the mean pool representation at the end and you train a classifier on top and you can create a lot of data for your training tasks because you can


sample a lot of different batch grades from a lot of images and see how good the features are so there are a couple of the details in in in getting this right which were extremely crucial and which have been adopted in every follow-up paper almost which is making sure that you jitter both spatially and color wise so firstly you should make sure that to pick the patches don't


overlap or else it's so easy to get the well which branches like to the ref to the right just by looking at the boundary pixels so that's one obvious but but but highly non-trivial at the time but considered obvious now so you make sure that the patches don't overlap second thing is you jitter you jitter the patches to prevent chromatic aberration so so that the but by that


what I mean is you you sample you basically you you sample a particular random crop view you can you divide it into three by three grid of patches and then within each cell of the three by three grid you be you basically create another random crop and drop some color channels and so you basically do spatial and color jittering at every single patch to prevent the chromatic


aberration from happening and that way the neural network can cheat and both of these details will have a non trigger at a time and and they were very crucial and all the follow-up work so another version which is very similar to router position prediction is you know actually going all the way so rather to position prediction looks looks like a jigsaw task like like the


jigsaw processed children saw so why not actually do exactly that just just make you don't have to saw six of puzzles and that's what this paper is doing which is you similarly take a crop three by three grid of matches from a random crop of your actual image and you shuffle them and then you try to predict what is the correct order of the shuffling so in this case you it has to identify that


similar kind of spatial reasoning and associations so that that's really what works well here and and and and if the neural network is able to solve this task well it means it's able to understand what this comes to the right and left it's learning general visual reasoning so how is it implemented a very easy very implement this is you if you have a 3x3 jigsaw puzzle task then


there are 9 factorial possible permutations so instead of asking the neural network to output the exact order you can ask the numeron that will top with an index of that and you can hash the corresponding order in a hash table of all possible permutations indexed by like some simple declare category and you can just have the neural network to predict the category and that way the


neural network is the you know doesn't you don't need an RNN decoder or something like that and and and it just looks like a normal classification task where every single patch in that word is passed through the same shard see in an encoder the meaningful representations are taken and they're concatenated in some form and tried you just trying to predict this output category so a final


version of a final version of this idea of doing puzzle tasks creating data creating tasks from raw readers wrote the very simplest version is rotation prediction this is really really so simple that it's amazing a works over but but there is also a concrete argument as to why it works the idea is you take an image you rotate it by a random angle so in this case you're


rotating it by a multiple of 90 degrees and this C and you pass it through the convolutional neural net and the conversion here and that is asked to predict what is the angle you rotated the original image by that's that's really it so in this case for the first time as you would predict the conversion and that has to predict 90 degrees for the second it has predict 270 for the third


180 and so on for the fourth it just has to predict there's no rotation which is a zero so why does it learn something nice why does it have to learn anything at all so if you look at the 180 degree rotation like the only reason you're able to say it's 180 degrees is because there are human faces and you know that they're inverted right so that means to be able to say this 180 degrees you've


identified that there are humans faces in the image similarly like look at the first image there's a bird and there is it you know there is a tree and so on but but but you know that if the normal view would have been that the bird is like if it was tilted by 90 degrees the tree would have the bark would have been horizontal and the bird would be standing standing in the same vertical


position so this is basically trying to identify if there was a photograph for that captured the actual image how is the photograph of position right so so so that is an inductive bias that is physically or geometrically grounded which is Kam image formation is something that's fundamental and that's how we all record images and most of images on image net have been captured


where the object was very centralized and so there's a lot of information as to why the camera was placed at what what pose and so on to capture the actual object so because you rotated it you're actually trying to identify in some sense you're trying to do inverse graphics of the camera parameters here which the only parameter you care about is the rotation angle here but but since


it's physically grounded it's gonna learn something useful so here is how they implemented it which is you take an image be rotated by various possible angles and you can struck these rotated versions you pass them to the same convolutional neural network and it is pretty clear rotation angle so you just do it for all possible images in your dataset and you you


you would learn really good features so one interesting point is you might think hey the more angles you had to a dataset the more good features that you can learn that's not particularly true so the authors found that if you just use for rotation angle 0 90 180 270 the and you put a linear classifier on top and train it on C for this is a small data set you can get 89% op1 accuracy but if


you add multiples of 45 degrees which is like one level more fine-grain your performance drops by 0.5% approximately and if you just use two angles which is less fine-grain the performance drops even more by two percent and and if you think about it as vertical versus horizontal like 90 270 instead of 0 180 the performance drops like another two percent so it is particularly important


to use both horizontal and vertical and it is more important to use vertical and it is also not so important to use a lot of angles maybe the amount even that that's breathing eight rotations wasn't trained well but it's sufficient to predict four rotations you don't have to be so fine great so here the results that they have where they basically train an allit so at that time it will


be Alex net was the common backbone used for self provision studies so they basically trained an alux net on all these different tasks like like for instance rotation that is there rot net was there paper but the baselines are coming from the car existing substitution papers which we also covered so if you use the conlusion l 4 & 5 which are the fourth and fifth


collusion filters in alex net and put a linear classifier on top the the results from just using supervised learning features are the topmost row 59.7 percentage top one which is pretty close to what alex net gets in terms of top one accuracy so just using the random features gives you like 27 percent and 12 percent respectively and the context paper doors are always the paper we saw relative


position prediction and and that's able to get fork on forcibly at forty five point six percent which is which is way better than random but not as good as supervising internet and colorization work which we saw the Richard Zhang's work that's five percent lower than doing relative with a transposition prediction so you can clearly see that doing more puzzles like tasks is better


than doing colorization and the jig suppose a task is on par with the context the relative position prediction it's similar 45% and bygone which is a paper we already covered in the again lecture is not as good as these puzzle tasks but it's on par with colorization and finally this rot net paper has a substantial improvement over the state of the art at the time so basically the


state of the art self supervision method had forty five percent and rot net accrued it to fifty percent and so that that's real clearly good and even on the con filer it shows like for the doors at all the the numbers are really low like 30% and the rot net is like clearly better forty three percent and way better than the other methods as well and these are like more more detail


results for various convolutional layers and you can see that the rutland numbers are significantly better than the other self supervision techniques at the time and it also transfers well to pascal so on pascal classification and detection and segmentation it ended up being sphere the arts of supervision method and was significantly better then context prediction so but but still


the gap between rot net and imagenet labels is really large if you look at the detection results it's close down like like fifty-four point four fifty six point eight is pretty pretty small but on classification the significant gap was seven percent and on segmentation there's a significant gap of nine for me not being mean my intersection over Union so so while this


was a pretty promising technique it was still not there yet so that's it for like puzzle-based us next we'd actually get into this context based prediction techniques so predicting the neighbor context was the final line of work that we wanted to cover and again like I wanted to go back to this likkle slide where you you're basically interested in tasks in the neural network to predict


missing parts from given parts or like neighbors given the neighbors you're trying to predict the surrounding context so one idea which was explore way back in 2013 was this word to back and we're going to cover that for us because it's very foundational so this is a figure from the bar from the 2:00 to 4:00 in class with Stanford and where the goal is to learn good word


embeddings so we're the bearings are very fundamental like you have a lot of words in the vocabulary and you would like to represent them vectors so that similar words are having similar kind of vectors either directionally or they are very close to each other in the high dimensional space and the name word embedding you can use this is one hard encoding and that's hardly infinitive of


any similarity across words so let's say that you have a bunch of sentences and you create account matrix of what words are occurring how many times so in this case you're basically trying to do some co-occurrence matrix if that that's very popular in NLP which is how many times each word co-occurs with the other and that that's usually used to construct these similarity matrices and


so this is how the count matrix looks like so I in like are occurring together because there are two sentences with them but but IND don't go together like like basically once this matrix is constructed you can think of applying single value decomposition to this matrix and this is really how recommender systems have been built in the sense that you will have a


history of what uses goodbye war items and you would construct a user ID matrix and then you would do an SPD on this matrix and you would get a user embedding in an item embedding and you just cluster or similar users in similar items and you will use it to build a recommender system so similarly you can think of building a term frequency resistant frequency matrix here in a low


peak and constructing a sweetie and get word embeddings so that's that's precisely what's being done here and you can get these U and V vectors so so what is the problem with the SVD approach this video prods the first one sparsity right so obviously there may be very related words which should be not necessarily co-occur with each other because you just not have that possible


sentence so sparsity is a big issue the resulting matrix you construct is very likely to be sparse and the computation cost right so SVD is a third order to compute so it's not going to be easy to optimize and there's also this problem of infrequent words which is when you have any dope when certain words are not particularly frequently present they're going to be hard to optimize because the


word embeddings for them will not be very accurate and you can also have noise from very frequent words like like like you know words like a or D or articles and these are going to be very frequently present so you have to use some heuristics like inverse document frequency to make sure that they don't corrupt your bearings and it's however all of this sounds like a


very hockey engineered pipeline and it's not particularly efficient and it's not going to scale well with like larger larger the vocabularies or like larger data sets it's it's very hard to scale this approach so that comes the idea of using Engram language models which is hey you just have if you just have a bunch of terms and in a sentence you say that the probability of sentence is


probability of individual trans present in it and that's the unigram model but a bigram model is like the problem you you you bet you basically try to say that you take into account the previous word percent in the sentence and say that the probability of a word is conditioned on its previous word and that's saddam v you can start counting pairwise occurrences of words instead of just you


know frequencies of single words so an Engram model basically generalized stuff so so let's let's actually go to the word Tyvek idea which is going to clearly generalize all these things and this is first proposed by niccolò back in 2013 and so we're two back what's the area here you're going to have a bunch of surrounding words and you're going to try to predict the center word so you


have a sentence you're you pick a particular word and treat that as a center word make all the surrounding words treat them as surrounding words and embed each of them and try to identify what is the center word from the surrounding woods so that is referred to as the Cibao model and the script skip grandmothers exactly the mirror image of this model which is it tries to


take the center word and it tries to predict all the surrounding words so one way is you embed all these individual one heart and coatings of the original words into a word embedding matrix which is basically like the word embedding matrix would be the number of resistance embedding dimension so by embedding one heart word it's basically if we went looking up what what what the


corresponding word embedding is and the C bar model would just average the embeddings of your surrounding words and try to identify the embedding of your missing word and the skip grammar model would basically try to use the word embedding and do an individual softmax over all the surrounding words so let's actually look into the math of how this works out so consider the Cibo Matto and here


you're trying to maximize the log probability of the surrounding word or the context word given the neighboring words and so you can think of this locked bu WC given all this WC minus ni for different values and so so so the way it's actually constructed is huge is average the embeddings of your neighboring words it's a very simple model so once your average that you just


have a single vector and now you just say that this is a nonparametric softmax over all possible words that that you can have for your Center word and so that way you don't have to explicitly take a soft max you just optimize for the dot products of the averaging words from the neighboring words and the possible product of you are sent over and that's what this loss amounts to and


oh and this way the parameters of your loss function end up being your word embedding matrices and all you need to do is take lots of different chunks of text pick particular sent over to pick some respondent neighboring words embed them average neighboring words try to maximize the dot product of the actual central word relatively all the other words in your vocabulary and if you do


this for waste chunks of text and optimize for a while you're going to end up with a relatively good word of any so that is the idea of what effect the SIBO model and the Skip Graham model is exactly the mirror image of this model where you're going to try to predict every independent sounding word with the key assumption it makes us that given the center where the surrounding words


are all independent of each other that's in the probability of a surrounding word given the Center word is independent of the other surrounding words so that's kind of similar to the Navy base assumption which is given the class the term frequencies are independent of each other so it's similar assumption to simplify the computation and so then you just have a similar kind of


nonparametric softmax over the possible word embeddings of the surrounding words and you just perform a similar optimization so a main idea here in the nonparametric softmax is you need to optimize over all possible word embeddings in your vocabulary and that could be very computationally inefficient especially back in that time when GPUs were not the go-to mechanism


for deep neural nets and the main aim of where Tyvek was to have some software that where you just feed in a chunk of text and it could just run and spit out word bearings for you but this whole process can just run on a very simple lightweight CPU so the authors went for very clever techniques like using negative sampling and not normalizing over all the words in the vocabulary in


the denominator so the partition function is now going to cover the entire vocabulary and as long as you pick really negative samples you can be very efficient in terms of the kind of embeddings you learn and you don't really and and and so we won't really go into the details of that but you can refer to the paper in terms of what how hierarchical softmax is and negative


sampling was used to make word to that really efficient and in terms of results the authors had really good results for the resulting word embeddings that we learned for instance here if you look at the word embeddings of different countries and their capitals you can see that the difference vectors are all quite parallel alright so the vector from China to Beijing and Russia to


Moscow they're almost parallel so it means that the relationships captured between the aspera the words are easily lay you can translate these relationships easily to like another capital and you basically have to translate the country by the same translation vector so so it's geometrically very consistent and you've also seen earlier about how DC Gann was able to do vector arithmetic on like


Sarah Brady faces it was all pretty inspired from word to back because you can because it's geometrically translation consistent you can think of taking the vector of Portugal and you know you can add a you and you can take the vector of Spain you can subtract those two vectors and you can think of the difference vector being similar to the difference vectors between the


capitals Lisbon in Madrid there are difference in the position vectors so so here are like various different you know clustered word embeddings for categories of different different categories of words okay so in word to back well the authors tried to look at how the word embeddings cluster together and you can see that various newspapers are clustered together


and various NHL teams and ba teams you can actually see that if you take the nearest neighbor of if you take a nearest neighbor of Detroit you get Detroit Pistons or Oakland and get Golden State Warriors off you take the nearest neighbor of Steve Ballmer you basically get Microsoft for Larry Page look at Google for Airlines you basically see that Spain and Spain are


closes to each other are same for Greece and egg and Airlines so so you basically can please see that the word embeddings because they have looked at what terms are good to next to each other and so forth they've understood relationships between companies and their CEOs or like you know Airlines and the countries that they operate in and so forth and that's that's true that's really interesting


I'm just going to switch to PowerPoint okay so so so next thing is how does the clothes and tees relate for different short phrases using the vertical bearings so here is one thing about various explorers like let's go to gamma and you can see that there's a relationship between Italian Explorer and for chess master chess grandmaster there's a relationship between Garry


Kasparov and and so so even these sharp phrases the closest entities that are being very exactly relevant and you can also see how it's relevant to the airlines and you know or or if you add two different embeddings what is the closest entity you get so for instance if you add Vietnam embedding for Vietnam the embedding for capital you're ending up getting the


closest and the closest nearest neighbor and bearing up that of Hanoi which is exactly right and similarly if you had the embedding for German and Airlines and it flew it searched for the entity or the word with the nearest embedding you're getting airline Lufthansa which is really cool and similarly Russian plus River get Volga River Moscow and you get various


French actresses and and you also get the currency for Jack so basically it's understanding relationships at phrase level not just word level and understanding relationships between multiple phrases so another interesting example is how can get all these different skip phrasing bearings so you can see that the closest tokens for skip grammar models are like way better


compared to the other models so these are like different models and like me at all is another example of noise contrasting model those strain on words and skip phrase model too McCollum or Oh like like a basically the nearest neighbor to the topmost row redmond is they the most relevant in the skiptrace model compared to the other models for instance redmond Ross Redmond


Washington Microsoft and you know graffiti he get spray paint and graffiti taggers etc whereas for the other things for instance for graffiti the nearest neighbors from these model is basically things like anesthetics monkeys Jews which doesn't really make sense at all so so that way you know it's understanding the actual things in the script our model so next we look at this


paper called representation learning contrast to predictive coding in some sense the best way to understand CPC is if someone were to do work back on all the modalities and not just text how do we go about it right so you remember that word to back has a very interesting model but it's also very primitive that is you if you look at the c ba model you know you're averaging the embeddings of


your neighboring words and then try to predict the context word so but average this is like a rape a you know you don't it's only important if you really care about the simplest possible linear model that you want to get working but in principle you can aggregate context using neural networks right like we have really powerful neural networks for context aggregation like continents or


transformers or LST amps so why not actually try to put this all together use the contrast loss that were to accuse is to predict the neighbors in a nonparametric softmax but try to replace all these embeddings of individual words and surrounding words with very powerful expression you know networks so that forms the basis for contrast the predictive coding then by Ironman the


New York so here's the idea let's say you have an audio signal the raw audio signal and let's say that you're trying to predict the future audio signal from the past or you're trying to for relationships between the future audio in the past audio so call the past as see a context see and call the future as X and instead of predicting the actual audio like a vein net what do you want


is to just operate at the latent space so let's say that you encode the context in with an encoder and you also encode a future audio chunk we call you can call it as a target and you encode the target at the same encoder so now our goal is to maximize the mutual information between the context and target don't really worry about like what mutual information well why does it come out of


nowhere that's not really the objective here the goal is we want to make sure that we learn a representation the encoder sits that its maximally predictive of the actual future when contrasted with with some fake futures so so that's that's really what's going on in CPC which is hey you produce your neural network a bunch of targets where your actual target is also present but


you could also have some fake targets so imagine that you could sample random audio chunks from totally different waveforms or you could also sample audio chunks which are not exactly corresponding to that future time step and you could present the inure network various different alternatives of what the true audio chunk should be and based on the past context the real truth and


all these other possible alternatives the neural network is supposed to pick or classify which is the right future and similar to word to back this can be done in a nonparametric softmax fashion where instead of actually decoding the audio chunk you're just trying to make sure that the embedding of your context and the embedding of your true future should correlate the most when


contrasted with the other fake audio chunks so so that so that way becomes the softmax or your number of negatives and and and and you can actually use any kind of a score function for how to assign the score between two embeddings it could be a simpler product in contrast pre-recording they make use of the bilinear dot product which is which which is also been used in the past in


past work so that's a little more expressive than just using a regular dot product and it doesn't require you to normalize the vectors so so that way you can think of the W matrix in the bilinear product is learning some kind of Association matrix that figures out some property that helps you to correlate two different things so that's really what's going on in CPC


you're basically going to take a huge audio chunk you are going to use audio signal you're gonna split it into small chunks you're gonna include each of these small chunks with a shared encoder it could be straight conversation on your network in this case and you could take a bunch of past audio chunks pass them through it gru any auto regressive model would do and you could use the


final gate in state of the gru to predict all the future Leyton's of true audio chunks but predict I just mean that your contrast elite trying to maximize those embeddings through the true futures when contrasted with the fake futures and you can sample negatives by using other time steps within the same audio waveform or you could use other audio waveforms and


depending on this negatives you're learning different things for instance if you're trying to collect audio signal from different speakers the negatives that come from other speakers lets you learn representations that allow you to identify the speaker while the negatives that are across the same waveform within the same speaker that you let's you learn like more fine-grained phoneme


features which are useful for like phoneme classification so depending on the downstream task the kind of negatives you pick are going to be crucial so here's basically CPC at a high level the diagram is very very very clear you're just basically gonna try to do this across various audio waveforms various different numbers of time steps you should be very careful in picking


what is the gap between the future and the context so there are overlapping audio chunks just like how you saw in the door shadow work where if you had overlapping patches it's going to be very easy to do these jigsaw puzzles Dallas CPC basically also suffers from the same problem so you should make sure that your negatives are come or your actual prediction tasks are not so


trivial enough that you could just look at the whatever is overlapping and just try to predict that so that's one really important thing about CPC but the more interesting thing is that you can actually perform all these tasks in any kind of fashion you can't you like you can so that the order of time is not so important you can pick anything as the context and


anything is target so you can actually predict from the future and go to the past as well or you could even mask something in the middle and use everything else as to context so it's it's totally up to you to how to frame what is the context and what is the target in CPC but based on how you frame it you should make sure that the negatives and the targets are chosen in


a non trivial fashion so you can clearly see that CPC is trying to generalize the older ideas like puzzle tasks and were to back together it's basically a framework in which you can perform all of these different tasks within one one particular architectural variant and and various different hyper parameters will correspond to various different versions of these different tasks so you can


think of CPC is trying to do something like multi-step prediction tasks so if you look at these multiple predictions emerging out of CT vector in the diagram you can think of a different W matrix being used for each of these different prediction time steps and each of them corresponds to say hey predict one step ahead or parade two steps ahead predict three steps and so forth and you can


think of all them is trying to make the representations learn different things and because you are optimizing all of them at once you're trying to learn a really rich representation that is able to do lots of different sub supervised dance at once so it constructs a whole variety of pretax tasks within a single loss a single framework and it's railing and then appealing that way


so one figure which is really nice to understand what CPC is trying to do is how it basically is doing something like slow feature analysis but what I mean by that is you your audio waveform is really high-frequency fast wearing and you're actually information that you care about for downstream towns is basically the slow bearing like hike signal content like phonemes because


that's what really allows you to use CPC features for something like speech recognition so what what CPC is trying to do is its trying to make predictions in the latent space at the levels of phonemes instead of variety of audio waveforms so as you keep processing these are your signals the information becomes more more semantic and so if you're trying to predict something


that's not overlapping in terms of a target if you're trying to predict your target that's reason to be a few time steps ahead you're trying to go more towards these slow varying phonemes that are that you know they would only change if the time steps are sufficiently further apart and so therefore if you're doing predictions are an appropriate offset by offset I just mean that the


gap between City and zt+ cave are the case like a number of time steps between the sufficiently high sister the phonemes actually change then you're going to end up learning really rich features so this is a really nice visualization of the representations learned on CPC audio audio tasks we're basically collecting a data set of various speakers as audio waveforms and


just performing the CBC optimization and your for and then you're taking the embeddings out and doing a piece near visualization in 2d and you can clearly see that different speakers having clustered out in separate blobs so it's clearly garden the speaker initially and you can also see the accuracy of predicting the the you can also see that the accuracy of predicting the positive


sample and the contrast of loss you it basically is so high in the beginning but it keeps going down touristy and by that what I mean is it's much easier to perform this contrast to task of identifying what is the right future when the prediction task offset is not much when you're actually trying to predict much closer to the future but as you keep moving further and further away


the mutual information between what you already have and what you're trying to predict is much lower there's much more entropy so you're not actually would optimize those future time steps as well because the context is not sufficient so the accuracy drops exponentially as you keep increasing the number of time steps you're using to predict the future so here are the CPC audio results in terms


of the downstream tasks on the left you see before the phoneme classification and speaker classification results and for phoneme classification there are four even possible classes so basically the way it works is you take the CPC features you freeze them out and you just put a linear classifier on top of these CPC features and you try to perform the task but what it look like


which is to say you try to identify the phonemes in the audio chunk or you try to identify who spoke this particular thing and you have labels for that but you're not gonna change the features you're just going to keep them frozen so for that version CPC speaker classification gets ninety 7.4 percent accuracy it is very close to what you'll get by just doing supervised learning


whereas if you just use something like m of CC features which are which is very engineered you're not able to do that well you're just able to get 17.6% so these features that you learn in a completely unsupervised fashion are v better way more semantic than something engineered with domain knowledge and also for phoneme classification which is actually even higher than speaker


classification CPC features without any fine-tuning just linear classifiers able to do is able to get 64 point six percent way better than MCC features which is this forty percent and we're better than rounding random initializations 30 percent and supervised learning gets seventy four point six percent which is 10% better than just linear classifier on top of CPC features however the


authors found that if you put an MLP instead of a linear classifier you can actually get pretty close to 74% with just CPC features no fine-tuning so this means that the information may not be linearly separable but all the useful information for performing foreign classification is there the CPC like the uncovered features and on the right you see a positions for phoneme


classification experiments and the the point I mentioned earlier is are really illustrated well here where depending on where you take the negative samples from it you're going to get different levels of results so if you if you if all if your negatives are all coming from the same speaker the accuracy is like sixty five point five percent that's that's basically for that that means that the


models like learning only phoneme relevant features it's not trying to do speaker and if occation whereas if you are if your negatives are all coming from mixed speaker so that that's going to get sixty four point six percent which is the result on the left table so that means that if you are more clever about sample negatives we already know what your downstream task is you could


prioritize something negatives in a fashion that will incentivize CPC to learn the features that will be more relevant for your downstream tasks so if you're if you don't really care about speaker identification you could just make sure that all the negatives are constantly coming from the same speaker and so that's really interesting way to illustrate this point and and and and


the second ablation that they did is the number of steps you predict into the future and namely you would imagine that predicting all the way into the future is going to be really helpful but it doesn't turn out to be the case so if you predict only up to twelve steps instead of predicting 16 up to all the way up to sixteen steps the downstream accuracy is better so this means that


the right way to predict it is such that the targets that you're trying to predict should share so amount of information with the context that you already have you go further and further into the future the entropy is higher and so the amount of actual information amount of bits that are shared between the two entities are not much so predict like for making a neural


network focus on those stars may actually end up encoding not very useful features so the hard part about CPC is trying to pick the right number of time steps to predict into the future or like they're how you sample the negatives but if you get those details right the features learnt are really useful and on par with supervised learning so one motivation for CPC was that they wanted


a single framework to work on any modality any kind supervised learning any modality you should just be able to use the same framework so that's a lofty goal so let's see how they actually instantiate it for image not so here is the image snap numbers for CPC where the framework how they actually executed is as follows you take an image you take these overlapping patches so you grid your


image into a bunch of overlapping patches so in this case the image is 256 by 256 from image net and you're basically taking a 64 by 64 patch laid across the image and you take matches with 50% overlap so 32 by 32 is a stripe and that mean you get a sent by seven grid of patches and you would encode each patch it with the same rest net so think about it as a rest at 101 or less


than 50 and you would get an embedding the meaningful embedding at the end and what you do with that is now that you have an embedding at every single patch this will form a grid of embeddings and you can perform predictive tasks 2d predictor tasks and on top of this grid so you can treat this great as your sequence just like the audio sequence there's a bunch of overlapping audio


waveforms audio chunks from your actual raw ear signal in this case it could be a bunch of were lapping patches in this 2d transcript and the task that the artists construct is to predict the future parents from the top row of patches so in this case you basically using the first three rows let's say and then you're trying to predict the bottom three rows you try to


predict every single patch in the bottom three rows by using the row the row of matches from the top three rows so in order to aggregate the context of the top few patches you would want to use some kind of a model that can take a bunch of embeddings in two-dimensional layout and try to summarize what is embedding screen at every single spatial location and that's


and you also want to do it in such a way that you don't want the information from the top to leak into the bottom because you're trying to predict the bottom for the top so we already know of one model that allows us to do this very efficiently it's called the mask convolution or the pixel RNN fix the scene and style models and and and and it also makes sense because CNN's were


also invented by the same first author so he just use pixel cnn's to aggregate the context of the top few rows of batches and once you lay that out on top of the grid of matches you can predict the bottom patches in a very parallel fashion so here is an example of how it would look like for an actual image when you grid it into patches so take this dog and you're like constructing all


these patches overlapping patches and then you let's say you you take the first three rows of embeddings and you try to predict the last two rows or which oh this is dispatch belonging to the last row second column or not it's this patch belonging to the last row third column or not so you would perform all these predictive tasks once once you get all these embeddings of individual


patches in a pixels in and on the top so how does the accuracy work out for like say you you're doing something similar to the audio experiment where you trained all this and a lot of images and then you take just the rest net encoder out and you put a linear classifier on top of it and see how well it performs on an omission air classification which is also the standard test being used in


previous self supervision methods though initially they were all attempt with Alex net and but but the baselines for rest net existed this table so we've already seen relative position we've seen by gal in synchronization and and jigsaw puzzles so all these methods when you just use a rest net encoder and you put a mean pool at the end and you put a linear classifier at the end D we get


you a top one accuracy like not more than 38% jigsaw works the best and the rot net numbers are not there on this but they're out net numbers are not are not higher than the CPC version of the CPC numbers so so if you look at the Aleks net results they are like around 38% and if you use a resident we do every single baselines numbers goes up so relative position gets a 6% game by


just using a rest net video instead of an alux net and colorization goes to 39 from 35 so the baseline for jigsaw doesn't exist here but I would imagine it getting to somewhere in the early 40s so CPC gets forty eight point seven this is really an old result now we will see in the next few slides how the state of the art has been pushed up way further but at that time this was a pretty big


jump from the existing state of the art and it also works really well if you look at if you look at competitive approaches of similar nature like you know a relative position of jigsaw is like kind of performing these spatial Association tasks too but doing it in a contrast of fashion and doing lot like a family of tasks within one parametrize model gets you much better numbers and


these are the standard reason we should visualize what kind of features that these models learn which is you take a particular feature there and you just see what new what what what kind of input maximally activates a particular neuron and you lose for a bunch of neurons and you see that you know like maximally activating neurons are the corresponding to different classes in


this case the first row is corresponding to leaf like patterns and textures calculator textures computers keypads and then skies and baby faces dogs so so there are clearly capturing all these high-level omission of features so another version of CPC was to try it on language will not really go into the details here but it but at that time the it was competitive


with skip top vectors which had similar similar ideas like predicting the future sentence when we give him the pass so CPC was able to somewhat be competitive or match those numbers but not not really that good finally it was also applied in reinforcement learning so in reinforcement learning you can think of accelerating your data efficiency by allowing your model or


agent to learn way faster by performing these unsupervised auxilary tasks in parallel along with your reward optimization and the authors try to use contrast of losses as the auxiliary losses and they were able to see some gains on sample efficiency will not really go to the details here as well so that's basically it for a CPC version 1 or CPC as it is called because the


fundamental ideas were put forth in this paper but like like like I said the numbers are not that great yet right like if you look at these numbers the linear classifier with the rest net gets 48.7% whereas a supervised learner with the rezident we do architecture typically gets like something like 76 percent top one so the gap was like really high the like around twenty eight


twenty nine percent and and and and so that needs to be addressed so until that's address self supervised is not really worth doing for practical amused classification or practical like this lofty goals we started off with which is hey we just want to learn features from data with our labels is that you can get similar quality of features so this is clearly far away and that was the


version that was the goal in in CPC version two to address that gap and this is a work I did during my internship at equine well with Ironman dinner where we basically took CPC version one and kept on hacking the architecture the the different type of parameters needed to get get all various patches and like add a lot of details in terms of data augmentations and and see


how far we could push the number so and what we ended up doing was like like we actually were able to match or sometimes beat supervised learning on various downstream tasks and I want to go through the details here so you've already looked at this where you grid an image into a bunch of matches and you encode every single pass using a really deep rest net so earlier you saw that


arrests at 101 we to the first three snacks were being used in the first CPC version this mist net is much deeper arrests at 161 with 2x wit and your third snack so it's it's having 4,000 features at the end and once you do that you get the embeddings for every single patch and you process that with pixel CNN this Pixar CNN is 2x wider than the pixel CNN use in the original work and


now you're trying to predict the bottom future like just like an original work but here we just use one offset we only predict two rows below and nothing else so like it's like you already saw in the audio experiment doing lots of predictions can hurt you if the amount of information shown art is much lower so we and also doing predictions when the information overlap as much closer


will also hurt you so you need to pick the prediction step very carefully depending on your Kratz eyes and so forth so we only predict two rows below and nothing else and so these are your context and latent vectors and you have the same kind of scoring function which is by linear product and you optimize this with a nonparametric softmax over there negatives and like I said the


naked in see how you sample the negatives is really crucial so you can sample negatives by taking negatives from other patches within the same image or you can take patches from other images and we have a version called all neg which is basically taking all the possible negatives you can you can construct which is all the patches in in your whole mini batch which is your mini


batch will be a bunch of images and each image is now a bunch of patches you just take all of them as your negatives in this particular loss so that way you get a lot of negatives this whole stack of optimization is like in general it doesn't matter how you construct the negatives of positives whether they use patches or not but just this whole framework because in general refer to as


the info and see a loss like you very basically construct contexts and targets and try to use contrast objectives to optimize for the associations and the implementation is really parallel because we can just use a pixel CNN with mass convolutions and you do the predictions at every single local position using a one by one composition so the recipe for CPC v2 is this train


on unlabeled image net train as long as possible so we trained for 500 bucks and this basically takes you like approximately a week and you augment every single local patch with a lot of species and color augmentations so like I already mentioned in the doors work on relative position prediction making a lot of spatial jitters is really useful so we take that to the extreme in this


work and use all possible augmentations and the effect a number of negatives that you have is number of instances in your mini batch times the number of patches for instance so unlike the earlier work which gridded the image into 7x7 grid of 64 by 64 we actually use much bigger patches and much bigger images like we used to 80 by 280 images and 80 by 80 crops so that that gave us


6 by 6 grid and with an overlap of start of 36 or something is that and so so that way the number of negatives is approximately 600 so so the fact is like we don't have a lot of negatives but all these negatives are really hard because they're coming from other practices within the same image so it's it's a mix of like instance negatives as for the spacial negatives


and it learns both kind of discriminative features so this is basically a diagram that illustrates this whole pipeline where you perform you you you have this feature extractor which is the rest net1 61 running on patches of images and you train the cells provision objective which is CPC and you do that for like a lot of long time once the mod of the strain there


are various ways in which you can evaluate the kind of features that you learn one is to take put a linear classifier as was already done in the past the other is you just take the rest net that you uncovered and instead of freezing those features what if we can actually find unit on a classification task so which is to say that instead of training a linear classifier and all


possible available labels what if you're allowed to put a small model on top finding the entire stack when the situation is you're not going to be given all the labels you're just going to be given a few percentage of the labels so you're allowed to train on all the unlabeled data you have but when you when you're beginning to perform supervised learning you're you're gonna


be given label data on different shards you're not gonna be given all of them but we also have benchmarks where you have all of them but in general as imagine the scenario where you can perform classification even with like 1% of the labels which is like 10 images per class and imagenet and you can also do a transfer learning which is you you're given a completely different data


set now you take this rest net throw it on that data set and perform new tasks so which could be something like pascal where you just take that rest that you've got and CPC and you throw it on object detection just like regular computer vision in benchmarking so so that's basically the goal and we will see how all these things work out if you do a lot of engineering so CPC we


do linear classifiers core remember that CPC v1 got 48.7% so just look at CPC v 1 and CP CV to CPC v2 gets 71.5% which is significantly larger than 48.7% and around that time a lot of competitive approaches were published with really good linear classifier scores as well like big buy again which is a largest Caleb I can push it up is 61% I'm Tim was another technique which is


also using something very similar to CPC that push it up to 68% and all these different methods just try to go for the same approaches just make your models as wide as possible or as deep as possible or a combination of both and optimize really long and use a lot of augmentations and careful engineering and CPC was the first method a shape improve and get to all the way up to 70


70 plus and the top rows where the models were different models were using different encoders so it's really hard to see what is helping there so on the bottom you like you can see that you just use the same rest at 50 encoder and then you compare across methods and CPC is better than all the existing methods this was the story when CPC version 2 was published but it's no longer the


case there's also a base line here in this table called momentum contrast which we'll cover as the next topic but note that momentum contrast has also improved a lot from the numbers that are presented on this table so on efficient image recognition which is you take the CPC features and you find unit for supervised learning where you can actually control the amount of label


data you have CPC version 2 is able to perform significantly better than just doing supervised learning so the red line is basically like for that corresponding percentage of label data you just do supervised learning so you just take a rest net you've trained it on whatever labels you have so you can clearly see that that would really well if you have all the labels but as


you reduce the amount of label data the rest net performance keeps going down so supervised learning is really really data Hungry's as far as the number of labels your house is concerned whereas if you do unsupervised learning and all the unlabeled data you have but you only collect label data for that corresponding percentage you can see how much gain it's giving you especially in


the low data regime so you basically just need eighty percent fewer labels to match the same amount or accuracy that supervised learning gets so with just ten images per class you're almost close to 80 percent top five accuracy on image classification which is the standard set by alx not and and and you can also see that the supervised state of the art is matched with around like twenty twenty


to thirty percent of the labels that you need so this way sub supervised learning is actually letting you to be very very data efficient for supervised learning and it also means like you need to hire very few data annotators now like instead of collecting 10,000 labels here you're collecting something like 2,000 naples or fewer right so your data annotation is much faster because you


already have a great set of features and your and-and-and the most interesting thing is even on even when you have all the labels that is when you have 100% of the labels your performance from free training and then fine-tuning is better than just doing supervised learning so there is no argument to not use unsupervised learning because even if you have all the labels the performance


you get by doing unsupervised learning and then performing supervised learning as a fine tuning step is higher than what do you get by just doing supervised learning and and it's it's uniformly consistent across all the label data regimes so here's like a good graph to understand how CPC version 1 interpolated to version 2 and you can see how each bottom improvement axis on


the x-axis added as far as the linear classification accuracy goes and basically Ln mr. layer nom in the clay layer nom helps a lot and you know like bu refers to like bottom-up predictions instead of just using top-down predictions which is basically saying that hey instead of just relating the bottom rows from the top why not do the other way and that also helps a lot


augmentation that every single patches which is referred as PA that helps a lot and so so so various different improvement axes which you can actually refer to the paper like like for instance be using bigger patches of the lot and so on so so you just from 48 49 percent you able to now go close to 72 percent but by just focusing on the engineering details and and doing large-scale


optimization really well and that that's really the success story of sub supervised learning just is just do the simple things right and you'll get really good numbers this is a table that shows the whatever graph you saw in numbers and you can see that on every single data data regime even if you train the deepest possible architecture for supervised based language is arrest


in 200 you're able to improve by 1% in the top five or more than that 1.3 percent if it with with cell supervisory training and you can also see that the low data regime your your your top fire accuracies are so good that they are even better than methods that have used very very engineered semi-supervised pipelines which which we'll cover in a future lecture


so using unlabeled data to improve the data efficiency of label data label they like like low rate a regime supervised learning it's not it's not just specific to cell supervised learning like it can also be done using other methods like semi-supervised learning and and and those numbers representing the that the thick methods that just use label propagation pseudo labeling data


augmentee unsupervised at augmentation and so forth will not cover that today but it will be covered in a future lecture and those are also very interesting but they involve more more handcrafted details as to how you go about doing simultaneous loss optimizations where it's in several ways be training is small like just think it moral destroy it once and then use it


everywhere so it's much more elegant that way so here is the final Pascal vo C numbers which is also a benchmark that people have cared about in in terms of transfer learning for self supervised learning there's always been this thing that some supervised learning will is only Garant considered to work if if the features you get from self supervised learning are gonna transfer to something


other than classification like like something like object detection on it on a data set where you don't have a lot of labels like pascal and for a long time people believe that you could never be the supervised baseline so if you look at the supervised baseline mean average position with the rest at 152 backbone you get seventy four point seven pascal walk just two thousand the data set with


2007 version and you can look at all these south supervised methods in the next row including momentum contrast which which was the first method which god a ball supervised and got got got to seventy four point nine and and and and and and they fast now seen and trained on the CPC version two features get seventy six point six as the mean average position which is which is even


better so so that goes on to say like well you know well that self supervised learning can actually work even better than supervised learning for downstream tasks so even if you collect a lot of label data you may not actually be able to get these numbers because these models are actually learning a lot more about the data so now that you looked at the principle of contrast of predictions


and contrasted learning and and and and seen the benefits and of it actually working at scale people start like like more people interested in just the image domain started looking at contrasted learning and people asked us question hey this contrast learning is cool but then do we actually need all these patches like like inherently patches is hard to deal with because when you


actually drew it an image into patches you're increasing your bat size by a lot and so even if you reduce your image size it's not it's not particularly good to pre-trained with much smaller images and find you in with larger images and secondly you also want to make sure that you use Bosch norm during your pee training and when you do something I see PC using Bosch norm is much harder


because yeah you don't want information to mix in your pixel cnn's so so therefore people want that to have this version of contrast to learning that just worked at an instance level where the the context is just one version of your image target is another version of the same image and negatives are just any other images so in this case it could be like hey you just take a


picture for dog you perform one data augmentation to it which is just apply a grayscale perform another data augmentation to it which is you flip it and you take a particular random crop so and any other image would be a negative for this particular anchor positive there so what does this actually learn them you're basically trying to learn that hey this particular you know the


legs are absent and the other version and so you're trying to learn that there's a dog here depending on the amount of random cropping and data augmentation you use the level of cheating you can afford to identify the two things with the same is lower lower and therefore you're forced to learn good features to make sure that you identify that two different images


presented you are actually fundamentally the same thing compared to any other image and that way you're able to learn really rich features it's very similar to the CPC area done at a patch level except that you're not actually doing any spatial prediction you're all you're trying to do is identify another version of the same image and this is in general referred to the principle of instances


combination and to reason people have really taken this far one paper is called Boco or momentum contrast and other paper is called sim clr or simple contrast of learning for representations for vision and we're just going to look at these two papers though there are like a lot of other papers that have competed with diesel vapors in the recent past but these two


papers are like the simplest and cleanest and also the most well functional in terms of state of the art metrics so first let's look at momentum contrast for unsupervised visual representation learning this is a paper by claiming he was also the inventor of rest nets and faster CNN masks are seen and and so forth so the late works as as false you you basically characterized


contrast to learning as a dictionary lookup task where you're saying that okay hey I want to identify if two things are the same so I present one thing treated as a query and whatever I want to pair with is also present and a bunch of keys I have and there are also lots of other keys which could serve as negatives and I want to identify what is the right positive among these bunch of


keys so you in Korea query you encode all your keys and you compute the pairwise similarities you know that true target which is because you know the ground truth for what is real other augmentation of the same image and then you just build this contrast loss and back prop so that does basically the idea of the instance discrimination so where does momentum come in here so the


idea of momentum is to make sure that you're not you can use a slowly varying encoder of the same and you basically have an encoder which is used to encode your queries but your keys are going to use a pol yet a historically average version of the same encoder and this gives you lots of different benefits so one benefit is that it can let you use a lot of negatives without using your current


mini packs which is to say that hey if you have a mini batch and if all your negatives are coming from your same mini batch then your number of negatives is limited by the batch size you have so that means you you you require a large batch size to use for your foot for being really efficient because you need a lot of negatives now that you're doing things at the instance


level you don't have patches so what if you use a memory bank where you have a buffer of previous embeddings and then you just use that buffer as your negatives so for that buffer to function well it needs to be reasonably historically average version of your current encoder so that it can contrast well if it's just your current encoder then the previous embedding stored in


the buffer or not rather than anymore so that basically so the idea for moko where you basically take an original image oh and you split it into queries in keys which is the two different augmentations and you encode both of them you encode the targets using your momentum encoder and you construct these query in keys you know the true target and you just optimize with the contrast


loss note that this contrast loss in vocal doesn't use the bilinear product it just uses a unit norm vectors of all these in bearings with the temperature softmax and that that version also works pretty well as long as you can pick the right temperature and this is actually the pseudocode for vocal written in rape in a Python style and I think it's best understood through this so fq and FK our


encoder networks with a query and key and your cue you have a dictionary of cues which is your memory bank of negatives and you have a momentum is a temperature parameter so initially you make the key encoder and the query encoder the same to start with and every time you load a mini batch you construct two different augmentations of it the query and the key and you fast forward


the query in the key using the you get the embeddings using the query and key encoders and you who'da stop gradient on the key encoder you're not going to back probe through the momentum encoder and now you're computing the logits of the positives which is you have the same mini batch so you constructed or different augmentations for the same image so that means that the Pirates products of the


corresponding batch indices like if you have a batch of 16 images and you created another batch of 16 images every particular index is basically another augmentation of itself so you can actually take the paralyzed or product for that and get get get the scores for the positives and for negatives you just use your cue or the memory bank of negatives that you have and you just


take your query encoder and you just compute the paralyzed our products with all of this memory bank negatives and now you know that you have the positives and the negatives for your influency loss you just concatenate them and then throw across and repeat laws by creating the true labels and users JT to optimize and and finally you have to perform the momentum update for your key encoder so


that your momentum encoder is slowly changing over time so that's the idea in moko and your cue or the memory bank is a first in first out queue so every time you are your plura upload a new batch of negative in bearings you are you also have to like take it take the the least reason keen a bad-size number of negatives out of the buffer and as long as you can do all


this without any mistakes this will work and these are the different ways in which you can do instance contrasted learning one is you just do it end to end don't care about this momentum just use your current mini box as your negatives use the same encoder for the queries in the keys and back prop the gradients to every everything so that is the end of a notion the other version is


hey I don't you know you you just say I count referred a really large bat size I want to use a memory bank of negatives and that would stay that would keep changing dynamically but then I can I'm going to use a lot more negatives that way so that's interesting and then you you cannot backprop to the memory bank but what you can do is you just applaud your most reason negatives every time


you pass forward a new mini batch you just collect those and bearings and you just thank you that your memory bank so that has a problem because like I said if it's not just if it's not changing over time dynamically then then it's possible that your turn encoders in bearings can be correlating very easily with the most recent negatives and you can just ignore all the other negatives


on your memory bank and so that way you're not actually taking true advantage of the number of negatives and the final version is the mokele illusion where you basically use a momentum encoder and then you use that for in queueing and D queuing and you use the you don't actually back prop to it and you only back for up to your queries and you have a lot of you you get the best


of using you know just a regular end-to-end version but you also make sure that you can afford a large batch size effectively so here's the local plot where the number of negatives basically is increased over time like is increased a different iteration increase for different runs of the same algorithm like we have three different versions the moco memory bank and n 2n and


increasing the number of negatives and enter in the authors only did it up 2024 because increasing it beyond that needs a lot more GPUs a lot more TP of course you know because it's basically your maths at global back size so having a global back size a thousand 24 is the maximum you can afford right now in 8/8 GPU Volta dgx so the authors didn't expand further on that but as we see


very immediately after this the sims ii as our technique is basically the same thing as n 2n technique in moco but the only difference is they were they scale the top to use bigger batch sizes and use a lot more TPU course so local scales gracefully with the number of negatives which is basically the size of your memory bank and you can see that the benefits are there and as you keep


increasing number negatives the linear classification accuracy and the y axis keeps going up so that means you're learning better representations so when mocha was published it was the state of the art for linear classification like CPC version two was not updated yet so and and so the updated results were already presented to you in the previous slide so Mokomoko had really good linear


classifier accuracy where when they made the rest net 4x wider they got 68% top one and and and it's a nice plot of how the number of parameters plays a significant role in giving you like better top-1 accuracy of the linear classifier so finally we look at this paper called zoom CLR or Sinclair simple framework for contrast learning by think chance I mean complet momen Eruzione geoff hinton


so the best way to understand since IAM clear now that you already know CPC and moco is it adopts instance discrimination it and it goes for the end-to-end mechanism proposed in moco so if you look at internment plot and figure a here you back propagating through both the query and the keys and you sharing the same encoder for the queries in the keys and your negatives


are just coming from your back so and and and that's really this exact same thing adopted and Sinclair which is you just take X your input your image mini-batch you perform two different augmentations to it get exciting the UNIX j2d you pass them to the same rest that encoder and you get hecha in history and there is another encoder or another MLP head which takes these mean


pool rest net embeddings and puts it into a lower dimension for the contrast of loss and then you just operate the same influency loss with unit vectors similar to moco so the new innovation in Sim CLR is this G functioned the the demo the the there is a transformation G that takes the rest net embedding and puts it into Layden's rather contrasted losses perform so


early earlier versions like CPC and moko do not make use of any depth from when you Benny when you take the embeddings of your context to the targets it basically just uses a one by one convolution to reduce the channel dimension and perform the contrast of the last in a lower dimension in sims CLR you are using a few layers of MLP to transform 2048 dimensional vector that you get from


rest net to something much smaller like 128 dimensions to the contrast to optimization so that's basically the difference and that really helped a lot so here is Sim Sim cleaners main algorithm another interesting version another interesting fact and sim clear is that if you're going to use your own batches your positives and negatives you can you can basically flip what is a


query and what is a key so if you take a batch you do two different augmentations one of them becomes the query and one that becomes the key but then what which one is query image from this key is totally arbitrary so why not just do both ways so they actually do both ways and so that is the determine the loss which is L of 2k minus 1 comma 2 k plus L of 2 K comma 2 K minus 1 CD they just


flip their order and then you can just use a large large batch gradient descent they use a large optimizer and use bad sizes of 2048 or 4096 on a cloud TPU and and and they're able to perform this optimization really fast and they also train much longer and just like CPC was trained for 500 a box they train actually 4,000 bucks and they add a little more data


augmentations like Gaussian blur that helps a lot and randomized the order of data augmentations so simply errand was published just two weeks ago actually 2 to 2.5 weeks ago it got stained the art performance on imagenet linear classifier so you can see that just with the resident 50 whatever CPC version to God was 63.8 and moco had like 60 point 1 sim clear just took it all the way to


69 percent a huge jump and when they made the resonance wider which is making the rest at 2x so 4x wider you get more features linear classification now gets all the way up to 76.5% which is as good as a supervisor learner can be at that it's it is using more features it's using more parameters so it's in the same parameter scenario is not as good as supervised but if you just have a


confirm for more parameters it's almost as good as supervised and this this kind of result was lacking for a long long time image image classification sub supervised learning was always lagging behind in supervised learning so so simple clear was the first to ensure that something can exist in this level so here is a more detailed study of how how you know you could ask a question


hey what if you made supervisor also wider would supervise increase as much as supervised it turns out to be not true so like a regular supervised resident get 76% but a wider doesn't actually improve a lot more it doesn't prove it gets to 77 78 but not not not more so the gap between self supervised and supervised release is further and number of parameters is more so the hypothesis


of the authors is that supervised models need me you know they cannot particularly benefit from more data augmentations or more parameters where sub supervised models can actually benefit from that because they're trying to do something much harder so when you actually can afford bigger models you would rather want to go for something like self supervised learning instead of


supervised learning if you can continue making more improvements so after Sinclair was published the fact that the MLP has helped a lot made the moko authors consider using MLP heads as well and also doing a little more engineering more hyper parameter sweeps and also train longer and use the same kind of data augmentations that simply are used but and so the moko authors


came back came up with reply to Sinclair in some sense and call their model this moko version to say on the left you can see what it basically is simply R which is the end-to-end version of loco on the right you can see the mocha version exactly the appealing thing about moko is it basically can let you use a lot of negatives with our naxi using large batches because of its memory buffer and


so when you add when you add these ideas so MLP had which is the G function aaaghh plus which is the Gaussian blur augmentation used in simpler and another detail that was used in sim clear was the cosine learning rate decay which moko didn't use if you add all these details then mokos accuracy linear classifier just arrests in 50 encoder improves all the way up to 71 point one


percent that just which is which is two percent better than simply results and it also improves the detection results in Pascal and so and so just by training longer and getting all these extra details right the moko authors were able to get very very impressive self supervised results with just using a GPUs and a bath size of 256 so that's that's that is really the state of the


art technique right now and here's also the ablation between moko version one vocal version two and simply ER and you can clearly see like mocha version one from sixty point six it went all the way from sixty point six to sixty seven point five even without training longer like the seven percent improvement just by adding the same details that simply are had which is using MLP heads


cosine learning rate DK and you the extra data augmentations and from 67.5 in it can have another 3.5% improvement by by training longer so so at this stage South supervised learning is really at a stage where it the amount of engineering detail you pay attention to and the amount of trickery you add in terms of model optimizations and like using clever techniques like memory


banks and doing very well on large batch training what data augmentations you use what is the learning rate decays or optimizers those are really the most important parts and getting state-of-the-art numbers and it's only a matter of time before like it's gonna match the results of supervised learning on all these benchmarks like seventy one point files still not as good as 76% so


that's still some gap to close in but the the rate of progress is because it's just been two or three months since all these papers came out and it's been rapidly exploring so so that's that's pretty much it for self supervised learning like we haven't really covered language like subsequent learning for language which is actually are arguably the domain where unsupervised ourselves


provides free training has really taken off even before all these revision successes and some like bird which is really famous and has 4,000 citations in a year like it it basically is already productionize dan google search already uses it so that those are bigger successes than CPC or instance contrast to learning or moko so so but we didn't really cover that


this is Earl Nightingale the purpose of this recording is to tell you about and try to condense one of the most amazing books ever written Think and Grow Rich by Napoleon Hill without question this single book has had a greater influence on the lives accomplishments and fortunes of more individuals than any other work of its kind all over the free world there are literally thousands of

Napoleon Hill's Think & Grow Rich  Condensed and Narrated by Earl Nightingale


successful men and all lines of work who are where they are today because they once picked up and bought a copy of Think and Grow Rich and they'll be quick to tell you so I first discovered this remarkable book in the fall of 1949 it was an enormous help to me it helped me decide once and for all how I was to accomplish my goal it unified my thinking and gave me a straight clear


Road to the point I had decided to reach one of my closest friends found the book several years ago and stayed home for three days reading and digesting its material and he then went on to reach the top in his industry i sat in richly paneled carpeted executive offices and listened to world famous business leaders some of them old enough to be my father tell me that


everything worked out fine after they had read Think and Grow Rich now what's the secret of this amazing book why has this book out of all the thousands of self-help books remain the one towering giant I think to understand this you have to know Napoleon Hill as I do he certainly was not the first man to be appalled at the poverty and seemingly endless struggle and lack of direction


he saw about him as a boy and as a young man nor was he the first to write on the subject but he possessed two unique highly developed abilities seldom found in one man the first was in the manner in which he approached his subject Napoleon Hill went after the answers to achievement in the same way a scientist seeks to open to the light of Reason a secret of nature he went after the