imagine a few ways around this one is you could use the reinforce algorithm to do policy gradients and use that to Train the generator network there's also the recently introduced techniques based on the Gumbel distribution for doing relaxations that allow you to Train discrete variables or finally you could do the old-fashioned thing that we used to do I saw geoff hinton on thursday and he was mentioning to me how this reminds him a lot of the way that Boltzmann machines were really bad at generating continuous values so what we do there is


we would pre process continuous values to convert them into a binary space and then we'd use Boltzmann machines from there so you could do the same thing in Reverse with genitive adversarial nuts you could have a model that converts these binary values to continuous values and then use generative adverts or networks from there you could for example train a word embedding model and then have a generative adversarial network that produces word embeddings rather than directly producing discrete words one very interesting extension of

Ian Goodfellow: Generative Adversarial Networks (NIPS 2016 tutorial)


the discriminator is to actually make it recognize different classes and this allows us to participate in an important research area of semi-supervised learning with generative adversarial networks originally generative adversarial networks used just a binary output value that said whether things are real or fake but if we add extra outputs saying which class they belong to and then having one fake class we are able to then take the and use it to classify data after we finished training the whole process and


because it's learned to reject lots of fake data it actually gets regularize drooly well using this approach tim Salomon's and and i and our other collaborators in open area we're able to set the state of the art on several different recognition tasks with very few labeled examples on em nist c fart n @ sv hn another important research direction is learning to make the code interpretable Peter Chen's info Gann paper here at nips actually shows how we can learn a code where different elements of the code correspond to


specific semantically meaningful variables like the position of an image another research Direction is connections to reinforcement learning recent papers have shown that generative address cell networks can be interpreted as an actor critic method or used for imitation learning or interpreted as inverse reinforcement learning finally if we're able to come up with a good algorithm for finding equilibria in games we can apply that algorithm to many other places besides generative adversarial networks things like robust


optimization literally playing games like chess and checkers resisting adversarial examples guaranteeing privacy against an attacker who wants to thwart your privacy and all of these different application areas are all examples of games that are rise in artificial intelligence and might be improved by the same kinds of techniques that could help us to improve generative adversarial networks we're very close to out of time but I'll give you five minutes to do this exercise and I'll answer the last set of questions during


the exercise this exercise is jumping back a little bit to earlier how I described that there's a different cost function that you can use for maximum likelihood and generative adversarial networks and I think this is a really good closing exercise because it really drives home the point that the key mathematical tool generative a versatile networks give you is the ability to estimate a ratio and to see how the estimate ratio estimation works you are going to derive the maximum likelihood learning rule in particular


we have a cost function for the generator network which is an expectation of X sampled from the generator and then applying f of X and we want to figure out what f of X should be to make this cost function give us a maximum likelihood as a hint you should first start by showing the following that the derivatives with respect to the parameters of the cost function are given by this expectation of f of X multiplied by the derivatives of the likelihood and if you'd like you could actually just take that as a given and


skip to the last step at the very end what you do is you should figure out what f of X should be given this fact about the gradients if you can choose the right f of X you can get the maximum likelihood gradient so I'll give you a few minutes to work on that and I'll take a few questions and then and then I'll conclude so in your previous about as a generative Network are you missions out there is a important assumption that is a function should be differentiable what if the function is not differentiable because in some area such


lacto Informatics the data is some categorical categorical label not numerical value so it's 94 insurable so in that use that acceleration how to generate a synthetic data and you using GT and network so there's there haven't been any papers actually solving that problem yet I talk about this a few slides earlier and my recommendations are to try the reinforce algorithm to do policy gradients with discrete actions to try the concrete distribution and Gumbel softmax which are two papers that were recently released about how to


train models with discrete outputs or to convert the problem into a continuous space where a generative address donuts can be applied so the variance and ganz is that it's very powerful in capturing the modes of the distribution right but it's not really truly understanding what images are as in disease you know you start from zero to generate X right so the question is you know if you increase the systems increase the image size assumably the modes of the distribution going to increase exponentially so ultimately you


know if you have a you know practically this may not be solution this may not be a problem maybe we just care about hundred by hundred pixel images but this assume I'm interested in two thousand by 2,000 pixel images you know if I truly understand what images are how images are generated you know there is no difference between a hundred by a hundred and two thousand by two thousand I can you know that ultimate machine my question is about like way down the future I mean at the end of the day you are capturing modes of the distribution


but this mode is going to explode if you go to larger images so at some point you know the the modes of the model also have an exponential explosion as you use a bigger convolutional net so if I mean I mean I don't want to repeat the same structure I mean the question is the modes of the distribution right at the end at the end of the day you are capturing the modes of the distribution yeah but a larger model can capture more modes I guess the nice thing about natural images is that when you increase the resolution you're looking at a


different level of detail but within the same level of detail the same structure is repeated all across the image so let's say that we've been studying 64 by 64 images and we couldn't really see the individual firs ah like Harrison and animals fur and then we move up to a higher resolution we can see their fur at the higher resolution we don't need to relearn the distribution over images of fur at every pixel separately we we learn one level of detail that can be replicated across the whole image and we generate different Z values at every X


and y coordinate the that randomly decided you know fine details of the fur like which angle it should be pointed in and things like why do you think in practice ganz don't ask a love well when you go to larger images oh well you might be surprised by what comes in a few slides yeah I think I should probably move toward the conclusion now so recalling exercise 3 we're looking to design this f of X this cost function that's applied for every example generated by the generator in order to recover the maximum likelihood


gradient we start by showing this property that we can write down the gradient of the generator in terms of an expectation where the expectation is we've taken with respect to generator samples and we multiplied f of X by a likelihood gradient that's relatively straightforward to show the basic step is to turn the expectation into an integral use leibnitz's rule which means you have to make a few assumptions about the structure of the distribution involved and then finally we take advantage of our earlier assumption that


the generator distribution is nonzero everywhere that allows us to say that the derivatives of pg are equal to P G times the derivatives of log P G so that gives us this nice expression where we can get a gradients of the likelihood in terms of samples that came out of the generator but what would you really like is gradients of the likelihood in terms of samples that came from the data so the way that we're able to do that is important sampling we have this f of X coefficient that we're able to multiply by each of the gradients and we can fix


the problem that we're sampling from the generator when we want to sample from the discriminator by setting f of X to be P data over P generator and this means that we'll have kind of bad variance in our samples because we're sampling from the generator and then rewriting everything to make it look like we sampled from the discriminator but in theory this is unbiased from there it takes a little bit of algebra to figure out exactly how we should take the discriminator and implement this ratio we recall that the optimal


discriminator gives us this ratio of p data over p data plus p generator and doing a little bit more algebra we can rearrange that to say that we need to set f of X to negative e to the logits this is maybe a lot to absorb right now but I think it's it's pretty intuitive once you've worked through it slowly on your own once and it gives you an idea of how you can take this ratio that the discriminator gives you and build lots of other things with it so to conclude the talk I'd like to show you some really exciting new


results that came out using generative adversarial networks and that kind of addressed the last question we had about whether a generative Ebersole networks scale to very large images a new model just came out last week I seem to have this curse that every time I have to give a talk about something an important new result comes out right as I have finished my slides so I desperately made some new slides on the plane on the way here plug and play generative networks or generative models sorry make 256 by 256 high-resolution images


of all thousand classes from imagenet and have very good sample diversity the basic idea is to combine adversarial training moment matching in a latent space do you know atom encoders and Monte Carlo sampling using the gradient and the really cool thing is they also work for captioning or inverse captioning where you generate images by giving an input senate's that describes the image overall the basic technique is to follow a Markov chain that moves around in the direction of the gradient of the logarithm of P of x and y with with Y


marginalized out you can use denoising auto-encoders to estimate the required gradient but to make the denoising auto-encoder create really good images the auto encoder needs to be trained with several different losses one of those losses is the adversarial networks loss and that forces it to make images that look very realistic as well as images that are close to the original data in l2 space this confirms some of the tips that I gave earlier in the talk for example on the tips and tricks section I said that you often get much


better results if you include class labels we see here that plug-and-play generative models don't make nearly as recognized full of images if we generate samples without the class we also see that the adversarial loss is a really important component of this new system if you look at the reconstructions of the denoising auto-encoder we begin on the left with the raw data in the middle we share the reconstructed image and on the right we show the reconstruction that you get if you train the model without the adversarial Network loss so adversarial


learning has contributed a lot to the overall quality of this current state of the art model so in conclusion I guess I'd hope that everyone remembers that generative adversarial networks are models that use supervised learning to approximate in intractable costs by estimating ratios and that they can simulate many different cost functions including the one that's used for maximum likelihood the most important research frontier in generative adversarial networks is figuring out how to find Nash equilibria in high


dimensional non convex continuous games and finally generate veteran-owned networks are important component of the current state of the art in image generation and are now able to make high resolution images with high diversity from many different classes and that concludes my talk and I believe that we're out of time for questions because we already took several of them in the exercise brakes and I think Erin will now announce that we are headed to sign textbooks and I hope you know what room were sending them


lecture seven of deep unsupervised learning today we'll be talking about self supervised learning and it is going to be pretty different from the previous lectures you've heard so far so far you've been looking at a lot of generative models how to use various classes of generative models to generate high dimensional data like images audio text and so forth however unsupervised learning is a much broader goal than just being able to generate data and one of the goals of unsupervised learning is to be able to learn rich features from raw unlabeled

Lecture 7 Self-Supervised Learning -- UC Berkeley Spring 2020 - CS294-158 Deep Unsupervised Learning


data such that they can be useful for a lot of downstream tasks and this lecture is going to get at that and recently people have started calling this stuff supervised learning where the data creates its own supervision and so we are going to look at all the various classes of techniques that are allow us to do stuff supervised learning so so far we've seen density modeling where we've covered Auto regressive models flow models and we also talked about variational inference and we've also looked at implicit generative models


implicit density models like jion's and energy based models and both these classes of techniques allow you to learn generative models which is you are going to be able to generate images and be able to report lac courte scores and so on but other than that we mainly looked at applications of generative models to various modalities of data we haven't actually seen how to use unsupervised learning to learn features so that's the motivation for today's lecture how do we learn rich and useful features from raw unlabeled Ino such that it can be useful


for a wide variety of downstream tasks and we're also going to ask ourselves the question of what are these various pretext or proxy tasks that can be used to learn representations from raw unlabeled data and if we are able to learn good representations how can we leverage that and improve the data efficiency and performance of downstream tasks with a good pre training model so here is a figure from in Goodfellows deep learning textbook the focus here is how do we learn good representations and here's a simple case study of why representations


matter see how a bunch of points two-dimensional and if you visualize this an XY coordinate the Cartesian coordinate the they're clearly two separate clusters but it's harder to visualize how to linearly separate them but the moment you visualize them in the polar coordinates you can clearly say that there are two different radii and a lot of different angles and so you can have draw linearly separable hyperplane between them so it's clear that representation matters so once you move from the Cartesian representation to


polar coordinate representation things become a lot easier to handle as well you know you can actually use this linear SVM so logistic regression to learn a classifier on this particular polar coordinate representation so what is deep learning doing deep learning is basically using depth and repeated computation to iterating we will find the features as you move thereby they over there so the bottom was there is using the raw pixels as input and here it's trying to make sense that there is a person in the photograph and you know


if it's looking at all the background pixels and understanding that there is a phase so the way it starts it starts with the highest frequency information at the bottom and it refines at the next level to edges and it refines the next level two corners and contours and then the next load actual object parts and finally it's able to figure out the identity of the objects present and the actual image so just like how we saw representations matter so representations are the higher levels are more semantic representations are


the lower levels are more fine-grain and detailed and high frequency so the deeper on that can be thought of as writing its own representations that every layer every successive there is written on top of the previous layer which is more abstract than the raw input and that allows you to do downstream tasks if you take the topmost layers so here's the Venn diagram that being good for suggests for deeper how to think about deep learning so deep learning is a subset of representation learning which is a subset of machine


learning which which can be considered as sort of AI in general and so the goal of deep learning itself nothing to do with unsupervised learning is to learn good representations of raw data so what is deep unsupervised learning so unsupervised learning is concerned with learning these representations without labels so it can be considered as another subset of deep learning which is doing deep learning without labels basically so we are gonna get at the goal of representation learning without labels and that's deepens promise


learning so it sort of gets at the core goal of the class and recently it's been called a self supervised learning and it's used interchangeably with unsupervised learning the exact terminology of what itself and what is undoes not matter it's basically concerned with learning representations with our labels whether one self usually refers to the scenario where you can create your own supervision based on the data but at the end of the day it it can be considered as as another way to reprime tries unsupervised learning


so why self supervised learning the expense of producing a new dataset for each task this is really high so there are actually billion dollar startups just doing data annotation for people who can just upload their images say what kind of labels they want and overnight or within an hour or fortnight you can get like high quality labels created by humans who would annotate this data on the client side so sorry on the server side so so basically you you need to prepare labeling manuals you need to figure out what categories of


objects you want you need to have someone else hiring humans you need to hire your humans to annotate data and whoever is doing that job you need to create good graphical user interfaces so that it's the process of annotation is really fast you also need to create good storage pipelines so that every you know let's say people are annotating usually it's like a lot of mouse clicks per minute or second and every mouse click is automatically recorded and converted to prompt rate data storage formats and stored efficiently into the cloud so


there are lots of back-end engineering you need to do not that this is a bad thing to do it is it is good unlike if we really need to work on better and better pipelines for data creation however good supervision may not be cheap for example annotating what are the objects contained in an image is probably something that you can take for granted now because people have created a lot of datasets but if you move to another domain like medicine or legal creating another data set may actually be pretty hard so taking advantage of


vast amount of unlabeled data on the Internet is something supervisor learning cannot do so no matter how much you can appreciate the success of supervised learning there is still a lot more unlabeled data then there is the amount of label data and it would be nice if we can leverage the of unlabeled data to further improve the performance of systems that work on label data so it doesn't have to be a dichotomy between hey we just want to do unsupervised learning or we just wanted to supervise for anything but rather we


want to figure out how to take advantage of large amounts of unlabeled data order billions of images for lots of text or audio samples or YouTube videos and learn amazing features and then make the process of doing supervised learning much more cost and compute and time efficient and finally there's this cognitive motivation which is how babies or animals learn in that like when they mostly learn by experimenting and without actually having a labels so a child can just look at other people doing things or its own experience


moving its own hands or looking at other people around in the house or like you know no modern days children grow up with gadgets so they can look at videos and already start learning good features without actually knowing this is it this is a cat this is a cat this is a cat like hundreds of times that's how emission a classifier is learned so that was a really nice code by Pierre cermony who's one of the leading researchers in the field which is give a robot a label and you feed it for a second but teach a robot a label


and you feed it for a lifetime what what he means by this is that if you taught the robot the underlying aspects of various objects in a completely Sal supervised fashion it knows what like a cat or dog is without actually being taught so that means that it can actually generalize much better so labels are cheap but then it may not be the most optimal way to learn representations so so what exactly is self supervised learning it's it is a version of unsupervised learning where data provides its own supervision so in


general the way it could work is you withhold some part of the data and you task a neural net to predict a bit with health portion from the remaining parts so this could be like you occlude some part of the image you look at the remaining pixels and you try to predict the occluded version or you have a video and you just hide some frames in the video you have the other frames and you try to fill in the blanks of the missing frames or you have a sentence and you mask out some words and you ask the neural network to fill in


those words or you just have to predict the future from the past or the past from the future or present from the past like loss a various different versions depending on the mask so this way the data is creating its own supervision and you can perfect you can ask a neural network we'll learn a lot more than just predicting labels so the details obviously decide what is a proxy loss or what is the pretext ask you can think about all these withhold and predict classes some kind of pretext ask and you can think of whatever details you use


whatever loss functions or whatever tasks you create depending on that the quality of the task could be different and therefore the quality of the underlying representation study uncover could also be different and so that is basically this whole topic which is how can we create these really good tasks which make the neural network to learn a lot of useful things and therefore be very useful in downstream tasks so the motivation another motivation of why we want to learn good features is one of the biggest reasons for supervised


learning to really take off not just as a research topic but also as an industry of practice is that the you can use a pre trained classifier for a lot of commercial a downstream tasks so a pre trained imagenet state-of-the-art emission a classifier like a rest at 50 can just be taken and the same backbone can be taken and put into a faster or CNN or a mass for CNN or a retina net and can be used for object detection or instant segmentation or it can also be used in a fully compositional neural net with the backbone as the rest in 54 a


semantic segmentation so this way you're able to solve a lot of harder computer vision problems with black collecting label data is much harder and you can actually just retrain a good classifier take those features and start the underlying downstream tasks with a much better prior and you don't have need so much label data now and you can also converge much faster on these harder problems so that way the recipe is very clear so you just collect a large table data set your trainer model you deploy and as long as you have a lot of good


data and you have sufficient data it is basically all you need in terms of getting some autom automation on on production so most of your industry or usage of computer vision like in video surveillance or in robotics where people have to detect objects or in shopping automated shopping where people you want to detect what people pick or which objects people pick it's basically just object detection and to get a very good object detector all you need is a lot of labels and a lot of good patron features so what is the goal of self-professed


learning the goal is to learn equally good if not even better features without supervision and be able to deploy similar quality systems as what is currently in production without relying on too many labels so what if instead of collecting 10000 labels now you could just collect thousand labels or hundred labels that makes the process of production much faster much more efficient and you don't have to spend as much and it's also much simpler to maintain and you can keep on bootstrapping more and more data you


don't have to rely on high quality expert labeling and you can still uncover the same level features as you have currently with all the effort had to collect labels so it could also generalize much better potentially because by doing some harder pretext asks than just predicting labels you are expected to learn more about the world and therefore generalizing in the longtail scenario is likely to be better so that is the hope and that's why people want to make self worth learning really work so this has been really you


know very nicely put together as a more inspiring slide by young Nicole and it's often referred to as the lake which we saw in the introduction to the class which is you can think of self supervised learning as the cake if intelligence of the cake you can think of sociable as learning as the cake and you can think of supervised learning as the icing on the cake and you can think of reinforcement learning as the cherry on the cake and there the argument is that most of the useful bits can come from doing really hard pretext asks just


from data and where the machine is predicting some part for missing parts and you get millions of bits that way whereas in supervised learning consider imagenet you have thousand classes so that's ten bits per image and if you have a million images you have basically a million times ten bit space key that's basically your whole data set and whereas if you're just doing generative modeling you're modeling all possible bits in your data set so that that's that's too huge right so some supervised learning is trying to find a middle


ground between these two and it's possible that the bits you get from sabra was learning or more useful that said there is a caveat that subscrube is learning the bits you get from there or not as high quality bits is the bits you get from supervised tasks when human is telling you that there is a cat here or there is a dog here or like there there is a cat exactly at this coordinate there's a dog there's a bounding box around a human that's very higher quality bit than saying these two pixels are of the same


color or like these this is a flipped version of that image or this image is a 90 degree rotated version of the other image things like that so it's not just a number of bits that matter the quality of the bits is equally important so you should take this slide with not not too seriously it's just for inspiration making the bits argument as a way to like work on unsupervised learning is fundamentally flawed because the label data bits are much much much more useful much more grounded in the real world you to behave so here is the aleck own


suggestion for how to do unsupervised or sub supervised learning which is creating your own proxy tasks and you can think of various different versions of that let's say that there's a video you could predict the top from the bottom bottom from the top left from the right right from the left it's basically masked some part of your input predict the mass part from the unmask part and obviously depending on the mass the mother's going to learn something trivial or non-trivial so usually like like for instance in a


video if you're just masking a couple of frames in between it may be very easy to fill it up by just using the optical flow information and just using the reference frames and just interpolating between the pixels the model doesn't necessarily have to capture what an object is and whereas if you have a sentence and if you are predicting the missing words or sub words in a sentence it's possible that the model learns a lot more about the grammar and the syntax and semantics of collective language because it's it's not possible


to just copy-paste previous words to fill in the fill in the sentence because language is already syntactic and grammatical and separate like every cent every word is conveying something new whereas pixels are more high-frequency there are more natural signals and so there is an spatial temporal correlation this is already available naturally so the model may not really learn the actual high-level information that you wanted to learn unless you carefully engineer what are the master you want to use so for the actual technical content we


are going to separate it into three parts so the first part is we're going to learn about how to do that we're going to separate the various cognitive principles basically the first principle is you corrupt your data and you try to predict the actual data from the corrupt aggression and the corruption can just be like you add some noise to your input or it could be like you hide some part if you input an you predict the missing part or it could be like you take your data and you basically do some signal separation so it could be like hey an


image is basically a grayscale in the color so you could predict the color from the grayscale or you have you have the depth image then you have the color image and you could try to predict the depth image from the color image where let's say you're recording everything from your Kinect so it could be source separation and then you try to predict the separate equation so that is the first principle the second principle is we're going to do something like visual common sense to us where it's more ad hoc and you just trying to create tasks


from data in a very creative way and see what kind of features the model can learn and there we are gonna look at three different techniques relative patch prediction jigsaw puzzles and rotation and finally we are going to look at contrast learning which is really the version of supervised or unsupervised learning that's been taking off very very recently and we're going to look at a foundational work over to back which explains a lot of these foundational ideas like nice contrast loss and then we're going to look at a


version that's been used in images called CPC or contrast to predictive coding and we're also going to look at follow-ups to that that made the CPC pipeline much simpler like instance discrimination and where is state of the art instantiations of that note that in this lecture we are not going to cover the more popular sub supervised techniques like or any anything to do with the latest language retraining pipelines arguably sub supervised learning has taken off way more language than computer vision but the focus of this lecture is going


to be more on computer vision because language pre-training will be covered separately in a guest lecture by Alec Radford and we're also not going to look at how unsupervised learning helps for reinforcement learning that will also be covered separately by Peter in another lecture ok so now let's go to denoising auto-encoders so do isaac autoencoder the basic idea is add some noise to your input and try to remove the noise and decode the actual image so here you see an amnesty j't and you and you see the noisy input on the on the left and you


see the demo image on the right and the end coder takes in the noisy image puts it into a smaller latent representation which is the features that we care about and the decoder is trying to use these features to get back the original input so you hope that the encoder gets the high level details removes the noise and the decoder can up sample that and get get you back to the actual image so depending on the kind of noise you add you've want to learn more non-trivial things if you don't add any noise you're just going to learn an identity function


that's just an or encoder but if you add some level of noise it's possible that you learn more useful features because you're learning to separate noise from the actual signal right and if you had too much noise then it may actually be a really hard task because the signal-to-noise ratio will you really low so this is the general computation graph of the denoising auto-encoder where x tilde refers to the noise version and F data basically at the ground truth x and you're trying to figure out how to reconstruct that back


and get the Layden's in a very useful way so there are various different versions of noise that you can add to in put in in a denoising auto-encoder so in the original denoising auto-encoder paper they considered the task of Ensenada they considered three different noises additive isotropic Gaussian noise where you basically just add Gaussian noise to the pixels and another version is the masking noise where you basically some fraction of your input pixels I just chosen at random and you just forced into zero which K you just mash


them out and they're going to be a black and finally there's the salt and pepper noise where some fraction of the elements the chosen at random and you can basically set them either to the minimum possible value or maximum possible value so instead of basically it's it's a version of masking where instead of just assigning masks to be 0 you can randomly assign the mask to be 0 1 so these are three different noises to consider in the paper you can note that as pixel level noise so you can think of denoising auto-encoders basically


learning a tangent hyperplane on your data manifold where around every input x there is a distortion radius created around it based on the noise that you're trying to add so it's very easy to understand in the case of additive Gaussian noise because you can think of Gaussian is a spherical distortion around your every input and you can think of the decoder as trying to put the distorted version back onto the tangent hyperplane so you can think of the whole denoising auto-encoder pipeline is trying to learn this tangent


hyperplane that describes the data manifold so that it's able to put back to distortions around the hyperplane back to the correct and like back to the data manifold and that way it uncovers the shape of the data manifold by operating at these local hyper planes at every individual point so here is the loss function of the denoising auto-encoder you can clearly see that there is a version where you can use the reconstruction error for the available pixels which are not having any noise and the reconstruction error for the


pictures that have been noise and you can also rate them based on you know like you can you can prioritize the reconstruction of the noise versions as compared to the versions that have not been noise so so if you have an amnesty Majin your dick map adding noise like 10% of the pixels you could prioritize the reconstruction error of those pixels more than the other pixels so that the model is not incentivized to learn identity function or around what is like already available without noise well and the models actually striving hard to


like we're getting get the details right at the noise pixels so and you could also imagine optimizing to different versions of the loss one version of the losses using the mean squared error and the other words for the loss is using a binary signal across central P loss and both both are equally good and endless it makes more sense to use the cross entropy loss but the mean square error loss is also very likely to work well if as long as you let it train with the right set of hyper parameters so stack denoising auto-encoder is basically the


version of denoising auto-encoder where you're going to do this layer by layer by layer so you take your original amnesty which you have one hidden layer and you run it in using or encoder and you get that feeling layer now you can take that hidden layer as you're like version of the image that you want use instead of the actual pixels you can add noise at the hidden feature level and learn a denoising auto-encoder for that feature right so and if you do this iteratively the denoising auto-encoder is now operating on more more abstract


inputs instead of a raw pixels so that's basically the idea in a stack denoising auto-encoder and the hope is that as you keep stacking more layers the higher layers get more more semantics but you should also be careful in thinking about what kind of noise you can add to the features so if back in those days people used to use neural networks with Sigma nonlinearities so in that case it's it's easy to add a noise like masking because Sigma can be considered as you know the neurons firing or not firing but but now


neural nets and like like the way they did the ear and that's a design is much different like the kind of nonlinearities used are very different so so this may not be particularly in a appealing idea to use in the current current infrastructure so finally one utility of the denoising auto-encoder is that once you've learned sufficient layers with the stark denoising auto-encoder you could basically have a target like a class table and just freeze all the features that you've learnt from the auto encoder and have a supervised layer on top make


a single linear layer that just predicts the class logit and you could use that perform a classification task and and this was particularly appealing back then because back then it was really hard to train deep neural networks to just do supervised learning even if you had a lot of data because directly training deep neural networks was not something that was particularly working with and like innovations were needed in terms of using momentum optimizers and bigger math sizes and etc convolutional neural network so as far as hidden here


look like feed-forward neural networks goes this was a standard recipe to even do supervised learning back then because you needed some reasonably high level features to operate well to train a good supervised model so here are the filters that you learn with a denoising auto-encoder for various levels of noises and you can clearly see that the ones where you actually add more noise is learning more of these digit edges whereas the ones you don't have any noise you're hardly learning anything because it's mostly gonna do an identity


map and this is also visualized for a particular neuron in magnified and you can see that filters are like more visible for the higher masking ratios you can also see the you can also see that there's something like a six visible towards the right and and and it's getting it's getting the notions of digit edges or strokes at the hidden level so these are various classifiers that it can train on top of these denoising auto-encoders the features that you get from stacking nosing order into rows and these are the error rates


in M this classification it's not particularly relevant now because M nest is considered solve but then you can clearly see that you know it's getting like 97% accuracy or something that range with putting SPMS and top so this is like a cool result at the time so here is another version of corrupting your image and trying to predict corruption or like trendy you try to hide some portion of your image and trying to predict the hidden portion so this is a people context encoders from the potato from work from Alyosha afer this group here


at Berkeley so the wait works since it basically takes an average mass out a rectangular region and encodes that image now with the mask and has a big order that tries to reconstruct the actual image now so that way the model is filling up the details of what's missing in the mask and supervision basically can be constructed from your data itself because you actually knew what part you masked you because you had the complete image so the model is able to learn without any labels by creating its own supervision so that that's


that's why it's called supervised learning so you can have various instantiations of this where you could mask out only the central region and try to fill up the central region or you can mask out various square blocks across spread across the image much smaller but lots of mass and you could try to fill them all or you could if you have access to a segment the segmentation mass of actual objects in your image you could segment out particularly the pixels belonging to one particular object like in this case the baseball player and you


could just try to fill in those pixels and that assumes access to label data of segmentation mass so that's not something that is completely self supervised but the other two versions are completely self supervised so they've made the reconstruction last good work is you you have a masking region and you have a ground truth for that and you could just apply the reconstruction error on the pixels that have been masked so you take your decoded image and you apply the inverse of the mask and you get all the other pixels out and


and then you can just mask out those pixels for the reconstruction here so there are multiple losses that you can use for the reconstruction objective one is one is you could use just a mean square error that you saw in the previous slide of diagnosing or encoder or one problem that's usually a common with the mean square error which was also mentioned in the Gant lecture is that they often tend to be blurry so because you don't want Larry reconstructions you actually want sharp predictions of all these missing pixels you can


actually think of using a gann loss which is you have a discriminator and that discriminators you behaving going to behavior I could learn a loss function and you can think of using the discriminator objective and the reconstruction objective together because back back in those days training just a discriminator and using adversary losses wasn't particularly easy so the authors ended up using a combination of the regular reconstruction objective and the and the adversarial a discriminator objective so so this is the architecture


they adopted where you have your original image one finis 8 by 128 and you just use strided 4x4 cons and down sample spatial resolutions and you also up keep up sampling your channel resolutions and you get a flat hidden vector of 4,000 dimensions and then you up sample using transpose convolutions back into the actual original image and you can use the reconstruction error it could be an l1 or l2 objective and I think the authors tried both both l1 and l2 and found l1 to be working slightly better as far as reconstruction goes and


they also have the discriminator that takes in real data as well as your predicted missing patches and then you classifying if it's a real or a fake image so you can see that the l2 loss is producing a blurry pixel interpolation basically scrapping all the neighborhood pixels the vaping workers you just have the missing square and you can just fill up like the borders based on what based on the pixels that are available immediately to the left or the top of the right at the bottom respectively and once you filled it up you can just fill


in the missing regions of the square based on what you've already filled up within the square on the edges and that will look like a reasonable completion very blurry though and if you look at the adversarial loss it's introducing artifacts that are completely new so as long as the discriminator thinks like it's a real object it would still work but it may not particularly have a coherence with respect to the actual background image and you've seen this problem in pics to Pyxis will vary for a conditional ganyo to provide the context


in addition to what you're trying to translate to so so that's clearly what's going on there and the joint loss is trying to use both the reconstruction objective and the and the adversary's objective and it's producing something sharper than just using l2 loss so so here are some results for what what happens okay let's say you finish this retraining process now you take the encoder out and you you want to use it for a bunch of downstream task so the downstream task could be a classification in detection or semantic


segmentation and classification and detection are done on a Pascal dataset which is much smaller than imagenet so you can think of the advantage of P training as hey if you don't have too many labels you know you really need some kind of features to start with to be able to perform the task and semantic segmentation are also on Pascal VLC but a different version of the dataset 2012 and that uses another architecture like fully convolution that for doing the semantic segmentation part so using the feed train part of context


encoder as a backbone our part in the FCN so so here the results are reasonably good so if you just use image net features which is you pre train a classifier or an image net and then you fine-tune it on Pascal the results are like seventy eight point two percent for classification and 56 twenty-eight percent for deduction forty eight percent for segmentation and context encoder the fine tuning to pascal classification is not that good it gets only fifty six point five percent which is way lower than supervised but it's


when it's reasonably good in the sense that it's able to perform on par with with a or are actually quite quite better than an auto encoder an auto encoder gets fifty three point eight percent on pascal classification and detection in case forty two percent so so they get two to three percentage points more than doing just a regular encoder and other other other self supervision methods that were available at the time so this was a reasonably interesting result at the time so next we look at the principles of predicting


one view from another where you basically do some source separation and you're trying to predict the separate parts from each other so this is a slide for Richard Zhang who is also the first author of this line of work so we already saw what a denoising auto-encoder is it basically takes raw data corrupts it and tries to reconstruct the original data now imagine that you can separate the raw data in two different views the best way to understand this is an image can be separated into a color image and the grayscale and you could try to predict


one view from another so you could try to predict the color image from the grayscale putting the grayscale from the color doesn't actually need any deep neural net all you need to do is average two pixels and quantize and you're going to get something that is reasonably grayscale and in exactly the conversion is done it's a weighted average of your RGB pixels so but the other version which is predicting the color from the grayscale means that you have to add some new information to what's already there because you don't have any information


about the color so that way you do have to understand that hey if you have a tree you know the leaves are green and like the bark is brown so you have to identify some of the objects and try to like learn features about like edges and so on so so so that's the goal of this this line of work so it's best visually illustrated here so grayscale so an image can be just like you have RGB there are like various different color channels parameterizations of an actual image and instead of using the RGB color space you can use the L a B color space


and the L channel will behave like a grayscale image and the a B channels behave like the color image and you can use the L Channel encode it and predict the a B channels and that is basically the tasks consequently considered in the learning to colorize work so you can see that those yellow light pixels are identifying the eyes of the fish of the body and but you can also see that because the background is Queen starting with the color of the body of the fish it's it's not able to separate it out so uniformly color is the background but


it's able to color the coral reefs around so incident francium is green so you can see that it's able to understand some high-level aspects of the image by doing this task so so that's basically what's going on and to visualize what how the actual image looks like you can just concatenate the two channels and see how it looks like so that is the ocean and here the author's first tried the new idea which is take your draw a ground truth then you just do a mean square between your predicted and the ground truth and that would


producing a very degenerate like degenerate like colorization of the actual bird and the ground truth is made more colorful and diverse so what the author has realized is instead of treating the prediction as a mean square a regression task what if you treated as a classification task where you quantize the pixels so you quantize the a/b channel information into a button and and and and bit it into a bunch of categories and now instead of predicting some value for the a/b Channel and just regressing to the ground truth you're


going to actually output a distribution over the possible values because it won't test and we can then do a softmax puzzle to be lost instead of mean square error loss and it's in general it works out really well and deep learning to do this you've also seen how it worked out really well in pixel RNN and where where all the pixels were quantized to discrete categories instead of using a Gaussian mean square error going to use the process to be lost and that produces sharper images so here here's how it goes you you basically take an image you


have the a be channel prediction you quantized it and now you're going to use Krauser to be lost to predict your actual a B channel information so that is basically how the colorization work is done another version of this done by Richard Zhang was trying to do something like a split brain or encode or a split view or encoder which is you've separate the channels in your source you have encoders that try to predict the other channel from the current channel so where X 2 hat is the other channel prediction and X 1 hat is the prediction


for from X 2 and now you concatenate these two channels together and you get your actual input again and you want to make sure that this version matches your original version so so this way it's like ensuring a backward consistency in some sense because it's not just about predicting the color from your grayscale it's also like hey the a B channel should also make sense like whatever you predict from your a B channels your L channel that the and what are we predicting I'll channel to a beach on together if you look at it it should


make sense and it should look like my actual image so this can make more sense if you're looking at other kinds of views like depth image and color images so here's here's like one way to implement this which is you you separate out you have two different encoders you predict the other channels missing channels you concatenate and then you have a loss on the actual predicted image so and this is how it would work for a color and depth information so so so these are all interesting ideas and we're not gonna really look into the


metrics that these methods have because more than the metrics these papers are more famous for the input-output version itself which we looked at pics depicts where the fact that even colorization can work so well is so peeling but but but in terms of the metrics in terms of the numbers will we look at them much later when we look at contrast Aloni so here here is here is the other final the second version that we wanted to see which is introduce some kind of common visual common sense tasks so so so here we are going to look at relative patch


prediction this was an idea put forth by Kadosh abhinavagupta ala al Alex EE froze and so in some sense this was one of the first papers to do cells crew as learning on images at a larger scale and it's considered one of the foundational papers for a lot of the ideas put forth so basically what is the task the authors considered here the task was given two patches try to identify the relative position of the two patches which is to say that if you have a center patch and if you have a patch linked to its immediate right and


given these two patches to a neural network the neural network should say hey this patches to the right of this reference patch so look at look at its best looked at from this figure so you take an image you take an approximately three by three grid of non-overlapping patches and now given the blue patch you're trying to predict that the yellow patches to the top right corner so you can number the surrounding patches from one to three so on until eight so and you basically have eight categories for a classification


task given the reference center patch and this way you can take any you know different regions so the same image or like lots of different images you just have to take an approximately 3x3 grid of non-overlapping patches and give it to your neural network select two of them give to your neural network and create these labels for free from your data so it is it is another version of supervised learning where you you are actually adding creating a task like a jigsaw task sort of not exactly jigsaw but you can you can think of it learning


spatial associations and then you know then it has to understand hey if you give me the year of the cat and the eyes of the cat it's then then it's likely that the year is lying on top to the right or to the left and and it also has to understand that what is what is left in right here so so so that means that it's learning these low-level features and as well as high-level associations and it's way that could be useful for a downstream task so that's pretty much it you you you share the CNN in chorus for the two


patches you use the mean pool representation at the end and you train a classifier on top and you can create a lot of data for your training tasks because you can sample a lot of different batch grades from a lot of images and see how good the features are so there are a couple of the details in in in getting this right which were extremely crucial and which have been adopted in every follow-up paper almost which is making sure that you jitter both spatially and color wise so firstly you should make sure that to pick the patches don't


overlap or else it's so easy to get the well which branches like to the ref to the right just by looking at the boundary pixels so that's one obvious but but but highly non-trivial at the time but considered obvious now so you make sure that the patches don't overlap second thing is you jitter you jitter the patches to prevent chromatic aberration so so that the but by that what I mean is you you sample you basically you you sample a particular random crop view you can you divide it into three by three grid of patches and


then within each cell of the three by three grid you be you basically create another random crop and drop some color channels and so you basically do spatial and color jittering at every single patch to prevent the chromatic aberration from happening and that way the neural network can cheat and both of these details will have a non trigger at a time and and they were very crucial and all the follow-up work so another version which is very similar to router position prediction is you know actually going all the way


so rather to position prediction looks looks like a jigsaw task like like the jigsaw processed children saw so why not actually do exactly that just just make you don't have to saw six of puzzles and that's what this paper is doing which is you similarly take a crop three by three grid of matches from a random crop of your actual image and you shuffle them and then you try to predict what is the correct order of the shuffling so in this case you it has to identify that similar kind of spatial reasoning and associations so that that's really what


works well here and and and and if the neural network is able to solve this task well it means it's able to understand what this comes to the right and left it's learning general visual reasoning so how is it implemented a very easy very implement this is you if you have a 3x3 jigsaw puzzle task then there are 9 factorial possible permutations so instead of asking the neural network to output the exact order you can ask the numeron that will top with an index of that and you can hash the corresponding order in a hash table


of all possible permutations indexed by like some simple declare category and you can just have the neural network to predict the category and that way the neural network is the you know doesn't you don't need an RNN decoder or something like that and and and it just looks like a normal classification task where every single patch in that word is passed through the same shard see in an encoder the meaningful representations are taken and they're concatenated in some form and tried you just trying to predict this output category so a final


version of a final version of this idea of doing puzzle tasks creating data creating tasks from raw readers wrote the very simplest version is rotation prediction this is really really so simple that it's amazing a works over but but there is also a concrete argument as to why it works the idea is you take an image you rotate it by a random angle so in this case you're rotating it by a multiple of 90 degrees and this C and you pass it through the convolutional neural net and the conversion here and that is asked to


predict what is the angle you rotated the original image by that's that's really it so in this case for the first time as you would predict the conversion and that has to predict 90 degrees for the second it has predict 270 for the third 180 and so on for the fourth it just has to predict there's no rotation which is a zero so why does it learn something nice why does it have to learn anything at all so if you look at the 180 degree rotation like the only reason you're able to say it's 180 degrees is because


there are human faces and you know that they're inverted right so that means to be able to say this 180 degrees you've identified that there are humans faces in the image similarly like look at the first image there's a bird and there is it you know there is a tree and so on but but but you know that if the normal view would have been that the bird is like if it was tilted by 90 degrees the tree would have the bark would have been horizontal and the bird would be standing standing in the same vertical position so this is basically trying to


identify if there was a photograph for that captured the actual image how is the photograph of position right so so so that is an inductive bias that is physically or geometrically grounded which is Kam image formation is something that's fundamental and that's how we all record images and most of images on image net have been captured where the object was very centralized and so there's a lot of information as to why the camera was placed at what what pose and so on to capture the actual object so because you rotated it


you're actually trying to identify in some sense you're trying to do inverse graphics of the camera parameters here which the only parameter you care about is the rotation angle here but but since it's physically grounded it's gonna learn something useful so here is how they implemented it which is you take an image be rotated by various possible angles and you can struck these rotated versions you pass them to the same convolutional neural network and it is pretty clear rotation angle so you just do it for all possible images in your


dataset and you you you would learn really good features so one interesting point is you might think hey the more angles you had to a dataset the more good features that you can learn that's not particularly true so the authors found that if you just use for rotation angle 0 90 180 270 the and you put a linear classifier on top and train it on C for this is a small data set you can get 89% op1 accuracy but if you add multiples of 45 degrees which is like one level more fine-grain your performance drops by 0.5% approximately


and if you just use two angles which is less fine-grain the performance drops even more by two percent and and if you think about it as vertical versus horizontal like 90 270 instead of 0 180 the performance drops like another two percent so it is particularly important to use both horizontal and vertical and it is more important to use vertical and it is also not so important to use a lot of angles maybe the amount even that that's breathing eight rotations wasn't trained well but it's sufficient to predict four rotations you don't have to


be so fine great so here the results that they have where they basically train an allit so at that time it will be Alex net was the common backbone used for self provision studies so they basically trained an alux net on all these different tasks like like for instance rotation that is there rot net was there paper but the baselines are coming from the car existing substitution papers which we also covered so if you use the conlusion l 4 & 5 which are the fourth and fifth collusion filters in alex net and put a


linear classifier on top the the results from just using supervised learning features are the topmost row 59.7 percentage top one which is pretty close to what alex net gets in terms of top one accuracy so just using the random features gives you like 27 percent and 12 percent respectively and the context paper doors are always the paper we saw relative position prediction and and that's able to get fork on forcibly at forty five point six percent which is which is way better than random but not as good as supervising internet and colorization


work which we saw the Richard Zhang's work that's five percent lower than doing relative with a transposition prediction so you can clearly see that doing more puzzles like tasks is better than doing colorization and the jig suppose a task is on par with the context the relative position prediction it's similar 45% and bygone which is a paper we already covered in the again lecture is not as good as these puzzle tasks but it's on par with colorization and finally this rot net paper has a substantial improvement over the state


of the art at the time so basically the state of the art self supervision method had forty five percent and rot net accrued it to fifty percent and so that that's real clearly good and even on the con filer it shows like for the doors at all the the numbers are really low like 30% and the rot net is like clearly better forty three percent and way better than the other methods as well and these are like more more detail results for various convolutional layers and you can see that the rutland numbers are significantly better than the other


self supervision techniques at the time and it also transfers well to pascal so on pascal classification and detection and segmentation it ended up being sphere the arts of supervision method and was significantly better then context prediction so but but still the gap between rot net and imagenet labels is really large if you look at the detection results it's close down like like fifty-four point four fifty six point eight is pretty pretty small but on classification the significant gap was seven percent and on


segmentation there's a significant gap of nine for me not being mean my intersection over Union so so while this was a pretty promising technique it was still not there yet so that's it for like puzzle-based us next we'd actually get into this context based prediction techniques so predicting the neighbor context was the final line of work that we wanted to cover and again like I wanted to go back to this likkle slide where you you're basically interested in tasks in the neural network to predict missing parts from given parts or like


neighbors given the neighbors you're trying to predict the surrounding context so one idea which was explore way back in 2013 was this word to back and we're going to cover that for us because it's very foundational so this is a figure from the bar from the 2:00 to 4:00 in class with Stanford and where the goal is to learn good word embeddings so we're the bearings are very fundamental like you have a lot of words in the vocabulary and you would like to represent them vectors so that similar words are having similar kind of


vectors either directionally or they are very close to each other in the high dimensional space and the name word embedding you can use this is one hard encoding and that's hardly infinitive of any similarity across words so let's say that you have a bunch of sentences and you create account matrix of what words are occurring how many times so in this case you're basically trying to do some co-occurrence matrix if that that's very popular in NLP which is how many times each word co-occurs with the other and that that's usually used to


construct these similarity matrices and so this is how the count matrix looks like so I in like are occurring together because there are two sentences with them but but IND don't go together like like basically once this matrix is constructed you can think of applying single value decomposition to this matrix and this is really how recommender systems have been built in the sense that you will have a history of what uses goodbye war items and you would construct a user ID matrix and then you would do an SPD on this


matrix and you would get a user embedding in an item embedding and you just cluster or similar users in similar items and you will use it to build a recommender system so similarly you can think of building a term frequency resistant frequency matrix here in a low peak and constructing a sweetie and get word embeddings so that's that's precisely what's being done here and you can get these U and V vectors so so what is the problem with the SVD approach this video prods the first one sparsity right so obviously there may be very


related words which should be not necessarily co-occur with each other because you just not have that possible sentence so sparsity is a big issue the resulting matrix you construct is very likely to be sparse and the computation cost right so SVD is a third order to compute so it's not going to be easy to optimize and there's also this problem of infrequent words which is when you have any dope when certain words are not particularly frequently present they're going to be hard to optimize because the word embeddings for them


will not be very accurate and you can also have noise from very frequent words like like like you know words like a or D or articles and these are going to be very frequently present so you have to use some heuristics like inverse document frequency to make sure that they don't corrupt your bearings and it's however all of this sounds like a very hockey engineered pipeline and it's not particularly efficient and it's not going to scale well with like larger larger the vocabularies or like larger data sets it's it's very hard to scale


this approach so that comes the idea of using Engram language models which is hey you just have if you just have a bunch of terms and in a sentence you say that the probability of sentence is probability of individual trans present in it and that's the unigram model but a bigram model is like the problem you you you bet you basically try to say that you take into account the previous word percent in the sentence and say that the probability of a word is conditioned on its previous word and that's saddam v you can start counting pairwise


occurrences of words instead of just you know frequencies of single words so an Engram model basically generalized stuff so so let's let's actually go to the word Tyvek idea which is going to clearly generalize all these things and this is first proposed by niccolò back in 2013 and so we're two back what's the area here you're going to have a bunch of surrounding words and you're going to try to predict the center word so you have a sentence you're you pick a particular word and treat that as a center word make all the surrounding


words treat them as surrounding words and embed each of them and try to identify what is the center word from the surrounding woods so that is referred to as the Cibao model and the script skip grandmothers exactly the mirror image of this model which is it tries to take the center word and it tries to predict all the surrounding words so one way is you embed all these individual one heart and coatings of the original words into a word embedding matrix which is basically like the word embedding matrix would be the number of resistance


embedding dimension so by embedding one heart word it's basically if we went looking up what what what the corresponding word embedding is and the C bar model would just average the embeddings of your surrounding words and try to identify the embedding of your missing word and the skip grammar model would basically try to use the word embedding and do an individual softmax over all the surrounding words so let's actually look into the math of how this works out so consider the Cibo Matto and here you're trying to maximize the log


probability of the surrounding word or the context word given the neighboring words and so you can think of this locked bu WC given all this WC minus ni for different values and so so so the way it's actually constructed is huge is average the embeddings of your neighboring words it's a very simple model so once your average that you just have a single vector and now you just say that this is a nonparametric softmax over all possible words that that you can have for your Center word and so that way you don't have to explicitly


take a soft max you just optimize for the dot products of the averaging words from the neighboring words and the possible product of you are sent over and that's what this loss amounts to and oh and this way the parameters of your loss function end up being your word embedding matrices and all you need to do is take lots of different chunks of text pick particular sent over to pick some respondent neighboring words embed them average neighboring words try to maximize the dot product of the actual central word relatively all the other


words in your vocabulary and if you do this for waste chunks of text and optimize for a while you're going to end up with a relatively good word of any so that is the idea of what effect the SIBO model and the Skip Graham model is exactly the mirror image of this model where you're going to try to predict every independent sounding word with the key assumption it makes us that given the center where the surrounding words are all independent of each other that's in the probability of a surrounding word given the Center word is independent of


the other surrounding words so that's kind of similar to the Navy base assumption which is given the class the term frequencies are independent of each other so it's similar assumption to simplify the computation and so then you just have a similar kind of nonparametric softmax over the possible word embeddings of the surrounding words and you just perform a similar optimization so a main idea here in the nonparametric softmax is you need to optimize over all possible word embeddings in your vocabulary and that


could be very computationally inefficient especially back in that time when GPUs were not the go-to mechanism for deep neural nets and the main aim of where Tyvek was to have some software that where you just feed in a chunk of text and it could just run and spit out word bearings for you but this whole process can just run on a very simple lightweight CPU so the authors went for very clever techniques like using negative sampling and not normalizing over all the words in the vocabulary in the denominator so the partition


function is now going to cover the entire vocabulary and as long as you pick really negative samples you can be very efficient in terms of the kind of embeddings you learn and you don't really and and and so we won't really go into the details of that but you can refer to the paper in terms of what how hierarchical softmax is and negative sampling was used to make word to that really efficient and in terms of results the authors had really good results for the resulting word embeddings that we learned for instance here if you look at


the word embeddings of different countries and their capitals you can see that the difference vectors are all quite parallel alright so the vector from China to Beijing and Russia to Moscow they're almost parallel so it means that the relationships captured between the aspera the words are easily lay you can translate these relationships easily to like another capital and you basically have to translate the country by the same translation vector so so it's geometrically very consistent and you've also seen earlier about how DC Gann was


able to do vector arithmetic on like Sarah Brady faces it was all pretty inspired from word to back because you can because it's geometrically translation consistent you can think of taking the vector of Portugal and you know you can add a you and you can take the vector of Spain you can subtract those two vectors and you can think of the difference vector being similar to the difference vectors between the capitals Lisbon in Madrid there are difference in the position vectors so so here are like various different you know


clustered word embeddings for categories of different different categories of words okay so in word to back well the authors tried to look at how the word embeddings cluster together and you can see that various newspapers are clustered together and various NHL teams and ba teams you can actually see that if you take the nearest neighbor of if you take a nearest neighbor of Detroit you get Detroit Pistons or Oakland and get Golden State Warriors off you take the nearest neighbor of Steve Ballmer you basically get Microsoft for Larry


Page look at Google for Airlines you basically see that Spain and Spain are closes to each other are same for Greece and egg and Airlines so so you basically can please see that the word embeddings because they have looked at what terms are good to next to each other and so forth they've understood relationships between companies and their CEOs or like you know Airlines and the countries that they operate in and so forth and that's that's true that's really interesting I'm just going to switch to PowerPoint okay so so so next thing is how does the


clothes and tees relate for different short phrases using the vertical bearings so here is one thing about various explorers like let's go to gamma and you can see that there's a relationship between Italian Explorer and for chess master chess grandmaster there's a relationship between Garry Kasparov and and so so even these sharp phrases the closest entities that are being very exactly relevant and you can also see how it's relevant to the airlines and you know or or if you add two different embeddings what is the closest entity you get so


for instance if you add Vietnam embedding for Vietnam the embedding for capital you're ending up getting the closest and the closest nearest neighbor and bearing up that of Hanoi which is exactly right and similarly if you had the embedding for German and Airlines and it flew it searched for the entity or the word with the nearest embedding you're getting airline Lufthansa which is really cool and similarly Russian plus River get Volga River Moscow and you get various French actresses and and you also get the currency for Jack so basically it's


understanding relationships at phrase level not just word level and understanding relationships between multiple phrases so another interesting example is how can get all these different skip phrasing bearings so you can see that the closest tokens for skip grammar models are like way better compared to the other models so these are like different models and like me at all is another example of noise contrasting model those strain on words and skip phrase model too McCollum or Oh like like a basically the nearest neighbor to the topmost row


redmond is they the most relevant in the skiptrace model compared to the other models for instance redmond Ross Redmond Washington Microsoft and you know graffiti he get spray paint and graffiti taggers etc whereas for the other things for instance for graffiti the nearest neighbors from these model is basically things like anesthetics monkeys Jews which doesn't really make sense at all so so that way you know it's understanding the actual things in the script our model so next we look at this paper called representation learning


contrast to predictive coding in some sense the best way to understand CPC is if someone were to do work back on all the modalities and not just text how do we go about it right so you remember that word to back has a very interesting model but it's also very primitive that is you if you look at the c ba model you know you're averaging the embeddings of your neighboring words and then try to predict the context word so but average this is like a rape a you know you don't it's only important if you really care


about the simplest possible linear model that you want to get working but in principle you can aggregate context using neural networks right like we have really powerful neural networks for context aggregation like continents or transformers or LST amps so why not actually try to put this all together use the contrast loss that were to accuse is to predict the neighbors in a nonparametric softmax but try to replace all these embeddings of individual words and surrounding words with very powerful expression you know networks so that


forms the basis for contrast the predictive coding then by Ironman the New York so here's the idea let's say you have an audio signal the raw audio signal and let's say that you're trying to predict the future audio signal from the past or you're trying to for relationships between the future audio in the past audio so call the past as see a context see and call the future as X and instead of predicting the actual audio like a vein net what do you want is to just operate at the latent space so let's say that you encode the context


in with an encoder and you also encode a future audio chunk we call you can call it as a target and you encode the target at the same encoder so now our goal is to maximize the mutual information between the context and target don't really worry about like what mutual information well why does it come out of nowhere that's not really the objective here the goal is we want to make sure that we learn a representation the encoder sits that its maximally predictive of the actual future when contrasted with with some fake futures


so so that's that's really what's going on in CPC which is hey you produce your neural network a bunch of targets where your actual target is also present but you could also have some fake targets so imagine that you could sample random audio chunks from totally different waveforms or you could also sample audio chunks which are not exactly corresponding to that future time step and you could present the inure network various different alternatives of what the true audio chunk should be and based on the past context the real truth and


all these other possible alternatives the neural network is supposed to pick or classify which is the right future and similar to word to back this can be done in a nonparametric softmax fashion where instead of actually decoding the audio chunk you're just trying to make sure that the embedding of your context and the embedding of your true future should correlate the most when contrasted with the other fake audio chunks so so that so that way becomes the softmax or your number of negatives and and and and you can actually use any


kind of a score function for how to assign the score between two embeddings it could be a simpler product in contrast pre-recording they make use of the bilinear dot product which is which which is also been used in the past in past work so that's a little more expressive than just using a regular dot product and it doesn't require you to normalize the vectors so so that way you can think of the W matrix in the bilinear product is learning some kind of Association matrix that figures out some property that helps you to


correlate two different things so that's really what's going on in CPC you're basically going to take a huge audio chunk you are going to use audio signal you're gonna split it into small chunks you're gonna include each of these small chunks with a shared encoder it could be straight conversation on your network in this case and you could take a bunch of past audio chunks pass them through it gru any auto regressive model would do and you could use the final gate in state of the gru to predict all the future Leyton's of true


audio chunks but predict I just mean that your contrast elite trying to maximize those embeddings through the true futures when contrasted with the fake futures and you can sample negatives by using other time steps within the same audio waveform or you could use other audio waveforms and depending on this negatives you're learning different things for instance if you're trying to collect audio signal from different speakers the negatives that come from other speakers lets you learn representations that allow you to


identify the speaker while the negatives that are across the same waveform within the same speaker that you let's you learn like more fine-grained phoneme features which are useful for like phoneme classification so depending on the downstream task the kind of negatives you pick are going to be crucial so here's basically CPC at a high level the diagram is very very very clear you're just basically gonna try to do this across various audio waveforms various different numbers of time steps you should be very careful in picking


what is the gap between the future and the context so there are overlapping audio chunks just like how you saw in the door shadow work where if you had overlapping patches it's going to be very easy to do these jigsaw puzzles Dallas CPC basically also suffers from the same problem so you should make sure that your negatives are come or your actual prediction tasks are not so trivial enough that you could just look at the whatever is overlapping and just try to predict that so that's one really important thing about CPC but the more


interesting thing is that you can actually perform all these tasks in any kind of fashion you can't you like you can so that the order of time is not so important you can pick anything as the context and anything is target so you can actually predict from the future and go to the past as well or you could even mask something in the middle and use everything else as to context so it's it's totally up to you to how to frame what is the context and what is the target in CPC but based on how you frame it you should make sure that the


negatives and the targets are chosen in a non trivial fashion so you can clearly see that CPC is trying to generalize the older ideas like puzzle tasks and were to back together it's basically a framework in which you can perform all of these different tasks within one one particular architectural variant and and various different hyper parameters will correspond to various different versions of these different tasks so you can think of CPC is trying to do something like multi-step prediction tasks so if you look at these multiple predictions


emerging out of CT vector in the diagram you can think of a different W matrix being used for each of these different prediction time steps and each of them corresponds to say hey predict one step ahead or parade two steps ahead predict three steps and so forth and you can think of all them is trying to make the representations learn different things and because you are optimizing all of them at once you're trying to learn a really rich representation that is able to do lots of different sub supervised dance at once so it constructs a whole


variety of pretax tasks within a single loss a single framework and it's railing and then appealing that way so one figure which is really nice to understand what CPC is trying to do is how it basically is doing something like slow feature analysis but what I mean by that is you your audio waveform is really high-frequency fast wearing and you're actually information that you care about for downstream towns is basically the slow bearing like hike signal content like phonemes because that's what really allows you to use CPC


features for something like speech recognition so what what CPC is trying to do is its trying to make predictions in the latent space at the levels of phonemes instead of variety of audio waveforms so as you keep processing these are your signals the information becomes more more semantic and so if you're trying to predict something that's not overlapping in terms of a target if you're trying to predict your target that's reason to be a few time steps ahead you're trying to go more towards these slow varying phonemes that


are that you know they would only change if the time steps are sufficiently further apart and so therefore if you're doing predictions are an appropriate offset by offset I just mean that the gap between City and zt+ cave are the case like a number of time steps between the sufficiently high sister the phonemes actually change then you're going to end up learning really rich features so this is a really nice visualization of the representations learned on CPC audio audio tasks we're basically collecting a data set of


various speakers as audio waveforms and just performing the CBC optimization and your for and then you're taking the embeddings out and doing a piece near visualization in 2d and you can clearly see that different speakers having clustered out in separate blobs so it's clearly garden the speaker initially and you can also see the accuracy of predicting the the you can also see that the accuracy of predicting the positive sample and the contrast of loss you it basically is so high in the beginning but it keeps going down touristy and by


that what I mean is it's much easier to perform this contrast to task of identifying what is the right future when the prediction task offset is not much when you're actually trying to predict much closer to the future but as you keep moving further and further away the mutual information between what you already have and what you're trying to predict is much lower there's much more entropy so you're not actually would optimize those future time steps as well because the context is not sufficient so the accuracy drops exponentially as you


keep increasing the number of time steps you're using to predict the future so here are the CPC audio results in terms of the downstream tasks on the left you see before the phoneme classification and speaker classification results and for phoneme classification there are four even possible classes so basically the way it works is you take the CPC features you freeze them out and you just put a linear classifier on top of these CPC features and you try to perform the task but what it look like which is to say you try to identify the


phonemes in the audio chunk or you try to identify who spoke this particular thing and you have labels for that but you're not gonna change the features you're just going to keep them frozen so for that version CPC speaker classification gets ninety 7.4 percent accuracy it is very close to what you'll get by just doing supervised learning whereas if you just use something like m of CC features which are which is very engineered you're not able to do that well you're just able to get 17.6% so these features that you learn in a


completely unsupervised fashion are v better way more semantic than something engineered with domain knowledge and also for phoneme classification which is actually even higher than speaker classification CPC features without any fine-tuning just linear classifiers able to do is able to get 64 point six percent way better than MCC features which is this forty percent and we're better than rounding random initializations 30 percent and supervised learning gets seventy four point six percent which is 10% better than just linear classifier


on top of CPC features however the authors found that if you put an MLP instead of a linear classifier you can actually get pretty close to 74% with just CPC features no fine-tuning so this means that the information may not be linearly separable but all the useful information for performing foreign classification is there the CPC like the uncovered features and on the right you see a positions for phoneme classification experiments and the the point I mentioned earlier is are really illustrated well here where depending on


where you take the negative samples from it you're going to get different levels of results so if you if you if all if your negatives are all coming from the same speaker the accuracy is like sixty five point five percent that's that's basically for that that means that the models like learning only phoneme relevant features it's not trying to do speaker and if occation whereas if you are if your negatives are all coming from mixed speaker so that that's going to get sixty four point six percent which is the result on the left table so


that means that if you are more clever about sample negatives we already know what your downstream task is you could prioritize something negatives in a fashion that will incentivize CPC to learn the features that will be more relevant for your downstream tasks so if you're if you don't really care about speaker identification you could just make sure that all the negatives are constantly coming from the same speaker and so that's really interesting way to illustrate this point and and and and the second ablation that they did is the


number of steps you predict into the future and namely you would imagine that predicting all the way into the future is going to be really helpful but it doesn't turn out to be the case so if you predict only up to twelve steps instead of predicting 16 up to all the way up to sixteen steps the downstream accuracy is better so this means that the right way to predict it is such that the targets that you're trying to predict should share so amount of information with the context that you already have you go further and further into the


future the entropy is higher and so the amount of actual information amount of bits that are shared between the two entities are not much so predict like for making a neural network focus on those stars may actually end up encoding not very useful features so the hard part about CPC is trying to pick the right number of time steps to predict into the future or like they're how you sample the negatives but if you get those details right the features learnt are really useful and on par with supervised learning so one


motivation for CPC was that they wanted a single framework to work on any modality any kind supervised learning any modality you should just be able to use the same framework so that's a lofty goal so let's see how they actually instantiate it for image not so here is the image snap numbers for CPC where the framework how they actually executed is as follows you take an image you take these overlapping patches so you grid your image into a bunch of overlapping patches so in this case the image is 256 by 256 from image net and you're


basically taking a 64 by 64 patch laid across the image and you take matches with 50% overlap so 32 by 32 is a stripe and that mean you get a sent by seven grid of patches and you would encode each patch it with the same rest net so think about it as a rest at 101 or less than 50 and you would get an embedding the meaningful embedding at the end and what you do with that is now that you have an embedding at every single patch this will form a grid of embeddings and you can perform predictive tasks 2d predictor tasks and on top of this grid


so you can treat this great as your sequence just like the audio sequence there's a bunch of overlapping audio waveforms audio chunks from your actual raw ear signal in this case it could be a bunch of were lapping patches in this 2d transcript and the task that the artists construct is to predict the future parents from the top row of patches so in this case you basically using the first three rows let's say and then you're trying to predict the bottom three rows you try to predict every single patch in the bottom


three rows by using the row the row of matches from the top three rows so in order to aggregate the context of the top few patches you would want to use some kind of a model that can take a bunch of embeddings in two-dimensional layout and try to summarize what is embedding screen at every single spatial location and that's and you also want to do it in such a way that you don't want the information from the top to leak into the bottom because you're trying to predict the bottom for the top so we already know of one model


that allows us to do this very efficiently it's called the mask convolution or the pixel RNN fix the scene and style models and and and and it also makes sense because CNN's were also invented by the same first author so he just use pixel cnn's to aggregate the context of the top few rows of batches and once you lay that out on top of the grid of matches you can predict the bottom patches in a very parallel fashion so here is an example of how it would look like for an actual image when you grid it into patches so take this


dog and you're like constructing all these patches overlapping patches and then you let's say you you take the first three rows of embeddings and you try to predict the last two rows or which oh this is dispatch belonging to the last row second column or not it's this patch belonging to the last row third column or not so you would perform all these predictive tasks once once you get all these embeddings of individual patches in a pixels in and on the top so how does the accuracy work out for like say you you're doing something similar


to the audio experiment where you trained all this and a lot of images and then you take just the rest net encoder out and you put a linear classifier on top of it and see how well it performs on an omission air classification which is also the standard test being used in previous self supervision methods though initially they were all attempt with Alex net and but but the baselines for rest net existed this table so we've already seen relative position we've seen by gal in synchronization and and jigsaw puzzles so all these methods when


you just use a rest net encoder and you put a mean pool at the end and you put a linear classifier at the end D we get you a top one accuracy like not more than 38% jigsaw works the best and the rot net numbers are not there on this but they're out net numbers are not are not higher than the CPC version of the CPC numbers so so if you look at the Aleks net results they are like around 38% and if you use a resident we do every single baselines numbers goes up so relative position gets a 6% game by just using a rest net video instead of


an alux net and colorization goes to 39 from 35 so the baseline for jigsaw doesn't exist here but I would imagine it getting to somewhere in the early 40s so CPC gets forty eight point seven this is really an old result now we will see in the next few slides how the state of the art has been pushed up way further but at that time this was a pretty big jump from the existing state of the art and it also works really well if you look at if you look at competitive approaches of similar nature like you know a relative position of jigsaw is


like kind of performing these spatial Association tasks too but doing it in a contrast of fashion and doing lot like a family of tasks within one parametrize model gets you much better numbers and these are the standard reason we should visualize what kind of features that these models learn which is you take a particular feature there and you just see what new what what what kind of input maximally activates a particular neuron and you lose for a bunch of neurons and you see that you know like maximally activating neurons are the


corresponding to different classes in this case the first row is corresponding to leaf like patterns and textures calculator textures computers keypads and then skies and baby faces dogs so so there are clearly capturing all these high-level omission of features so another version of CPC was to try it on language will not really go into the details here but it but at that time the it was competitive with skip top vectors which had similar similar ideas like predicting the future sentence when we give him the pass