is standardized you don't have to care about that right and the score that they measure at the beginning of training and what you can see is that there is a linear relationship sort of like a sort of from from these experiments what you'll get is like this sort of feeling what they're gonna propose is that you should take that score as a as a measure and here again also sword of sword sword of there is a there is a clear trend as you can see right here though yeah though this as you can see this sort of spreads out and the most right one is


imagenet which is the most difficult one of course so and this is C for 100 which is more difficult than C for 10 so we can see that this sort of relationship at the top it doesn't really hold anymore if the task gets difficult and this is so what I think is happening this is kind of an interjection of my own opinion what's happening here is that this score that they discover allows them pretty efficiently to see which networks are just degenerate and and cannot be trained like if you try to train them they just perform really

Neural Architecture Search without Training (Paper Explained)


poorly okay that it's probably a very good score for weeding those out and that would mean if you kind of barrier here somewhere right you could just discard a whole lot of this crap or even even here right you could just discard a whole lot of this crap and also now here just you know all of this crap yeah whereas here as you can see some of this score sometimes it's higher then these ones even though they perform better and again you could probably discard a lot of the crap but it's not as distinctive for the well performing


networks because these here are all not the degenerate version right they're not degenerate in the sense that they're they have some fundamental flaw where the function lakhs now expressivity from the very start so you can't train it and then probably other factors come into play other factors then you can simply determine with this particular score but you know there there is this relationship that's that's you know you can see that and they do some ablations on this here for example or your score is a proxy for a number of parameters


and they say no the number of parameters works way worse than this particular score which it always is a cool thing then how important is a specific mini-batch and initialization and they say look right here we for some architectures we do different mini batch sizes and you can see each of those groups they don't vary too much in how their it influences their score this is I believe this is the same architecture so it's always an architecture that achieves in this case for example wow that's a not a straight line 77% or so


and you can see if you go for different mini batches the score varies only minimally initialization is a bigger variance inducing thing but also here the scores don't vary too much but it is interesting that the different initialization to get you to different score because it would directly support kind of my hypothesis now what's going on here is that you sort of measure initial degeneracies and you can sort of make up for these initial degeneracies in the architecture sometimes with sort of a different initialization so the


different initializations give you differently performing networks we already know this from things like you know a lottery ticket hypothesis and so on that the initialization can matter to some degree in these types of things now that being said they always train to the same it seems but their their score varies so I might be backwards correct here or not correct but in any case the initialization here matters more but also you can still see this linear relationship and this is particularly interesting this is even the case when


you just input white noise so instead of the data you measure that score by just inputting noise that I guess has some sort of the same magnitude as the data would have but it's just noise and you can still sort of see this linear relationship which is very interesting and that I think also shows some that you what you'll find what you find is a property of the network itself and the fact that it is it is initialized and built in such a way that it allows you to train it in a very in a sort of a benign manner it has no degeneracies


okay so in last experiment they go here and they say we evaluated the score on initialized networks in the PI torch CV library so they go to this library that has a lot of these networks but these networks are not the same as this benchmark this benchmark is specifically designed to do architecture search now the networks in this library they are all designed to perform really well some are designed to be quite small some are designed to be quite fast and so on but in general they're all of their goal is to perform well and they have been sort


of found by humans to perform well so they take now these networks on C 410 and they test them so as you can see here here is the test accuracy again and here is their score that they give it and they say rip it up put it up now I can't move this anymore hello well okay they say that this linear relationship still sort of holds it doesn't it doesn't hold super super well but you can still sort of if you squint if you squint hard you can see that it sort of goes upward though you really have to squint hard like what are these things


right here and what again what's the case is that if the score is low you will sort of be able to cut off the cut of the worst-performing ones but really at the top here it doesn't seem like there is a particular relation between between these networks and this initial score which sort of strengthens my hypothesis that what this does is just kind of weed out the bad ones but it's pretty cool because you can weed out the bad ones without any training right it's simply forward prop backward prop there you have it so cool now they come they


here is the experiment where they now really do this na s benchmark and they compare with other methods so some of these other methods are designed to do they call it weight sharing which basically is a technique where you can sort of speed up the speed up the algorithm as compared to non weight sharing and the non weigh cheering that's one of these we have discussed initially that was my initial example with the controller and so on where it takes super long so here you see the method and how long each method takes


now the best ones as you can see already the best ones here are these these methods right here are the best ones they score somewhat like a 93.9 or so on C 410 whereas these weight sharing ones they don't perform too well except this one seems to perform quite well and in this hour's case they perform worse than that but they still perform better than a lot of the weight sharing once so what their point is basically is that they get a pretty good score which is a ninety one point five on C for ten which is you know it's at least not


degenerate it's a it's a good accuracy they score that with simply evaluating ten architectures right and as n goes up as they evaluate more and more architectures they do they do get better but not much so they have a discussion here I'm having trouble moving this all right so we'll sort of go through the discussion we report results yada yada yada yada as the set up the non weight sharing methods are given a time budget of twelve thousand seconds for our method and the non weight sharing methods are averaged accuracies or averaged over 500


runs for weight sharing methods accuracies are reported over three runs with the exception of G das our method is able to outperform all the way chairing methods while requiring a fraction of the search time and that you maybe see at the table this is the real I mean this is the real deal here they only use here one point seven seconds compared to the twelve thousand seconds of the other methods and you reach almost the same accuracy now to be said two percent in this particular regime on C 410 is still a sizable difference and


that's the same benchmark right with the same sort of the same training schedule and so on so there's not too much room to tune here you simply have to find a better architecture so these things are still sizably ahead of this and what it appears to me that these methods here that don't perform well they're they're simply crap it seems they're simply I don't I don't know but they might be trying out something or you know doing something researchy or whatnot but it seems like if you're well able to weed out the bad architectures you might be


getting to a score like this and then if you are actually performing a search to find the best one then you might be getting to somewhere like this and you can see this here throughout so in C for 100 they achieve a better score than these things but a worse score than the non weight sharing method and an image net it gets even the difference is even larger so again what I can see here is that there's is a good method to maybe get you like let's say 90% of the wear of the way you want to go and what's interesting is that here they say we


also show the effect of sample size we show the accuracy of the network's chosen by our method for each n so that's the sample size we list the optimal accuracy for sample sizes 10 and hundred and random selection over the whole benchmark so in this case they have the the optimal one which I guess they just draw 10 samples and then take the best one so they train all of them and then take the best one you can see that already gets you to the 93 and whereas in their case sometimes when they add more they get worse so here


they get better but then they get worse again so they comment on this right here we observe that the sample size does not have a large effect on the accuracy of our method but note that as sample size increases our method suffers from a small amount of noise increasing the gap between our score and the optimal result and of course the key practical benefit is execution time so again they are massively faster than the other methods but to me it seems you could just think of combining these methods right you combine this with this in that what you


want to do is actually actively search for the best ones but by doing so you could if you could pretty quickly weed out the bad ones using this method down here you might already have like a big speed up because again with comparison to this random ones what appears to happen is that the get good at finding you know you're 90% architecture but then they fail to differentiate the top performance performers from each other where you'd really have to train the network to find out what's you know which one's better


so yeah here they say they visualize the trade-off between search time and accuracy for C for 10 for different NES algorithms on dnas benchmark by removing the need for training our method is able to find accurate networks in seconds instead of hours and here you can see the accuracy and here you can see the time and all the the good ones are either way over here or here and there's is almost at zero while being quite close to the accuracy of the other ones all right yeah that was that was this paper again I think this is pretty


valuable if you are especially if you're in a new domain where you might not know what kind of network to build you might just be able to write a little script that generates networks run it through this algorithm and at least you get an idea of which ones are certainly not worth considering and then you can simply select one of the other ones it doesn't you know often it doesn't need to be the best ones and you can then tweak it a little bit manually the ones you found may be you see some regularity and yeah that was my two cents on this


hi welcome to lecture nine of CS 294 158 deep unsupervised learning spring 2020 I hope everyone had a good spring break despite the rather unusual circumstances today we will be covering two main topics semi supervised learning and unsupervised distribution alignment before diving into that couple of logistics so common situation and a quick mid semester update well first we hope everyone and their families are able to keep healthy during these pretty unusual times please practice your health and well be and accordingly

L9 Semi-Supervised Learning and Unsupervised Distribution Alignment -- CS294-158-SP20 UC Berkeley


please don't hesitate to let us know if this class would interfere with that and we'll be happy to figure something out when the kiss by kids faces since water just past the middle of the semester and says there's some replanting paired the current situation here's a quick overview of what's still ahead in this class today we have lecture 9 which will cover semi supervised learning and unsupervised distribution alignment next week we'll have lecture ten on compression which will be a live zoom lecture then at the end of next week


your final project three-page milestone reports argue this is not graded but it's a great way to get feedback and make sure you're on track for a good final project and we'll try to give you feedback into your Google Doc that you share with us on pretty short turnaround then we'll have lecture 11 on language models with a guest instructor aleck Radford from open AI then we'll have our midterm which will adjust to the current circumstances we'll see how we do it but the high-level model will promised a similar that we want to cover the main


derivations that we've seen in this semester and have you be able to read arrive those then we'll have our final regular lecture lecture 12 on representation learning in reinforcement learning that also be a live zoom lecture and recording then as our our week which hopefully gives you time to catch up on a lot of things you know hopefully including them making the extra push in the final project for this class and then during finals week on the 13th of Wednesday there's final project presentations we'll see how to do that


with the new situation and final project reports will be Q at that time also with all the logistics covered let's dive into the technical accountant for today so we'll cover first semi-supervised learning which Harvin along will cover and then we'll cover unsupervised situation alignment which will be covered by Peter Chen and Ashley will use the lecture he gave last year also for this year welcome to lecture 9a of deepens for us learning in this part of the lecture we'll be covering semi-supervised learning so first to


understand what is amiss for us learning let's look at supervised learning in supervised learning you have a joint distribution of data points and labels and use sample from the Joint Distribution in expectation over two samples from the Joint Distribution your goal is to maximize a lot of probability of the classifier love P of Y given X we all know how to do this as far as procedurally how it's done the basically sample image label or like sequence and particular label or any pair of x and y from your data set assuming they're all


independent identically distributed and you don't know what the analytical following the distribution is where you assume that it is some complicated distribution and you just keep sampling multiple points repeatedly and from stochastic rain descent optimized objective now what is semis corresponding assume that you have an unlabeled data set tu where X is sample from P of X which is a marginal responding to the Joint Distribution from the label data set sample from so you have D you the unlabeled data set and D s the label ta set and your goal


is to perform the same thing earlier which is supervised learning on label data set but with access to this extra unlabeled data that's coming from the same marginal so that's the assumption you make semi-square is learning which is that the unlabeled data is coming from the margin of the correspond to the same Joint Distribution that supervised materials coming from and assistant mathematical assumption in practice you can't really ensure that but your goal is to make sure that you can use this extra unlabeled data to perform your


supervised learning objective even better on the label Deana so not mathematically here is the summary of what we just described take a task like classification a fully supervised scenarios where you have every single data point given to you in the form of image from a label they try and predict a label for new images that's your task that for a supervised learning now the scenario is you're going to be given a few on label samples but multiple you're going to be given a few label samples but you're also gonna


be given a lotta on label samples now labeling is a time-consuming process and potentially very expensive and actually pretty hard in certain domains like medicine right so or detecting rare events and sub driving for them for that matter so if you have a lot of unlabeled data and your training data set can be reprime tries now is having some lis pairs of label data points image command label and also a lot of other data points where you just have the image and your goal is take this extra data extra data points where you don't have labels


and try to improve our classifiers it just works with the label data and how you do it is totally up to you and that basically decides what kind of summarized algorithm you're gonna come up with to use and we're gonna look at in this lecture we're going to look at how these algorithms can be designed what are the mathematical or intuitive aspects of these algorithms and how you compare each other and how they can scale the larger data sets like image that and beyond so as to why we even interested in this problem semi-supervised learning is


really important because they believe us even though you can collect label data very easily these days with a lot of data annotation startups it's still expensive in terms of hiring people to write annotation manuals for actual data AM traders and preparing graphical user interfaces so that all this is done really fast and making sure it's stored on the cloud efficiently and syncing from the browser to the cloud there are lots of engineering challenges involved in setting up a good data on editing tool now that's not to say we're never gonna


do it we are still gonna do it but the goal is to make sure that we don't do it as much as we're doing it right now because we also have access to full-on link data that we can potentially exploit and they maybe even improve the performance or legal data systems this is similar in spirit to our goals for Salesforce learning and service learning is a different take on this subspace learning can work with just unlabeled data whereas my learning needs some label do a lot of unlabeled data that's the key difference so here is a slide


from Tung Wong who took this particular picture from Vincent Van Hook on a blog post called the quiet some expose learning revolution where the belief of many more practitioners at least until recently was that semi-square is learning will be really useful in a low data regime where it's really really going to be better than normal supervised learning when you hardly have any label here however once you collect sufficient amount of labels supervised learning we catch up and eventually be much better this is why a lot of


startups don't care about since Portland because the amount of effort needed to do research and engineering together since we're learning working especially even that it's a brand new fear this lot whenever needed to collect the external data points and you're guaranteed a better performance anyway so that's the rationale so as far as the left bloc glows but look at the plot on the right the dream of many times players learning researchers is that it not only is gonna be super useful in a loliter regime but it's gonna be extremely useful even in


the high energy because it's still gonna give you that final few percentage extra performance because of access to a lot of unlabeled data and learning much richer or more fine-grained detail classifiers because of that and that's basically what's happened recently and we look at the history of the field in recent times so the core concepts needed to understand Sims quest learning are very few and we're just gonna look at them at a very high level and it's really really intuitive and not hard to understand so the first principle we


look at is confidence of classifiers versus minimal entropy on unlabeled data it's it's it's a very nice coupling that the disciplines will shows and it's being used to very good effect in recent recent work and mathematically there have been papers on these two ideas which is first ideas entry minimization which we look at just the idea of taking on label data and making sure that the classifier training or label here has minimal entropy on unlabeled data so that way you're making sure that the classifier is confident even on


unlabeled data it's a useful bit a regularizer classifier the second idea is your labeling where you take your classifier you asked to classify to predict what the labels are for unlabeled here and you take the confident directions and convert them to extra they ask 'but was the ground truth and train the model on those data points so this idea is also referred to literature as sub training the area of training the model of its own predictions if the model is confident enough and expanding your data set and regularizing your model further


this is a little tricky because it has this reinforcement reinforcing effect of the model using its own predictions so it needs to be done very carefully so that's the caveat there and the other way to add noise to a model to regularize the model is virtual adversarial training which we really look at in detail but the idea is similar to how Ibis your training is performed for and you know absolute examples and images where you have a particular label and which you're trying to fool the classifier and believing


that this image corresponds that label and you're trying to add a particular noise in the direction of the gradient of the output with respect to the input Society model starts producing some other label similarly here you want to make sure that the model the same squared learning model is regularized in the directions around which tomorrow on unlabeled data you want to make sure find directions in which the classifier is likely to be confused and you want to make sure that the model is not confused in those directions so that's the area


in which we'll have a single training and there are also areas like label consistency which is make sure that the augmentations of the sample have the same class so you have an image we know that we use a lot of the documentation and regular supervised learning but in semi-supervised learning you have a lot of unlabeled data and you can't apply the augmentation to them if you're not passing them to the classifier but instead what you can do is you take an unlabeled sample it create two different orientations of it


and you see it you tell the classifier that it's predictions on these two different augmentations of the unlabeled data should be roughly similar because even though you don't have a label you tell the classifier that whatever it's printing it should be similar and this way the classifier gets a lot of structure and loss function and parameters that are being learned from unlabeled data a lot of constrains are being imposed and therefore it's going to be much more regularized than just training on the label data so this is a


very neat idea and it's also similar in spirit to things you've seen in subsidising which is the idea of taking two different views of an image and trying to make sure that they attract each other relative to another image so basically consistency constraints embedded into your encoder and raise different ideas in the past have attempted to do this and we are going to look at them in the PI Maru temporal ensemble and mean teacher finally we're going to look at regularization which is the idea of taking a model making sure


that it's generalizing well to a new unlabeled data set or new validation set so typically people use weight decay dropout data argumentation for making sure that the classifiers generalize well and those are also pretty important in semi-supervised learning and methods that identity use these are unsupervised it augmentation or UDA and mix-match which we look at in detail but there are also other papers that we can't really cover in the scope of the structure but you should check out other people's that are related


I mentioned this related work on these papers finally we look at the area of cool training or a self-training or pseudo Laban all these are ideas that have already been mentioned in this list of bullet points but there is a particular paper on his student which has taken these ideas to a whole new level in terms of performance and so we look at that in a little more detail so entropy minimization it's a very simple idea you have a lot of unlabeled data and you have your label here your training and classifier on the label


data but you want to make sure that the unlabeled data is also influencing the classifier in some way so one simple idea is you take your classifier and ask ask you to predict on the unlabeled data and you want to make sure that the classifier is pretty confident on the unlabeled data or rather it's entropy of the class probabilities it outputs on the unlabeled data is small enough and this way you ensure that the structure is this way you ensure that the classifier is understanding the structure the unlabeled data we're


trying to be confident about it so it's trying to find a solution for the label data in such a manner that it will be pretty confident on the unlabeled data as well so this this is one way to do something special so very old idea and pseudo label this is a very similar idea and being see how it's actually similar but the goal here is to take your classifier that's being trained on label data and ask you to predict on unlabeled data and you pick the most confident predictions and you turn them into extra label layer as if that were the ground


truth and you train a classifier or its own prediction so the classifier is making a bunch of predictions and those are being converted to ground truth proxy 100 data for itself and it's going to train again on new things new data sets created from the unlabeled data based on itself and this principle is also referred to as self training and there is a connection to end musician so here is the connection so consider an image X and classes y1 y2 y3 and let's say you're doing a classification problem and let's say


there is a classifier a with probabilities for the output classes being point one point eight point one and that's why at B with the probability is being point one point six and point three so that's why a clearly has lower entropy and you can say it's more confident it's more confident that the true ground truth is y2 and its score for y2 is much higher and scores for the other two classes are more similar and lower compared to Class B so there is clearly a connection to be made in terms of a classifier being more confident and


therefore having lower entropy in the output probability distribution of the classes and therefore minimizing the entropy of your classifier on unlabeled data I can do taking a classifiers outputs an unlabeled layer if they are confident enough you're trying to train on its own predictions so it has a similar effect and mathematically it's shown in these older papers that have been linked here so you can go and check it out and the next thing we're gonna see is later augmentation for label consistency so you take an image let's


say it's whether it's unlabeled or labeled you're given an image and now you've created it for an organisation's of it so similar this is the same picture we used in Sinclair and moco so I'm just using it so that you can relate this concept with the earlier lecture where we talked in cells for learning about data augmentation consistency using contrast losses so similar ideas have been used in semi-supervised learning as well so like I said you're already using the documentation for label data so it doesn't matter if you


enforce consistencies there but for unlabeled data if you take two different views and make sure that the logits are close enough for the classifier that's being trained on label dia that enforces a lot of structurally believer so you just make sure that the predictions are roughly similar and if you do this for a lot of legal data with a lot of different data occupations then your classifier is getting very much regularize in generalizable even those training on very little data so that's the idea of label label consistency constrains using


the augmentation and the more they documentation you use the better it is so that's it for the foundational material next we'd actually look at different semi-supervised learning algorithms like the PI model temporal ensemble in virtual adversarial training and so on but we also look at how the algorithms compare to each other and this particular paper from Google brain realistic evaluation decent spread learning algorithms compares these various different semi-square learning techniques on the C fart and svh and


data set which are reasonably small and you can run a lot of prototyping experiments with that so the four algorithms we're going to be looking at our PI model temporal ensemble main teacher and virtual adversary training so basically let's look at the PI model basically the idea is pretty much whatever we talked about our legal consistency you take your image you are creative different views using the stochastic a dog hunter but stochastic it could be at a random crop or a sequence of data augmentations whose sequence is randomized or but


there you apply it running grayscale so on and now you take it out you pass it to the model and the model itself could be stochastic it could have drop out so every forward pass could give you a different output even for the same image and you get two different latent variables for these different we'll turn the input down in the model so every time you make a forward pass you label data you can enforce it regularly the supervised crossing could be lost and if you throw unlabeled data you can enforce the square or square


difference between the outputs of the model or dutiful views and for label either you can force both of these losses well for unlabeled data you can you just enforce this label consistency loss which is you just take your output before the softmax or even after the softmax it depends on how you want to implement it but you take a particular layer at the end and you make sure that the layers is similar for two different views and you weigh both these losses together so one is gonna be unsupervised on same squares loss and the other is


supervised loss and you can actually control loose loss actually dominates the training the beginning at the end so for instance one reasonable motivation is you can make sure that the supervised lost dominates in the beginning so that the model already learns how to classify images and then you can ramp up to wait for these cells for the semi polarized or unsupervised loss of that it's learning to structure the unlimited on similar fashion so this idea is called pi model and this is the zero code for pi model excise


your training wise your labels WT is your rammed up wait for your hands provides function f theta faxes your neural net that's the classification task and it could have some drop are in that world to be stochastic G of X is your data argumentation is also stochastic and you basically perform two different augmentations of your mini-batch get two different output CI and Zi to D and you make sure that Zi and Zi absolutely are close to each other using a square error loss or some some distance metric and you also make


sure that the predictions of the classifier or matching the true ground truth whenever you have labels so that's that's really it that's as simple as I can get the PI model basically it's using the label the distance e principle so temporal augmentation there's something slightly different which is it says hey I don't want to do four passes of two different views all the time because it's expensive why not let me just keep a moving average of these sample embeddings for every single sample and make sure that the consistencies and


falls across time so I would still do a stochastic the augmentation stochastic Network I would get an embedding every time in the forward pass but I would say that those embeddings should be close to some historical a version of the same same same samples embedding the past and so that way gives to enforcing some kind of data augmentation constraints because you would have done a different organization in the past but you're going to keep an estimator of it for every single sample separately so this is very similar to those ideas we talked


about in again let chosen your improve gang where there was a constraint on the parameters to be closely extort conversions so this is that at a sample level and other than that it's pretty much the same as the PI model there's a cross between laws there's a roundup function for the unsupervised objective and we'll the objectives are only optimize together so this is the serial code for tempora ensemble is proposed in the same paper we're fine models pose so one negative thing about imported ensemble is it's


not going to scale with the size of the dataset you're not gonna be able to maintain a separate moving average embedding for every sample if you have if it is that is so big enough like a million or billion images so mean teacher basically amortize that and said that hey if you want to keep an exponential moving average for n bearings why not just keep an expansion moving average for parameters so that you used to take two different views but make sure that the embeddings match that of the moving average versions rather


than creating separate moving average embeddings for every sample so you take your model Tita there's also en male version of Tita and you make sure that the embeddings that you get for one view matches the in bearings very good for the other view but with different encoders basically so that's the idea of this main teacher approach where the teacher can be considered as the AMA version and a you think about it as a teacher because it's giving you these constraints and you also perform the classification task as


fication lost in peril sometimes they finally let's look at virtual art whistle training so in adversarial training you're create using this fast sine gradient method where you basically calculate the gradient of your input image basically the gradient of your output with respect your input image this is a huge dimensional vector or matrix depending on what your input is and you move your input in the direction where you basically get this gradient you get this sign and you move your input in a small epsilon in that


direction and that lets you fool your classifier and you want to make sure that the in your classifier is not full if you for these perturbations and so you would perform at with stable training to make sure that the classifier is not full at these data in these different rotor patient directions now in Sammy squirrel learning you don't have the labels for unlabeled data so how would you address your training there so the idea is to do virtual address zero training where you look at the distribution of your classes instead


of a particular class because you don't you you don't want to make take a particular class and see the perturbation direction for that class you take a distribution of classes and you take something a distance metric between your unperturbed data point and your perturb data point in trying to figure out the direction that maximizes this scale and it turns out if you linearized scale term you can actually solve for this direction are using power iteration and once you get the direction and you can make sure that you info


structure your classifier around unlabeled data we're trying to make sure that on these perturbations are unlabeled here the classifier is still not fooled even though you don't have access to a true label so that's why it's referred to as virtual data Co training it's not actually adversarial training but it shares principles with annual training done cleverly with some mathematical tricks and this is basically the pseudocode for the power iteration method which i mentioned because it works because you linearized


the k ho term so that's that's it for the techniques by model temporal on some going mean teacher and questionable training of that these are the four technique considered in this comparison paper and they make sure that they use the same architecture for all these techniques because prior work did not do that so they use a wide wrist not and the idea and white resonators your normal resin goes through a bottleneck and the wireless and doesn't do that it just uses three by three cons and does know one by one confer down sampling SAS wide


as possible so given all these constraints even all these similar architecture similar hyper parameters for various different semi-square learning algorithms it turns out that which will address your training performs the best if you look at C 410 which is four thousand labels so you see pretend originally it has 50,000 images so you basically just use 46,000 unlabeled data points and four thousand label data points which is like four hundred labels for class so that's really tiny compared to what the ocean is that is and virtual abyssal training


gets an error rate thirteen point eight six percent which is the best among all the other methods and which I was so training plus entropy minimization together it's even lower error rate of thirteen point one three percent and the trends is similar for the svh and I said where virtual artists are training plus entropy immunization outperforms all the other methods and the authors also say that the report much better piece like stand prior work like for instance in prior work the baseline supported had much higher error


rates than what the authors report so this paper actually took the effort to make sure that all the ablations are done carefully and one negative thing about semi-supervised learning on c4 is that if you use something like a preacher image it you take an image Tantalus pre-trained on image netlabels and then you find unit on c4 you actually get better numbers than using the unlabeled data on c4 itself even though see far the unlabeled data on c4 us coming from the same underlying distribution and image that is a


completely different distribution completely different image sizes so that's slightly bad so and there's a significant difference as at least one point one just one one plus percentage difference and even if you address the class overlaps and remove the classes overlapping and see far it's - you still get a lower error it and just using some experimenting on c4 and the author is also analyze each things like hey if your unlabeled data as soon see further Oerlikon ten classes and if you if you assume that the


unlegal leader doesn't have the uniform class distribution its label there and you can play around what the distribution of unlabeled here points are as far as a class overlap with Lolita is its it it is clear that as you as the distribution response increases which for a personal training is the most persistent compared to all the other approaches similarly if you vary the number of legal data points obviously the test error is going to be lower as the number of legal data points increases because that's slowly getting


to the supervised learning regime and all the meant is roughly performed similar but in the extreme scenario value are very few labels 215 up to 50 label data points and so on which will have a certain training significantly the best in this vhn and it is through the best on see for our notes other methods are competitive as well the lessons from this paper are when you use the standard when you compare different algorithms and semi squares I mean you should make sure that you use a standard architecture and equal training


budget which is you should spend equal amount of time tuning hyper parameters for all of them and if your unlabeled data is coming from a distribution that is a necessary overlap with your label data points then the benefits of something surprise learning will not be there thirdly most methods that likelike don't work well in a very very low data regime so this is not true right now but this was true when that paper was published and we look see how it changed over time and transferring 320 machine had produce better error rates but again


it's not true right now so this is an old paper but the main reason for going through that is to introduce all these different techniques like the PI Maru and virtually I was still training and temporal on something in the mean teacher so the agenda for the less rest of the lecture is to cover three very recent papers and some surprise learning that actually have taken some space learning to a whole new level unsupervised the augmentation makes match and noisy student so before that let's actually take a break


you you okay resuming so first let's look into unsupervised data augmentation for at for a consistency training in same as for a Fleming so this is a paper from Google brain from Cork lace group and these slides are from tong long was one of the authors in this paper so we've already seen how important data augmentation is and it's it's it's been significantly useful in supervised learning the high data regime but if you just do supervised learning if you don't have a lot of labels just data augmentation isn't gonna get you very


far even if you use extremely aggressively documentation like Auto augment which is shown here where you basically can rotate shear and add colors to some scene and create a lot of different views of the same image similarly in language you can create different versions of the same phrase or sentence using the technique called back translation so what it does is basically they cannot take and take a sentence in particular language translate it it to another language and then translate it back from the other language to the


existing set first source language so you go from a source language you go to a target language and you come back from the target language to the source language so you hope that this entropy in the decoder and the encoder will result in a in in a new version of the same sentence and examples are here so so the source sentence here is given the low budget and correctional limitations the movie is very good and if you look at it three different Mac translations since it was highly limited in terms of the budget and the production


restrictions the film was cheerful there are a few budget items with production limitations to make this film a really good one dude is a small dollar amount and recommendations Neos for them is very beautiful so the first and third versions are particularly really good well the second conveys a slightly different meaning but it's more or less there so this is giving a lot of diversity and based on which language you move to you're gonna get very different outputs and also the same language you're gonna get different


outputs for different decoding so the key idea and UDA or unsprayed augmentation is apply the state-of-the-art data augmentation techniques on unlabeled data for consistency training in semi-supervised learning you've already seen that PI model are like like Supai model was basically doing consistency training but it was a pretty old paper and data augmentation through these numeral architectures were not as devil back then so you can think of UTA is doing pi model right by using a lot of data augmentations on the right


architectures so here's a nice way to understand UDA so think about label data and you have X and your ground through the Y star and your training a classifier at P feet of Y given X and you have your standard supervised cross-entropy laws that make sure that the logit so the true class are really maximized and you also have unlabeled data so this is the situations and it's quite funny and models like virtual address zero training add noise to regularize to models predictions and the noises between the virtual address


retraining direction very calculated gradient in approximate fashion next you have this thing called unsupervised consistency loss which is you take the noise from the model and your original model and you make sure that the logic seriously similar unlabeled data so this is something you already know and a final loss is combination of supervised and unsupervised consistency loss and you can see that virtual adversity of training actually works pretty well so this is a green eyes illustration of which I do sell training


even though it's being presented in the UTA strikes so the uncolored data points are unlabeled data by the green and pink data points with a label here you only have roughly eight data points which are labeled but after performing virtual adversity of training after imposing the consistency between the model and the noise version of the model with a noise comes from that you can see how this labels propagated and covered up this federal so that's really cool so that's the goal of semi-square learning to do this really well in high dimensions when


you have a lot of data and a lot more parameters so you can think of UDA as creating this noise at the input level using various different data augmentations and depending on the two modality for instance you use Auto Alcuin for images you would use TF idea word replacement or back translation for NLP and based on that you create different organizations at the same image or the same sample and enforces consistency laws and the unsupervised consistency but the last part and you also do supervised cross and freelance


and you together out twice this so that's basically UDA and augmentations provides a diverse and valid perturbations for your input so like I said back translation produce three different versions that look very different from each other in converted roughly the same meaning as the original source sentence so in this case they actually went from English to French and back to English but you can also think of doing it to other languages and you can increase the diversity by playing around the temperature various different


sampling techniques like beam search on nucleus sampling and you know whenever you do use a software so you can always use a temperature there you're gonna get tablet samples if you use high not high enough temperature and if use a low enough temperature you're going to get the most confidence after product samples with less diversity but high quality so you can control for that similarly in images you can use argument but depending on the type of argumentation you can control the strength of the argumentation and get


sufficient distortions based on what one level of distortion eclair bar so let's look at the experiments carried out on this paper so in language they experimented with document classification or sentiment classification or review review sentiment analysis so if you look at the size of the data sets they have like twenty five thousand or four five sixty thousand and six fifty thousand samples and so on and the error rates before birth and after birth are mentioned different lines and you can see how Bert has significantly improved the error


rates so what they did in the UDA papers for various different initializations random bird base birth large and bird fine-tuned they have numbers for whether you use UDA or whether you don't use EDA but you notice how the number of label data points are changed by orders of magnitude three or four higher so if you look at IMDB you earlier had twenty five thousand label data points but the authors run his experiment trending labeled eight points which is three orders of magnitude lower similarly for yell quits 2.5 km this is


650 K and so on and you can see how the performance especially when you use word fine-tune is on par with the birth large that strain on all the label data points so the performance you get from taking birth large and fine-tuning on the fully supervised baseline is actually on par with what do you get from you da plus using bird fine-tune but just on like thousand x few labels which is incredible so it means that the consistency loss is using back translation is actually working really well secondly they have this idea called


training signal annealing which is you want to prevent overtraining on the label here so you have a label data so you don't want to make sure that you want to make sure that your classifier doesn't over train on it and for that they actually have something like a threshold a procedure where they take the classifier and if it's sufficiently confident they don't train the classifier points and this threshold is actually varied over time so you have an indicator variables of whether the classifiers output logic score is less


than threshold and only if it is you're gonna have the model trained on those data points or else you're just not gonna back up those gradients and this threshold has the you can play around with different schedules you could it could be really confident the beginning at the end you want to make sure that the thresholds are actually high enough so that model is not training and in the beginning you don't want to have high enough threshold it so because your mom could be erroneous so they play around with linear or exponential and log and


the paper finally this is a really cool plot that shows the dream of something which is the benefits hold even in the high rated regime even when you how to twenty five thousand label examples the performance that you get is from semi-supervised learning is better than supervised bird find here so it's actually able to take advantage of all the unlabeled I have so next let's look at the computer vision experiments carried out on the UDA paper so basically they use the standard benchmarks on C for Tanana speech and semi-supervised learning


which you saw in the prior work on realistic evaluation of same exploiting algorithms so a lot of these baselines are from that paper where there is a virus net with 28 letters and you can see how the parameters are controlled for all the different algorithms and the numbers are reported for C flower with 4000 labels innovation with thousand labels and UDA is actually the best algorithm in this setting with an error rate as low as 5% on c4 and 2.5% on s vhn and with architectural changes like shake shake and shake drop and permanet


they get all the way down to 2.7 percent which is significantly lower so just four thousand samples they're as good as ninety 90 97 either better than nice and accurate on C far which is kind of accuracies you usually get with using all the label data points so that really means to do domain consistency that the data augmentation label consistency is really really helping them and the models are the scaling was with larger networks so when you moved away from 1.5 million parameters it was typically used in semi-square is


learning to see far earlier to a model that's big enough they passed 26 million parameters the error rates are getting significantly lower so this shows that this technique scales with the number of parameters use next is how you can actually match the complete supervised baselines with just using order of magnitude fewer table data points so the complete supervised baseline uses 50,000 labels and this one uses 4,000 labels so that's 10x R or at like more than 10x fewer and and and if you look at the numbers you basically


get 5.4 percent error rate with supervised and 5.3 percent with UTA for the virus that's 28 architecture and for the other models even though UDA is slightly higher for shake-shake-shake drop it actually matters to supervise baselines though it's not as good as the Auto argument abortion it's still very very close finally they also ablated for how much data mutation matters and it seems to matter really it's really the biggest deal as we would expect because we've seen that in subsequent learning as well and that's what they observe even here


so the summary of UDA is that it's a data augmentation technique that's applied on unlabeled data to improve the performance of a classifier that's trained on very few labels their points and data augmentation is that for a critical part of this pipeline and it's an effective perturbation technique for us any surprise nothing so it's even it's it's even more effective than perturbing the model and Yui is significantly improves for both language and vision by 10x or 100x a thousand X fewer label data requirements and


combines very well to transfer learning like Bert and scales for the model parameters and model sizes so they also experimented with imagenet where they take unlabeled imagenet of 1 million examples and 1 million unlabeled data points they use 10% label data which is 100,000 labels or 100 labels per image and pure supervised gets 55% whereas their model you DEA gets 68.9 close to 69 percent accuracy which is at least 30% better and this shows the benefits are using more unlabeled data points so another thing that they tried


is having a data centers even larger than Internet which is the jft layer set from Google it's an internal data set it's on a scale of Google photos and they'd use 1.3 million images from JFK to just to see how much the domain is much works out [Music] Huli obtaining extra in domain unlegal data actually the auto domain on legal deer herd state performance and so you can see that it's not actually working as well so using more future days that actually works better for them and they also oblations on different


schedules for the thresholding and you find out the exponential scale it's better for language but linear scale works better for images that's that's something very empirical diversity constrains how did how do you control the diversity constraints for making sure that you have effective data augmentation they use a lot of hacks like minimizing the entropy and your decoding using the softmax temperature controlling and confidence based masking for your image image net case where they found the outer distribution dataset


unlabeled unlabeled I said like JT wrote the performance in that for they would take the most confident predictions from your label reader and you use that to filter the habibu data and tenday would find gains so domain relevance based data filtering is like a critical aspect of using own label data for improving performance of label data and semi-square learnings we already saw most of the mathematical or intuitive foundation system is for learning make the assumption that the unlabeled data is coming from the same data


distribution as to marginal corresponding to the joint distribution that you use for label data points but that's often not the case in practical scenarios because today said one label unlabeled data is usually something coming from some other data set or some other data source and you want to make sure you still transfer knowledge therefore that that's a suspect to assumption to make but in practice it can be there's a workaround using various kinds of filtering techniques that this paper proposes so next look at mix-match which is another very


interesting the similar paper similar spirits so the key idea in mismatches you take on label data you perform a lot of different augmentations and now you have you run all of these different augmentations to the same classifier and you get the predictions and you average two predictions across these augmentations and you end up with a bunch of class probabilities you can sharpen those class probabilities with soft max temperature controllers and once you get a sharpened distribution you you you you have an idea of what the


classifier would have guessed for the unlabeled data point and now what are you going to do is you're gonna take this and use it as part of your label Diaz for every label data updated new SEM is worth learning so it's that simple only thing is you you're making sure that your guess for unlabeled data comes from averaging or multiple augmentations and sharpening the distribution so there it's confident enough so it's called mix-match because it uses to mix up trick firstly for if you're not familiar with mix up the idea is you


take your input X and your output Y and let's say you're trading a model to predict Y from X you basically create convex combinations of two pairs of X for my one next to I do and create a new data file so for images it should be it would be something like for every pixel you take the weighted combination the pixel from the first image and the pixel from the second image and you which is average those two pixels and create a new image and similarly you would average the corresponding and ground route labels in Korea target ground


truth for your cross country with large its loss and this this technique is called mix-up and it's a documentation technique so mix-match basically the following it takes a bastion label data and imagine unlegal data and produces a new batch of process legal examples with this guessing technique and mixing up so let's look at the English part of this pseudocode apply data argumentation to X B so basically our badgerlink data points X X B PB and unlabeled data points you be so you apply the data argumentation to X P and


you apply Cayetano data augmentation to you B which is the only with a point and you compute the average predictions across all the augmentations of UB in practice they just use cake with a 2 but it can be really large if you want it to be and you apply temperature sharpening which is soft max temperature to make sure that the average predictions are picky enough and after the peaks are obtained you can take the our Max and get the corresponding classifier corresponding data point for proxy data point for the classifier so you Ottoman


the legal examples with these guest labels and using these guesses and the original labels you create an Augmented mini batch and you shuffle will combine these mini batch data points using the mix up trick and once you apply mix up to the labeled and unlabeled data you can just train a regular service learning model that has domain consistency losses and the supervisors cross will be lost treating this new batch as this mix match so mixed match is producing this processed label plus on people Taylor batch from


two different data batches that are coming independently so the last function for mixed match is going to be the regular cross country loss in a classifier there's some kind of consistency laws for unlabeled data points that you normally use in semi-supervised learning with some waiting constant and in practice it works really well earlier you saw in the realistic evaluation of learning algorithms that all these techniques were not really working well and this was concurrent work with UDA and you can clearly see


how its really into the performance and see for at an SV hm so here are the numbers you can see that it's not only working in the regime where you have 4,000 label examples but it works all the way well even when you have 250 labels mixed masses able to get all the way down to 10% error rate on CFR which is a which is very impressive so 250 images for class this means when two to three images of labels just means 25 images per class and you stood ability gets a classifier that gets around 90% accuracy so that's it for mixed match an


expansion UD ever like conquering work and I'll raise similar ideas different kind of implementations and UDA is probably more broader in terms of applications to NLP as well make sponsors well analyze internally ablated foresee for RNs future so let's look at the final paper in this agenda sub training with my student this is the largest scale since Freud learning experiment conducted and machine learning so far these are also sliced from tango Wong was one of the authors in this paper so as I said there's the


semi-square learning had a background just the dream is to make sure that unlabeled data improves the performance of supervised learning or label data even when you have a lot of labels and that's what this paper tries to achieve so you you already remember how unfiltered aft was not able to give sufficient gain on imagenet for the UDA paper and they actually use clever filtering techniques noisy student is actually a larger scale version of that so the way it works is as follows you train a teacher model with long label


data trying to really really good classifier and asset classifier to predict an unlabeled here and you infer the cereal labels on illegal data and you train a student model with the combined data of the teacher teacher of the original data label data that you use for the teacher as well as the guest labels on our label data and you add noise to this process through data augmentation dropout and stochastic depth which is another version of skip connections with stochastic and create a lot you know noisy student that's why


it's called noisy student here you're gonna have a lot of data augmentation to this process and once you do that your student model is pretty good it's it's highly regular I stand also trained on a lot more data and therefore it can over train on anything but still it rains on a lot of different proxy proxy later you generated from label data and which has already been filtered because it's only taking the confident predictions so now you can treat that student as a new teacher and repeat this process multiple times so


you take the new student as a teacher if you you basically train it on label data you infer the pseudo you infer the zero labels unlabeled data again with this new teacher and you create a new student and so on and repeat this process like multiple times and you get a really good for Adam so here are the experiment settings the architecture that you use is efficient at and the model noises they use is dropout of stochastic tap the input nicely uses Randolph which is a version of auto valve it's more efficient and for zero labels they use


the soft seal enables continuous values they don't actually use the one harden coatings and the label data said they uses a ridge net which is 1.3 million images and the unlabeled data said they use is jft which is 300 million images and they basically do iterative training where they take the biggest efficient at model possible and actually make it wider for the next next scenarios so the original teacher could be b7 which is the widest and deepest efficient net that exists and the student model the trains next could be another bigger


version of that model which is the L tomorrow the Dakar so in terms of results they actually had the state-of-the-art numbers for an image not already here two eighty eight point four percent top one accuracy which is significantly better than any other model and the previous best was eighty six point four percent which is actually trained on 3.5 billion labeled images from Instagram so with just one point three million labeled images and 300 million unlabeled images you're actually able to surpass those numbers


by two percentages significant especially in those those regimes and they do that with one order of magnitude fewer labels and they actually do that with twice few as that smaller in terms of number of parameters because they use efficient mass which are much more efficient in terms of parameters and flops and rest net or rest next so the improvements are also exists without iterative training so it's not that they actually need it right of training so even without iterative training they get significant improvement which is one


iteration they can get minimum of one percent improvement for all the different model regimes b0 b2 b5 b7 improvement of 1% is pretty standard and which is pretty pretty nice because this means that the filtering mechanism actually works for finally they also show really good robustness results on a mission net because of training on a lot of different data augmentations and model noises and water unlabeled data in addition to label data you would expect the resulting classifier to actually be good on a new server robustness


benchmarks which happens to be the case so they actually beat the state of their army robustness benchmarks on each net a which is a corrupted version of image net where usual classifiers fail over there and their model actually gets 83 point once and 83 point seven and top one which is unprecedented and they also do really well on emission at C and P where they get very very competitive very very very good numbers on the top one accuracy over their 77227 eighty-six percent which is significantly better than the


models that did not use this noisy student process so here's like examples where the model that they trained on these harder versions of imagenet ended up making the right predictions which is in black well the baseline models ended up making the wrong predictions so the baseline models are focusing on aspects that we use the car up and there's actually your amazing car or like for instance they're actually looking at so the models able to capture the basketball in the photo on the bottom room where there's a man holding in


basketball whereas the baseline models are not able to do that so so that shows this models very fine-grain in terms of its recognition abilities and here are a bigger example is this a dragonfly at the right but but the baseline models of food and thinking it's a bullfrog and similarly parking meter was his vacuum swing was his mosquito net so this mod is actually very very good at details and they also have ablations for how much the noise matters in the in this process and it seems to matter significantly enough when he used all


the different kinds of noises like grappa stochastic depth and data augmentation you get the best possible numbers so in summary we looked at semi-square is learning and it's a practically important problem in the industry for two different scenarios one is when you have a lot of label data and a lot more on label data like for instance image ladder and j ft and you're trying to improve the performance of image net the other is when you have very little label data and you have plenty of unequal data which is usually


the case in medicine or finance or something like that so the promise of semi-supervised learning is always existed for the second second scenario but there have been very good results in the the last few months or like last year or so in both these scenarios which is the noisiest student model really helping in the scenario where you have a lot of legal data but you have a lot more and label data but then unsupervised data augmentation or mix-match is really very good at the low Reiter regime where you have unlabeled data but very little


label data and you're able to do really well so this means that when you have the scenario are you using unlabeled data to improve the performance of supervised learning systems self supervised learning is not necessarily the only option semi-supervised learning is asked lucrative or probably even better because its ability to improve the performance even in the high data regimes and make it possible for building emotional classifiers that have an unprecedented top on accuracies like noisy student that's it for the lecture


thank you very much all right so let's get started welcome back to the lecture nine I'm actually not sure Oh keep unsupervised learning something lecture something and so we will have a two part lectures today the first part we would look at something called unsupervised distribution alignment it also goes by a lot of other names and then the second part would be a guest lecture by Professor Alyosha to talk about some of I guess the works from his lab so any logistics questions before we dive into your lecture mouse tone can we


have late days for mouse tones all right there you go which no one is working on all right so so let's get started so in this lecture we will look at unsupervised distribution alignment so what does that even mean so let's remove the part unsupervised let's just first look at a distribution alignment problem so a lot of problems in image to image translation take this form so let's say I want to go from semantic mask to RGB images then this is a distribution alignment problem because we can think of us having a distribution over mask


here and then we also have a distribution of like just regular images here and then they would like co-occur with certain join probability distribution right it's mostly for one image there is only one correct semantic mask but for one mask there could be many corresponding images and the goal would be like how can you align these two in such a way that when I give you an image on the right you can generate the mask or the other way if you want to generate more training data and then the more image problems that takes this form


let's say I want you know an image what does it look like in the day time and what does it look like in night time and then again we can think of it as having a distribution of images of day time and then also a distribution of images in night time and then you want to align them in certain way so you say why is this helpful like one way that this could be helpful is say if we want to train at home as vehicles to drive safely at night but then it's harder to collect data at night so is there a way for us to collect corresponding images during


daytime and then find a way to find their nighttime counterpart that would be useful if we can do this and then there's somehow a lot of other problems also fall under this formulation like black and white images to color images and for basically everything that we have seen they are relatively tractable because like I can totally just take a color image and then convert it into black and white and that gives me a lot of pairs that I can train on and similarly to the semantic mask and Street View RGB images as well as


daytime and nighttime for a lot of those you can actually find natural pairs so these are some of the problems distribution alignment problems in image space and this kind of distribution alignment problems also happen in the attacks analog of that and most straightforward example is really just machine translation how do you translate a sentence or a paragraph from one language to another that again you can think of as a distribution alignment problem you can think of there was a distribution of English text and then


there's a distribution of Chinese text and then the question here is how do you align these two things together so when this kind of distribution alignment problem when is supervised then they are relatively easy so when in the in the case of form an image goes to semantic mask it's basically just a Mantic segmentation problem when it's like other image to image translation there's this text effects work that's done here at Berkeley and then for text to text domain alignment when you have the supervised pairs it's just machine


translation and when you want to go from image to text again when you have supervised pairs they are just like captioning tasks and a lot of things so in the end of it it really just borrowed down to feeding a certain conditional distribution like your given image B what the correct mask a and the you have this luxuries when you have kind of a and B pairs that co-occur in the wheel world either through your annotation effort or by taking an image at a time and it take the same image at nighttime as long as you can gather this kind of pairs is


somewhat trivial at least from a formulation perspective it's really just fitting a conditional distribution and we have talked about all sorts of ways to feed distributions in this class can auto regressive model or whatever and but the question becomes interesting if this kind of pair of data what if they are expensive to obtain or they just don't exist then we are basically going out of the range of this supervised distribution alignment problem like you have one distribution you have another but you don't have any pair of data then


like can you still do this or even do it at all so I'm taking some examples from a paper called cycle game like what if you want to turn the painting into a photograph or turn a photograph into a painting the second one might be more tractable because like you could possibly say I take a picture and then I hire someone to paint it for me but if I want to do it in a very specific style by a specific artist and you really couldn't do that so in a sense the natural pairs don't even exist in the real world similarly like if you want


you for whatever reason if we want to turn a zebra into a horse or turn a horse into a zebra then it would be very difficult to force a zebra in a horse to take up exactly the same pose and take a picture of them having the exact correspondence so these are the kind of pair of data that would not exist in the real world and there are a lot of other applications so let's think back to machine translations so if we want to if I want to translate between Chinese and English or English and Germany that's relatively easy because there are


large demand of those language uses and it makes it economical to annotate a lot of data of basically supervised language sentences pairs but then like it's not economical to do it for the other probably hundreds of languages that exist in the world it just doesn't make sense to annotate that much data and and if we can make this kind of distribution alignment to work without any supervision then it could be used as a lot more it can be used as a way to augment label examples in a kind of semi-supervised way we can be also used


to do style transfer like some of the things that we have seen that basically had no ground truth in the real world yes yeah well it's just okay so if I have a good translation model between two languages then the value of that model is kind of proportional to the usage that you can get from it let's say to train any pair of languages you need the same amount of investment that's called fifty million dollars probably on the lower side then like if I throw in this fifty million dollars for between Chinese and English


you probably get a ton of usage and you get ads or like what other revenue but then I have like English - like whatever language that probably only a hundred thousand people speak then you get drastically less usage of the model that means for the same investment you get a lot less out of it so it's not that they are more expensive to label it's just like that value it doesn't make it justified so okay so let's look at this problem again so it would be of course it would be a nice thing to be able to achieve to give me two distributions and


then find a way to align them but this is even a feasible problem right so if we look at the problem statement we are basically told to do two things one thing is we have distributions a we have two random variables a and B two distributions and then we get access to the samples from them like we get a bunch of samples in one domain we also get a bunch of samples from the other domain that's all great but what we crucially don't have is we don't have any samples from the pairs yet we need to estimate how they are related to each


other so this problem form a high level seems pretty hopeless because you have really given too little information to to tacko it so what we would look at next kind of certain so basically the crucial problem now is like where do we even get any training data like if I don't have any supervised pairs like what do I even train the model on so the the way that people have been doing this is they try to rely on certain invariants that are true for kind of any pairs of distributions and then somehow you could get some meaningful learning


signal out of it so the first kind of invariance that we can rely on is something called marginal matching so there's some math here but then the brief idea is really like if I want to translate one distribution from one distribution to another after the translations the distribution should still look like each other and more precisely what it means is that there was some fundamentally unknown coupling there's some fundamentally unknown relationship between these two random variables a and B that we are trying to


approximate but we don't have access to them so let's call that our approximation Q so given B like what what a is most likely well this is the other this should be so basically we were trying to learn two mappings or two conditional distributions and when you specify this kind of conditional distribution distribution Q of a given B you implicitly induce a marginal distribution so when I specify Q of B given a I implicitly specifying a marginal distribution on B and the way that you compute it is if I sample a for


my one true distribution of a and then you essentially average out this conditional distribution that would be the marginal distribution of p and ideally I warn my Q to be close to the ground truth conditional distribution P of P given a and that means if I sample a lot of a and then I map it through my conditional distribution the outcome of that should map back to the original ground true of distribution and similarly I can do that for a so I sample from B and then I would from from this B samples I would calculate my


approximate conditional distribution and after those transformations they should be the same as original marginal distribution as a and oftentimes in literature this conditional distribution is just a deterministic mapping so I would say give me any sample from a I would map it to its corresponding sample in domain B so far so good so the question is basically we have stated the question to be in the end we are trying to match this marginal of be approximate B to the ground truth B which we have access to but in nopon you are saying


the new net Q needs to look at a essentially so that is a true statement so if my Q if my Q is so powerful that it could just represent the whole marginal distribution of B so let's say let's call if we have Q of P given a that is equal to Q of B for all a and B pair then your statement would be true basically you can just like approximate the marginal distribution without even doing any meaningful work so that's why like in practice people would have a fairly restrictive mapping so like that's why like in most of the works


that we would look at like it usually takes the form of a deterministic mapping so when when Q of B given a is deterministic then like you don't you don't get to represent the whole marginal distribution unless P of P itself is only a pawn mass but that is a correct observation but like you said like I suggested like I mean this is a very weak learning signal like if you don't correct you if you don't construct your model in the right way like you you could extract nothing from it so let's see some examples of like how like how


does this how does this in an ideal case work at all right so let's say I have two distribution a and B and they are very simple categorical distributions that only have three possible values a 1 a 2 a 3 I'm just going to draw some frequency here right so and then I'm going to do the same thing for B so this is probability mass function all right so so based on let's say we have a deterministic mapping that means like each of the a1 has each of the a has to map to some B and each of the B has to map to some a then like based on


the marginal meshing like this can seemingly gives us a way to recover the crunch of Correspondence let's say the ground shook of respondents is between a 2 and B 1 and a 3 and B 2 and a 1 and B 3 right so I would argue this would be the only correspondence that satisfied the marginal matching constraint well it could be not a by direction but then like it might not fulfill the properties until unless some of the unless some of the values have like no probability mass then like you can do whatever right so argue this is the only way that you


could make the marginal meshing look the reason is that because each value has a distinct frequency here so if you match it in the wrong way the marginal distribution of the induced mapping would no longer be measuring the original one but there are still a lot of ambiguity right so if we imagine a distributions that's like the most kind of the most difficult one let's say I have a uniform distributions over two random variables then this is kind of hopeless because all kind of mapping could work so let's first look at the a


to be mapping so a one can map to B 1 a 2 map to beat you but then it's also possible that a 1 can be mapped to beat you a to map to b1 and will still be fine form a marginal matching perspective but then the problem is so well then that means this thing is ambiguous and this is not just hopes sorry mainly joyed the other way so there are two set of mappings that we are we need to learn here one is G a B which is mapping from A to B and then another set of things is another set of things that we need to learn is GPA and


basically you would need to learn the product of these two different possibilities B 1 2 A 1 B 2 2 a 2 B so how many like totally possible solutions are there to this problem if we just use marginal matching yeah so basically each direction there are two possibilities and then if you multiply them together there are four possible solutions in this problem and they are basically totally ambiguous so this is one of the thing that we are seeing here is you can have your objective function that induce a really large solution set really in


this case is almost all of the solution set and then people realized this can potentially be a problem so they introduced another technique to try to at least restrict the solution set a little bit so this thing is oftentimes referred to as cycle consistency but it has also taken a lot of other names in literature called dual in learning back translations and really the core idea is that if I if I take my so basically the whole idea is that my apartment mapping should be similar to the ground truth mapping and if those mappings are


deterministic then what that means is that if I step through my mapping I should get back my original sample so if we think about the case of P of a be given a given of a would be this would map aid to his correspondence in B and then if you apply that again from the other direction B to a mapping you should get back a and this should hold you in both directions so this gives you another invariance so if you say that the relationship between these two distributions are indeed deterministic then I would say these


things should hold you for all possible pairs and if we step back to you the example that we just look at so if we impose psycho consistent see what would be the number of possible solutions now why is that no longer four yeah so like we can see that in this case the original total solution set is four but after you impose this cycle consistency constraint you can reduce the solution set to exist like some of the some of the mapping are no longer valid so let's say if I pick this and then take this this would be no longer


valid because an a.1 would get translated into B 1 and then P 1 according to this would get translated into be a 2 so this is G a B this is G B a and this is this no longer satisfy the cycle consistency constraint so that means like I can use this constraint to reduce the possible solution set in my in my search but still like we can see that it's still fundamentally under defined like we are still left with two possible mappings and we are not sure which one is correct but it at least exponentially shrink the space that is


possible so so far we have seen the two core in variances that people have used and these are invariants that's true for all alignment problems and then we can use them as learning signals and again like obviously like we just look at like even in a extremely low dimension basically categorical examples like it's still not going to work so there are there are definitely problems that this cannot solve but then in practice like people can find problems that this kind of search is amenable to you and then you can oftentimes ensure that there's


no biases in your system by selecting the right architectures loss functions and etc and you can actually get to a certain level of success with this yeah so they expect this this one basically says for an arbitrary data porn in its it's just a generalized version of the cycle consistency thing so for any of the data porn a if I draw samples for my approximate conditional distribution and then I translate that B back to a using my approximate once the the distribution that induce should be similar to if you do it with the real-world one and what


we're saying here is just that like when in the case of both of when we say we know the P and the Q are both deterministic then it reduces to the deterministic mapping example but it could exist in a more general form so probably the best-known example that uses those learning signals are cycle again so psycho gains laws essentially consists of two parts one part is this marginal matching so meaning after I translate my data from one domain to another the marginal of them still match with each other and you can kind of see


it here so essentially my generator no longer takes a mapping from Z to X instead this thing is trying to translate from X to Y and I want to do it in such a way that it looks like my target image so this is just a standard gain training loss where you're trying to say my mapping from X to Y it should look like just like Y so that's fairly straightforward and so but it's actually instead of looking at frequency you use again to help you do the marginal matching and the second dimension of this is you can achieve cycle


consistency by an l1 loss essentially what there's if we unpack this up this objective function is your sample for data in one of your domain probably your source domain and then you map it to you map it to a cycle then it should look like itself in an l1 sense so this is I think what they call forward cycle consistency because it's going from X to Y to X and then they also have a backward one where you essentially kind of think of a sample from oil labels and then you map it through this thing again Y dou X dou Y


should be similar to yourself in an l1 sense so that's the lost function and then you essentially would combine these two things together and then train it I think in practice they use at least I think the in practice they use at least square again instead of the original gain objective but probably it doesn't make too much difference so they reported a couple numerical results and the first results that they look at is so in the case of going from photo to semantic mask you can actually calculate the accuracy so we can get a


quantifiable notion of how well the method is doing so the method that we just introduced is called cycle again and it's basically unsupervised so you give it a bunch of distressing images and then a bunch of semantic masks and then you hope them that they somehow align each other and what this shows that they actually do pretty well so a peg to peg is a fully supervised model so the last rode means you get you actually get pairs of image and their corresponding label so the last rows should be read as basically an upper


bound on the performance and then the cycle again by using no labels at all you can actually do pretty fully so like you can roughly say that 60% of the pixels are labeled correctly with the right with the right class we actually know even with no information on how they should be related to each other know so there's there's no pairs so you you train you train the whole system with just a bunch of unordered images and then a bunch of an order masks and then it's learning to align them good good question I don't think


they have that level of analysis but I think that would be interesting to see like what are the kind of things that are easier for you to align what are the things that are not like a mushrik yeah yeah yeah yeah so the so the question is there was a lot of inductive bias going from one image to another using a conf net and then also using a certain discriminator that operates on a patch basis so you kind of like do a kind of domain alignment patch like so you can think of it as having its there's really a lot of training signal that is not


captured by the loss function at all so unfortunately we don't know the answer to that so I guess what what I know for sure is like if you just scramble the image like like I mean just permute the dimensions in your image tenser then I'm pretty sure you would do full but then like is does that mean like this is no useful probably not but this still means that we don't fully understand what are the inductive biases they're helping us but that's a good question yes like right so I guess so so the common is around like the a lot of these


translation problems like operate in a very local manner like you're kind of like saying I just need to change my local pixels like when you go from zero to you to two holes like you're just kind of changing a local texture as opposed to something that is global which is presumably much harder I think that's likely the case I have no idea you cannot you can ask Alyosha who will be here soon so I I think this is a long way from supervised learning so I believe supervised learning like this thing should like I don't know but


I I imagine this to be at least 95 percent plus but again like this is not a correct comparison I think the correct comparison is to compare cycle again with takes two pics because they use similar architecture except one is supervised the other is unsupervised all right so they have some Appalachians in terms of loss function or don't know in terms of architecture the Appalachians basically tell you what we sort of expect that like for one like if you can alone if you just you can along this means you just do marginal matching


so which is actually not bad already and then you can see that if you add psycho consistency in there it helps you and there's something that's really puzzling like I'm really confused by what is happening here it just kills everything I like I have no idea what is happening there so I don't know and and what's also interesting is that it doesn't always help so what we're looking at we're looking at going from photos to labels and then they have another experiment that is going from labels to photos so this is a much higher entropy


mapping whereas they still use just deterministic mapping so you can imagine some there might be something that is playing with that in here that I guess we don't we don't fully understand what's interesting here though is the evaluation map metric is pretty interesting so remember here we are evaluating it from label to photos so basically is give you a semantic mask how well you can generate the scene but then how do you even evaluate that so they actually have a pretty clever way of evaluating that so what they would do


is they would run another pre-trained semantic segmentation now walk fully convolutional network they will run it on the generated image and then they use step to quantify the results so that is a pretty interesting trick to evaluate this mapping kind of like the inception school except like in this case like we you kind of yeah it's kind of like Inception school but I think it's better than inception in school in this restricted domain these are some of the other codes the first cases where you translate from I guess a schematic


annotation of a facade facade going from address to Shu going from shoes to address most of them make sense they applied this to a wide variety of different problems where you it's just impossible to get label pair it's just like I guess somewhere Yosemite and winter with somebody you can get pears although not exactly the same and translating apples to oranges like just like we said like this is it's it's not like this is not supervised and it's not fully unique they have their set of failures and in this one what the


authors explain the paper is they're like I mean when you train on how movies are success I don't know when you train on horses in imagenet they decide like they're like they would they have not seen a human riding on it and as such like you would just classify or similar texture to be horses and then you just translate that so like this this I guess goes back to one of one of the question that someone mentioned it's like what are the failure cases so like I think this is one good example of like what it fails on I think this is a good example


of what the model is doing is it's trying to find like yellowish pattern and then change that yellowish pattern to stripes so that's apparently what the model is doing so that's that for cycle gain so essentially it's pretty surprising that it can work on certain domains and when it works I think it's very reasonable so the next thing that we would look at is we look at improving cycle gain in certain dimensions so the crucial dimension that we would look at here is that remember when we talked about the cycle again the cycle


again has this deterministic mapping so you give you an image X which translated into domain Y and but this translation is deterministic but that is fundamentally not correct at least for a lot of the alignment problems that we care about so let's say if I want you go from mask to image semantic mask to image like there was a lot of different ways to satisfy the same semantic mask there are a lot of different ways to generate that image like semantic mask only tells you there's a car here but what does the car look like what's


inside the car what color it is like it's simply specifying none of those so basically there's this high entropy mapping going from semantic mask to image and that is apparently not deterministic so you can say oh one straightforward way to extend cycle again is to say cycle gain is essentially this right so you take in an image a and then you're trying to map it to image B so one straightforward way to extend that would be to make this mapping taking an additional noise sauce just like in it typical again so you


could take in an image a and then you can also take in a noise sauce that probably described like what does the car look like what is the color other than the contour everything other than the contour and then from that noise source you can map to some image P and then if you sample different Z's hopefully you get different cars is that some motivation make sense so that's all good so and in fact like this has been done concurrent you cycle again there's another paper called do you again where this is essentially the architecture the


mapping would take in both an image from source domain as well as a random noise source however this is not enough to just change your architecture because if you changed if you even if you change your architecture the noise are doomed to be ignored and to see that the reason is essentially our lost function the l1 law the l1 cycle consistence loss carry to do the following so if my map my a with certain Z this this produce like some kind of B for me and then if I met my be with another Z Prime I should get back to a and we can see


that in this whole mapping the choice of z and z prime are essentially ignored so you can choose different Z or Z Prime you still need to satisfy this mapping so what that means is that the noise source is necessarily ignored when you impose a cycle consistency loss and when you optimize it to is fixed port so that's not good and then there was this augment excite Oakham paper that proposed a way to solve it so you would augment the noise to your architecture but you will also learn so instead of only learning the mapping from A to B


and B to a you will learn an encoder of each of the noise source not the similar to how an encoder is used in a variational method and it's actually pretty interesting so the way that you would go is I have some ground Shu image a and I have then what I'm going to say is that my ground truth image a would comes from a corresponding B and it's corresponding noise sauce za and then I would have this blue arrow which is a network that infer what's the a it is so basically I'm trying to infer what Zee produce my a and I'm going to


infer what P produce my a so instead of only inferring what is my corresponding B I infer that as well as what is the noise source that produced me now with both the noise source and the corresponding B I can use that to map it to an a prime using this color which is the mapping coming back from B to a and in the end I can say that a and a prime should be similar in l1 in l1 lost sense so now it's okay because I'm choosing a specific Z for each particular data point so if we think about it from an information theoretic sense whatever


information that is not captured in B you push it in to Z that allows you to perfectly reconstruct the original image as well as maintaining the ability to have diversity of mappings because a different egg come from different see yes so the question is how do you prevent the model from putting everything into Z so you could but in my fail the marginal matching criteria right so I guess the statement is like the Amb relationship could could become decoupled right like from a I would just match an arbitrary B that actually has


no corresponding with a from the beginning but then like remember that is always the problem like even with the original cycle again you could still produce an arbitrary mapping that this consistent but it's not the ground truth mapping so this I guess what I'm saying is this doesn't make it worse yeah so we can say it again so I was saying like basically you can play this multiple steps and then like the evolution of them should still match the original marginal distribution yeah so many ways that you can play with this