made a new holdout set let me explain how so if here is progress on SIF r10 over time so if you just use raw pixels and do linear classification you get thirty seven percent accuracy I just realized we've switched from error to accuracy hopefully it will be clear from context of the same accuracy now we have 97.1% in 2017 and 2019 what is it lou big-nose these numbers


ninety-nine point what nine point zero so we got you know there's still time to write more iclear papers everybody we've got ten more ten more ticks so this is like our deep revolution here all right this is the deep revolution happened in 2012 and you know we just keep making progress make them bigger get them more capacity make these models really large the shake-shake models are you.why

Training on the Test Set and Other Heresies


rednecks also or just ones right I try to get them to fit brushes is this overfitting right is this overfitting right because this early part we can match with shallow Methos yeah actually you can get even get to like eighty I can't remember the number but there's there some work by chumki cada aleck agrawal greg valiant and leigh song like white 13 so if you just do random


features just you can get to about 85% accuracy using just dumb random features shallow stuff can get it was big so it's also a large shallow model can get to about 85 and so the question is are all we're doing here is just overfitting to the test set by graduate student design so we're gonna check and we're gonna check by building a new test set now what does that mean so it turns out that


the syph are ten creation process is super well documented and it was documented by the folks at Toronto who made it in the first place and so there there is a lot of details about how they did it and in particular where they got their images to begin with and that comes from one of my favorite data sets ever it's called the 80 million tiny images the tiny images


data set and it's a was curated by Antonio - Rob Rob Ferguson bill Freeman and they made beautiful like scatter plots like all the images on the internet and these like mosaics and just seeing like how everything varies it's very cool kind of looking at what kind of things are out there that you can get to get off the internet in 2008 so the reason why there were thumbnails is


because they wanted to make something that you could store and then do all sorts of visualization and studies with so from these thumbnails they subsample 60,000 using a process that was very well detailed supply well detailed with human not experts but human laborers and they tried to get down to these ten classes so can we get iid resampling well there are you know there's out of


80 million images we only took 60,000 out so the hope is maybe we could sample some more maybe not even that many more how many do we need to actually get something that would believe so I think Ludvig did his error bar calculation that said mm that's what he wanted 2,000 new ones and so and work that I did not want to get involved with Ludvig Becca and VY salt labeled I say mostly just


Becca right this one for this one was looting in Becca labeled tens of thousands of images in in the tiny images data set and we got a new test set of size 2000 and again there find details about like what counts as a boat what counts to the car and this kind of thing we drive to match as closely as possible so what did we see okay so the first thing we see is


we take vgg 16 everyone's favorite network all right I guess anyway this is a big one and what we saw is a huge drop in accuracy right there's an 8 percent drop in accuracy from the first test set to the second test set which is much bigger than you would expect by a 1 over root N and some some reasonable capacity that's big mean this is we put a little confidence interval about where we


should be this dashed line is the reproducibility you'd hope that the confidence interval would hit the dash line that would mean okay there's you know these we'd say hey we're not having been overfitting at all well that's great that's great so it's clear what's gonna happen cuz right 85.3 was right in the ballpark of what people had seen you could do with just showering clearly


what's gonna happen let me just I don't know if anybody's read this paper yeah but clearly what I thought would happen what I put my money on was that yeah now the shallow learning stuff will just be here and we'll just see that we'll just get a saturation so we've just been adapting and we'll just see a saturation up here and everybody will be equal and right and that was like this big drop


and so here here here's our random features that's not what we saw at all not what we saw at all I lost money I didn't lose money because all everybody else got the same it's good there was no house in this case right bigger drop 12% drop so that went from eighty five point six to seventy three point one bigger job present and the shake-shake model only had a 4% drop


this one that had super-high Agra so we saw exactly the opposite of what our hypothesis said right our hypothesis suggested that maybe the big models were able to just fluctuate themselves to overfit to there's one particular holdout set and if we draw a new holdout set we'll see some sign that they had adapted to it but we saw the opposite in fact the ones that have the better test


error on the original test set have a better accuracy on the new testament oh did you have the original weights for these models or did you have to retrain them we did well there was that we did retraining in the paper but in this case this was just original weights this is some summary training but again this is why github as an experimental resource is so powerful people put post the


weights in the repos saves you tons of time so it actually is the contrary it's very you don't even need a GPU right thank you you can just download stuff that people already done change a parameter here do some kind experiments you already picked out like got a big problem here right so there there was clearly this is not an iid resampling it's not ID resampling because that can


Ludvig or not we don't know who the toronto folks were probably some paid undergraduates and they're not that kind of like who have never been to Toronto right yes so there is a there is a yeah so it's not perfectly ID right this is a great question but we'll come back to it let's come back on the empirical training possibly yeah but no you can't tell a difference if


you train a classifier do you sample test what white features oh just individual features to see if you see significant things we didn't do we did not do any of this yeah but let me keep going well then they're not from the same distribution right Marshall over it yeah we would hope so right I mean I that that you could probably check because there were some sub selected to


begin with right so if too many were sub-select 'add then they're not going to be iid and that's something I think we could probably the dead part didn't vote human so that part did not involve humans but not he's asking is just that probably the only thing that changed probably again being generous to us the only thing that's changing here is the labeling function


not that of the human the thing is producing the why not the thing that's producing the X actually sight under why you only cycle you just want to know if you could do read you interleave the two test sets if you or sorry if you can interleave some of the stuff into the training data so we're gonna be then the entire and actually the fact that better models have a smaller drop suggests that maybe


this boils down to the test and it could be I mean what well so sorry what so what were you what are you suggesting can we just quit crossing it so this is actually really important this is really important the fact that you've already moved on from the fact that there's no adaptive overfitting is shocking look there's no adaptive overfitting we took this is rule one don't look at


the holdout set more than once that's rule one we look at the holdout set fifty thousand trillion times at Google every day fifty thousand trillion times and yet it doesn't matter the better you do on this holdout set this one stupid fixed holdout set like it doesn't matter that I think really before we talking to about why the drop happens just the fact that we don't see adaptive overfitting


isn't it blew my mind I did not think that would happen I did not think that was or whether you see the single noun to productive or fitting into some others and so those are two different things which so far from your results in love wait what does that mean might be your video yeah I cut her with Zico there's no adaptive or control and this gap is only because of the difference in the


distribution yeah the thing is to be fair gonna be hard to get it's gonna be well I think so but right listen one second what is that I think it's gonna be hard to tell from safar 10 and I think what's interesting is if we go to this other data set that's more interesting anyway I mean maybe we can't a little bit but I think it's much more competitive yeah yeah I'm getting there


maybe I might be gay there we'll see how a lot of people want to stay through lunch yeah tell you it's not his hand up for a second here go ahead what it's gonna have a quick propose also epub Bank I knows we can have like 50 km yeah all these new data set and then test on all superpower tested and I don't know if we do get at leats I I don't think we get too concerned about mm emerges and


some of 50 was a basically exposed so if you wanted to get another 50,000 mm just would actually be different I think we could probably do more in 2000 but I'm not sure what the limit is yeah yeah that's actually in the paper you can see at the end but but also also the thing is I only have two Becca Becca and living now they have limited cycles you know this is a problem it's a


problem it's a problem and certainly certain and I lost both of them that's really sad actually that's really sad yet both well I know loot big sting sorry good okay that was okay no first thing we have to do though we did these Becca she's graduated this year okay also who cares about as if our time so far ten is this fun little thing that we all kind of like this we can train it


and it certainly it's interesting because it's like the first non-trivial thing we can get to but you know what captured people's imagination was this image net data set questions do we do that one now that's harder because that's bigger there's more labels there's more and even like even here like this 1.2 million training images were not labeled in the lab they were labeled using Mechanical


Turk that's opportunity right so now we could try to reproduce this data set and this procedure using Mechanical Turk which makes it much more scalable and we can make much larger who can do much larger experiments so how do we get the xiety resampling of image that in the water but it's not as surprising as you say given the fact that we can we can drive particular models to zero laws


something which we thought was impossible and that's okay this is like the community drive in there it's like community doing or repeating on the test and it's kind of a same effect no I don't think so like why do it why don't we see a plateau anyway that's right that's right that's why anyway let's this is more interesting Misha the zipper tempting is so boring that meets


let me show you this one this is much more interesting much more interesting I promise I promise much more interesting as if you want we can go into some more nitty-gritty with this one I have far more that I'm gonna be able to get through which is fine it's about no equations I'm no proof assemble class like this so this this is kind of the interface that you would see if you were


a mechanical turk being paid to label images on the internet which is basically how all of our everything is made to work now inside all the big tent companies they hire people and have them watch horrible disturbing content and flag it for their their algorithms is great wonderful world we live in so hey so we have so here's our images right these are supposed to be bow


everybody can read the top a bow is actually is not quite clear a weapon for shooting arrows composed of a curved piece of resilient wood with a top cord to propel the arrow and the task is click on every single image that has a boat that's what the Turker is asked to do click on every single image that has a bow so for example these three have both these ones don't actually these


were three that were adversarially placed in here to make sure that the people that that the turgor z' weren't cheating by the so what we were eight what we did was we actually flag all of these nuke so we come in reproducing and reproducing image that we tried to reproduce the query that was used to collect images reproduce the things that were allowed like including the date


range from flickr reproduce the way that these things were labeled and we also through the old test set in here to see how those were labeled by these Turner's which is much much nicer than what we're able to do with with safar tender and what's interesting here is that there's super high variability in what everybody's like not everybody says the same thing obviously there's gonna be


disagreement right there's one you have this is site anybody who work in crowdsourcing knows this right you have the people who do these tasks see through they work too quickly or there's just new us right so for example this is this bow was selected a hundred percent of the time by every turn this one was 70% of our heroine from brave I mean there's a bow there it's a little


bit hard to see you on this blown off screen right so there's a loss of reason it's not Center frame so the lot of reason to miss it this one now becomes a metaphysical conversation about what is the bow good friend Conrad Curtin says yes so I mean neuroscientist so you know I don't know everybody has a good opinion about what that is and this one's wrong and then


20% of the people the boat it's a boat it's not the boat we want we want right ok so I think that's that's our issue right so this is not the boat we want and so we actually had like the histograms for every class for every class thousand classes we had histograms of the selection frequencies and so when we actually sampled our new test set we tried to match the statistics of the the


label from the original set with the new set also the big back end vise all looked at every single image and made sure it was correct in the new test which is clearly creating a distribution shift yes Alex told me it's Berg told me but I'm not anyway it's not true oh yeah you could go look look that's actually the other amazing thing about all these data sets how many people who ever


looked at the images name isn't it I mean you don't need to you just watch the curve go down but like you know it's like actually especially though it's like we kind of have decoupled ourselves from actually any of the domain expertise it's actually quite entertaining what you do yes it's quite a JD what do you do I grew up right ok so punch line right we see exactly the same thing we see


exactly the same thing we saw in separate n a bigger drop technically speaking is the tempers dislike it 10 percent drop at the top here for the best ones but we still see a positive slope so the models that have that models that fared better on the original set fared better on the new set but a little bit it's not for analysis it was first two part 10 but we definitely see


a positive slope in our fit but we do still see the significant drop so what is how is the new legally we need you said you also labeled the ultimately just we labeled the old imagenet just to match the selection frequencies from our so okay there's was there a distribution shift already you are you already pointed out that Ludwig and Becca are not the same and then we


matched their sampling frequencies but you also in the process you also saying we sampled from we have a bigger pool we didn't have our new test set with a bigger pool of images and we sampled from that pool to match the statistics of the old Tessa oh just the statistics yes no no no not using in this evaluation say there are no old images in here really yeah yeah yes and there


isn't same neighborhood yeah you can measure how will the old labels this might labeling take those images on those images take the predictor defined by the old labels yeah accuracy on old accuracy images it doesn't write it has seven foundation just old validation is yeah it's noisy its noisy no man it's noisy it's still noisy again so the orange right here


nothing too big nor think it's then but even if it doesn't matter it says you're taking the label that was as it appears in the chain sent every image on these the label yeah it was generated using some procedure yeah measure of how well those predictions predicted a pose that was assigned to me for your process that's not how these sets are built at all hey Louie let me I got that I've got


the okay it's okay I got the forest but I know I'm also used to this it's a procedure it's a procedure so so it's also true right that that none of there's no these data sets are not made by labeling images again like what happened right you show people you you query and then you test correctness as to whether or not they go in or out it's a weird it's a weird process the other


thing that's really interesting by the way the other thing it's really interesting we take for granted the question you asked the Turkish is which which of these images contains at least one type of object Bo that is not the classification problem that everybody is now weightlifting against each other for the question we have now is just label the damn image it has one label and that


label somehow comes out and we know that like why is this a book I mean this contains a bow but this is the the I forgot her name what's her name she's from brave okay it's not Elsa that's frozen I know that one yeah great so again like I asked you what is that image you it you could describe so much it's Elsa in a forest with her bow on the branch here anyway there's like I'm


sorry Meredith excuse me anyway right so it's like a much more complicated thing and what we evaluate is very different let me do I I didn't make the original image that day is that so I I just illusion equals ask the question was we asked exactly the same question that the imagenet people asked I'm just saying it's the exact same question oh my god maybe you should redo it


$1,000 it's really expensive it's really expensive it's so at anytime you want to say you want more right I mean just note that this is a very expensive project and thank you to Microsoft for Darren's leave funding part of the lately anyway can I just let me just blaze for to cut late so they run Mont I have to like two more examples do you want I mean everyone's goal oh yeah okay


okay here two things I do think and yeah you could fight with me maybe try to find more evidence that we do not see adaptive overfitting or this is not obvious but we do see significant fragility from distribution shift and the distribution shift is here is just humans disagreeing with each other barely I mean just so sorry Simon humans disagreeing with each other it's just


that there were much chemikal Turkish in 2011 and now they're mechanical turkeys and 2019 and that's different people most likely I don't think people Turk for that long oh geez no no retraining no nothing is just take the thing take the weights don't have to retrain because otherwise that would be really expensive again unless they were tearing with the dawn


bench some of them are really slow you know some of them come from Google so they're really really slow and so you have like these things that are huge but yeah the trend is the trend is so the question is can we find more evidence I just want present more data and then I'll stop go ahead I know man well it's the small that it feels like it's the small distribution


shift and just manifested in a huge error so I guess the other way I'd say that is small distribution shifts and seem to propagate into large like imagine what happens in reality right I mean this is like one of these so then I say I mean this is trying as hard as we possibly could to match the statistics maybe we could've tried harder it's it's yeah it's a small distribution shifting


reduces the large error and we should be worried about that I'm more worried about that than the diminishing returns reversing but what's also surprising that in what flux you're just like you would imagine that the fragility increases as it were it doesn't right so they clearly all we have to do is go from 600 600 million parameters to 600 billion parameters and we'll be up on


the line and then it'll be fine right I guess then we'll have cheese self-driving and full self-driving will happen once we get up there but um yeah I don't know I mean I don't know I don't know how to extrapolate this any further because I we've already even even at Google they've run out of resources actually even iid know and actually I think the test set okay validation set


commissioner and folded into the training set and so on you can basically the same actresses yeah evidence video yeah you want lunch you don't see plots 10:00 you want to see Plus man you want to fight with him I have a date bar in my back right so this is this is a cool one same time this is a curated data set for 2015 by the original image net folks right here there was like


there were 4,000 videos but then they just rendered them down as 1 million JPEGs and then presented to you as JPEGs and each of these corresponds to some classes it's a 30-30 classes the subset of imagenet so what we know exactly where these images came and it's kind of it was supposed to be for video but you could just use this for detection and classification and what we did is we


invented a metric that's fairly reasonable metric which is you teach you cheat each video as a set of similar images and then for every frame you pick a K and you look in the neighborhood of that K and you see if you could find one where you get a miss classification this just allowed us to prune through that data set pretty quickly so remember that these are all mostly 30 frames per


second videos so 10 frames is about a third of a second ok so so here's some cool pictures you see these cotton pickers on Twitter all the time where we go from a domestic cat and within 10 frames it's called a monkey oh yeah I think it's cuz now I see a monkey this what I don't see it goes from bird to domestic cat I guess that's what is he eating I don't know this one goes from turtle


to lizard this one goes from dog to horse what's amazing it's easily these images to you look the same I mean I have to tell you to squint to see why they're different right again within a third of a second of each other and of course Jason can't see you that's right to Jason that jason has a good filter that makes them all look to say they all are cats and we made a lot of effort to


like make sure when you're going through here that the kinds of things we were pruning when we actually get to this next plot were like deep so the ones we saw before would look really similar these look really different right these are very different we prune those they did not go into the data set when we were doing curation so we did this again we I didn't do this


Ludvig if I saw I saw Dave we incorporeal Ike like we recruited people from CMU for this I mean it's like a lot of work to label and we see the exact same plot man we see the exact same plot so that you yeah again like just using this metric you see exam this big shift this again it's again it's a small distribution ship is within four point three frames this is 10 this is PM 10


this is in 0.3 seconds of each other you see it's a small distribution shift and again everything you don't see adaptive overfitting you do see some sensitivity to distribution shift let me skip this go I'll talk about the end finally kaggle two more things Kaggle CAG will released a nice meta-analysis that were metadata set of all their competitions with a lot of information not all the


information we would have liked but a lot of information everybody knows on cavil you have a public and a private leaderboard the nice thing is almost in every competition here those are a ID split from each other this is cool because now what we should see is that if you train the public leaderboard and you train and you somehow evaluating the on the private you should just see


clustering around the y equals x and that's exactly what we see clustering around y equals x and it's not just on two it's on like B's how many do we do look I forgot the number 117 I don't have all of them in here but really key in this case because you have iid splits you just see clustering around the y equals x so now so again evidence that distribution shift because


these are these are iid evidence here is that the distribution shift is really what's causing us to be ok I'll stop there um oh sorry the most important slide of course is that you because no one day right the data set to make all data sets and we see Leon Batu wrote this beautiful oral history of the M this data set and also managed to reconstruct a bunch more examples and


it's very nice fun read and again we see the distribution shift again between these two tests we were doing accuracy so it should be negative right yeah yeah I didn't want to I just cut this out of their paper I should already made it that all right so I'll stop there it sounds like we've seen this before we know this we knew this was true in boosting the interpolating training data


we knew that was fine and it did seem to always make you like building bigger models seem to have better test error making your models big doesn't hurt but definitely does seem to be some other issues that are going to really be the pressing ones that we need to deal with moving forward I'm not sure Jonathan and I will talk about how we teach this to our undergrad machine learning class is


very gently I think so how but I do think that for us the researchers and and for anyone in industry the big issues the bigger issues are this distribution shift is a real dangerous thing right if you're putting it in a car or you're making health care decisions and like that we I can show you if you're interested afterwards I could show you a paper which which


demonstrates that actually a lot of the there's huge distribution shift effects in medic with all of the radiology there's a deep learning for radiology huge things where you're basically overfitting to the machine that took the image so that's dangerous if that's a life-and-death situation we already know that Tesla cars kill people what's amazing about Tesla cars


two people have died driving under trucks with their autopilot on in Florida and what's nice about that is that machine learning generalization you should never make the same twice I showed you your corner case and yet two years apart same errand same accident so this is a real thing that's something we have to think about but you know I think that's also cool there are


you haven't heard this yet so I'm Gordon our facilitator is Florian so so today the paper we're going over is unsupervised data augmentation the primary author qi j xie and several other co-authors from google brain and carnegie mellon this is we're gonna go over motivation so deep learning is sorry typically requires a lot of labelled data in order to succeed and

Unsupervised Data Augmentation | AISC


and label data is very expensive so that's one of the main motivations for this paper some of the less costly ways of applying improving deep learning are using unlabeled data which is much more abundant and and easy cheaper to accumulate and data augmentation which basically stretches your supervised labeled samples further and as well are we good with the sound data augmentation

Unsupervised Data Augmentation | AISC


has mostly been applied in the supervised setting and so we want to see if it can be applied in the unsupervised setting as well the main contributions which we'll get into are applying state-of-the-art data augmentation to semi-supervised learning a training technique called TSA training SiC leg kneeling that effectively prevents overfitting when you have much more

Unsupervised Data Augmentation | AISC


unsupervised data than supervised data and they they achieve performance improvements on multiple text and vision benchmarks and then they also introduce a method to even the prediction distributions over across classes for unlabeled and labeled data semi-supervised learning I hope people online know what this is I probably won't explain it again so we'll get into

Unsupervised Data Augmentation | AISC


smoothness in forcing okay so this is one approach to semi-supervised learning and the general idea here is you you try and regularize a models prediction to be less sensitive to small perturbations applied to to the the input data so that in potato can be label or unlabeled and when we say perturbations we're basically talking about adding some sort of noise to the

Unsupervised Data Augmentation | AISC


inputs ampuls and so yeah you want your model to given a sample and an Augmented sample or perturbed sample you want the models predictions to be similar on both I think that's what I just said so enforce the predictions to be similar and in general you want a good model should be invariant to small perturbations on this input data that don't actually change the nature of the

Unsupervised Data Augmentation | AISC


example and yeah so data augmentation is a technique to boost your training data size and the diversity of it so the general idea is you're augmenting in some way again adding some noise to your input samples so that you cannot that you can both get more training data and have more diverse training data and I guess what we'll see some examples of what diversity means so yeah basically

Unsupervised Data Augmentation | AISC


you apply some sort of transformation you have a transformation function the apply to your input data and in data augmentation there's always this trade-off of diversity and validity that's being managed so so yeah you want to create novel and realistic training samples without augmenting them so much that you change their underlying inherent label so diversity it means

Unsupervised Data Augmentation | AISC


growing the the reach of your data set or making your data set more broad and validity is making sure that you're not blowing up your samples so much that they are no longer recognizable or they're not related to the label that they should have assigned any questions so far so this this is what supervised data augmentation looks like here so here Q is is our transformation function so you

Unsupervised Data Augmentation | AISC


can see it's conditioned on an input example X and then X hat is the Augmented data sample and so so basically we're trying to minimize the log likelihood the negative log likelihood of the true ground source ground truth label which is y star given an Augmented sample X hat and and so yeah you can see this as basically an additional training signal that's being

Unsupervised Data Augmentation | AISC


sent to the objective function that that is hoping to yeah that is that is just using the the Augmented samples and then very similarly or actually slightly differently unsupervised data augmentation so this is when you have unsupervised unlabeled data this is a common way to to use that data for data on plantation you can basically take examine the output distribution

Unsupervised Data Augmentation | AISC


prediction probability distributions for an unlabeled sample so here that's X and then an Augmented unlabeled sample X hat again and and you've got the same transformation function Q that you had in the supervised setting and so really what you're trying to do is in this case minimize the divergence or the the difference between these two probability distributions so you're trying to

Unsupervised Data Augmentation | AISC


normalize or regularize the predictions on the Augmented samples to have similar class distributions to to the unag mented unlabeled data does anyone have any questions here doesn't minimizing this minimize they are therefore as well minimize this minimize the other in this case I'm sorry maybe I misspoke earlier in this case these are X's all labeled labeled samples here ordinary boys why

Unsupervised Data Augmentation | AISC


is so why here is not the ground truth label y here so this is y star that's the ground truth annotated label and here we just have the the output prediction distribution or from the model for both the unlabeled and the Augmented unlabeled sample okay what's the difference oh yeah good question so theta it is implying that these are parameters that are being updated so the gradient is

Unsupervised Data Augmentation | AISC


passing through these I'm pretty sure that the idea with theta tilde is those parameters are frozen so they're they're not they're not updated in the objective function data here yeah yeah so this is the instead of thing yeah these would be these are two different settings but but yeah in both cases they're the model parameters question here is the transformation function so so the

Unsupervised Data Augmentation | AISC


augmentation function basically the way that you're adjusting your input samples so the first version sees X's or actual the data that you have that you have labels for in this one yeah so in sorry in this case the unsurprised a documentation you're assuming that you have so yeah I guess you can see here you've got a so in U and U is an unlabeled set whereas in the

Unsupervised Data Augmentation | AISC


first one you have x and y star in in a labeled set so yeah there's no labels in this data at all so this this actual approach there are a few different ways you can do this this specific approach of using the KL divergence between the unlabeled or started the Augmented and unadmitted was from a paper in 2018 VA t believe was the mall so so they borrowed that

Unsupervised Data Augmentation | AISC


approach here and really the main difference they applied in this case is the the transformation function so what they did with Q and so that's what we're going to talk about now so this idea of targeted data augmentation so so over conventional methods such as adding Gaussian noise or affine transformations perturbations like that if there are a few advantages applying targeted

Unsupervised Data Augmentation | AISC


augmentation so one is that they give a valid and realistic perturbation so so the idea is when you apply some of these state-of-the-art data augmentation methods the the output augment example it is still very much in in the same distribution as the the sample it was transformed from so so these are sort of realistic augment augmented examples whereas when you just apply

Unsupervised Data Augmentation | AISC


some random Gaussian noise it can often make the the data point if you apply too much make it sort of very unrecognizable and not realistic so so not really as we say valid it also applies a diverse perturbation so again if you want to use those other methods just adding Gaussian noise you're usually not going to change you're not going to be able to change your input samples significantly enough

Unsupervised Data Augmentation | AISC


and so you just end up with sort of local changes to the samples whereas with targeted augmentation you can really generate diverse diverse samples that are much more useful and in growing your training set and then as well we'll see what some of the methods they had a targeted targeted inductive bias so so yeah you can actually apply approaches that are optimized for the tasks that

Unsupervised Data Augmentation | AISC


you're that you're solving in the particular data set and so so we'll see we'll see an example of that in the augmentation strategies they applied yep so this is the the training set up that they applied so on the left hand side we have the labeled data and it's so split up to x and y star part of me x is fed in through m that's the model and then we just have the standard supervised

Unsupervised Data Augmentation | AISC


cross entropy loss being calculated here that's feeding up into a final loss on the right hand side is where they take the unlabeled data and and they do two things so so one they feed it through the same model and and then as well they take that sample the unlabeled sample and then they apply some augmentation to it so we'll get into all these different segmentation

Unsupervised Data Augmentation | AISC


strategies so that results in X hat so this would be their their Q the transformation function so they get X hat and then they feed that through the model as well and then they take the the output of the model on X and on X hat and they feed that into the unsupervised loss function and that's the the KL divergence that we saw here so basically they just take a weighted sum of the

Unsupervised Data Augmentation | AISC


supervised and unsupervised loss and and combine those together question yeah what is supervised course and complete loss this is just the standard loss function for okay across the so you take the the ground truth label and and compare that to the output prediction probabilities yeah Joey so none of these data seem to have two tilde over it does that mean that oh yeah that's a great

Unsupervised Data Augmentation | AISC


point I pretty sure they forgot to apply a theta on this this prediction distribution so the idea is when you when the gradient propagates back through the through the model it will run through the supervised portion and it will also run through the Augmented application up here but but I'm pretty sure it's not flowing through just the unag mented unlabeled data good question

Unsupervised Data Augmentation | AISC


I can't remember if there was a justification for that what's up and okay we're in this flow does the targeted or the diversity augmentation where does that come to play in this flow the target okay so so the idea is that based on the type of input data that you have for the unlabeled data they'll apply a specific augmentation to that sample so so here they've listed the ones that they've

Unsupervised Data Augmentation | AISC


used but the idea is they only use one at a time so based on the type of data you're using and the particular data set they'll apply a particular augmentation strategy so so I think in the next slides it'll show the particular policies they just they just sort of listed them all here on the left hand side is a leap of data so you have the ground truth the white star the right I

Unsupervised Data Augmentation | AISC


know if you don't have one right so the lesson1 the left hand side is your doing the prediction training and the right hand side you do the prediction infant yeah so so on on the right side yeah the loss function does look different than then on the left but in both cases you have you have a version of the model and with its parameters that that can be updated with signal from from the loss

Unsupervised Data Augmentation | AISC


function so so in the unsupervised case this is the this is the function that is leading to the unsupervised or signal flowing through the unsupervised portion yeah so there's just two different loss functions one looks like this and then the other is the center oh my oh my okay scheduling of training do they train all supervise first then on label or are they training all mixed together that's

Unsupervised Data Augmentation | AISC


a great question and that relates to the big contribution that I mentioned earlier TSA the training signal and annealing so we'll probably wait until that two to explain that yes so for the unscrew right side it doesn't matter what the label and what the model what with the label is not about the model is right like the lost isn't it yeah really what you care about here is

Unsupervised Data Augmentation | AISC


that the the amount that you're changing or sample by is not differing significantly from the the same model prediction on the sorry the model prediction on the same unlabeled sample okay so so going into the augmentation strategies the first one we'll talk about is Auto and all so Auto augment learning augmentation strategies from data the general idea here is they have input

Unsupervised Data Augmentation | AISC


data on the left and they basically have a model that will automatically search through multiple different augmentation strategies so you can see here in policy one there they're making a transformation rotation to the input data the other ones are mostly changing the color of the samples and and so the idea is the the model will automatically select the policy that is adding the

Unsupervised Data Augmentation | AISC


most novel signal to to the yeah to the training so for example if and they find that in different in different data sets different policies are optimal so in in one data set you might need to modify the color of your images a lot to get more diversity and improve your training training set and in another you might need to rotate the images etc so basically it's a again coming back to

Unsupervised Data Augmentation | AISC


the idea of being a targeted policy and this is something that you can vary on a task by task basis so even on a particular data set you can see what's the optimal augmentation approach any questions on this one so that was the one that they applied for vision and then they have to for text one is back translation so the general idea here it's pretty intuitive

Unsupervised Data Augmentation | AISC


you take a question in one language in their case they used English to French so they trained a machine translation model between those two languages then they translate the English sentence into French they then take that French translation from the model and they translate that back into English and and then they use that as the Augmented sample in the model so obviously any

Unsupervised Data Augmentation | AISC


machine translation model is going to have some some loss and it's not going to exactly translate something the same way back and forth so in this case you can see a lot of it has stayed the same this is Google Translate but but this word crankily was previously grin jingly and I think spoof spoof gets translated to tragic travesty so so this is one way that you can augment your samples and

Unsupervised Data Augmentation | AISC


then the other one that they have for text is this tf-idf based word replacement so and they use this for text classification so so the idea here is sometimes in back translation the the Augmented transformation might actually miss translate some of the key words for that sample and in the classification tasks and so here they basically assign an IDF score to each word in the in the

Unsupervised Data Augmentation | AISC


sample and then they randomly sample or randomly swap out words and giving a higher likelihood to swap words that have a low IDF score so here you can see I've just sort of created this example and so in this case the words this in decides to etc are transformed or swapped but but the words that are a little bit more rare and therefore have a higher IDF score such as pathetically cringing

Unsupervised Data Augmentation | AISC


lease poof those are more likely not to be swapped out and so that's yeah based on the intuition that certain keywords sometimes are really useful for text classification any questions on the strategies those are the three okay so so now we come back to the question of how do we balance the need for having a large model so so when we're dealing with unlabeled data of which we have a

Unsupervised Data Augmentation | AISC


really often you have much higher volume you have much more unlabeled data than say labeled data and so you generally would need a very large model to to train on that data so but but you may have a small amount of labeled data so you wanna the question is how to balance a need for a large model while preventing overfitting at the same time and so they're their answer to that

Unsupervised Data Augmentation | AISC


question is to gradually release the training signals of supervisors examples as the model is trained on more and more of the unsupervised examples so I'll show the the equations for for all this okay so here let's see so B here is the batch that is just a renormalization constant and the key is is this part over here so again this is sorry this is the objective function and

Unsupervised Data Augmentation | AISC


and so really what they've introduced is this portion on the right hand side where they say and this is all with regards to the labeled samples so n or what's this constant in Ada Adah yeah so so a dub G is is a threshold that varies with your training progress so we'll see that in the next slide but basically it's a threshold and and if the if the models prediction

Unsupervised Data Augmentation | AISC


probability on the on the correct class for a labeled sample is above that threshold then then that this will evaluate to zero and therefore that sample signal will not get propagated through to the loss function at that time and yeah so so the constant will will change over time but generally the idea here is is that one yeah when ADA is is small then you'll

Unsupervised Data Augmentation | AISC


pretty much be rejecting most most signal from labeled samples from going to the loss function so you'll be preventing the model from overfitting on on say a small set of labeled data so that was the that was the reason they introduced this any questions question it is I the indicator function yes sorry I didn't say that I is the indicator function here and and and Zed is just to

Unsupervised Data Augmentation | AISC


rebalance or renormalize the effect from that during this rain ah ADA goes from it's always increasing okay so I'll show the schedule so they they introduce a few different schedules for for ADA so here's the equation for ADA on the right K is the number of classes in the classification example and lambda of T you can see in in the plot so lambda of T varies with the training progress and

Unsupervised Data Augmentation | AISC


and as lambda of T increases or sorry as training progress increases lambda increases and as well the threshold also increases so at the beginning lambda will be zero and ADA will be one over K so 1 over K is 1 over the number of classes which is just the random chance of predicting a sample and then at the end ADA will be 1 and so coming back to this equation if ADA is 1 that means

Unsupervised Data Augmentation | AISC


every single sample practically will be will be carry forward towards the the loss function and and at the beginning when it's 1 over K only the predictions that only the samples that the model is very unconfident on will actually be used in the loss function and so so the idea here the intuition is that if you have a small labelled set so you have only a few examples that have labelled

Unsupervised Data Augmentation | AISC


labels you want to avoid your model overfitting on those at the beginning of the training so they suggest using this exponential schedule for for the case where you have a low number of labeled examples and so yeah the idea is at the beginning you you won't be feeding as much signal from those samples to your loss function but by the end you can start releasing more and more of it once

Unsupervised Data Augmentation | AISC


more of the unlabeled data has been incorporated and then conversely if you have say a large number of labeled samples then you can use a log schedule the green line here and so that will release a lot of the supervised signal at the beginning and less at the end any questions on this think that's it before the break so we'll just take a 5-minute break and then afterwards we'll go over the

Unsupervised Data Augmentation | AISC


results and have some discussion come back people online the sound is working sounds good okay great so we're gonna do a five minute break now we just did a five minute break we're gonna do experiments so basically they they applied this method to two different types of tasks Texas classification and then image recognition so a couple of vision benchmarks these are the actual data

Unsupervised Data Augmentation | AISC


sets that they used so there's a mixture of binary and five class text classification most of it is sentiment I except for dbpedia which is I believe categories and I think dbpedia actually has fourteen classes or ten classes the two image benchmarks both have ten classes they also use image net they test on image net and they do some ablation studies for TSA and for the targeted

Unsupervised Data Augmentation | AISC


augmentation so for text classification and the settings for the labeled data so the goal here was to use a small number of labeled samples so for binary classification that means twenty just twenty labeled samples and the rest of the data coming from the unlabeled and for the five class classification they found they needed to use a bit more so here they use twenty five hundred total

Unsupervised Data Augmentation | AISC


samples and and so that's five hundred per class so when speaking to the author the goal here was to find out how low could they go so I was asking him did you experiment or how did you stumble across these numbers and really they wanted to determine how few examples they could labeled examples they could keep while still achieving really strong results from the unlabeled data for IMDB

Unsupervised Data Augmentation | AISC


which is one of the the binary classification they use the concatenation of the training set that they didn't use so the training data that was not used for super as supervised they use that as unlabeled and then they use the rest of the unlabeled set so I believe the total training set is 25,000 and the unlabeled set is about 50,000 so that's how much unlabeled data they're using there and

Unsupervised Data Augmentation | AISC


for Yelp and Amazon they obtained some really large review datasets and I believe there was something like 6 million samples in in the unlabeled and so for the most part again another choice was that I think for the most part they used one Augmented sample per unlabeled sample so so but but he said that that would be something that might be a task specific parameter and that

Unsupervised Data Augmentation | AISC


you could adjust so for some tasks for each unlabeled sample you might want to make a couple of augmentations and use both and as for the model they try a few different initialization schemes so all are working on the transformer architecture applied in Beart so they have just a random initialization then they have bird base bird large and Bert large fine-tuned on on the unlabeled

Unsupervised Data Augmentation | AISC


data the in domain on little data so the same unlabeled data that they're using and and for each setting they compare the performance for each of these settings with and without the unsupervised data augmentation method UDA so here are the results for for the text benchmarks so you can see at the top the data set name and then below that the number of supervisors examples that

Unsupervised Data Augmentation | AISC


exist in that data set the the top two results are the pre bert state-of-the-art so that's that's how influential Burt was there's a before bird and after bird so they report both in some cases Burt was better than the state of the art in other cases I I guess it's always okay and then their results the results from their paper are in the bottom in the semi-supervised

Unsupervised Data Augmentation | AISC


setting again they've got the different initialization strategies on the on the left and you da X or check indicates whether or not they use u da and and so these are error rates here so lower is better and below the the data set name in the bottom you can see the number of labeled samples that they used so for IMDB they literally only use 20 examples and and when they apply when

Unsupervised Data Augmentation | AISC


they use an initialization from fine-tuned Burt model they're they're able to beat the state of the art and the pre bird state of the art so that literally means they're just using 20 labeled examples along with Burt which is fine-tuned on unlabeled data and and then they use augmented samples from the unlabeled unlabeled data set so that's that's one of the most significant

Unsupervised Data Augmentation | AISC


improvements and they found across the board they they got very close it or actually beat the the the state of the art for these tasks I think the one that they found the most difficult and they perform the the worst on was the de 5 class classification for both Yelp and Amazon you can see that their results are still a bit a bit off from from the baselines so yeah a really big finding

Unsupervised Data Augmentation | AISC


here is that obviously just with using Burt and know you da you still get 6.5 error with just 20 labeled examples so a lot of this is just indicating how much information is being contained in Burt but but it's quite significant that this shows that you can use pre-trained language models along with UDA so it's it can be complimentary to pre-trained language district models yeah that's

Unsupervised Data Augmentation | AISC


right on the 20 labeled samples plus fine tuning for the last case but but yeah that's it yeah they do they do do some studies on the vision benchmarks that investigate whether or not the whether or not doing augmentation on the unlabeled data is actually more advantageous than doing it on the labeled data so but they don't think they did that for for this so yeah it

Unsupervised Data Augmentation | AISC


could be a good unless I'm wrong that could be a good feedback they might have their I should say that they did do something speaking with you out there they did do other experiments that they didn't report here and I think I might have asked about that I just can't remember if if that was something they did but if they did it wasn't as as performant so for the vision benchmarks yeah so

Unsupervised Data Augmentation | AISC


they wanted to use the the same model that was used for prior semi-supervised work and this was the wide residual networks with depth 28 and with 2 and and so yeah they use the same exact labeled samples that Auto augment use to find its optimal policy so for C far 10 which is a 10 10 Way image classification task they have 4000 samples and for this is Street View

Unsupervised Data Augmentation | AISC


house numbers it's a digit recognition data set they used a thousand labeled examples and so they do 10 10 runs with this model and and calculate the average and the standard deviation so here are the results here these on the left you see the the fully supervised setting at the top so with no augmentation and then following that you see different different augmentation techniques that

Unsupervised Data Augmentation | AISC


that that are applied to to the unlabeled data so I think for I can't remember how many samples but there are roughly maybe 50,000 unlabeled samples for 4c far 10 yeah I'm not sure of the numbers right now but I think you might remember earlier I mentioned V 80 which was the the paper that they got the the idea to take the KL divergence between the distribution of unlabeled and an

Unsupervised Data Augmentation | AISC


Augmented unlabeled sample so really what you're seeing here so obviously they performed the the previous state of the art method but between UDA and v80 the only real difference is the perturbation or transformation function that they applied since they're both using the same KL divergence technique and so so this is indicating that that targeted data augmentation strategy was was

Unsupervised Data Augmentation | AISC


helpful so is there any metric or measure that we can look up into in order to see like they are saturated and why not using hundred-thousand right so we would say morning that's better but there's a point of a saturate there like the diversity that they don't were like the distribution or being the new thing that you're defining in our data we are like many in the data which doesn't

Unsupervised Data Augmentation | AISC


matter after that right so for example Wendy chose to see a thousand there's a reason for that certain metric for us to see that's level oh you know I don't think they provide our metric there might be a good discussion for at the end though when we get to the discussion points like so so yeah keep that out of minds but but as far as I remember there wasn't anything

Unsupervised Data Augmentation | AISC


provided Hockman Auto admins will find the optimal policy but I didn't read the paper in depth I'm not sure if it optimizes anything else such as the number of samples as far as I know it's just finding the optimal policy for a particular data set the optimal augmentation transformation and okay so they also perform some experiments on imagenet and the the motivation here was

Unsupervised Data Augmentation | AISC


they the first the initial data sets they use all had between two and ten classes so they wanted to and and and they all had a low number of supervised examples four thousand or less so they wanted to use a data set that had a hot much higher number of classes it's a bit of a harder task and much more supervised examples and see if this approach was still applicable or if it

Unsupervised Data Augmentation | AISC


was really only sort of a niche improvement for for smaller data sets and then they also wanted to see if they could make use of out of domain unlabeled data that had different class distributions so keep in mind in all of the previous examples the unlabeled data sets that they were using largely were just the the actual labeled data without the labels being used so so it's very

Unsupervised Data Augmentation | AISC


much in domain literally samples that have a true label they just didn't tell the model what the label was so they wanted to see if this would be applicable if you didn't have yeah if your labels coming from out of domain and so imagenet overall has almost 1.3 million images and and about a thousand different classes so the settings they do a couple of settings so one is image

Unsupervised Data Augmentation | AISC


they call image net 10% and this is where they they take roughly 10% of all image net data and use that as labeled samples and they use all of the the rest of image net as unlabeled data so ten percent would be I guess one hundred and thirty thousand samples and and then they would have over a million unlabeled or about a million and then the the other one is the fully supervised

Unsupervised Data Augmentation | AISC


scenario so this is where they use the entire image net data is supervised data and they obtain extra unlabeled data from data set called jft it's another image I believe it was automatically generated image data set so they essentially train a model on image net and they use that model to to source out the most relevant samples from the jft data set for each class in the image net

Unsupervised Data Augmentation | AISC


so they basically for each class they take the I think it was thirteen hundred most most relevant examples from jft and they and they use that as the unlabeled set for for their experiment any questions okay the baseline model that they used was ResNet fifty here so they did encounter some some errors some issues with with image net so they observed that they had flat class

Unsupervised Data Augmentation | AISC


distributions so the prediction probability distributions across classes was pretty flat or uniform for the unlabeled and the Augmented unlabeled samples so there really wasn't much signal coming through from from the unlabeled part of the the training set up and yeah so there's there's a probably to do with the fact that there are so many more classes and and there's also

Unsupervised Data Augmentation | AISC


so much supervised data available here actually sorry no but the amount of supervised data wasn't an issue I think this was more of an issue even for the image net 10% so where they only had 10% of the training data as supervised this was this was an even larger issue so this led to the unsupervised training signal being pretty much dominated by the the supervised signal and so that

Unsupervised Data Augmentation | AISC


their solution they another one of the contributions I mentioned in the beginning was to sharpen in a few different ways the prediction the predicted distribution produced on unlabeled samples so that there would be basically to encourage the model to use the training signal from from the unlabeled samples so the specific techniques that they use were entropy minimization so

Unsupervised Data Augmentation | AISC


they added an over an entropy term to the overall objective to regularize the predicted distribution on the Augmented examples to have a low entropy so again to discourage these uniform distributions of probabilities they also did soft max temperature control and here this is to control the temperature of soft max when they're computing the prediction on the original example and

Unsupervised Data Augmentation | AISC


then confidence based masking so this was where they basically removed any samples that the unlabeled samples that the model was not very confident on so all of these approaches were were to try and sharpen the probability distribution of the on the unlabeled Augmented samples for these sessions but the car if the actual distribution of your training data your related training data is

Unsupervised Data Augmentation | AISC


uniform and then you try to sort of force it to take a different shape on your unlabeled a of what we shouldn't that prepared for that why should that work so the the number of samples is uniform across the classes but here this is this is saying the problem was that the the models predictions prediction probabilities across the different classes for an individual example were

Unsupervised Data Augmentation | AISC


more uniform so so the example has some category let's say it includes a giraffe so that should be the true label but the actual output prediction distribution was was pretty no uniform so they wanted to find ways to encourage it to be sharper to - yeah to be less less prone to just being killed divergence per batch no I believe it would be calculated per sample pretty sure it

Unsupervised Data Augmentation | AISC


would be each sample would look at the figure so their first average error and then the unsupervised error and then it will do back propagation right hmm I did do it like separately but the way I add necessarily do it all together so the mass calculation kind of patchwork L the Virgin's okay yeah so I suppose that they do these are the results here that the left

Unsupervised Data Augmentation | AISC


is the image net 10% again with the baseline its resonant 50 and the right is is the fully supervised image net setting and top 1% is the the models accuracy on for its first top prediction top 5 is its accuracy looking within the first 5 samples and and yes so you can see that it improved on on the baselines in both cases I guess the smaller improvement for the fully supervised

Unsupervised Data Augmentation | AISC


setting where they're using an additional 1.3 million samples from jft yeah over the the previous Auto augment policy I think for image net 10 was it for this one yeah I think they also so someone was asking earlier about why not using the why not use the labeled examples for augmentation for the baseline and I think they also did run that experiment here so they they used

Unsupervised Data Augmentation | AISC


the 10% labeled examples as for augmentation and so that I think got something like 58 accuracy so still significantly below and yeah moving on to the ablation studies so they want to do they did a couple the first one is for TSA so to determine if this training signal annealing actually made a difference and so they did it for Yelp 5 and see far 10 in the Yelp case they

Unsupervised Data Augmentation | AISC


didn't use Bart pre-training just to really make sure that there was all the information was coming from from the data use and not from the pre train distribution or language model and yes so you can see in the first case where there's an X that's where they're not using any tsa and so you can see in both cases applying some schedule does improve the results and you can also see

Unsupervised Data Augmentation | AISC


that it's it's slightly different for the two cases and I think for CFR 10 there were 4,000 examples and so that's optimal with the linear schedule and Yelp five was about 2,500 and so that's that's one of the lower data settings that are that they hypothesized would be best with the exponential schedule and then the other one was for a question yes you say race when TSA work for pure

Unsupervised Data Augmentation | AISC


supervised learning that's a good question I'm not sure I don't think there's a clear reason that it wouldn't seems like you could apply it so the idea would be that you would [Music] yeah so remove yeah I don't know if there would be as much motivation to do that when you're not incorporating additional data but but that's a good question maybe someone in the discussion

Unsupervised Data Augmentation | AISC


will have some thoughts and for for the actual data augmentation policy so they wanted to see here did the targeted data augmentation that they applied actually make a difference so they have the two image data sets here and some on the left you can see different augmentation strategies Auto augment was they what the one they actually used and switched augment implies they they took the

Unsupervised Data Augmentation | AISC


optimal policy for from one data set and applied that on the other data set so you can see that it really does perform best when you take the optimal augmentation policy the most targeted policy and and apply that to the data set I think that's mostly it just some conclusions so in summary they introduced a method called UTA unsupervised state augmentation that used targeted data

Unsupervised Data Augmentation | AISC


augmentation to generate diverse and realistic perturbations to input samples and so this enforces the model to be smooth the way they applied it with respect to these perturbations they also introduced TSA training signal annealing and this was to prevent UDA from overfitting when a lot more unlabeled data is available compared to the amount of labeled data and their results showed

Unsupervised Data Augmentation | AISC


that they achieved state-of-the-art on IMDb with only 20 labeled examples and they were able to reduce vision error rates by more than 30 percent in the in the semi supervised setting and they were also able to leverage unlabeled data to improve performance on on image net even though what's a maybe a small improvement and so some overall findings are that data augmentation and

Unsupervised Data Augmentation | AISC


semi-supervised learning are well-connected topics that should be explored further and you da+ unsupervised representation learning in this case Bert can compliment each other and work work well together and so that's that's the presentation we just have some discussion points over class that has weight less data right someone was asking about that earlier I

Unsupervised Data Augmentation | AISC


don't think no they didn't do that they specifically in the case where they were using image net and they were boosting from the the jft data set they specifically took an even number of samples across all the classes from jft and I don't think there was any case where they had an imbalance across classes oh yeah that would be an interesting problem to solve for sure

Unsupervised Data Augmentation | AISC


okay so fluorine and I or fluorine and the the author so TJ had had one discussion point that he wanted to to pose to the audience which was so to some extent this type of approach the ability to use more on labelled data can can help to make machine learning more University accessible because it's much more affordable to train models on unlabeled data and so what applications

Unsupervised Data Augmentation | AISC


do you guys think stand to benefit the most from from semi-supervised learning that that haven't already been tried in this paper big problem is you have like a cooking video where you want to get the recipe use automatic machine translation to get to rest there are like small data sets available and it's a very complex problem because of just how recipes are organized right so if

Unsupervised Data Augmentation | AISC


you have a small training data that can really work there are tons of unlabeled data for like videos for recipes right hey there's a gift recip so whatever where you can get large amount of data very easy it's just everything sound a little hmm so so video yeah yeah interesting take a video and you just make a description of what is happening in the video right yeah I don't know if

Unsupervised Data Augmentation | AISC


that's been and tried or or not and then I think so as well what are some other targeted augmentation strategies for text so they introduce back translation and this word replacement I'm sorry they didn't introduce those those those have already existed at least the first one are there any other strategies for for text augmentation that anyone's used or think

Unsupervised Data Augmentation | AISC


could be could be valuable is UI regression when you're trying to compare your interface with a new interface and you don't want to label everything and in the test cycle you want it to be on labeled potentially measures of potential here again you are it hasn't gained much traction because you'd have to label every part of that that's not enough for agile so there was some and

Unsupervised Data Augmentation | AISC


so part of the reason that I asked this question is I felt like the approaches for text augmentations still felt like they were may be most useful for for a text classification case but they're there many other case many other tasks to solve in in NLP and and so I think that they said back translation has been shown to be successful for other tasks but I think that the word replacement

Unsupervised Data Augmentation | AISC


one might not be as relevant so I was curious if anyone had had tried any others and then Florian I think you had a few here so so yeah what are some other ways to automatically optimize data augmentation strategies the idea here instead you see they have like some kind of targeted augmentation strategy right but instead of having some kind of time use augmentation start sheet let's put in

Unsupervised Data Augmentation | AISC


kind of like a neural network right so I kind of like again and then at the same time you optimize scan to make the best possible kind of transformations right so the kind of the gang would be optimized to help the other neural network to get to better care older versions right I wouldn't know if doesn't make sense so it would work this is beginning like if you want to have

Unsupervised Data Augmentation | AISC


minimum corralled emergency we just do the identity right so no but I think is an idea I can throw in a room koc is like some kind of variation of attack order right it doesn't need to be again some generative model to come it with a policy yeah I think it ties in with some of the other points so you're asking you thinking about should we compare UDA or combine it with automatically generated

Unsupervised Data Augmentation | AISC


data sets this is like an active field of research as well yes so so data sets that are completely fabricated without any annotation is this something that competes head-to-head like it's is are they strictly one or the other or can it be combined and another food for thought is will in the long run UDA be a cost effective alternative to to cheap manual labeling so if if manual labeling is I

Unsupervised Data Augmentation | AISC


think you throw out a number like 140 per hour with yeah we're pretty us us 140 US per hour will this actually be a cost alternative method if if the costs are that low this type of approach would face and that's some Angela data privacy I'm not sure if there's an easy easy way to build on something like differential privacy or something like that in order to ensure

Unsupervised Data Augmentation | AISC


that when we're gobbling out unlabeled data that we're not you know accidentally throwing in user data that might even wonder like maybe they could be picked him over let go malicious attacks rates so something to keep in mind with one applying this method the network model is still located present data but you just don't have the neighbor yeah so if we have all

Unsupervised Data Augmentation | AISC


the customer data right and you want to do something you should I mean you still have all the Congress customer looks at all the customers who change much and you just doesn't have some kind of decision [Music] we applied like we were saying the case about the credit scores so like maybe there's a lot to inconsistency in like the ones who retro actively got approved

Unsupervised Data Augmentation | AISC


for credit so like there might be the human error inconsistency or just like the fact that there is self selection or something so like this could kind of help with that right like you're saying there are some people who maybe fit the traits of being approved but then they just somehow didn't get approved so that they're actually being not approved right yeah there's some there's some

Unsupervised Data Augmentation | AISC


bias of some sort in the labeling process the annotation process then definitely that could mean that the the samples in your unlabeled set are valid but but just not being included for one reason or another so I think yeah those kind of cases are really good candidates for for this kind of work any other thoughts or questions in case of a supervisor provide sort of

Unsupervised Data Augmentation | AISC


clustering job and using the sort of the model only on the label to maybe if you have to do a job I'm very nervous very rare that said could you actually refers to a cluster job on a much smaller data set and use those as the label and service supervisor set and apply this for the classes of all the others and see if those classes would actually fit in the initial

Unsupervised Data Augmentation | AISC


foster class and whether it be actually faster than doing the big massive across from John just okay so so the idea would be to take a subset of the unlabeled data you're strictly unlabeled data take a subset train a clustering model on that and use the clusters as ground truth labels that sounds to me like a cool way of starting from scratch like you have no labels I think that's been

Unsupervised Data Augmentation | AISC


that's been done before but in conjuncture with with this type of approach would be interesting yeah I don't see why you couldn't do it I think you might you might find that I don't know how well it would perform yeah as opposed to just doing all once I have like one criticism about this paper I only skimmed it and I wasn't able to follow your entire presentation so crab

Unsupervised Data Augmentation | AISC


me if I'm wrong but what he liked it seemed they called unsupervised data augmentation but it seems that like especially in the case of text they use the model to translate and translate back and there's a very right it's a it's a separate model they train a separate machine translation model yeah to do the back translation but they do utilize that model to to port data

Unsupervised Data Augmentation | AISC


augmentation right they use that machine translation model just to generate just to generate the Augmented samples oh yes yeah so I guess my criticism is that they say it's unsupervised but this is not exactly distillation but there is some kind of because that translation translation back that's supervised and there is some a lot of like that kind of signal that's kind of being learned and

Unsupervised Data Augmentation | AISC


distilled by doing this approach even the tf-idf frequency approach so there's a very strong prior that you built into the into the augmentation part so this doesn't feel like you see like exactly unsupervised in the sense that you just give it a lot of data but there's some kind of implicit supervision I don't think it exactly distillation but some something there is some minute I think

Unsupervised Data Augmentation | AISC


it's acknowledged they call it I mean there isn't a language of targeted data augmentation so you're you're picking an augmentation strategy that is targeted to the particular task and in some case to the data set there you're working on so I think it's not it's not an unacknowledged part of the paper and like they are it's it's yeah on purpose but they're doing that way nonetheless I

Unsupervised Data Augmentation | AISC


do agree that I don't know how well some of those that was my question here how some of those augmentation strategies would apply to other for example text tasks so so yeah it remains to be seen how much work you would have to do to come up with an augmentation policy that is successful for a particular task and data set I think that would probably be very interesting future work something

Unsupervised Data Augmentation | AISC


to add on to that something I think would be interesting is what happens if you have a worse translated back translation because you're adding in creative for the patient to it right but does your unit become better because you have worse back translation for that oh if you if you if your machine translation all was less accurate yeah yeah I think that comes back to the

Unsupervised Data Augmentation | AISC


trade-off between diversity and validity if your translation model is terrible then the actual sample that you generate is probably not going to be a valid sample for the label that it's supposed to be assigned to but if you don't change it enough then you're not really adding any new signal so so yeah there's some inherent trade-off there I think so yeah maybe using a slightly worse

Unsupervised Data Augmentation | AISC


translation model would be effective definitely not using a terrible one but yeah it's a good point the other scenario and that's something the various amid sedimentation so that's means they had a like the jury instructed the different level of the target from the background so in that of Seven AO how can we use the you dated to did up automatic when you have different social

Unsupervised Data Augmentation | AISC


services for image segmentation yeah object detection or yeah because this scenario is one Operator for the pilot but the other some cannot do you know network about the model the end is a the outputs will be the images most sorry at the end the output of the ornamentation yeah hmm right so that's a good question I'm not too sure I'm sure there's been work done

Unsupervised Data Augmentation | AISC


on augmentation for object detection and and segmentation but I'm not I'm not too familiar with that work yeah so I don't know exactly how its applied settings is that the question is having work before you leveled the the trailer the car view yeah yeah but for this one you it's it's a it's a prepared via the distribution of the white and something else buddy in

Unsupervised Data Augmentation | AISC


the sedimentation but how we prepare of the year I mean I think you can still some augmentations are pretty clearly still valid if you just changed the color for example the object detection is still going to be valid you're not transforming or rotating removing anything outside of a box so I think that would that's one approach you could take and and I think color augmentation

Unsupervised Data Augmentation | AISC


[Music] [Music] because the shift from one zodiacal house to another takes more than two millennia scholars wondered how and where Hipparchus could have learned of the precession in the second century BC it is now clear that his source was Sumerian professor Langdon's findings revealed that the Nepean calendar established circa 4400 BC in the Age of Taurus reflects knowledge of the


precession and the shift of zodiacal houses that took place 2160 years earlier than that professor Jeremiah's who correlated Mesopotamian astronomical texts with Hittite astronomical texts was also of the opinion that older astronomical tablets recorded the change from Taurus to Aries and he concluded that the Mesopotamian astronomers predicted and anticipated the shift from Aries to


Pisces subscribing to these conclusions professor Willie Hart nur suggested that the Sumerians left behind plentiful pictorial evidence to that effect when the spring equinox was in the zodiac of Taurus the summer solstice occurred in the zodiac of Leo partner drew attention to the recurrent motif of a bull lion combat appearing in Sumerian depictions from earliest times and suggested that


these motifs represented the key positions of the constellations of Taurus bull and Leah lion to an observer at 30 degrees north such as at a circa 4000 BC most scholars consider the Sumerian stress of Taurus as their first constellation as evidence not only of the antiquity of the zodiac dating to circa 4000 BC but also as testifying to the time when Sumerian civilization so


suddenly began professor Jeremiah's found evidence showing that the sumerian zodiacal chronological point 0 stood precisely between the bull and the twins from this and other data he concluded that the zodiac was devised in the age of Gemini the twins that is even before Sumerian civilization began a Sumerian tablet in the Berlin Museum begins the list of zodiacal constellations with that of Leo


taking us back to circa 11,000 BC when man had just begun to till the land professor HV Hill Precht went even farther studying thousands of tablets bearing mathematical tabulations he concluded that all the multiplication and division tables from the temple libraries of nipper and sipar and from the library of Ashurbanipal in Nineveh are based upon the number 12 million


960,000 analyzing this number and its significance he concluded that it could be related only to the phenomenon of the precession and that the Sumerians knew of the great year of twenty five thousand nine hundred and twenty years this is indeed fantastic astronomical sophistication at an impossible time just as it is evident that the Sumerian astronomers possessed knowledge that


they could not possibly have acquired on their own so is there evidence to show that a good deal of their knowledge was of no practical use to them this pertains not only to the very sophisticated astronomical methods that were used who in ancient Sumer really needed to establish a celestial equator for example but also to a variety of elaborate texts that dealt with the


measurement of distances between stars one of these texts known as a oh six four seven eight lists the 26 major stars visible along the line we now call the Tropic of Cancer and gives distances between them as measured in three different ways the text first gives the distances between these stars by a unit called Mohnish oculto measured and weighed it is believed that


this was an ingenious device that related the weight of escaping water to the passage of time it made possible the determination of distances between two stars in terms of time the second column of distances was in terms of degrees of the arc of the skies the full-day daylight and night time was divided into twelve double hours the arc of the heavens comprised a full circle


of 360 degrees hence one Barrow or double hour represented 30 degrees of the arc of the heavens by this method passage of time on earth provided a measure of the distances in degrees between the named celestial bodies the third method of measurement was Barrow Enosh ami length in the skies after o dangun pointed out that while the first two methods were relative to other


phenomena this third method provided absolute measurements a celestial Barrow he and others believed was equivalent to ten thousand six hundred ninety-two of our present-day meters eleven thousand six hundred ninety three yards the distance in the skies between the twenty six stars was calculated in the text as adding up to six hundred and fifty-five thousand two hundred Barrow drawn in the


skies the availability of three different methods of measuring distances between stars conveys the great importance attached to the matter yet who among the men and women of Sumer needed such knowledge and who among them could devise the methods and accurately use them the only possible answer is the Nephilim had the knowledge and the need for such accurate measurements capable


of space travel arriving on earth from another planet roaming Earth's skies they were the only ones who could and did possess at the dawn of mankind's civilization the astronomical knowledge that required millenia to develop the sophisticated methods and mathematics and concepts for an advanced astronomy and the need to teach human scribes to copy and record


meticulously table upon table of distances in the heavens order of stars and groups of stars heliacal risings and setting a complex Sun Moon Earth calendar and the rest of the remarkable knowledge of both heaven and earth against this background can it still be assumed that the Mesopotamian astronomers guided by the Nephilim were not aware of the planets beyond Saturn that they did not


know of Uranus Neptune and Pluto was their knowledge of Earth's own family the solar system less complete than that of distant stars their order and their distances astronomical information from ancient times contained in hundreds of detailed texts lists celestial bodies neatly arranged by their celestial order or by the gods or the months or the lands or the constellations with which


they were associated one such text analyzed by Ernst F Widener has come to be called the great star list it listed in five columns tens of celestial bodies as related to one another to month's countries and deities another text listed correctly the main stars and the zodiacal constellations a text indexed as B M 8 6 3 7 8 arranged in its unbroken part 71 celestial bodies by


their location in the heavens and so on and gone and on in efforts to make sense of this Legion of texts and in particular to identify correctly the planets of our solar system a succession of scholars came up with confusing results as we now know their efforts were doomed to failure because they incorrectly assumed that the Sumerians and their successors were unaware that


the solar system was heliocentric that earth was but another planet and that there were more planets beyond Saturn ignoring the possibility that some names in the star lessons may have applied to earth itself and seeking to apply the great number of other and epithets only to the five planets they believed were known to the Sumerians scholars reached conflicting


conclusions some scholars even suggested that the confusion was not theirs but a Chaldean mix-up for some unknown reason they said the Chaldeans had switched around the names of the five known planets the Sumerians referred to all celestial bodies planets stars or constellations as mule who shine in the heights the Akkadian term kkuk AB was likewise applied by the Babylonians and


Assyrians as a general term for any celestial body this practice further frustrated the scholars seeking to unravel the ancient astronomical texts but some mules that were termed Lu bad clearly designated planets of our solar system knowing that the Greek name for the planets was Wanderers the scholars have read Lu bad as wandering sheep deriving from Lu those which are shepherded and bad high


and far but now that we have shown that the Sumerians were fully aware of the true nature of the solar system the other meanings of the term bad the olden the foundation the one where death is assume direct significance these are appropriate epithets for the Sun and it follows that by Lu bad the Sumerians meant not mere wandering sheep but sheep shepherded by the Sun the planets of our


Sun the location and relation of the Lu bad to each other and to the Sun were described in many Mesopotamian astronomical texts there were references to those planets that are above and those that are below and Kugler correctly guessed that the reference point was earth itself but mostly the planets were spoken of in the framework of astronomical texts dealing with mule


mule a term that kept the scholars guessing in the absence of a better solution most scholars have agreed that the term mule mule stood for the Pleiades a cluster of stars in the zodiacal constellation of Taurus and the one through which the axis of the spring equinox passed as viewed from Babylon circa 2200 BC Mesopotamian texts often indicated that the mule mule included


seven loom ash seven Wanderers that are familiar and the scholars assumed that these were the brightest members of the Pleiades which can be seen with the naked eye the fact that depending on classification the group has either six or nine such bright stars and not seven posed a problem but it was brushed aside for lack of any better ideas as to the meaning of mule mule Fromm's Kugler


reluctantly accepted the Pleiades as the solution but expressed his astonishment when he found it stated unambiguously in Mesopotamian texts that mule mule included not only Wanderers planets but also the Sun and the moon making it impossible to retain the Pleiades idea he also came upon texts that clearly stated that mule mule rules shul 12 mil mule is a band of 12 of which 10 formed


a distinct group we suggest that the term mule mule referred to the solar system using the repetitive mule mule to indicate the group as a whole as the celestial body comprising all celestial bodies Charles viral au transliterated a Mesopotamian text that describes the members of the mule mule or cacao boo or Kaka boo group the texts last line is explicit kkuk Abu Kakha boo the number


of its celestial bodies is 12 the station of its celestial bodies 12 the complete months of the Moon is 12 the texts leave no doubt the mule mule our solar system was made up of twelve members perhaps this should not come as a surprise for the Greek scholar dye odorous explaining the three ways of the Chaldeans and the subsequent listing of 36 celestial bodies stated that of those


celestial gods 12 hold chief authority to each of these the Chaldeans assign a month and a sign of the zodiac ernst Weidner reported that in addition to the way of on you and it's 12 zodiac constellations some texts also referred to the way of the Sun which was also made up of 12 celestial bodies the Sun the moon and 10 others line 20 of the so-called T tablet stated 'no far 12


sheer mesh Halle Chaka Cobb Lucia sinew shamash inna Libby yaku which means all in all 12 members where the Moon and Sun belong where the planets orbit we can now grasp the significance of the number 12 in the ancient world the great circle of Sumerian gods and of all Olympian gods thereafter comprised exactly twelve younger gods could join this circle only a folder God's retired likewise the


vacancy had to be filled to retain the divine number twelve the principal celestial circle the way of the Sun with its twelve members set the pattern according to which each other celestial band was divided into twelve segments or was allocated twelve principal celestial bodies accordingly there were twelve months in a year twelve double hours in a day each division of Sumer was


assigned 12 celestial bodies as a measure of good luck many studies such as the one by s Langdon show that the division of the year and 12 months was from it's very beginnings related to the twelve great gods fritz Hummel and others after him have shown that the 12 months were closely connected with the 12 zodiacs and that both derived from 12 principal celestial bodies


Charles F Jean reproduced a Sumerian list 24 celestial bodies that paired 12 zodiacal constellations with 12 members of our solar system in a long text identified by F Thoreau dangun as a temple program for the New Year Festival in Babylon the evidence for the consecration of 12 as the central celestial phenomenon is persuasive the Great Temple the s Aguila had 12 gates


the powers of all the celestial gods were vested in Marduk by reciting 12 times the pronouncement my lord is he not my lord the mercy of the God was then invoked 12 times and that of his spouse 12 times the total of 24 was then matched with the 12 zodiac constellations and twelve members of the solar system the boundary stone carved with the symbols of the celestial bodies


by a king of Susa depicts these twenty-four signs the familiar 12 signs of the zodiac and symbols that stand for the twelve members of the solar system these were the twelve astral gods of Mesopotamia as well as of the hurryin hit IDE Greek and all other ancient Pantheon's although our natural counting base is the number 10 the number 12 permeated all matters celestial and


divine long after the Sumerians were gone there were 12 Greek titans 12 tribes of Israel 12 parts to the magical breastplate of the Israelite high priests the power of this celestial 12 carried over to the Twelve Apostles of Jesus and even in our decimal system we count from 1 to 12 and only after 12 do we return to 10 and 3 13 10 and 4 and so on where did this powerful decisive number


12 stem from from the heavens for the solar system the mule mule included in addition to all the planets known to us also the planet of on you the one whose symbol a radiant celestial body stood in the Sumerian writing for the god on you and for divine the cab of the supreme scepter is one of the sheep in mule mule explained an astronomical text and when Marduk usurped the supremacy and


replaced on you as the God associated with this planet the Babylonian said the planet of Marduk within mule mule appears teaching humanity the true nature of earth and of the heavens the Nephilim informed the ancient astronomer priests not only of the planets beyond Saturn but also of the existence of the most important planet the one from which they came the 12th planet on most of the


ancient cylinder seals that have been found symbols that stand for certain celestial bodies members of our solar system appear above the figures of gods or humans an Akkadian seal from the 3rd millennium BC now at the Vorta Asiata shop Tai Lung of the State Museum in East Berlin departs from the usual manner of depicting the celestial bodies it does not show them individually but


rather as a group of 11 Globes encircling a large raid star it is clearly a depiction of the solar system as it was known to the Sumerians a system consisting of twelve celestial bodies the ancient depiction shows a planet unknown to us considerably larger than Earth yet smaller than Jupiter and Saturn which clearly follow it farther on another pair perfectly matches our


Uranus and Neptune finally the smallish Pluto is also there but not where we now place it after Neptune instead it appears between Saturn and Uranus treating the moon as a proper celestial body the Sumerian depiction fully accounts for all of our known planets places them in the correct order with the exception of Pluto and shows them by size the forty five hundred year old


depiction however also insists that there was or has been another major planet between Mars and Jupiter it is as we shall show the 12th planet the planet of the Nephilim if this sumerian celestial map had been discovered and studied two centuries ago astronomers would have deemed the Sumerians totally uninformed foolishly imagining more planets beyond Saturn now


however we know that Uranus and Neptune and Pluto are really there did the Sumerians imagine the other discrepancies or were they properly informed by the Nephilim that the moon was a member of the solar system in its own right Pluto was located near Saturn and there was a twelfth planet between Mars and Jupiter the long-held theory that the moon was nothing more than a frozen golf


ball was not discarded until the successful conclusion of several US Apollo moon missions the best guesses were that the moon was a chunk of matter that had separated from earth when earth was still molten and plastic were it not for the impact of millions of meteorites which left craters on the face of the moon it would have been a faceless lifeless history less piece of matter


that solidified and forever follows earth observations made by unmanned satellites however began to bring such long-held beliefs into question it was determined that the chemical and mineral makeup of the moon was sufficiently different from that of Earth to challenge the brakeaway theory the experiments conducted on the moon by the American astronauts and the study and


analysis of the soil and rock samples they brought back have established beyond doubt that the moon though presently barren was once a living planet like Earth it is layered which means that it solidified from its own original Moulton's staged like Earth it generated heat but whereas Earth's heat comes from its radioactive materials cooked inside earth under tremendous pressure


the moon's heat comes apparently from layers of radioactive materials lying very near the surface these materials however are too heavy to have floated up what then deposited them near the moon's surface the moon's gravity field appears to be erratic as though huge chunks of heavy matter such as iron had not evenly sunk to its core but were scattered about by


what process or force we might ask there is evidence that the ancient rocks of the moon were magnetized but there is also evidence that the magnetic fields were changed or reversed was it by some unknown internal process or by an undetermined outside influence the Apollo 16 astronauts found on the moon rocks called breccias that result from the scattering of solid rock and its


reweld extreme and sudden Heat when and how were these rocks shattered then refused other surface materials on the moon are rich and rare radioactive potassium and phosphorus materials that on earth are deep down inside putting such findings together scientists are now certain that the moon and earth formed of roughly the same elements at about the same time evolved as separate