question you asked the Turkish is which which of these images contains at least one type of object Bo that is not the classification problem that everybody is now weightlifting against each other for the question we have now is just label the damn image it has one label and that label somehow comes out and we know that like why is this a book I mean this contains a bow but this is the the I forgot her name what's her name she's from brave okay it's not Elsa that's frozen I know that one yeah great so again like I asked you what is that


image you it you could describe so much it's Elsa in a forest with her bow on the branch here anyway there's like I'm sorry Meredith excuse me anyway right so it's like a much more complicated thing and what we evaluate is very different let me do I I didn't make the original image that day is that so I I just illusion equals ask the question was we asked exactly the same question that the imagenet people asked I'm just saying it's the exact same question oh my god maybe you should redo it $1,000 it's really expensive it's really

Training on the Test Set and Other Heresies


expensive it's so at anytime you want to say you want more right I mean just note that this is a very expensive project and thank you to Microsoft for Darren's leave funding part of the lately anyway can I just let me just blaze for to cut late so they run Mont I have to like two more examples do you want I mean everyone's goal oh yeah okay okay here two things I do think and yeah you could fight with me maybe try to find more evidence that we do not see adaptive overfitting or this is not obvious but we do see significant


fragility from distribution shift and the distribution shift is here is just humans disagreeing with each other barely I mean just so sorry Simon humans disagreeing with each other it's just that there were much chemikal Turkish in 2011 and now they're mechanical turkeys and 2019 and that's different people most likely I don't think people Turk for that long oh geez no no retraining no nothing is just take the thing take the weights don't have to retrain because otherwise that would be really expensive again


unless they were tearing with the dawn bench some of them are really slow you know some of them come from Google so they're really really slow and so you have like these things that are huge but yeah the trend is the trend is so the question is can we find more evidence I just want present more data and then I'll stop go ahead I know man well it's the small that it feels like it's the small distribution shift and just manifested in a huge error so I guess the other way I'd say that is small distribution shifts and


seem to propagate into large like imagine what happens in reality right I mean this is like one of these so then I say I mean this is trying as hard as we possibly could to match the statistics maybe we could've tried harder it's it's yeah it's a small distribution shifting reduces the large error and we should be worried about that I'm more worried about that than the diminishing returns reversing but what's also surprising that in what flux you're just like you would imagine that the fragility increases as it were it doesn't right so


they clearly all we have to do is go from 600 600 million parameters to 600 billion parameters and we'll be up on the line and then it'll be fine right I guess then we'll have cheese self-driving and full self-driving will happen once we get up there but um yeah I don't know I mean I don't know I don't know how to extrapolate this any further because I we've already even even at Google they've run out of resources actually even iid know and actually I think the test set okay validation set commissioner and folded into the


training set and so on you can basically the same actresses yeah evidence video yeah you want lunch you don't see plots 10:00 you want to see Plus man you want to fight with him I have a date bar in my back right so this is this is a cool one same time this is a curated data set for 2015 by the original image net folks right here there was like there were 4,000 videos but then they just rendered them down as 1 million JPEGs and then presented to you as JPEGs and each of these corresponds to some classes it's a 30-30 classes the subset


of imagenet so what we know exactly where these images came and it's kind of it was supposed to be for video but you could just use this for detection and classification and what we did is we invented a metric that's fairly reasonable metric which is you teach you cheat each video as a set of similar images and then for every frame you pick a K and you look in the neighborhood of that K and you see if you could find one where you get a miss classification this just allowed us to prune through that data set pretty quickly so remember that


these are all mostly 30 frames per second videos so 10 frames is about a third of a second ok so so here's some cool pictures you see these cotton pickers on Twitter all the time where we go from a domestic cat and within 10 frames it's called a monkey oh yeah I think it's cuz now I see a monkey this what I don't see it goes from bird to domestic cat I guess that's what is he eating I don't know this one goes from turtle to lizard this one goes from dog to horse what's amazing it's easily these images to you look the same I mean I


have to tell you to squint to see why they're different right again within a third of a second of each other and of course Jason can't see you that's right to Jason that jason has a good filter that makes them all look to say they all are cats and we made a lot of effort to like make sure when you're going through here that the kinds of things we were pruning when we actually get to this next plot were like deep so the ones we saw before would look really similar these look really different right these are very different we prune those they


did not go into the data set when we were doing curation so we did this again we I didn't do this Ludvig if I saw I saw Dave we incorporeal Ike like we recruited people from CMU for this I mean it's like a lot of work to label and we see the exact same plot man we see the exact same plot so that you yeah again like just using this metric you see exam this big shift this again it's again it's a small distribution ship is within four point three frames this is 10 this is PM 10 this is in 0.3 seconds of each other you


see it's a small distribution shift and again everything you don't see adaptive overfitting you do see some sensitivity to distribution shift let me skip this go I'll talk about the end finally kaggle two more things Kaggle CAG will released a nice meta-analysis that were metadata set of all their competitions with a lot of information not all the information we would have liked but a lot of information everybody knows on cavil you have a public and a private leaderboard the nice thing is almost in every competition here those are a ID


split from each other this is cool because now what we should see is that if you train the public leaderboard and you train and you somehow evaluating the on the private you should just see clustering around the y equals x and that's exactly what we see clustering around y equals x and it's not just on two it's on like B's how many do we do look I forgot the number 117 I don't have all of them in here but really key in this case because you have iid splits you just see clustering around the y equals x so now so again


evidence that distribution shift because these are these are iid evidence here is that the distribution shift is really what's causing us to be ok I'll stop there um oh sorry the most important slide of course is that you because no one day right the data set to make all data sets and we see Leon Batu wrote this beautiful oral history of the M this data set and also managed to reconstruct a bunch more examples and it's very nice fun read and again we see the distribution shift again between these two tests we were doing accuracy


so it should be negative right yeah yeah I didn't want to I just cut this out of their paper I should already made it that all right so I'll stop there it sounds like we've seen this before we know this we knew this was true in boosting the interpolating training data we knew that was fine and it did seem to always make you like building bigger models seem to have better test error making your models big doesn't hurt but definitely does seem to be some other issues that are going to really be the pressing ones that we need to deal with


moving forward I'm not sure Jonathan and I will talk about how we teach this to our undergrad machine learning class is very gently I think so how but I do think that for us the researchers and and for anyone in industry the big issues the bigger issues are this distribution shift is a real dangerous thing right if you're putting it in a car or you're making health care decisions and like that we I can show you if you're interested afterwards I could show you a paper which which demonstrates that actually a lot of the


there's huge distribution shift effects in medic with all of the radiology there's a deep learning for radiology huge things where you're basically overfitting to the machine that took the image so that's dangerous if that's a life-and-death situation we already know that Tesla cars kill people what's amazing about Tesla cars two people have died driving under trucks with their autopilot on in Florida and what's nice about that is that machine learning generalization you should never make the same twice I showed you your corner case and


you haven't heard this yet so I'm Gordon our facilitator is Florian so so today the paper we're going over is unsupervised data augmentation the primary author qi j xie and several other co-authors from google brain and carnegie mellon this is we're gonna go over motivation so deep learning is sorry typically requires a lot of labelled data in order to succeed and and label data is very expensive so that's one of the main motivations for this paper some of the less costly ways of applying improving deep learning are

Unsupervised Data Augmentation | AISC


using unlabeled data which is much more abundant and and easy cheaper to accumulate and data augmentation which basically stretches your supervised labeled samples further and as well are we good with the sound data augmentation has mostly been applied in the supervised setting and so we want to see if it can be applied in the unsupervised setting as well the main contributions which we'll get into are applying state-of-the-art data augmentation to semi-supervised learning a training technique called TSA training SiC leg

Unsupervised Data Augmentation | AISC


kneeling that effectively prevents overfitting when you have much more unsupervised data than supervised data and they they achieve performance improvements on multiple text and vision benchmarks and then they also introduce a method to even the prediction distributions over across classes for unlabeled and labeled data semi-supervised learning I hope people online know what this is I probably won't explain it again so we'll get into smoothness in forcing okay so this is one approach to semi-supervised learning

Unsupervised Data Augmentation | AISC


and the general idea here is you you try and regularize a models prediction to be less sensitive to small perturbations applied to to the the input data so that in potato can be label or unlabeled and when we say perturbations we're basically talking about adding some sort of noise to the inputs ampuls and so yeah you want your model to given a sample and an Augmented sample or perturbed sample you want the models predictions to be similar on both I think that's what I just said so enforce the predictions to be similar

Unsupervised Data Augmentation | AISC


and in general you want a good model should be invariant to small perturbations on this input data that don't actually change the nature of the example and yeah so data augmentation is a technique to boost your training data size and the diversity of it so the general idea is you're augmenting in some way again adding some noise to your input samples so that you cannot that you can both get more training data and have more diverse training data and I guess what we'll see some examples of what diversity means so yeah basically

Unsupervised Data Augmentation | AISC


you apply some sort of transformation you have a transformation function the apply to your input data and in data augmentation there's always this trade-off of diversity and validity that's being managed so so yeah you want to create novel and realistic training samples without augmenting them so much that you change their underlying inherent label so diversity it means growing the the reach of your data set or making your data set more broad and validity is making sure that you're not blowing up your samples so much that

Unsupervised Data Augmentation | AISC


they are no longer recognizable or they're not related to the label that they should have assigned any questions so far so this this is what supervised data augmentation looks like here so here Q is is our transformation function so you can see it's conditioned on an input example X and then X hat is the Augmented data sample and so so basically we're trying to minimize the log likelihood the negative log likelihood of the true ground source ground truth label which is y star given an Augmented sample X hat and and so

Unsupervised Data Augmentation | AISC


yeah you can see this as basically an additional training signal that's being sent to the objective function that that is hoping to yeah that is that is just using the the Augmented samples and then very similarly or actually slightly differently unsupervised data augmentation so this is when you have unsupervised unlabeled data this is a common way to to use that data for data on plantation you can basically take examine the output distribution prediction probability distributions for an unlabeled sample so here that's X and

Unsupervised Data Augmentation | AISC


then an Augmented unlabeled sample X hat again and and you've got the same transformation function Q that you had in the supervised setting and so really what you're trying to do is in this case minimize the divergence or the the difference between these two probability distributions so you're trying to normalize or regularize the predictions on the Augmented samples to have similar class distributions to to the unag mented unlabeled data does anyone have any questions here doesn't minimizing this minimize they

Unsupervised Data Augmentation | AISC


are therefore as well minimize this minimize the other in this case I'm sorry maybe I misspoke earlier in this case these are X's all labeled labeled samples here ordinary boys why is so why here is not the ground truth label y here so this is y star that's the ground truth annotated label and here we just have the the output prediction distribution or from the model for both the unlabeled and the Augmented unlabeled sample okay what's the difference oh yeah good question so theta it is implying that these are parameters that

Unsupervised Data Augmentation | AISC


are being updated so the gradient is passing through these I'm pretty sure that the idea with theta tilde is those parameters are frozen so they're they're not they're not updated in the objective function data here yeah yeah so this is the instead of thing yeah these would be these are two different settings but but yeah in both cases they're the model parameters question here is the transformation function so so the augmentation function basically the way that you're adjusting your input samples so the first version sees X's or actual

Unsupervised Data Augmentation | AISC


the data that you have that you have labels for in this one yeah so in sorry in this case the unsurprised a documentation you're assuming that you have so yeah I guess you can see here you've got a so in U and U is an unlabeled set whereas in the first one you have x and y star in in a labeled set so yeah there's no labels in this data at all so this this actual approach there are a few different ways you can do this this specific approach of using the KL divergence between the unlabeled or started the Augmented and

Unsupervised Data Augmentation | AISC


unadmitted was from a paper in 2018 VA t believe was the mall so so they borrowed that approach here and really the main difference they applied in this case is the the transformation function so what they did with Q and so that's what we're going to talk about now so this idea of targeted data augmentation so so over conventional methods such as adding Gaussian noise or affine transformations perturbations like that if there are a few advantages applying targeted augmentation so one is that they give a valid and realistic perturbation so so

Unsupervised Data Augmentation | AISC


the idea is when you apply some of these state-of-the-art data augmentation methods the the output augment example it is still very much in in the same distribution as the the sample it was transformed from so so these are sort of realistic augment augmented examples whereas when you just apply some random Gaussian noise it can often make the the data point if you apply too much make it sort of very unrecognizable and not realistic so so not really as we say valid it also applies a diverse perturbation so again if you want to use

Unsupervised Data Augmentation | AISC


those other methods just adding Gaussian noise you're usually not going to change you're not going to be able to change your input samples significantly enough and so you just end up with sort of local changes to the samples whereas with targeted augmentation you can really generate diverse diverse samples that are much more useful and in growing your training set and then as well we'll see what some of the methods they had a targeted targeted inductive bias so so yeah you can actually apply approaches that are optimized for the tasks that

Unsupervised Data Augmentation | AISC


you're that you're solving in the particular data set and so so we'll see we'll see an example of that in the augmentation strategies they applied yep so this is the the training set up that they applied so on the left hand side we have the labeled data and it's so split up to x and y star part of me x is fed in through m that's the model and then we just have the standard supervised cross entropy loss being calculated here that's feeding up into a final loss on the right hand side is where they take the unlabeled data and and they do two

Unsupervised Data Augmentation | AISC


things so so one they feed it through the same model and and then as well they take that sample the unlabeled sample and then they apply some augmentation to it so we'll get into all these different segmentation strategies so that results in X hat so this would be their their Q the transformation function so they get X hat and then they feed that through the model as well and then they take the the output of the model on X and on X hat and they feed that into the unsupervised loss function and that's the the KL

Unsupervised Data Augmentation | AISC


divergence that we saw here so basically they just take a weighted sum of the supervised and unsupervised loss and and combine those together question yeah what is supervised course and complete loss this is just the standard loss function for okay across the so you take the the ground truth label and and compare that to the output prediction probabilities yeah Joey so none of these data seem to have two tilde over it does that mean that oh yeah that's a great point I pretty sure they forgot to apply a theta on this this prediction

Unsupervised Data Augmentation | AISC


distribution so the idea is when you when the gradient propagates back through the through the model it will run through the supervised portion and it will also run through the Augmented application up here but but I'm pretty sure it's not flowing through just the unag mented unlabeled data good question I can't remember if there was a justification for that what's up and okay we're in this flow does the targeted or the diversity augmentation where does that come to play in this flow the target okay so so the idea is that based on the type of

Unsupervised Data Augmentation | AISC


input data that you have for the unlabeled data they'll apply a specific augmentation to that sample so so here they've listed the ones that they've used but the idea is they only use one at a time so based on the type of data you're using and the particular data set they'll apply a particular augmentation strategy so so I think in the next slides it'll show the particular policies they just they just sort of listed them all here on the left hand side is a leap of data so you have the ground truth the white star the right I

Unsupervised Data Augmentation | AISC


know if you don't have one right so the lesson1 the left hand side is your doing the prediction training and the right hand side you do the prediction infant yeah so so on on the right side yeah the loss function does look different than then on the left but in both cases you have you have a version of the model and with its parameters that that can be updated with signal from from the loss function so so in the unsupervised case this is the this is the function that is leading to the unsupervised or signal flowing through the unsupervised portion

Unsupervised Data Augmentation | AISC


yeah so there's just two different loss functions one looks like this and then the other is the center oh my oh my okay scheduling of training do they train all supervise first then on label or are they training all mixed together that's a great question and that relates to the big contribution that I mentioned earlier TSA the training signal and annealing so we'll probably wait until that two to explain that yes so for the unscrew right side it doesn't matter what the label and what the model what with the label is not about the model is

Unsupervised Data Augmentation | AISC


right like the lost isn't it yeah really what you care about here is that the the amount that you're changing or sample by is not differing significantly from the the same model prediction on the sorry the model prediction on the same unlabeled sample okay so so going into the augmentation strategies the first one we'll talk about is Auto and all so Auto augment learning augmentation strategies from data the general idea here is they have input data on the left and they basically have a model that will automatically search

Unsupervised Data Augmentation | AISC


through multiple different augmentation strategies so you can see here in policy one there they're making a transformation rotation to the input data the other ones are mostly changing the color of the samples and and so the idea is the the model will automatically select the policy that is adding the most novel signal to to the yeah to the training so for example if and they find that in different in different data sets different policies are optimal so in in one data set you might need to modify the color of your images a lot to get

Unsupervised Data Augmentation | AISC


more diversity and improve your training training set and in another you might need to rotate the images etc so basically it's a again coming back to the idea of being a targeted policy and this is something that you can vary on a task by task basis so even on a particular data set you can see what's the optimal augmentation approach any questions on this one so that was the one that they applied for vision and then they have to for text one is back translation so the general idea here it's pretty intuitive you take a question in one language in

Unsupervised Data Augmentation | AISC


their case they used English to French so they trained a machine translation model between those two languages then they translate the English sentence into French they then take that French translation from the model and they translate that back into English and and then they use that as the Augmented sample in the model so obviously any machine translation model is going to have some some loss and it's not going to exactly translate something the same way back and forth so in this case you can see a lot of it has stayed the same

Unsupervised Data Augmentation | AISC


this is Google Translate but but this word crankily was previously grin jingly and I think spoof spoof gets translated to tragic travesty so so this is one way that you can augment your samples and then the other one that they have for text is this tf-idf based word replacement so and they use this for text classification so so the idea here is sometimes in back translation the the Augmented transformation might actually miss translate some of the key words for that sample and in the classification tasks and so here they basically assign

Unsupervised Data Augmentation | AISC


an IDF score to each word in the in the sample and then they randomly sample or randomly swap out words and giving a higher likelihood to swap words that have a low IDF score so here you can see I've just sort of created this example and so in this case the words this in decides to etc are transformed or swapped but but the words that are a little bit more rare and therefore have a higher IDF score such as pathetically cringing lease poof those are more likely not to be swapped out and so that's yeah based on the intuition that certain keywords

Unsupervised Data Augmentation | AISC


sometimes are really useful for text classification any questions on the strategies those are the three okay so so now we come back to the question of how do we balance the need for having a large model so so when we're dealing with unlabeled data of which we have a really often you have much higher volume you have much more unlabeled data than say labeled data and so you generally would need a very large model to to train on that data so but but you may have a small amount of labeled data so you wanna the question is how to balance

Unsupervised Data Augmentation | AISC


a need for a large model while preventing overfitting at the same time and so they're their answer to that question is to gradually release the training signals of supervisors examples as the model is trained on more and more of the unsupervised examples so I'll show the the equations for for all this okay so here let's see so B here is the batch that is just a renormalization constant and the key is is this part over here so again this is sorry this is the objective function and and so really what they've introduced is

Unsupervised Data Augmentation | AISC


this portion on the right hand side where they say and this is all with regards to the labeled samples so n or what's this constant in Ada Adah yeah so so a dub G is is a threshold that varies with your training progress so we'll see that in the next slide but basically it's a threshold and and if the if the models prediction probability on the on the correct class for a labeled sample is above that threshold then then that this will evaluate to zero and therefore that sample signal will not get propagated through to the loss function at that

Unsupervised Data Augmentation | AISC


time and yeah so so the constant will will change over time but generally the idea here is is that one yeah when ADA is is small then you'll pretty much be rejecting most most signal from labeled samples from going to the loss function so you'll be preventing the model from overfitting on on say a small set of labeled data so that was the that was the reason they introduced this any questions question it is I the indicator function yes sorry I didn't say that I is the indicator function here and and and Zed is just to

Unsupervised Data Augmentation | AISC


rebalance or renormalize the effect from that during this rain ah ADA goes from it's always increasing okay so I'll show the schedule so they they introduce a few different schedules for for ADA so here's the equation for ADA on the right K is the number of classes in the classification example and lambda of T you can see in in the plot so lambda of T varies with the training progress and and as lambda of T increases or sorry as training progress increases lambda increases and as well the threshold also increases so at the beginning lambda

Unsupervised Data Augmentation | AISC


will be zero and ADA will be one over K so 1 over K is 1 over the number of classes which is just the random chance of predicting a sample and then at the end ADA will be 1 and so coming back to this equation if ADA is 1 that means every single sample practically will be will be carry forward towards the the loss function and and at the beginning when it's 1 over K only the predictions that only the samples that the model is very unconfident on will actually be used in the loss function and so so the idea here the intuition is that if you

Unsupervised Data Augmentation | AISC


have a small labelled set so you have only a few examples that have labelled labels you want to avoid your model overfitting on those at the beginning of the training so they suggest using this exponential schedule for for the case where you have a low number of labeled examples and so yeah the idea is at the beginning you you won't be feeding as much signal from those samples to your loss function but by the end you can start releasing more and more of it once more of the unlabeled data has been incorporated

Unsupervised Data Augmentation | AISC


and then conversely if you have say a large number of labeled samples then you can use a log schedule the green line here and so that will release a lot of the supervised signal at the beginning and less at the end any questions on this think that's it before the break so we'll just take a 5-minute break and then afterwards we'll go over the results and have some discussion come back people online the sound is working sounds good okay great so we're gonna do a five minute break now we just did a five minute break

Unsupervised Data Augmentation | AISC


we're gonna do experiments so basically they they applied this method to two different types of tasks Texas classification and then image recognition so a couple of vision benchmarks these are the actual data sets that they used so there's a mixture of binary and five class text classification most of it is sentiment I except for dbpedia which is I believe categories and I think dbpedia actually has fourteen classes or ten classes the two image benchmarks both have ten classes they also use image net they test on

Unsupervised Data Augmentation | AISC


image net and they do some ablation studies for TSA and for the targeted augmentation so for text classification and the settings for the labeled data so the goal here was to use a small number of labeled samples so for binary classification that means twenty just twenty labeled samples and the rest of the data coming from the unlabeled and for the five class classification they found they needed to use a bit more so here they use twenty five hundred total samples and and so that's five hundred per class so when speaking to the author

Unsupervised Data Augmentation | AISC


the goal here was to find out how low could they go so I was asking him did you experiment or how did you stumble across these numbers and really they wanted to determine how few examples they could labeled examples they could keep while still achieving really strong results from the unlabeled data for IMDB which is one of the the binary classification they use the concatenation of the training set that they didn't use so the training data that was not used for super as supervised they use that as unlabeled and then they use the rest of

Unsupervised Data Augmentation | AISC


the unlabeled set so I believe the total training set is 25,000 and the unlabeled set is about 50,000 so that's how much unlabeled data they're using there and for Yelp and Amazon they obtained some really large review datasets and I believe there was something like 6 million samples in in the unlabeled and so for the most part again another choice was that I think for the most part they used one Augmented sample per unlabeled sample so so but but he said that that would be something that might be a task specific parameter and that

Unsupervised Data Augmentation | AISC


you could adjust so for some tasks for each unlabeled sample you might want to make a couple of augmentations and use both and as for the model they try a few different initialization schemes so all are working on the transformer architecture applied in Beart so they have just a random initialization then they have bird base bird large and Bert large fine-tuned on on the unlabeled data the in domain on little data so the same unlabeled data that they're using and and for each setting they compare the performance for each of these

Unsupervised Data Augmentation | AISC


settings with and without the unsupervised data augmentation method UDA so here are the results for for the text benchmarks so you can see at the top the data set name and then below that the number of supervisors examples that exist in that data set the the top two results are the pre bert state-of-the-art so that's that's how influential Burt was there's a before bird and after bird so they report both in some cases Burt was better than the state of the art in other cases I I guess it's always okay and then their

Unsupervised Data Augmentation | AISC


results the results from their paper are in the bottom in the semi-supervised setting again they've got the different initialization strategies on the on the left and you da X or check indicates whether or not they use u da and and so these are error rates here so lower is better and below the the data set name in the bottom you can see the number of labeled samples that they used so for IMDB they literally only use 20 examples and and when they apply when they use an initialization from fine-tuned Burt model they're they're

Unsupervised Data Augmentation | AISC


able to beat the state of the art and the pre bird state of the art so that literally means they're just using 20 labeled examples along with Burt which is fine-tuned on unlabeled data and and then they use augmented samples from the unlabeled unlabeled data set so that's that's one of the most significant improvements and they found across the board they they got very close it or actually beat the the the state of the art for these tasks I think the one that they found the most difficult and they perform the the worst on was the de 5

Unsupervised Data Augmentation | AISC


class classification for both Yelp and Amazon you can see that their results are still a bit a bit off from from the baselines so yeah a really big finding here is that obviously just with using Burt and know you da you still get 6.5 error with just 20 labeled examples so a lot of this is just indicating how much information is being contained in Burt but but it's quite significant that this shows that you can use pre-trained language models along with UDA so it's it can be complimentary to pre-trained language district models yeah that's

Unsupervised Data Augmentation | AISC


right on the 20 labeled samples plus fine tuning for the last case but but yeah that's it yeah they do they do do some studies on the vision benchmarks that investigate whether or not the whether or not doing augmentation on the unlabeled data is actually more advantageous than doing it on the labeled data so but they don't think they did that for for this so yeah it could be a good unless I'm wrong that could be a good feedback they might have their I should say that they did do something speaking with you out there

Unsupervised Data Augmentation | AISC


they did do other experiments that they didn't report here and I think I might have asked about that I just can't remember if if that was something they did but if they did it wasn't as as performant so for the vision benchmarks yeah so they wanted to use the the same model that was used for prior semi-supervised work and this was the wide residual networks with depth 28 and with 2 and and so yeah they use the same exact labeled samples that Auto augment use to find its optimal policy so for C far 10 which is a 10 10 Way image

Unsupervised Data Augmentation | AISC


classification task they have 4000 samples and for this is Street View house numbers it's a digit recognition data set they used a thousand labeled examples and so they do 10 10 runs with this model and and calculate the average and the standard deviation so here are the results here these on the left you see the the fully supervised setting at the top so with no augmentation and then following that you see different different augmentation techniques that that that are applied to to the unlabeled data so I think for I can't

Unsupervised Data Augmentation | AISC


remember how many samples but there are roughly maybe 50,000 unlabeled samples for 4c far 10 yeah I'm not sure of the numbers right now but I think you might remember earlier I mentioned V 80 which was the the paper that they got the the idea to take the KL divergence between the distribution of unlabeled and an Augmented unlabeled sample so really what you're seeing here so obviously they performed the the previous state of the art method but between UDA and v80 the only real difference is the perturbation or

Unsupervised Data Augmentation | AISC


transformation function that they applied since they're both using the same KL divergence technique and so so this is indicating that that targeted data augmentation strategy was was helpful so is there any metric or measure that we can look up into in order to see like they are saturated and why not using hundred-thousand right so we would say morning that's better but there's a point of a saturate there like the diversity that they don't were like the distribution or being the new thing that you're defining in our data we are

Unsupervised Data Augmentation | AISC


like many in the data which doesn't matter after that right so for example Wendy chose to see a thousand there's a reason for that certain metric for us to see that's level oh you know I don't think they provide our metric there might be a good discussion for at the end though when we get to the discussion points like so so yeah keep that out of minds but but as far as I remember there wasn't anything provided Hockman Auto admins will find the optimal policy but I didn't read the paper in depth I'm not sure if it

Unsupervised Data Augmentation | AISC


optimizes anything else such as the number of samples as far as I know it's just finding the optimal policy for a particular data set the optimal augmentation transformation and okay so they also perform some experiments on imagenet and the the motivation here was they the first the initial data sets they use all had between two and ten classes so they wanted to and and and they all had a low number of supervised examples four thousand or less so they wanted to use a data set that had a hot much higher number of classes it's a bit

Unsupervised Data Augmentation | AISC


of a harder task and much more supervised examples and see if this approach was still applicable or if it was really only sort of a niche improvement for for smaller data sets and then they also wanted to see if they could make use of out of domain unlabeled data that had different class distributions so keep in mind in all of the previous examples the unlabeled data sets that they were using largely were just the the actual labeled data without the labels being used so so it's very much in domain literally samples that

Unsupervised Data Augmentation | AISC


have a true label they just didn't tell the model what the label was so they wanted to see if this would be applicable if you didn't have yeah if your labels coming from out of domain and so imagenet overall has almost 1.3 million images and and about a thousand different classes so the settings they do a couple of settings so one is image they call image net 10% and this is where they they take roughly 10% of all image net data and use that as labeled samples and they use all of the the rest of image net as unlabeled data so ten

Unsupervised Data Augmentation | AISC


percent would be I guess one hundred and thirty thousand samples and and then they would have over a million unlabeled or about a million and then the the other one is the fully supervised scenario so this is where they use the entire image net data is supervised data and they obtain extra unlabeled data from data set called jft it's another image I believe it was automatically generated image data set so they essentially train a model on image net and they use that model to to source out the most relevant samples from the jft

Unsupervised Data Augmentation | AISC


data set for each class in the image net so they basically for each class they take the I think it was thirteen hundred most most relevant examples from jft and they and they use that as the unlabeled set for for their experiment any questions okay the baseline model that they used was ResNet fifty here so they did encounter some some errors some issues with with image net so they observed that they had flat class distributions so the prediction probability distributions across classes was pretty flat or uniform for the

Unsupervised Data Augmentation | AISC


unlabeled and the Augmented unlabeled samples so there really wasn't much signal coming through from from the unlabeled part of the the training set up and yeah so there's there's a probably to do with the fact that there are so many more classes and and there's also so much supervised data available here actually sorry no but the amount of supervised data wasn't an issue I think this was more of an issue even for the image net 10% so where they only had 10% of the training data as supervised this was this was an even larger issue so

Unsupervised Data Augmentation | AISC


this led to the unsupervised training signal being pretty much dominated by the the supervised signal and so that their solution they another one of the contributions I mentioned in the beginning was to sharpen in a few different ways the prediction the predicted distribution produced on unlabeled samples so that there would be basically to encourage the model to use the training signal from from the unlabeled samples so the specific techniques that they use were entropy minimization so they added an over an entropy term to

Unsupervised Data Augmentation | AISC


the overall objective to regularize the predicted distribution on the Augmented examples to have a low entropy so again to discourage these uniform distributions of probabilities they also did soft max temperature control and here this is to control the temperature of soft max when they're computing the prediction on the original example and then confidence based masking so this was where they basically removed any samples that the unlabeled samples that the model was not very confident on so all of these approaches were were to try

Unsupervised Data Augmentation | AISC


and sharpen the probability distribution of the on the unlabeled Augmented samples for these sessions but the car if the actual distribution of your training data your related training data is uniform and then you try to sort of force it to take a different shape on your unlabeled a of what we shouldn't that prepared for that why should that work so the the number of samples is uniform across the classes but here this is this is saying the problem was that the the models predictions prediction probabilities across the different

Unsupervised Data Augmentation | AISC


classes for an individual example were more uniform so so the example has some category let's say it includes a giraffe so that should be the true label but the actual output prediction distribution was was pretty no uniform so they wanted to find ways to encourage it to be sharper to - yeah to be less less prone to just being killed divergence per batch no I believe it would be calculated per sample pretty sure it would be each sample would look at the figure so their first average error and then the unsupervised error and then it

Unsupervised Data Augmentation | AISC


will do back propagation right hmm I did do it like separately but the way I add necessarily do it all together so the mass calculation kind of patchwork L the Virgin's okay yeah so I suppose that they do these are the results here that the left is the image net 10% again with the baseline its resonant 50 and the right is is the fully supervised image net setting and top 1% is the the models accuracy on for its first top prediction top 5 is its accuracy looking within the first 5 samples and and yes so you can

Unsupervised Data Augmentation | AISC


see that it improved on on the baselines in both cases I guess the smaller improvement for the fully supervised setting where they're using an additional 1.3 million samples from jft yeah over the the previous Auto augment policy I think for image net 10 was it for this one yeah I think they also so someone was asking earlier about why not using the why not use the labeled examples for augmentation for the baseline and I think they also did run that experiment here so they they used the 10% labeled examples as for

Unsupervised Data Augmentation | AISC


augmentation and so that I think got something like 58 accuracy so still significantly below and yeah moving on to the ablation studies so they want to do they did a couple the first one is for TSA so to determine if this training signal annealing actually made a difference and so they did it for Yelp 5 and see far 10 in the Yelp case they didn't use Bart pre-training just to really make sure that there was all the information was coming from from the data use and not from the pre train distribution or language model

Unsupervised Data Augmentation | AISC


and yes so you can see in the first case where there's an X that's where they're not using any tsa and so you can see in both cases applying some schedule does improve the results and you can also see that it's it's slightly different for the two cases and I think for CFR 10 there were 4,000 examples and so that's optimal with the linear schedule and Yelp five was about 2,500 and so that's that's one of the lower data settings that are that they hypothesized would be best with the exponential schedule and then the other one was for a question

Unsupervised Data Augmentation | AISC


yes you say race when TSA work for pure supervised learning that's a good question I'm not sure I don't think there's a clear reason that it wouldn't seems like you could apply it so the idea would be that you would [Music] yeah so remove yeah I don't know if there would be as much motivation to do that when you're not incorporating additional data but but that's a good question maybe someone in the discussion will have some thoughts and for for the actual data augmentation policy so they wanted to see here did the targeted data

Unsupervised Data Augmentation | AISC


augmentation that they applied actually make a difference so they have the two image data sets here and some on the left you can see different augmentation strategies Auto augment was they what the one they actually used and switched augment implies they they took the optimal policy for from one data set and applied that on the other data set so you can see that it really does perform best when you take the optimal augmentation policy the most targeted policy and and apply that to the data set I think that's mostly it just some

Unsupervised Data Augmentation | AISC


conclusions so in summary they introduced a method called UTA unsupervised state augmentation that used targeted data augmentation to generate diverse and realistic perturbations to input samples and so this enforces the model to be smooth the way they applied it with respect to these perturbations they also introduced TSA training signal annealing and this was to prevent UDA from overfitting when a lot more unlabeled data is available compared to the amount of labeled data and their results showed that they achieved state-of-the-art on

Unsupervised Data Augmentation | AISC


IMDb with only 20 labeled examples and they were able to reduce vision error rates by more than 30 percent in the in the semi supervised setting and they were also able to leverage unlabeled data to improve performance on on image net even though what's a maybe a small improvement and so some overall findings are that data augmentation and semi-supervised learning are well-connected topics that should be explored further and you da+ unsupervised representation learning in this case Bert can compliment each other

Unsupervised Data Augmentation | AISC


and work work well together and so that's that's the presentation we just have some discussion points over class that has weight less data right someone was asking about that earlier I don't think no they didn't do that they specifically in the case where they were using image net and they were boosting from the the jft data set they specifically took an even number of samples across all the classes from jft and I don't think there was any case where they had an imbalance across classes oh yeah that would be an

Unsupervised Data Augmentation | AISC


interesting problem to solve for sure okay so fluorine and I or fluorine and the the author so TJ had had one discussion point that he wanted to to pose to the audience which was so to some extent this type of approach the ability to use more on labelled data can can help to make machine learning more University accessible because it's much more affordable to train models on unlabeled data and so what applications do you guys think stand to benefit the most from from semi-supervised learning that that haven't already been tried in

Unsupervised Data Augmentation | AISC


this paper big problem is you have like a cooking video where you want to get the recipe use automatic machine translation to get to rest there are like small data sets available and it's a very complex problem because of just how recipes are organized right so if you have a small training data that can really work there are tons of unlabeled data for like videos for recipes right hey there's a gift recip so whatever where you can get large amount of data very easy it's just everything sound a little hmm so so video yeah yeah

Unsupervised Data Augmentation | AISC


interesting take a video and you just make a description of what is happening in the video right yeah I don't know if that's been and tried or or not and then I think so as well what are some other targeted augmentation strategies for text so they introduce back translation and this word replacement I'm sorry they didn't introduce those those those have already existed at least the first one are there any other strategies for for text augmentation that anyone's used or think could be could be valuable is UI regression when you're trying to compare

Unsupervised Data Augmentation | AISC


your interface with a new interface and you don't want to label everything and in the test cycle you want it to be on labeled potentially measures of potential here again you are it hasn't gained much traction because you'd have to label every part of that that's not enough for agile so there was some and so part of the reason that I asked this question is I felt like the approaches for text augmentations still felt like they were may be most useful for for a text classification case but they're there many other case many other tasks

Unsupervised Data Augmentation | AISC


to solve in in NLP and and so I think that they said back translation has been shown to be successful for other tasks but I think that the word replacement one might not be as relevant so I was curious if anyone had had tried any others and then Florian I think you had a few here so so yeah what are some other ways to automatically optimize data augmentation strategies the idea here instead you see they have like some kind of targeted augmentation strategy right but instead of having some kind of time use augmentation start sheet let's put in

Unsupervised Data Augmentation | AISC


kind of like a neural network right so I kind of like again and then at the same time you optimize scan to make the best possible kind of transformations right so the kind of the gang would be optimized to help the other neural network to get to better care older versions right I wouldn't know if doesn't make sense so it would work this is beginning like if you want to have minimum corralled emergency we just do the identity right so no but I think is an idea I can throw in a room koc is like some kind of variation of attack

Unsupervised Data Augmentation | AISC


order right it doesn't need to be again some generative model to come it with a policy yeah I think it ties in with some of the other points so you're asking you thinking about should we compare UDA or combine it with automatically generated data sets this is like an active field of research as well yes so so data sets that are completely fabricated without any annotation is this something that competes head-to-head like it's is are they strictly one or the other or can it be combined and another food for thought

Unsupervised Data Augmentation | AISC


is will in the long run UDA be a cost effective alternative to to cheap manual labeling so if if manual labeling is I think you throw out a number like 140 per hour with yeah we're pretty us us 140 US per hour will this actually be a cost alternative method if if the costs are that low this type of approach would face and that's some Angela data privacy I'm not sure if there's an easy easy way to build on something like differential privacy or something like that in order to ensure that when we're gobbling out unlabeled

Unsupervised Data Augmentation | AISC


data that we're not you know accidentally throwing in user data that might even wonder like maybe they could be picked him over let go malicious attacks rates so something to keep in mind with one applying this method the network model is still located present data but you just don't have the neighbor yeah so if we have all the customer data right and you want to do something you should I mean you still have all the Congress customer looks at all the customers who change much and you just doesn't have some kind of

Unsupervised Data Augmentation | AISC


decision [Music] we applied like we were saying the case about the credit scores so like maybe there's a lot to inconsistency in like the ones who retro actively got approved for credit so like there might be the human error inconsistency or just like the fact that there is self selection or something so like this could kind of help with that right like you're saying there are some people who maybe fit the traits of being approved but then they just somehow didn't get approved so that they're actually being not approved

Unsupervised Data Augmentation | AISC


right yeah there's some there's some bias of some sort in the labeling process the annotation process then definitely that could mean that the the samples in your unlabeled set are valid but but just not being included for one reason or another so I think yeah those kind of cases are really good candidates for for this kind of work any other thoughts or questions in case of a supervisor provide sort of clustering job and using the sort of the model only on the label to maybe if you have to do a job I'm very nervous very rare that said could you

Unsupervised Data Augmentation | AISC


actually refers to a cluster job on a much smaller data set and use those as the label and service supervisor set and apply this for the classes of all the others and see if those classes would actually fit in the initial foster class and whether it be actually faster than doing the big massive across from John just okay so so the idea would be to take a subset of the unlabeled data you're strictly unlabeled data take a subset train a clustering model on that and use the clusters as ground truth labels that sounds to me like a

Unsupervised Data Augmentation | AISC


cool way of starting from scratch like you have no labels I think that's been that's been done before but in conjuncture with with this type of approach would be interesting yeah I don't see why you couldn't do it I think you might you might find that I don't know how well it would perform yeah as opposed to just doing all once I have like one criticism about this paper I only skimmed it and I wasn't able to follow your entire presentation so crab me if I'm wrong but what he liked it seemed they called unsupervised data

Unsupervised Data Augmentation | AISC


augmentation but it seems that like especially in the case of text they use the model to translate and translate back and there's a very right it's a it's a separate model they train a separate machine translation model yeah to do the back translation but they do utilize that model to to port data augmentation right they use that machine translation model just to generate just to generate the Augmented samples oh yes yeah so I guess my criticism is that they say it's unsupervised but this is not exactly distillation but there is

Unsupervised Data Augmentation | AISC


some kind of because that translation translation back that's supervised and there is some a lot of like that kind of signal that's kind of being learned and distilled by doing this approach even the tf-idf frequency approach so there's a very strong prior that you built into the into the augmentation part so this doesn't feel like you see like exactly unsupervised in the sense that you just give it a lot of data but there's some kind of implicit supervision I don't think it exactly distillation but some something there is some minute I think

Unsupervised Data Augmentation | AISC


it's acknowledged they call it I mean there isn't a language of targeted data augmentation so you're you're picking an augmentation strategy that is targeted to the particular task and in some case to the data set there you're working on so I think it's not it's not an unacknowledged part of the paper and like they are it's it's yeah on purpose but they're doing that way nonetheless I do agree that I don't know how well some of those that was my question here how some of those augmentation strategies would apply to other for example text

Unsupervised Data Augmentation | AISC


tasks so so yeah it remains to be seen how much work you would have to do to come up with an augmentation policy that is successful for a particular task and data set I think that would probably be very interesting future work something to add on to that something I think would be interesting is what happens if you have a worse translated back translation because you're adding in creative for the patient to it right but does your unit become better because you have worse back translation for that oh if you if you if your machine

Unsupervised Data Augmentation | AISC


translation all was less accurate yeah yeah I think that comes back to the trade-off between diversity and validity if your translation model is terrible then the actual sample that you generate is probably not going to be a valid sample for the label that it's supposed to be assigned to but if you don't change it enough then you're not really adding any new signal so so yeah there's some inherent trade-off there I think so yeah maybe using a slightly worse translation model would be effective definitely not using a terrible one but

Unsupervised Data Augmentation | AISC


yeah it's a good point the other scenario and that's something the various amid sedimentation so that's means they had a like the jury instructed the different level of the target from the background so in that of Seven AO how can we use the you dated to did up automatic when you have different social services for image segmentation yeah object detection or yeah because this scenario is one Operator for the pilot but the other some cannot do you know network about the model the end is a the outputs will be the images

Unsupervised Data Augmentation | AISC


most sorry at the end the output of the ornamentation yeah hmm right so that's a good question I'm not too sure I'm sure there's been work done on augmentation for object detection and and segmentation but I'm not I'm not too familiar with that work yeah so I don't know exactly how its applied settings is that the question is having work before you leveled the the trailer the car view yeah yeah but for this one you it's it's a it's a prepared via the distribution of the white and something else buddy in the sedimentation but how we prepare of

Unsupervised Data Augmentation | AISC


the year I mean I think you can still some augmentations are pretty clearly still valid if you just changed the color for example the object detection is still going to be valid you're not transforming or rotating removing anything outside of a box so I think that would that's one approach you could take and and I think color augmentation was one of the strategies that that they used for the image data sets so maybe we'll cap it there thanks very much for your attention there's some pendeks that's all and that's it

Unsupervised Data Augmentation | AISC


I kept trying to get this secret to work for me for such a long time and it just wouldn't work until I changed one thing and I just want to show you what that one thing is are you ready let's go when it did work what happened is I manifested this no I'm only joking watch this because actually what I manifest was much more than just a five thousand dollar massage chair aside from a $20,000 watch I don't know if you can see out here and then this all became my lifestyle alongside that beautiful little boat there now your

Why 'The Secret' Won’t Work For You Until You Do This.. [Law of Attraction]


question is why am i showing all of this to you and what a show-off who is this guy anyway I just want to break it all down for you because it's not about me and I showed you all of this stuff it's because I want you to know that it is possible one of the biggest things is that we need to believe before we can manifest and that's the biggest problem that a lot of people don't know how to get so I want to break it down for you into three simple steps in this video how to actually manifest and attract what you like what you desire in life


using the law of attraction or the principles from the secret but before we get into it so real hit it Bob today isn't that they're struggling it's not they can't get out of this it's because the that they're in isn't big enough and why I say that is because if that was big enough right now if the people had a girl acquaintance or that head they would actually get that ass moving always understand that emotions is what these emotion is energy in motion so if your emotion or drive is not stronger you're not gonna do anything to get out of it


what's up guys this era coherence national speaker entrepreneur and best-selling author and in this video I'll break down for you the three keys to make the secret work for you finally while I relax on this chair love it so much so we're gonna get straight into it the first thing using the law of attraction is we must visualize and this what the secret talks about but you know just visualizing alone is really not going to be able to allow the manifestation to work for you the biggest problem is is because there's a


lot of that goes on through the day so let's say for example you spend I don't know three minutes five minutes visualizing in the morning the rest of the day you're visualizing something else so what that means is it kind of like puts us in a position where we're manifesting everything that we don't want and the small time and energy that we put into what we do want isn't manifesting first so what is the solution the solution is to understand a step number two and step number two is the principle of the law of attraction


we have Canada here say hi the principle of the law of attraction is to understand that it's all based around emotions the more you are able to magnify your emotions what happens is a speeds the whole law of attraction of but aside from that it makes it much much more powerful because vibrations if you think about it when you are very very emotionally charged up your vibration is kind of raised to a crazy crazy crazy vibrational frequency and the more crazy it is the more it sets it in there so even if you just do it for


three minutes five minutes in the morning hold stronger vibrations than your whole entire day so it's all about the second part raising those emotions and really getting into the group so step number one is to actually visualize that number two is to really amplify the emotions as if you're living it right now and finally step number three Aires with that step number three is the most important part at all which is to let go now you're thinking what do you mean like let go I think a lot of people recently have been getting really really


confused when I say set it and forget it that's what I write about in my book they say Eric if I said it when do I said it how long do I sell it for when do I let it go what's the balance between setting it and forgetting it well actually you said it early on in the morning when you're most at peace the rest of the time you've got to have trust I was doing this call yesterday for the whole superphone thing I know a lot of you a lot of you have been texting me over the last couple of days and I am receiving your messages by the


way it's just like there's thousands and thousands of thousands of requests coming in so it's very hard to actually reply to you all however one of the biggest questions that I keep getting from a lot of you is when you said it when do you forget it and also how do you actually trust the universe what if you have a lot of doubt happening within you because if there's doubt it's very hard to make the Law of Attraction work for you so the whole idea of it what I was teaching yesterday during the live call was that you need to get this into


your head and I want you to type this in the comments below those of you who are new to this channel we always type our learnings below to reaffirm the learning Who am I not to trust I've run my whole life based on this very very simple concept Who am I not to trust if you understand we were all created by something that's greater than us much powerful force a much magical force you can call it God you can name it the universe you can name it whatever you want to name it but we can't deny that thing exists now if that thing exists


who are we not to trust it has the decision to put you into this world when it wants to and it also has the decision to take you away and sometimes think about it how many times throughout your life so far have you felt totally out of control you don't know how to handle it you don't know how to deal with the situation and you felt so so challenged and you were just stuck but somehow you pulled through and that magic happened so if it's ever happened to you before whether in relationships and business and finances whatever it is then if nothing


actually happened before who are you not to trust because it always always the universe always always has your back and if you understand that and have that peace of mind then when you have that peace of mind everything starts shifting and Wow we start shifting we don't need to play your song I'm gonna play your song it's beautiful so I'm ready I made it for too long have you played [Music] a prize for whoever guesses what the song is called you can comment below anyway guys I'm gonna finish off for today because we have to go up to


Newcastle with ah okay you know where are you because today is dad's birthday so I'm a blog along the way I don't know it depends if we have enough time to do so but anyway guys if this video has been of any use to you whatsoever you know what to do hit that thumbs up hit the like button also comment below let us know where you guys have tuned in from and finally if you're new to this channel and have been any used to you remember remember to hit the subscribe button and the notifications button next year because every single day I'm


Germany's super-rich no other European country has as many billionaires and while their fortunes are growing more and more Germans are living under the poverty line set by the Organisation for Economic Cooperation and Development the OECD the press frequently reports on the country's high income inequality and low social mobility but little is known about the super-rich nate'd sure money attracts success and success attracts money I do believe that I've experienced myself you can suddenly connect with people

Germany: The discreet lives of the super rich | DW Documentary


previously out of reach who are the kind of shores that had - who are Germany's super wealthy how do they live and how do they see the country they live in the way wealthy people in Germany are talked about bothers me because it's sensationalizing in sycophantic and in no way reflects what wealthy people have done and continue to do for this country it creates this impression of rich people being like Scrooge McDuck that they had these money bins in which they wallow in their coins look Scrooge McDuck wants more more MORE that's not


my world I like earning money is less but not swimming in it we thought we wanted to get closer to the discrete world of Germany's ultra rich of company owners and heirs worth millions we wanted to find out what makes those on top of the world tick [Music] every year an exclusive event takes place in the Schloss hotel Kornberg near frankfurt to which the public is not invited it's the annual Hall of Fame evening for the business monthly manager magazine hardly any other occasion in Germany draws as many wealthy business


owners [Music] the Iggy had laudatory esteemed presenters dear jury members dear ladies and gentlemen welcome to manager magazine's Hall of faith when we first founded the Hall of Fame in 1992 we wanted to take a stand for excellence and unconditional entrepreneurship and against faintheartedness and averageness we begin our nominations today with Harv dama moot his company United Internet is valued at around 11 billion and on our list of Germany's richest people he ranks 25th with a personal wealth of 4.5 billion euros


collectively the guests of this elegant evening are worth billions of euros this is the face of wealth in Germany mainly male and although we're close to the super-rich here their world remains somehow out of reach Hamburg home to manager magazine's parent company the speaker group hockey any publication keeps closer tabs on Germany's ultra rich every year the magazine's team gathers information on wealthy Germans and using the Forbes model makes a special edition with a list of Germany's 1001 richest people it's painstaking


work how many billionaires are video last year it was a hundred and thirty six right no last year it was around 170 170 okay someone who wants to get on our list at 1001 richest Germans needs to have around 100 million that doesn't have to be money in the bank most people have it as assets or as property but that's the ballpark we're looking for to be on our list of the richest Germans the thousand vison contant so nice and our chances editor-in-chief Stephon clothespin has been around Germany's ultra-wealthy for


years what does it take to get on the list as it Amazon the top I'd say that the top 150 spots on our list will always go to company owners and heirs even if you're a chief physician you'll need to see a whole lot of patients to become a billionaire managers also have a hard time getting that high up here in Germany there's a debate about whether the heads of Dax companies earn too much but if you compare it to what people in similar posts in the US the UK or China earn its peanuts so that alone can't ever make you one of the truly ultra


rich in them it's a club des vins a new chef creating a list of the wealthiest people is especially difficult in Germany though not because they aren't enough of them [Music] he somehow managed to amass this huge Empire and fortune in just a few years his wealth is estimated to be around four billion he's the least known super-rich Germans would you let us interview him at home they prefer to fly under the radar to avoid envy the super-rich don't like to melt themselves so to speak you can normally only get interviews inside the homes


it's second-tier once there you'll always get guys who get a kick out of publicity like mr. Marsh - and such getting a good shot of mr. Marsh Mayer is never a problem but the real money keeps itself hidden the super-rich are trying to go unnoticed sometimes they even try to hide there are no photos of several people on our list you won't find a single picture if you go online and google them there hasn't been a photo of their hymens germany's richest family for decades many stay hidden because they want to live normal lives


and think that they won't be able to do so if they're known to be multimillionaires Lemkin for months our interview requests were rejected agreed film shoots were canceled last minute none of the rich wanted to talk to us about money finally we got lucky in the financial hub Frankfurt here in a prime location tucked behind the Bank towers sits the asset management company for come Foca managers German business family's fortunes its chairman Christian fanbase Todd's home provided some insight as to why rich Germans are so shy


the joy many wealthy Germans are reluctant about stepping out into the public eye because they're afraid they could be seen negatively they ask themselves what do I get from showing myself to the public it doesn't give me anything on the contrary it could lead to some crazy person taking note and breaking into my home or kidnapping one of my children and those fears are not unwarranted then there's also the fact is that many heirs are inheriting fortunes that are somehow tainted by or related to the Third Reich the Abbott


ear of rich Germans avoids publicity as if it were the plague but what are they so afraid of if I asked a number of my friends whether they'd like to be interviewed for this film and each one said no they'd say someone else can do that better than me I can't do it right and I might come across wrong they think they'd have much more to lose than to gain even calm [Music] after a lot of back and forth with his press team one ultra-rich German did agree to meet us Michele Otto is the chairman of the Supervisory Board of the Auto Group and


one of the ten richest Germans we asked him why Germans are reluctant to show off their wealth any wiring doing so would lead to envy in the u.s. achievement and wealth has seen in a highly positive way but here they carry a bitter aftertaste where does his fortune come from or how did he get his wealth that scares some artists whom with a shrine weren't you I think a lot of people find their own wealth a little nauseating I like it Pindos good dick Hoffman grew up poor his mother ran a small drugstore in post-war Hanover her


son had bigger plans in 1972 the idea of opening the first self-service drugstore in Germany came to him today he's a multi-billionaire why was he happy to step into the spotlight negan unfunctional in the early years it was just about getting the name rossmann out there so when I was invited to a talk show on a small regional channel I liked going because I thought free publicity for my company but then two or three years ago I started to understand that this slightly flabby balding man whose teeth aren't perfect was hungry for recognition


himself back then I thought I was stepping into the limelight to promote the company but everyone rationalizes their motives and I did too y'all stop adore within 40 years horsemen became the most profitable drug store chain in Europe with stores in six countries in 2018 a total of 56 thousand people were working on the chain Hoffman is active in other business areas as well and he speculates on the stock market that could not take any capitai legal debt so I have a couple of private equity investments the largest is palliated between 80 and 100


million depending on the spot rate so it's a fair sum there are also shares in different industries introduced a mob will be boring it's off to tear a little I'm fine you're still referred to as an SME because small medium enterprise right why do you think that so I don't really know with 55,000 employees you're not really a medium enterprise anymore you're in another league we're back at Manager magazine in Hamburg with photos with a special issue are being selected the list of the richest Germans includes a notably high


number of company owners from the so called medium-sized businesses it's a German peculiarity and it's not the only one the cover story for this current issue is that last year money rained down on Germany's super-rich up here we have the chef Lutz the matriarch and her son who now owns 80% of mom has kept only 20 on the chef's loose have been at the top of our list for many many years I'd have to check exactly how much they're worth but around 20 billion give or take a bit down here we have MS Basel toss in the head of the Hinkle class


she's the first and so far only woman chairman of the Supervisory Board of one of the 30 Dax companies Germany's economy is still extremely male-dominated what's interesting is if you compare our list here with a list of the ultra-rich in the US we have a lot of old money old companies that have been around for decades in the US you have all those lads from Facebook Google snapchat and so forth that have bubbled up to the top of the list we don't have that type of thing here and compared with other countries Germans are very


reticent about showing their wealth very few Germans sail around in boats like this usually that's Americans Russians Chinese and so on here you don't really show your money you might have various houses villas and such but there's likely to be a Fulks park and parked out front it seems the average ultra wealthy German Israeli conspicuous unlike in the US athletes actors and TV personalities rarely make it onto the German list important on guns even though we put a great deal of love in a sweat into estimating guests fortunes they're


probably much bigger especially if we're talking about urban real estate prices have exploded over the past 10 years a lot of people have doubled their property assets so if you started with five billion in that market you're likely to have ten or fifteen billion today money makes money but while rich Germans fortunes have exploded since the financial crisis due to the increase in value of real estate stocks and assets those with average incomes have had to swallow losses even liberal economics Institute's are concerned about social


inequality in Germany how do the super-rich see this disparity the headline in the newspaper reads the rich are getting richer and richer which is true the rich are getting richer yes it's true but it's also false there are 20 million citizens in Germany who have assets worth between 100,000 and 1 million so millions of people are getting richer now the rich are getting richer even faster because one factor is probability that they can dedicate much more time to increasing their wealth about initiative the Left Party would likely say split up


your wealth but my response is I also do things for the world which I live in I don't just take I'm not a socialist though I can only do things because I have things I can y'know Bastogne Posada Michel Otto is one of the rich whose wealth has increased he successfully transitioned his mail-order business into a digital enterprise over 20 companies now belong to the Auto Group we wanted to know how Otto sees the debates about rich and poor do the rich understand the worries of the poor in the tough one just one good when people


talk about those on top who don't understand those at the bottom I wouldn't say that applies to me because I was not born rich I came to Hamburg as a refugee from West Prussia and my father had to start from scratch that's why I absolutely do understand people living in poverty today on the other hand I'd also say that if Germany is getting more and more millionaires people with small or medium-sized businesses because generally the millionaires in question did build a business they now run then I think that's great


because they are the people creating jobs for me that's what should matter in this debate we should focus on that and not rich versus poor voted is good young ensuring that the wealth of rich families can increase despite current zero interest rates is the mission of Christian fund based on science company each day he and his employees sent out investment opportunities from banks and other entities given exclusively to his wealthy clients this is a way to present an offer to just a handful of valued clients and focus


Oh [Music] is it fair that rich Germans are able to increase their wealth while the rest of the population gets left behind [Music] I think it's difficult to apply terms like equity and fairness to the distribution of wealth I would say that here in Germany were better off than ever before and people here live in above-average circumstances nonetheless we have to make sure the gap between rich and poor doesn't get too wide because we don't want social conflicts like in the US or Latin America to happen here for better times company


Fulgham is a so-called multi-family office family officers take care of the needs of very wealthy families managing and increasing their assets it only makes sense to use family offices if your wealth is upward of 30 million euros who can afford such a thing well a family office like ours obviously can't discuss its clients we have well known German business families that's our typical client profile Mikasa she can take someone who is or was a company owner thinks differently to someone who has spent their life as an


employee or they are families who have had money for a long time so wealth is so to speak in their genes and then there are families who have just come into their wealth people who are still pumping with entrepreneurial energy they're usually quite different from heirs option Kaka I see it as a top top in my Stanford [Music] - Allah seems to have plenty of this entrepreneurial energy his company is McFate Europe's largest fitness studio chain its headquarters are in an old baking Factory in Berlin Hina Shula


started off small but today his wealth is valued at 250 million euros [Music] which bring me finish then they're from I was born near long back grew up in a small village normally in villages you do the sports available and when I was around 1516 my role models were Arnold Schwarzenegger and Stallone that's how I came to the fitness world Shahla went from secondary school to complete a Salesman apprenticeship and became the manager of three supermarkets then he decided to start something new - when I was 25 I decided to switch to the


fitness industry my idea was to open a gym where anyone could train the matter kids or her income that was the initial idea and I had big goals I wanted to be number one in Europe but that's all I had I didn't have financing okay good it was 1997 when shala opened his first fitness studio in verts book close to his home village he made use of some unusual methods on the monetary hoods book was a big step for me I opened my first fitness studio there under the slogan now also inverts book which people saw through as


a marketing gag because customers came to me and asked where else we had studios so I gave it some thought and came up with the next marketing gag soon also in ela anything that did the job and also put me on track to going from verse bog to Erlangen and then it grew from there cuza ten years later Heine Shula reached his goal he's number one in Europe and still expanding McFate now owns ten Fitness companies as well as its own model agency meanwhile Shanna is getting ready to open fitness studios in the US des moines severan is getting to be


number one is much easier than staying number one I think if you want to be successful you need to be a bit of an alpha animal inside the investor would probably pick a brand and say okay that could work I like it although it's probably two steps too far from any but if someone doesn't want to get involved with us because of it so what I'm convinced that you have an easier time if you fought your way to the top and to success lions feel it and so do partners I think that's our situation which is why I can also imagine that someone who


inherits something or takes over or even has to take over a company in the second or third generation will have a much tougher time and Montezuma Muhsin as CSV aha but there are plenty of heirs in Germany huge fortunes and thousands of companies have been passed from one generation to the next they are heirs who don't want to and others who shouldn't take over their parents businesses succession is hugely important among Germany's richest Michel Otto inherited the mail-order company from his father and successfully managed


it from when the kinder is this it will be harder for my children though because now the Auto Group has a hundred and twenty three companies in over 30 countries I know every single company either because I was involved when it was founded or because I led the takeover negotiation but my children don't yet know that many companies guns wunderkind otto's children have opted against direct succession I think it's important to give your children the option without pressuring them so you don't force them into a role I think that mistake is made


often and I'd say it's bad for both the children and the company when master kundo under Thunder Lim kaput so now we have one point of sale Berg Vader are you a bit tired yes it's for an exhausting couple of days in ten minutes I'll be fine again it was just Bam Bam Bam Bam dear horsemen also spent a lot of time considering who his successor would be it's now decided how old horsemen the younger of his two sons will take over as manager of the drugstore chain do we actually sell much yarn it's not exactly part of a


drugstores range the trend sort of over it was big in 2016 already fading in 2017 and it's been stagnating in 2018 but it's not the worst product we have and it still brings in some revenue rose father had to show him the appeal of being in charge when the boys got more engaged I thought oh now we can't look as if the drugstore business is only about making money so I showed them how we're also active in social issues in Africa and so forth I've always showed my sons that what we do isn't just about making money it's also about being


responsible for others as a child I wanted to become a film director that was always my dream it still is today you might be thinking no I don't want him to be a director I want him in the company so I said well go ahead and become a director but being a director of such a big company is also exciting I didn't manipulate him a beauty goddess but boy if you could fly off and asked whether I feel competitive toward my father and sure he built up this big company that I'll only take over but it's really difficult to keep something


going these days the founding period has its own challenges and just having the idea of founding a self-service drugstore was hugely innovative but the fight to survive has gotten tougher and that's the one I'm in a coma also now as a thought everyone in my family wants to be good at sports my father and I battle each other in tennis we all compete with one another and that's also shaped our view of life or mine at least as a tumulus miner despite their competitiveness the Osmond family reached a harmonious agreement


with regard to succession cling against Rommel wipes me out I normally prefer playing doubles that's much more appropriate for men my age I'm starting to worry a bit about you you worry about me with your inheritance I wouldn't worry I'd be looking forward to it selecting heirs and successes is usually not quite as amicable as at the Hoffman's christiane fan base time has seen many inheritance disputes in rich families it's his job to preserve the family's assets and protected from all sorts of dangers maintaining a family fortune


over several generations is incredibly difficult because it's under threat from being divvied up through inheritance from wealth disputes from expropriation from wars or simply from stupidity most families will have one or all of these happen to them only a handful of families have managed to stay more or less afloat over centuries but those right on top have been switched out again and again when based on time speaks from personal experience his own family's history dates back 900 years I have a horribly long name at least on my


birth certificate there's my six given names Christian Lawton Ludwig William and Maria as a good Catholic followed by BA horn phone Malkin Haim Jane and beshte Simon what at work are normally called mr. Phan Bettelheim and at social events baron or lord baron baron from back home is an indirect successor of the figures the richest family in German history when does wealth begin for you if you're asking me at what point I consider someone to be truly right I would say over a hundred son I am definitely not rich I've


escaped nakute but I'm comfortable you and our family is comfortable and I'm certainly not complaining when you have a family history as long as mine your family has seen everything near bankruptcy years overflowing with money inside and years when a lot was lost on the one wall news he takes us to the hunting lodge of the best horse hime family in Turlington it looked like the Lord had been lost forever during the division of Germany the house was built in 1892 was a hunting lodge for my great-great grand uncle's it's been in the family ever


since except for a short interval it was expropriated in 1952 and then restituted in 1992 and since then I've owned it the family's hunting lodge survived expropriation and socialism without much damage today from bechtolsheim also owns hundreds of hectares of forest nearby and regularly invites business acquaintances for hunts that'll say many of the trophies are mine but several also come from my father some from my great grand uncle leopard down there I didn't shoot him that was my great grand uncle and then my dogs chewed off his


ears so he no longer has his former beauty but he's too precious not to keep as we're sharpen a recurring topic in the special issues of the manager magazine are the super-wealthy snit works this gift goes alongside there are larger and smaller networks and there are a lot of them and most even we journalists don't know about in high society there are certain typical hobbies horse racing hockey okay a bit of tennis though that's almost old-school he'll meet in the boxes at major football stadiums because of course they're all football fans and


football is huge of that a lot of networking happens there that's like their marketplace they mingle and meet there more than its so-called parties for the rich as of zuga nantan i [Applause] Dork oh yeah good we always have a lot of employees here we have a lot of friends in the box because Jim Pfeifer is here today Germany's most famous criminologist sometimes you live for someone from politics comes by Christian Wulff and Bettina are part of my close circle of friends so there's always lots going on here [Applause]


in villain avinash allah is opening a new club a new branch of his fitness empire shallow also has become a member in the network of important people from the sports business and entertainment industries true it's a closed circle that's hard to get into sure money attracts success and success attracts money I do believe that I've experienced myself how you can suddenly connect with people previously out of reach I am in a different position the hot scene at Bouchon field for independence [Applause] Derk horseman is ready to leave his box


he wants to celebrate the victory with his friend the Berlin air and hearing aid company owner Martin Kent Kent is also co-owner and president of Hannover 96 his box is located on the other end of the exclusive VIP area I told you we'd win today I told him that if we didn't win he'd pay 10 million that was the bet right - halfway decent many networks are important for business but do the rich also have political clout can political influence be bought in Germany bingo is life lost in the best way available for


rich or super rich people in Germany to exert influence is the number of employees working in their companies someone who owns a company with a hundred thousand employees or let's say less maybe fifteen thousand employees can go to business associations and say finally go ahead and pass that law but that will cost me or an even better argument is that will cost you two thousand jobs in that area but there are no super rich people who regularly call up the ministers or ms merkel and say what needs to happen next


which tax laws they'd like and so on that's not how things work here in Germany the deadline for the special issue is approaching the heart of it is the ranking of the 1001 richest Germans what do the rich think of this ranking is a ranking suit those rankings are for entertainment all their scraped together sometimes using stock market quotations but they aren't reliable in any real way and five Gunson fontana giving and visit roughly and also craft as each house I don't think much of these rankings and I didn't want to be included because it


creates this impression of rich people being like Scrooge McDuck that they have these money bins in which they wallow in their coins the fact that I'm at the top of the rankings including of the wealthiest Germans does make me feel proud and Finnish naturally rich noise it is it's well read manager magazine every now and then but I've never read the rank I don't know whether I'm in it I don't need that I have other goals one of these is that Highness Allah now wants to open the world's largest Fitness Centre in North rhine-westphalia


and let me write the Mariah's Japanese and means the future we think it's a perfect fit for the whole concept and vision because what we want to create here is truly unique and has never been done before our goal is to become the world's fitness center fitness center embed bite severity Shanna thinks he's found the perfect location to realize his vision in Oberhausen he's rented an old factory complex at the moment the space is still being used to make steel parts but before long thousands of customers will be exercising here the


emergency exits path will be here and the offices on top seanny experienced his most traumatic experience to date in the area in 2010 21 people died and 54 were injured in a stampede at the Love Parade in duisburg shala had been the parade's organizer the cause of the panic has still not been conclusively determined how does an entrepreneur in the fast lane deal with that kind of tragedy an event like that will haunt you always for the rest of your life I've got a moral responsibility I was the organizer if I could turn back


time I would do it immediately before the scale of what happened but you can't make it undone you have to try to deal with what happened in the Musa Dagh Mitzvah Zulu once again dick Hoffman has also seen setbacks and crises in the 90s we expanded dramatically hinted the Czech Republic Hungary and Poland I was also speculating on the stock markets a little too much and neglecting the company then in 1996 we suddenly had a loss of 12 million Deutschmarks the banks don't joke around if you're highly indebted and then you come in with huge


losses that was critical and then I had a heart attack in 96 but everyone knows that life can get tough and things got very tough back then incurred odd nights what is saying so I dialed back a bit including stock speculation I sold them all and thought the only thing on my table now is pulling the rossmann drugstore business through and dumped death it was the right move to focus on one thing and not do so many different things horsemen emerged from that crisis stronger than ever he started speculating again but so he


assures us only with his private wealth so let's see where gas promise I don't have a laptop I normally do this via NTV text 254 yeah here it says gas promised it 375 I could already sell those now I bought 250,000 of those so 250,000 times 20 cents that would give me 50,000 euros profit but I won't sell I think it's gonna rise to 4 euros sometimes it works sometimes it doesn't but I enjoy that's why I don't play the lottery because I find that boring can a large fortune also be a burden the hoped it does I'd say that for most people


although they wouldn't voluntarily give away their money the fact that they wanted to grow can be a burden they are controlled by their own assets for example they'll move to Switzerland or somewhere to save on taxes and give up their entire circle of friends and basically become a slave to their fortune in my opinion that's absurd considering the conditions we currently have in German Finland conditions in Germany are currently more favorable than ever for the rich they pay significantly less tax than they did


25 years ago only minority still feels compelled to emigrate abroad article 14 of the german constitution states property entails obligations its use shall also serve the public good do the rich in Germany live up to this responsibility you finish fish dish I think it is important if you're successful if you're lucky enough to have reached a certain level of prosperity and wealth to give something back to society Michele Otto is one of Germany's biggest donors his money helps fund the environmental cultural and social


sectors like most rich people he prefers to decide himself what he spends his money on rather than leave that to the state Otto like many wealthy Germans donated millions for the construction of the air fill ammonia in Hamburg in Germany wealthy people like to donate and this makes important contributions to society and public life but generally they're against the proposal of redistributing wealth by a higher taxes for the rich in film German businesses would yield to all the demands of let's say miss varnish to the left party then