distance in that case you maximize the dot product between two things and the two things are two images that go through the same neural network as before right this and this now what's different here is that here input and one image of the data set that's the same as before okay so we input one image but here before in the self supervised learning we input an


Augmented version of that and now we input something else we input this K right here now what's K what K comes from this neighbor set of X okay this is the set of neighbors of X and these neighbors are determined with respect to this neural network right here okay so what you do after step one is you take your neural network with the good embeddings and here is your data set X

Learning To Classify Images Without Labels (Paper Explained)


your data set X that's really another your data set X is this list basically of all the images in your data set and what we're going to do is you're going to take all of them using that neural network that you just trained and embed them into a latent space right here okay this is the latent space where you have done this self supervised training and now for each image right here so if this


is X eye you're going to find its K nearest neighbors and they use I think they use five as a benchmark so you're going to find its nearest neighbors it's five nearest neighbors and you do this for each image so this image has these five nearest neighbors and so on so in step two what you're trying to do is you're going to try to pull together each image and its nearest neighbors in


that in this this not in this space directly but you determine which ones are the nearest neighbor from this neural net where can you keep it constant that's how you determine what the nearest neighbors are in the first task and that is your NX set for X I and in the second step you're trying to make the representations of any image and its nearest neighbors closer to each other


okay so with with this thing right here you maximize the inner product between X in after this neural network and a nearest neighbor of X that was was a nearest neighbor after the first task now the way they cluster here is not just again by putting it into an embedding space like we saw before but this thing right here this neural network as you can see here is is a C


dimensional vector in 0 1 now C is the number of classes that you can either know that so you don't know which class is which you don't have labels but you could know how many classes there are or you could just guess how many classes there are and as long as you as you over guess you can still like build super clusters later so this they simply say it's in 0 1 but they also say it


performs a soft assignment so we're also going to assume that this is normalized so for each for each data point X here you're going to you're going to have an image you're going to put it through this new neural network ok this new neural network new and it's going to tell you it's going to give you basically a histogram let's say class 1 2 or 3 we guess there are 3 class and


it's going to give you an assignment of the 3 and you also take a nearest neighbor here is your dataset you also take a nearest neighbor of that so you so you look for this set n of X and you take a nearest neighbor maybe that's that's a maybe that's a dog I can't I really can't draw a dog yeah that's the best I can do I'm sorry and you also put that through the same network and you're


saying since they were nearest neighbor and task 1 they must share some sort of interesting high level features because that's what the first task was for therefore I want to make them closer together in in the in the light of these of this neural network right here so this is also going to give you an assignment like maybe like this okay and now you object you you train this


network right here to basically match these two distributions okay so this is this is now a classifier into c classes but we guess c and we don't have labels we simply our label is going to be my neighbors from the first task must have the same labels that's our label now they say they also have this term right here which is the entropy over assignments okay as you can


see so they minimize the following they minimize this quantity which has a negative in front of it so that means they maximize this log inner product and they also maximize the entropy because sorry so they minimize this thing and but the entropy is a negative quantity right so they maximize the interview because here is a plus and now they minimize the entropy let's see what they


say by minimizing the following objective now entropy is the sum of the negative sum of P log P and this if this is P yes this is the probability that an image is are going to be assigned to cluster C over the entire dataset so they're going to mmm yes so it's negative this quantity negative minus P log P and this is the entropy so they're going to minimize the entropy let's see


what they say we include an entropy term the second term in equation two which spreads the predictions uniformly across clusters C ok so what we want is a uniform assignment over class which means we should maximize the entropy oh yes okay they minimize this thing and this here is the negative entropy right okay so they want basically what they want of over the


whole dataset that not all of the images are going to be in the same cluster well this is cluster one and then this is cluster two and then this is cluster three so that term counteracts that basically the more evenly spread the entire dataset distribution is the the the higher the entropy the lower the negative entropy and that's the goal right here I'm sorry this this was I was


confused by the too many negative signs and then you minimize the entire thing all right now they say they say different thing right here they say here this bracket denotes the dot product operator as we saw it's the dot product between these two distributions right here the first term in equation two imposes this neural network to make consistent predictions for a sample X I


and its neighboring samples the neighbors of X I and here is an interesting thing note that the dot product will be maximal when the predictions are one hot that means confident and assigned to the same cluster consistent so they basically say the objective encourages confidence because it encourages predictions to be one hot and it can Courage's consistency because it you


know that because the distributions need to be the same they should be in the same cluster right now I agree with the consistency like if you make the inner product high then of the up to date of two of these histograms of course they'll look the same right because these are ultimately vectors these are three-dimensional vectors let's call them two-dimensional vectors right so


here is class one here is class two if you you know make the inner product small or high they will agree on their predictions but I I disagree that this encourages anything to be one hot like in and if you have two vectors that are both zero one times zero one the inner product is going to be one and if you have two assignments that are 0.5 and 0.5 then it is also going to result in


an in an inner product of is it 0.5 right is also going to to be no so what's the inner product here the inner product is 0.5 times 0.5 plus 0.5 times 0.5 which is 0.5 am i dumb an embarrassingly long time later oh it's because the l1 norm okay okay we got it we got it I am I am okay I am too dumb yes of course I was thinking of these vectors being normalized in L 2 space


where their inner products would always be 1 but of course if you have assignments between classes and it's a probability distribution a histogram then all of the possible assignments lie on this on this thing right here now the inner product with yourself of course is the length of the vector and the length of a vector that points to one class or the other class is longer than a vector


that points in between so okay I see that's where they get this that's where they get this must be one hot from so okay I'll give that to them it is actually encouraging one hot predictions as long as these things are normalized in l1 space which they probably are because they're histograms right yes that was that was dumbness of me I was trying to make a counter example I'm


like wait a minute this counter example is a counter example to my counter example okay so yeah that's that's that so as you can see they are of course correct here and they now make the first experiments so they say basically after the first step of the self supervised training they can already retrieve sort of nearest neighbors and the nearest neighbors the


nearest neighbors of these images right here are the ones that you see on the right and after the self supervised one these nearest neighbors are already pretty good at sharing the high level features actually crazy-crazy good right this flute here is in different sizes as you can see the fishes aren't aren't all exactly the same the birds so you can see it really focuses on sort of higher


level of features but I guess it's really dependent on this higher level tasks and they what they also investigate this quantitatively but I just want to focus on how how good is this after only the self supervised thing and now they do this clustering and they can already sort they could already evaluate it right here because now they have a clustering right after


this step they've basically pulled together the neighbors and they have this neural network that is not assigning classes so they could already evaluate this and they are going to do that but that's not good enough yet then they do a third step which is fine tuning through self labeling now self labeling is pretty much exactly what it's what it says it's you label your


own data with your own classifier now that might be a bit outrageous and you basically saying wait a minute if I label my own data and learn a classifier on these labels isn't isn't it just going to come out the same and the answer is no right if you have a dataset because your classifier doesn't give you just first of all if your classifier is something like this right just happens


to be and you label and you learn a new classifier it is going to be more like this right because it sort of maximizes a lot of classifiers maximize these distances between the classes so even if it's like that and then the second step they do is they say okay there are some points where we are actually more confident about such as this one we're more confident about that one also this one


and then this one here is pretty close like we're not super neither this one but we're very confident about these two so we're only going to use the ones where we are in fact confident about to learn to learn the new classifier or basically we you can also weigh them and so on but they go by confidence right here as you can see in this final algorithm so this is the entire


algorithm and I got kicked away at our algorithm there we go all right so semantic clustering by adopting nearest neighbors they're scan algorithm so in the first step you do this pretext task this is the self supervision the representation learning okay for your entire data set no sorry this is this is this your optimized optimizer neural network with task T


that's just self supervised representation learning okay then the second thing we're going to determine the nearest neighbor set for each X now they also in that's that they also augment the data they do a heavy data augmentation and so on also in this in the third step in the self labeling they do date augmentation there's a lot of tricks in here but ultimately the base


algorithm goes like this so you find your neighboring sets for each X and then what you do while you're clustering loss decreases you update this clustering neural network by with this loss that we saw so this is the loss where you make the nearest neighbors closer to each other while still keeping the entropy high okay and then in the last after you've done this you go through when you say


while the length of Y increases what's why why is all the data points that are above a certain threshold now you going to filter the data set that is above a certain threshold and that's your data set Y and you Terrain this same neural network you basically fine-tune it with the cross entropy loss on your own labels so now you only have labels Y okay so it's not it's not labels you


have the cross entropy loss between the assignments of this and the assignments of your data set okay so you basically do the same task but you filter by confidence and they use a threshold I think of 0.7 or something like this now let's go into the experiments the experiments or look as follows so they do some ablations to find out where in their methods kind of the the gains come


from and we'll just quickly go through them if they just do these self supervision at the beginning and then just do k-means clustering on top of that that will give them on C for ten a thirty five point nine percent accuracy so not very good so the clustering you can't just cluster on top of these representations and then be done if they do what they say so this is sample and


batch entropy loss this basically means you do not care about the nearest neighbors you do this entire thing but you only make an image close to the prediction close to itself and it's augmentations so you don't use any nearest neighbor information also doesn't work like I wouldn't pay too much attention that the numbers are ten twenty or thirty it just it like doesn't


work now if you use the scan loss you all of a sudden you get into a regime where there is actual signal so this is um this is now significantly above the this is significantly above random guessing and if you use strong data augmentation as I said is a lot of this is has these tricks in it of what kind of data augmentation you do and so on so never forget that that these papers besides


their idea they put in all the tricks they can so you get 10% more and then if you do this self labeling step you get another 10% more and this is fairly respectable like 83.5 without ever seeing labels it's fairly good but of course there are only ten classes right here so keep that in mind but they will do it on image net later and they investigate what kind of self


supervision tasks at the beginning are important and they investigate things like rot net feature decoupling and noise contrastive estimation which noise contrastive estimation is the best a noise contrastive estimation I think is just where you as we said you input an image and then it's kind of noisy versions with augmented in various ways and then you classify them together and


this has been like this these methods have been very successful in the last few years yeah so this they they have various investigations into their algorithm I want to point out this here this is the accuracy vs. confidence after the complete clustering step so this is now the third step is self labeling and you can see right here as these confidence of the network goes up


the actual accuracy goes up as well so that means the network after the classroom is really more confident about the points that it can classify more accurately there's like a correlation between the network is confident and the actual label of the point which is remarkable because it has never seen the label but also see how sort of the range here is soup is quite small so with the standard


augmentation that goes like from here to here so where you set that threshold is fairly important that might be quite brittle here because you need to set the threshold right such that some points are below it and some are above it and you you don't want to pull in points where where you're not because if you pull in points from here you're only you only have the correct label for 75% or


something like them of them and that means if you now self label and learn on them you're going to learn the wrong signal so this this step seems fairly brittle honestly but I don't know of course they go on and investigate various things such as how many clusters do you need or how many nearest neighbors sorry do you need this number K here and you can see that if you have


zero neighbors then you're doing a lot worse than if you have let's say five nearest neighbors so the jump here as you can see is fairly high in all the data sets but after that it sort of doesn't really matter much so it seems like five nearest neighbors should be enough for most things and here they just show that when they remove the false positives that their algorithm


actually converges to the correct clustering the correct accuracy which is not surprising like if you remove the wrong samples that are wrong then the rest of the samples are going to be right I think that's just showing that it doesn't go into some kind of crazy downward spiral loop or something like this but still it's just kind of funny okay so they do you investigate how much


they improve and they improve by quite a lot above the kind of previous methods so they have a lot of previous methods but a manner this includes things like k-means and so on ganz deep cluster that we spoke about and this method it already gets as you can see fairly close to good accuracy so you have like eighty eight point six percent accuracy and that's you know


fairly remarkable on C 410 without seeing the labels but we'll go on and now they go into image net now image net of course has way more classes it has a thousand classes compared to see four tenths ten classes so if you if you think you know clustering ten classes might and they're fairly apart from each other might work with various techniques image net a thousand classes that's way


more difficult that I do subsample this to 50 100 and 200 classes and they get okay accuracy as you can see they they get 81 percent in for 50 classes where a supervised baseline would get 86 percent into 200 classes they get 69 percent where a supervised baseline would get 76 percent so it's fairly it's it's there and and that's quite remarkable for this low number of classes and they figure


out that if they look for these samples that are kind of in the most of the middle of their cluster they get these prototypes right here you can see all of these images I don't if you know imagine that some of the images we really only have the part of the object and so on so here with the prototypical things you really get center clear shot of the object with clearly visible features and


so on so this sort of re sort of repeats the fact that this clustering really does go on what sort of semantic information of course the labels here are you know from the from the test label set the network can't figure that out and and then they go for a thousand classes and in a thousand classes it doesn't really work because there might be just too many confusions right here but they do


have this confusion matrix of their of their method and it shows that the confusion matrix is pretty much a long like block diagonal along these super clusters right here so you can you can see the dogs the network confuses the dogs fairly often and then insects with each other but not really across here which is still quite remarkable but I mean that's you get the same thing for a


lot of these methods so I don't know I don't know how much different this would be in other methods but certainly it's interesting to look at now they go into one last thing and that is what if we don't know how many clusters there are right if we don't know anything so say so far we have assumed to to have knowledge about the number of ground truth classes the model predictions were


valid losing the Hungarian matching algorithm we already saw this in the DET är by facebook if you remember however what happens if the number of clusters does not match the number of ground truth classes anymore so they now say Table three reports the results when we overestimate the number of ground truth classes by a factor of two okay so now they build just twenty classes for C for


ten instead of ten classes and we'll going to look at table three real quick where's Table three this is Table three okay so when they over cluster you get the thing here on the bottom and you can see there is a drop in accuracy right here now what I don't actually they don't actually say how they do the over cluster matching so if you imagine if I now have I don't know six clusters but I any need


to assign them to three clusters you know here do I still use this most optimistic thing so they are still used I think they still use this most optimistic matching right where you assign everything to its best fitted cluster right you compute all the permutations and then you give it the best benefit of the doubt now if you imagine the situation where I over


cluster to the point that I have each image in its own cluster and I run this algorithm to evaluate my clustering I give it basically the most beneficial view then I would get a hundred percent accuracy okay so like in in in one of the in this over cluster approach I would sort of expect that you actually get a better score because you can like there is more generosity of the matching


algorithm involved now that's counteracted by the fact that you can't group together things that obviously have similar features because they are in the same class so there's kind of two forces pulling here but I was kind of astounded that it's going down and the evaluation method of this matching algorithm it sort of breaks down when you have more classes at least in my


opinion yeah but but it's interesting to see that you can just overshoot and but then you need some sort of heuristic to reconcile that in any case I think this paper is pretty cool it brings together a lot of things that we're already present and introduces this kind of this step approach but what you have to keep in mind and by the way there's lots of samples down here what you have to keep


in mind is there are a lot of hyper parameters in here there are like this threshold and you know the first of all yeah the number of classes the thresholds the architectures and so on and and all of this has been tuned to get these numbers really high right all of these steps all of the augmentations and so on the chosen data argumentations it has been chosen to get this number as high as


possible so you know to interpret this as oh look we can close classify without knowing the labels is you know if yes in this case but the hyper parameter choices of the algorithm are all informed by the labels so it is still very very unclear of how this method will actually work when you really don't have the labels when you actually have to choose the hyper parameters in


today I want to talk to you guys about how to get machine learning on to your physical devices when there is no network connectivity and convince you why you would want to do that why you would actually prefer even with network connectivity to leave it aside perhaps in certain situations but to get there I want to start back we're gonna go back in time back to when the computer first

On-device Machine Learning With TensorFlow


came about right that started this computing revolution that we're in today but today we're in what we call an AI first world how did we get here well first we had the computer and that gave us computing then the internet came right the internet connected all the computers together and I want to point out that the internet did not wipe out the computer right the computer is still


used today then came mobile the mobile revolution can brought the Internet to everyone it brought the connected computers into our pockets and again we see that mobile did not wipe out the Internet we still use the Internet it's still strong with an IP version 6 we're running out of IP addresses and so too will a I not replace mobile but build on top of it and so I see mobile has a key foundation


for the core of how we will interact with AI in the years to come and so to that end I want to show you guys one approach for integrating AI into mobile devices and bring amazing user experiences today over half of the fortune 500 globally have disappeared the company's kaput gone since 2000 and so how can we not have that situation first of all but also what will happen


to the other half right it's the companies who embrace AI these startups who are well nowadays all the startups are saying we are AI startup everyone is a start-up but a few short years ago machine learning and AI was something that companies would add on to their product they would say oh yeah we do a little machine learning here on the side right but now I root


everyone's doing it it's it's super popular and for the most part people trained on the server right server has a lot of compute power it makes sense and I'm now going to refute that because it makes sense Mobile is not a compute powerful platform so we train on the server that's fine but what if we did the predictions on the mobile on mobile what if we did inference there instead


of serving them from a server web server and in particular this approach can lead to amazing user experiences I want to just show with one illustration example this is a Google Translate and it has the ability to overlay the translation directly in the image that you see now you can tell just from the the speed that it can do this that it is definitely not happening over the


network right the video is not being streamed to Google servers and then sent back that's why it works when you are offline that works it works on a boat it works underwater well if your phone can be underwater it would even probably would work in space and having access to AI on your phone wherever you go that is responsive and accurate that can really change things on a global scale


for mobile for the internet and for computing so how can we make something like that right machine learning is hard enough by itself then put it on mobile I mean mobile apps aren't easy either and so combining those that can be a real challenge so let me start by showing you guys what a little demo of what I've kind of put together and then we'll talk through how


we might build something like this it's a simple demo just mainly just to demonstrate the core kind of functionality and what I've got here is a phone and I have a little app if we switch over to the camera here I have on the table a few different candies let's see what we see here oh great and so let's see you guys see this okay so what I'm gonna do is what we'll take some of


these away and if the lighting is good you know here we have a a Reese's Cup right and we can see here know that as the image kind of isolates down it recognizes that no I also have a smaller Reese's Cup which just fell on the floor okay so we have a smaller Reese's Cup here and you know and we it will switch over and recognize that now you might say well how do I know that he's not


cheating by sending all these images over the web right well let me first of all there's no Wi-Fi here secondly let's let's just go ahead and hit airplane mode right as well and we'll see here that you know everything continues to work just fine these are some more peanut butter cups I'm a big fan of peanut butter and so let's push these guys away and so here we have a shot of


the Justin's see here it says white chocolate peanut butter cups and you know one of my hand enters the frame it may get upset candy is a little bit bent out of shape from its days inside the suitcase but you know you can see that it clearly recognizes that versus a very similar packaging right but this is milk chocolate this is a milk chocolate peanut butter cup and and this one also


updates you know it updates right away you can see the the confidence and then we here we have some juicy fruit gum just for variety you know some people don't like peanut butter I understand that so that's that's the gist of the demo here and so if we switch back to the slides we can think about how we might build something like this how do we go from collecting data to having an app that


can recognize images in real time custom images right I just I chose these pretty arbitrarily if you were to take any generic machine learning visual model and pointed them at this it might take candy candy bar maybe it even says chocolate maybe it just says yellow but how can you get something to recognize something that's specific you know imagine having something recognize


your particular products your whether it's your brand things in your home and so this is my are kind of guidelines here this is what will follow this our little nap this is our road map and what we're gonna do is we're gonna try to fill in each of these blocks and go from gathering data to having an application so the first thing you got to do is collect data my data collection is as we


all know the most fun step of machine learning now I've found a bit of a shortcut for you instead of you know collecting lots of pictures and then trimming them down and then labeling them and it would just be a lot of work right it's a lot of pictures so maybe we can shoot some video we shoot some video and we only take the video of that particular object so I go through each


one now you can see there I have one of the the peanut butter cups and we go through each one and we capture a video and what's nice about that is that entire video every single frame is a picture of that object hopefully from a different angle so keep that camera moving and then what we can do is we can well we chop that up right there's a command line tool called ffmpeg and


there's lots of tools ways you can chop up a video that's a solved problem and we put those pictures all into one place all together in one folder for each of the objects you want to recognize so so one folder for your juicy fruit one folder for your milk chocolate peanut butter cups one folder white chocolate ones and so on and so now we've effectively just labeled all


of the images right we didn't have to come up with any sophisticated way to label it for us with some system so that's great so we have folders of images what's next well we take these pictures and we send them to training right so in my particular case I uploaded them to the cloud because my MacBook was running out of space form all the images and pictures and so I


zipped them up put them in the cloud and I happen to do my training in the cloud you can do them on your data center you can do them on your local machine if you have a lot of storage and the training we did is using transferred learning now this speaker before me mentioned transfer learning so I won't go too much into detail about it but I do have a kind of little story a little analogy


that I recently accidentally discovered I was playing with a puzzle this is a jigsaw puzzle right lots of pieces and and this is the final picture this is the box showing what I was supposed to build and I'm not a very good jigsaw puzzle person I kind of struggle with it but as I was struggling through this I realized you know there's some good tricks in here I could do first of all I


could separate the the pieces with no images the white pieces from the everything else right so I moved all of the white pieces to one side okay so that's the obvious step first step but then what how do we start from there I also noticed that in this image the roofs all the individual roofs are very distinctly patterned and so I said ah I know I can look for pieces with this


pattern and with that I can begin to put those together those were easy to find I could recognize those patterns so my brains neural network through my eyes could find those individual small pieces and combine them together and say ah there's a roof similarly I noticed in the stairs the stairs have a distinct pattern they're parallel lines there's some of those X's there and I could find


the pieces that kind of looked similar and begin to squeeze those together and eventually get that together and instantly everything else came together well there was a few more steps right but that that's as far as I got the edge pieces they're so hard so so a convolutional neural network kind of works in a similar way I use transfer learning with the inception model which


is a model that the Google brain team created a few years ago it has 48 layers and to give you some perspective on how big or how small that might be just a few years ago I think it was 2011 or 2012 it was Impractical to train a neural network that was more than four layers deep the year before inception v3 came out the model that won the International image recognition


competition called imagenet that had 22 layers so in the inception of e3 model really was a giant leap ahead right more than double the number of layers it really showcased the improvements in both computational power that was available as well as network design so we can use that wonderful research and take it for our advantage so we trained the last layer right of the network and


leave everything else intact this means that everything in the visual part of recognizing those pieces recognizing the little bits the principal pieces are already there in place for us so you have a great model you can train it but when you're done you look at your file system and you say wow this model is 84 megabytes I'm trying to put this in a mobile app


can you help me out sure thing we're gonna optimize it for mobile what can we do to shrink down a model well handily enough there is a graph transform tool and what that's gonna do for us there's a couple of steps in there that we can do the first thing is a technique called quantizing or quantization and the floating-point numbers those 32-bit floating-point numbers that are taking


up all this space we're gonna shrink that down to just eight bits how can we get away with this we're gonna lose so much accuracy right well not necessarily luckily neural networks are designed for fuzziness in their inputs so by quantizing down to eight bits and and that's just by rounding but like actually saying these numbers are close enough we're gonna make them all kind of say this number


and then this number so we then say the range we're gonna split that out but into 256 pieces so if the range of values is say negative ten to thirty within there when they divided up into 256 little steps and so that gives you a little more accuracy than just purely changing it directly to 8-bit and additionally that means when you do compression those values are the same


and so that's what it lets you go down literally for X so you go from 84 megabytes down to around 20 21 megabytes and one small additional thing you can do is take away the parts of the graph the parts of the graph that you don't need anymore for prediction there's some graph pete nodes which are only useful for training and there's also a tool that will prune that down for you as


well so that's also part of this graph transform tool is a whole suite of tools so that's really useful and I also want to call out that so far everything we've done is basically running existing code and existing tools you didn't have to custom write anything the only custom thing you had to do was shoot that video and run ffmpeg so this really makes it you know puts it in an immediate


possibility type of stage there is one more thing one more consideration to think about when looking at deploying a machine learning model to a mobile device and that is whether you package it inside the app or alongside the app you can make it a data file or you can make it integrate it into the app and some of the thoughts there are whether you want to be able to secure the model


whether you want to be able to download updates without pushing a new updated version of the app itself and whether or not you care about sizing and how whether or not you want to secure the the model from outside access so that's our our overall design right we gather it up we shoot up the videos slice it up and and train and optimize and then we can deploy so that that's


our of our finished model and this is a video of the the same thing so I won't show that and the final kind of point here is how are we going to be able to how have we done this right what makes it possible and that's tensor flow tensor flow is Google's machine learning library hopefully some of you have heard of it it's too dark so I can't do a show of hands but it's been incredible to see


the community adoption and the reception to the launch it was open sourced in November of 2015 and hit 1.0 this past February and with that we have support for not just these platforms right which we expect CPUs GPUs and of course Android but also iOS and Raspberry Pi so for those of you who like to tinker with IOT devices you can load a model onto a Raspberry Pi so that means you can


recognize things without any network traffic it can be handy and the community responds to tensorflow in the past while now more than fourteen months but in the first fourteen months there were over fourteen thousand commits hundreds of non-google contributors and now that it's 1.1 of 1.0 it is production ready the AP's api's are stable and backwards compatible so things won't change out


from under you so that's really quite nice and so in conclusion putting machine learning on mobile will just make that experience of mobile the internet and computing that much more powerful and usher in a new wave of innovation and open a whole new world of possibilities and moreover you can build one easily by gathering your own labelled data simply by shooting a video


and running basically a series of well-defined steps and having a trained model as the output so please use this to build magical experiences for your users and with that I want to thank you and have some resources for you here we have a code lab to help you do the inception retraining and as well as a sample app for loading models into tensorflow on github and so thank you


[Music] all right thank you very much for the introduction and I hope you had a nice lunch and welcome to this talk about self supervised deep learning towards autonomously learning machines as you already heard my name is Simon T ballina I'm head of yard craft works and also lectured a couple of universities and craft works well you might have heard about us we are a vienna based

Self-Supervised Learning - Towards Autonomously Learning Machines—Simon Stiebellehner


artificial intelligence and big data company specializing in solving hard industrial problems using artificial intelligence and of course data also most of our clients come from the industry ranging from the automotive sector all the way to the energy sector and pretty much everything in between the topic of this talk self supervised deep learning is also primarily


motivated by the work we do with our clients Before we jump right into the topic we first need to lose a few words about the current state of artificial intelligence though probably as most of you are aware of AI has come a pretty long way in the last couple of years in fact we have made massive progress for example when you think of popular voice assistance such as Syria or Cortana they


have become incredibly good at understanding human natural language just in the last couple of years similarly in a completely different area of artificial intelligence in computer vision we can now highly accurately segment even complex images into all of its part parts and not only that we can even automatically generate text that describes what's happening in an image


so called scene understanding so obviously great great progress has been made when it comes to artificial intelligence just in the last year's and it turns out that many of these breakthroughs many of these things you read about in the news are actually based on something that's called supervised deep learning and this brings us to the first part of this talk


supervised deep learning the good and the ugly we will find - I supervise deep learning is so great and so many breakthroughs are based on it but also we will get to know it's very ugly side one of its major downsides one of its major weaknesses subsequently we will find out how taking a self supervised approach to deep learning can help us overcome or at least mitigate that core weakness and


finally we will look into an industry case that hopefully gives you a good idea of how you can use self supervision in practice to make your models better and more robust but first let's look into supervised learning probably some of you or maybe if many of you know this data set it's an incredibly popular famous data set even containing images of obviously cats and dogs and this data


set is used by data scientists around the world to do the first steps in image classification using especially deep learning so usually the task at hand is building a classified a differentiates between cats and dogs based on images theoretically you could approach this problem from two sides you could approach it from an unsupervised side unsupervised learning we do not use


labels right and in supervised learning this would be the other side we do use labels so we do use textual or numeric information that tells us if there really is a cat or if there really is a dog in that image just a quick recap to bring us on the same page in unsupervised learning as I mentioned we don't use labels we just tried to detect similarities and dissimilarities in


these images and based on that form homogenous clusters that hopefully so we end up with a cat and the dog cluster of course you're not using labels here you lack supervision so usually results will not be optimal actually the web more choice for this kind of task really is supervised learning especially supervised deep learning because in supervised deep learning by using


different different deep learning techniques such as convolutional neural networks we can actually teach them what makes a cat a cat and what makes a dog a dog and they learn that in a fully automated fashion by providing them labels by providing them ground truth if the relays or if there really is a dog in that image and based on that they then make the classification and results can


actually be astonishing astonishing good right we can use we can achieve human and even super human performance especially for these types of tasks which are very specific and also based on images you can achieve astonishingly great results using supervised deep learning but well that great performance that you often read about in the news for reaching that for reaching human


level and superhuman level performance very often you're gonna need just tons of labeled data and really tons we are speaking about tens of thousands hundreds of thousands millions of labeled images and that is a lot and this is also a big big problem because obviously label data is so important for achieving this great performance but at the same time label data is scarce right


data we have a lot of it we have data lakes and data warehouse is full of data but actually getting the labels that you need for solving your specific problem that's something that's usually not existent of course at that point you could argue well you know if I don't have these labels I can just label it by myself I can sit down on my desk look look through these images of cats and


dogs and note if there is a cat or a dog on that image and I fully agree yes you can do that absolutely but you can imagine that this is quite some effort and that effort Rises very very quickly with the complexity of the image for example this image is an image we took from a real practical use case we did together with me burr moebus a large manufacturer of industrial parts and


that image actually shows one of these parts well on that image in theory you can recognize defects in the underlying part well it's not an easy dog because these images they are really large they are highly noisy and defects they can be incredibly subtle incredibly small so it even takes a human expert quite some time to see me reliably find and label mark all these defects on such an


image so assume you want to automate that by trading some supervised deep learning model and for that you need to build up a large label data set of let's say a hundred thousand images labeled images you can imagine that that's going to be quite some effort and you know time is money so it's also going to be expensive how expensive well it's actually not a too hard calculation to


to compute this assume you use some publicly available labeling tool such as Amazon Sage make a ground truth for example they charge you I think if I remember correct around four US dollars per labeled image then as we said before you can do 30 images per hour because you can do you need two minutes per image and then of course it's not going to be you labeling but probably you're


going to employ some working student for example it does the labeling and that working student charges you 15 US dollars per hour well and now this is critical highly critical but most people forget about that you do not only need one person labeling your images why well because a person can have a bad day a person you know might just not be too accurate on that task and and this


actually biggest problem these images you saw a highly complex these defects are so difficult to spot it's really really easy to oversee one so you need multiple people labeling the same image and then you need to aggregate these labels to actually end up with a really robustly labeled data set if you don't do that you're gonna end up with a garbage dataset and you know how it is in


machine learning garbage in garbage out your model is not gonna learn accurately what you wanted to learn and if you do the math behind this you will find out that building up that label data set cost you ten thousand hours work time and that amounts to more than a quarter of a million US dollars just labeling cost and this is significant right of course if you're big if your business


case is big quarter of a million might be nothing you might not even lose a single thought on that but for most companies in most departments spending quarter of a million I'm just building up a label dataset it's a significant challenge because at that point you haven't built a single classifier at that point you haven't deployed anything in production you basically haven't shown any value and


this is a problem and this is why we argue that supervised learning and especially supervised deep learning is just very often not feasible because first labels the labels you need to solve your problems probably don't exist and second if you want to create these labels well that's gonna cost you a lot of money and this is also by thought leaders of the fields argue that Dai


Revolution is not going to be based on supervised learning and I agree how can Dai Revolution be based on supervised learning if we don't have labels right if labels are not ubiquitous how can a I be ubiquitous it can't at least not when it's based on supervised learning and when I thought about this the first time I still found it a bit strange because well deep learning is based on


artificial neural networks and neural networks as the name already says are inspired at least inspired by the human brain but we humans we need so much less labelled data to learn a task highly accurately so where did we go wrong here what makes the difference and this brings us to the core of this talk self supervise deep learning when we talk about self supervise deep learning we


first need to take a step back and think about the human brain we need to ask ourselves how do we humans actually learn do we do supervised learning well absolutely yes sometimes for example in school or at university you have an explicit supervisor telling you what's right and what's wrong what is a cat and what is a dog so yes we do but of course not always you don't always have a


supervisor standing next to you telling you what's good and what's bad and even if we have a supervisor for example at school you don't need to be taught something 10,000 times before you understand to do a task actually just a few example as a fish so again we are we humans are such efficient and effective learners our supervised learning seems to be highly


different from the supervised learning between machine learning obviously well how about trial and error learning or reinforcement learning just to bring us on the same page what is reinforcement learning basically it's a subfield of machine learning where we try to teach an agent to learn a policy a behavior to solve a highly complex mostly sequential task such as driving a


car in a simulation or playing chess and that agent learns that by let's say smart trial-and-error basically we humans we also do trial and error learning absolutely sometimes we try something we fail we try again and then do better hopefully but of course we don't trial and error everything for example you don't trial and error learning how to drive a car you don't


just drive around randomly until you manage to stay on the road right this is not a good strategy how to how you should learn how to drive actually what we do is quite different we get into a car the first time and after a couple of hours we are actually reasonable drivers we can at least follow traffic in a basic way and this is fascinating because if you want to teach a machine


to follow a real-world traffic in a reliable way but this is a big big big problem you need a lot not only of examples but also engineering power behind this before as humans it's so easy it's oh it's so it's because we are such effective and efficient learners again so obviously also our trial and error learning that we humans do is fundamentally different from the trial and error learning we see


in machine learning so what makes us humans such effective learners what is this magic ingredient that allows us to only require very little label data to learn a task incredibly well well this magic ingredient is something that we call having a general understanding of the world it's something that we all some also called common sense we humans we just know how


things work right we just know how things work and we obtained this general understanding through observe raishin from the day that we are born until the day we die we humans we observe we observe what's happening around us with all our senses we smell we touch we see we hear and this continuous observation this taking the world around us as our supervisor this


is actually what forms our general understanding and all this observation lets us understand the true meaning of things what does it mean if something is heavy what does it mean if something is hot what are the implications what are the consequences also we start to understand abstract concepts and concept relationships such as friendship for example and all these forms our general


understanding and this general understanding makes us such effective and efficient learners because this means everything we learn we do not start from zero we always have a head start we always base everything we learn upon our understanding of how the world works and all our previously acquired knowledge and this is what makes us human such effective and efficient


learners well that's great for us humans right but this talk is not primarily about human intelligence but of course it's about artificial intelligence so the question remains how do we inject this general understanding into machines how do we make machines effective and efficient learners how do we allow machines not to need tens of thousand hundreds of thousands millions of label


data points anymore and still achieve great great maybe even human level performance well the answer is quite simple actually we just need to let them observe the world we just need to make the data and their supervisor well but how do you do that how do you force a machine to observe the world doesn't sound too easy well imagine you put up a video camera at this traffic junction and that video


camera just continuously records what's happening in that traffic situation so you basically end up with an endless and an endless sequence of images and now imagine you have never seen a car before you have never seen a motorcycle before you have never seen traffic before and somebody gives you this video and you watch that video I promise after a sufficiently long amount of time you


will have figured out what a car is what a motorcyclist and how traffic works when it's a car allowed to go right when it's a car allowed to go left when it needs to stops and so on you have understood the concept of traffic by simple observation well that's how we humans do it but again how do we frame this a machine learning problem now the good thing is the good news is that


supervised machine learning problems always somewhat the same they always have this very very basic structure that you can see here and I would say 95% of all supervised machine learning problems are framed just like that you have some kind of input which in our case of course is our video our sequence of images that goes into a model could be anything in our


case that lets say some type of neural network that model spits out a prediction and we then compare the prediction to the label to the truth based on a difference based on the error that our model makes we're going to take a step in our optimization procedure for example using gradient descent and the next time our model is going to perform better that's how supervised learning works at


the very foundations and it's quite clear about to use this input and also the model part is clear but what is our model gonna predict actually and what are the labels we don't have labels right we only have a video only a sequence of images so what should we actually teach our model to output and this is actually myself supervision comes in and this is also where your


creativity comes in you need to think about how do I shape and frame the data the unlabeled data that I have to form a supervised learning problem from it one example would be well you could simply chop up your endless video your endless sequence of images into smaller sub sequences and always take the last image of such a sub sequence and use it as label and then teach your network to predict


the future from the past you teach it to predict the end of that sequence from the beginning or the previous elements of that sequence of these images your model is going to learn to predict the future from the past it's going to learn when a car is going to move when it can drive forward when it has to stop when it's allowed to go left and went allowed to go allowed to go right and we


can do this in a fully automated fashion you know you can simply create a small Python program for example that chops up your video into small sequences always takes the last frame of such a sequence using it as label and then you train your network and you're gonna have an expert network when it comes to traffic basically and you can use this concept of self supervision of automatically


creating labels from unlabeled data also in other types of data it's very flexible for example you can use it on images you can just randomly crop out images of let's say faces and teach your model to predict these rectangles or you can also use it on text you randomly remove words from text and use the surrounding words the context words to predict the target word the missing word


that you removed well that's that's awesome but so what right what what are you gonna do with the model I predict the missing rectangle in an image that's usually not what you want to solve and that's true so what we actually gain from this what do we gain from a model that knows how to complete an incomplete face well what we gain from it is understanding that model by performing


this task by learning this task builds up an understanding of the concept of a face it learns very low-level representations about the face and also high-level representations such as it's gonna learn that usually areas one knows in a face there are two ears in a face these ears sits on the sides of your head and so on it will learn a general understanding of the concept of a face


and you can use this understanding you can use this knowledge for any other task you really want to solve that is somewhat related what does it mean well first let's say you do exactly what did you randomly remove rectangles for images of faces and if if you have little images of faces you can remove rectangles in a variety of different ways right so you can actually end up


with a large data set you train your model to predict these missing rectangles your model will then have an understanding of the concept of a face then you take the same model you take the model with all of its knowledge and use it for whatever task you really want to solve for example for predicting the age based on images you just fine-tune it a bit maybe modify the architecture


bit no big modifications and your model is not going to give you the missing rectangle anymore but it's going to give you the age of the person that's visible on that image and all that by simply using a model that has acquired general understanding through self supervision and then you just fine tune it with little label data for the tasks you really want to solve what you gain from


that is overall well first of all since your first building up general understanding and then fine-tuning your model to the task you want to solve and that means your model will need less label data to achieve better performance your model will converge faster because its weight already in a somewhat optimal position and also your model is going to be more robust because the features it


learned these important characteristics will learn on a very large data set on yourself supervised data set and you were just fine-tuning on your downstream tasks where you're potentially a very little labeled data this is actually really a win-win situation here but how does this work in practice instead really usable in practice well and this is why I brought this industry case with


me which we did together with upstream ability and our task at hand was estimating mode of transport from GPS movement data well GPS movement data is quite easily described you just have sequences of GPS coordinates and then associated timestamp and basically what we what we got was a data set of these trips across Vienna and that person here started his trip at the Green Dot ended it at the


red dot and every dot in between was a GPS signal our task was for each of these GPS signals estimate which mode of transport he was taking was it going by bus was it going the Metro was he walking cycling going the car and so on actually quite hard quite complex given the little information so the first thing we asked our self here was so how would we humans solve this problem well what I


would do is I would go on Google Maps and check out where transport lines are running right because if a GPS signal is nowhere close to a metro line probably he was not going by Metro and that's what we did we enriched this map of Vienna by adding image channels and each image channel represents a different transport network one represents the Metro network one the bus network and so


on this is awful for humans right this produces an image you cannot interpret but it is great when it comes to deep learning well and then we used that information to embark on building a super lies deep learning model with no self supervision at this point the way we framed this problem was well we framed it as a sequence learning problem what does this mean so basically we


thought our hypothesis was the movement history of a person provides us with important information also for estimating his current mode of transport because obviously if a person has been on the Metro for the last five GPS signals because his GPS signals were just on top of that metro line and his current trip ethical is again on top of that metro line it is quite likely that


he is still on that Metro because changing transport modes of transport is actually associated with costs in terms of time effort and maybe even money so our hypothesis was abused the previous movement history to estimate this current mode of transport so again how do we frame this a machine learning problem we have a big big image with dots on it and then we want to


incorporate the sequence of dots to make our estimation so you can do this in a variety of ways of course the way we did it was for each of these dots we want to incorporate in our for making our prediction we cropped out a small rectangle of the image around it so we end up with a sequence of image tiles each of these image tiles representing the path in total then goes into a


convolutional neural network a so-called CNN which is a type of neural network that is very good at extracting important information from images that CNN then produces what that CNN thinks is important for making the modality estimation so called feature vectors it learns feature vectors which is simply a numeric dense vector representation of that image and what is important in that


small image title in that sequence of image a feature vector s then goes into a recurrent neural network a so called R and n an errand n again is a type of neural network just a very different type which is great for modeling sequences from modeling temporal dependencies between elements of a sequence and since we are dealing with a sequence of GPS coordinates here or a


sequence of image tiles are actually now a sequence of learned image features an RNN is an obvious choice here and there are an N then finally it gives you in the end which modality the person was likely to be taking was he walking going by bus taking the metro and so on and that what you can see here all that was basically one single deep learning model and it worked out reasonably well


actually surprisingly well however you can imagine that this was not a small model actually we were dealing with I think more a bit more than hundreds of thousands of parameters in that model which is ok if you have sufficient labeled data and you can guess well we didn't of course we were dealing with rather little real-world data here which is a problem given if you use a


large network because the network is likely to overfit and so on so we thought about well how about self supervised deployer this is actually a very good very good case for that how about first letting the model develop general understanding and then taking this model with all of its understanding and using it for solving our very specific modality estimation tasks and


that's how we came up with that concept you could for example teach your model the entire Vienna transport network in a self supervised pre training stage just as we did before then take the not take that model with all of its knowledge and find unit to make modality estimation happen sounds good but how do you teach a model EBN a transport network and this is again


where it comes to self supervision where it comes to creativity how can you frame a problem from the data you have to make this happen one way is to for example randomly sample 32 by 32 pixels again these small image tiles from a map of Vienna which is publicly available of course and then you just automatically check which line of transport is running on that tile


it's all information is publicly available again you take that as labels and you teach a model to predict which line of transport is running on a given image tile your model is going to learn the Vienna transport network it's going to be an absolute expert and you can then take this model you can take this CNN that knows everything about the Vienna transport network and put it back


into our original architecture replacing the CNN from before the CNN that had to be trained from scratch our model now does need to be trained from scratch it has a big big head start it has a quiet general understanding so it just needs to be fine-tuned a bit and that's an end and passes on high-quality feature vectors again to RNN which learns the temporal dependencies and outputs the


predictions what you gain from this is you have a significantly smaller model because you do not need to train the CNN from scratch this means you can have many of the parameters frozen and you end up with a more with a model with trainable parameters going from a hundred thousand before to ten or tens of thousands now which is a lot better suited for little labelled data so again


what you're going to get is your model will be able to deal with little label data better it's gonna be more robust it's gonna converge faster because you simply have to optimize less parameters here and overall it's gonna make more accurate predictions given the same labelled data we have come a pretty long way now all the way from supervised deep learning why it's so great so powerful


but also we got to know its ugly side we got to know that it just did tons of label data to achieve this famous performance subsequently we found out how taking a self supervised approach can help us overcome that key problem of supervised learning and finally we're looking through the industry case that hopefully gave you a pretty good practical idea of how you can apply self


supervision to your problems to make your models work better on little label data I hope you enjoyed the talk feel free to ask any question you might have in case we are out of time for questions I will be hereafter talk as well so feel free to approach me and ask any question in private that may have been left unanswered also of course drop me an email and add me on LinkedIn


I'm always happy for for method for interesting message as I get and most of all visit our website craft works dot AI thank you very much okay um let's see I think we maybe have time for one question would you like to choose which one you think would be the best one the unites answer okay I think the second one what is the main difference between supervisors and self


supervised learning this is a important question because it's very fundamental so self supervised learning is supervised learning the concept behind self supervision is that you do not have a label dataset but you come up with a smart task how you can create labels from a previously unlabeled data set not needing any human labeling this is the concept of self supervision you use the


data that you have that video that we had before you do not have any labels you cannot just apply supervised learning because you don't have labels but you need to implement a self supervised task that where you first pull out these labels to actually make supervised learning happen and you're learning from a distribution that is within the data actually thank you and


hi and welcome to an illustrated guide to recurrent neural networks a Michael also known as learned vector I'm a machine learning engineering the natural language processing and voice assistance space if you're just getting started in machine learning and want to get some intuition behind recurrent neural networks those videos for you if you want to get into machine learning

Illustrated Guide to Recurrent Neural Networks: Understanding the Intuition


recurrent neural networks are a powerful technique that's important to understand if you use smart phones and frequently surf the internet odds are you use applications that leverages RN ends recurrent neural networks are using speech recognition language translation stock prediction it's even using image recognition to describe it content in pictures so I know there are many guys


on recurrent neural networks but I want to share illustrations along with an explanation of how I came to understand it in this video I'm going to avoid all the math and focus on the intuition behind RN ends instead by the end of this video you should have a good understanding of RN ends and hopefully have that light bulb moment so RN ends are neural networks that are good at


modeling sequence data to understand what that means let's do a thought experiment say you take a still snapshot of a ball moving in time let's also say you want to predict a direction that the ball is moving so with only the information that you see on the screen how would you do this well you can go ahead and take a guess but any answer you come up with would be that a random


guess without knowledge of where the ball has been you weren't having an update of to predict where it's going if you record many snapshots of the balls position in succession you will have enough information to make a better prediction so this is a sequence a particular order in which one thing follows another with this information you can now see that the ball is moving


to the right sequence data comes in many forms audio is the natural sequence you can chop up an audio spectrogram into chunks and feed that into RN ends text is another form of sequences you can break text up into sequence of characters or sequence of words okay so our ends are good at processing sequence data for predictions but how well they do that by having a concept I


like to call sequential memory to get a good intuition behind what sequential memory means I like to invite you to say the alphabet in your head go on give it a try that was pretty easy right if you were taught the specific sequence it should come easily to you now try saying the alphabet backward I bet that was much harder unless you practice the sequence before you'll likely have a


hard time here's a fun one start out the letter F at first just struggle with the first few letters but then after your brain picks up the pattern the rest will come naturally so there's a very logical reason why this can be difficult you learn the alphabet as a sequence sequential memory is a mechanism that makes it easier for your brain to recognize sequence patterns all right so


are n ends have this abstract concept of sequential memory but how the heck does it replicate that concept well let's look at a traditional neural network also known as a feed-forward neural network as an input layer hidden layer and output layer how do we get a feed-forward neural network to be able to use previous information to affect later ones what have we had a loop in a neural


network that can pass previous information forward and that's essentially what a recurrent neural network does an RNN has a looping mechanism that acts as a highway to allow information to flow from one step to the next this information is the hidden state which is a representation of previous inputs let's run through an art and use case to have a better understanding of how this works let's


say we want to build a chatbot they're pretty popular nowadays let's say the chatbox can classify intentions from the user's inputted text to tackle this problem first we're going to encode the sequence of texts using an RNN then we're going to feed the RNA and output into a feed-forward neural network which will classify the intents okay so a user types in what time is it


to start we break up the sentence into individual words rnns work sequentially so we feed it one word at a time the first step is to feed what into the RNA the RNA encode what and produces an output for the next time we beat the work time in a hidden state from the previous step remember that the hidden state represent information from all previous steps the RNN now has


information about the work what in time we repeat this process until the final step you can see about a final step the Arnon has encoded information from all the words in the previous steps since the final output was created from the rest of the sequence we should be able to take the final output and pass it to the feed-forward layer to classify in intent for those of you who like looking


at code here are some Python showcasing the control flow first you initialize your network layers in the initial hidden state the shape and dimensions of the hidden state will be dependent on the shape and dimension of your recurrent rail network then you loop through your inputs past a word and hence a into the artnet DRN and returns the output at a modified hidden state


this modified hidden state should now contain information from all your previous steps you continue to loop until you're out of words last you pass the output to the feed board layer and it returns a prediction and that's it the control flow of doing a forward pass of a recurrent neural network is a for loop okay now back to our visualization you may have noticed the odd


distribution of colors in the hidden states this is to illustrate an issue with our n ends known as short-term memory short-term memory is caused by the infamous vanishing gradient problem which is also prevalent in other neural network architectures so as yarn and processes more steps it has troubles retaining information from previous steps as you can see the


information from the word what and time is almost non-existent at the final step short-term memory and vanishing gradient is due to the nature of back propagation algorithm used to Train and optimize neural networks to understand why this is let's take a look at the effects of back propagation on a deep feet board neural network training and neural network has three major steps first it


does a forward pass and makes a prediction second it compares the prediction to the ground truth using a loss function the loss function outputs an error value which is an estimate of how badly the network is performing last it uses the error value to do back propagation which calculates the gradients for each node in the network the gradient is a value used to adjust


the network's internal weights allowing the network to learn the bigger the gradient the bigger the adjustments and vice versa here's where the problem lies when doing back propagation each node in a layer calculates its gradient with respect to the effects of the gradients and the layer before so the adjustments in the layer before it is small then the adjustments in the current layer will be


even smaller this cost gradients to exponentially shrink as it back propagates down the earlier layers failed to do any learning as the internal weights are barely being adjusted due to extremely small gradient and that's the vanishing gradient problem let's see how this applies to recurrent neural networks you can think of each time step and over current no


network as a layer to train a recurrent neural network use an application of backpropagation called back propagation through time the gradients value will exponentially shrink as it propagates for each time step again the gradient is used to make the adjustments in the neural networks weights thus allowing it to learn small gradients means small adjustments this causes the early layers


to not learn because of the vanish ingredients the RNN doesn't learn the long-range dependencies across time steps this means that there is a possibility that the word light and time are not considered when trying to predict a user's intention the network has to make its best guess with is it that's pretty ambiguous and would be difficult even for a human so not being


able to learn on earlier time steps causes the network tap short-term memory okay so RNN suffer from short-term memory so how do we combat that to mitigate short-term memory to specialized recurrent neural networks were created one called long short term memory or lsdm for sure the other is gated recurrent units or gr use LS TMS and gr use essentially fokin just like our meds but they're capable


of learning long-term dependencies using mechanism called gates these gates are different tensor operations that can learn what information to add or remove to the hidden state because of this ability short-term memory is less of an issue for them to sum this up RNs are good for processing sequence data for predictions but suffer from short-term memory the short-term memory issue for


vanilla arm ends doesn't mean to skip them completely and you still more involved versions like LST MS or gr use RNs have the benefit of training faster and uses less computational resources that's because there are less tensor operations to compute you could use LST MS or gr use when you expect a model longer sequences with long-term dependencies if you're interested in


Germany's super-rich no other European country has as many billionaires and while their fortunes are growing more and more Germans are living under the poverty line set by the Organisation for Economic Cooperation and Development the OECD the press frequently reports on the country's high income inequality and low social mobility but little is known about the super-rich nate'd sure money

Germany: The discreet lives of the super rich | DW Documentary


attracts success and success attracts money I do believe that I've experienced myself you can suddenly connect with people previously out of reach who are the kind of shores that had - who are Germany's super wealthy how do they live and how do they see the country they live in the way wealthy people in Germany are talked about bothers me because it's sensationalizing in sycophantic and in


no way reflects what wealthy people have done and continue to do for this country it creates this impression of rich people being like Scrooge McDuck that they had these money bins in which they wallow in their coins look Scrooge McDuck wants more more MORE that's not my world I like earning money is less but not swimming in it we thought we wanted to get closer to the discrete


world of Germany's ultra rich of company owners and heirs worth millions we wanted to find out what makes those on top of the world tick [Music] every year an exclusive event takes place in the Schloss hotel Kornberg near frankfurt to which the public is not invited it's the annual Hall of Fame evening for the business monthly manager magazine hardly any other occasion in


Germany draws as many wealthy business owners [Music] the Iggy had laudatory esteemed presenters dear jury members dear ladies and gentlemen welcome to manager magazine's Hall of faith when we first founded the Hall of Fame in 1992 we wanted to take a stand for excellence and unconditional entrepreneurship and against faintheartedness and averageness we begin our nominations today with Harv


dama moot his company United Internet is valued at around 11 billion and on our list of Germany's richest people he ranks 25th with a personal wealth of 4.5 billion euros collectively the guests of this elegant evening are worth billions of euros this is the face of wealth in Germany mainly male and although we're close to the super-rich here their world remains


somehow out of reach Hamburg home to manager magazine's parent company the speaker group hockey any publication keeps closer tabs on Germany's ultra rich every year the magazine's team gathers information on wealthy Germans and using the Forbes model makes a special edition with a list of Germany's 1001 richest people it's painstaking work how many billionaires are video


last year it was a hundred and thirty six right no last year it was around 170 170 okay someone who wants to get on our list at 1001 richest Germans needs to have around 100 million that doesn't have to be money in the bank most people have it as assets or as property but that's the ballpark we're looking for to be on our list of the richest Germans the thousand vison contant


so nice and our chances editor-in-chief Stephon clothespin has been around Germany's ultra-wealthy for years what does it take to get on the list as it Amazon the top I'd say that the top 150 spots on our list will always go to company owners and heirs even if you're a chief physician you'll need to see a whole lot of patients to become a billionaire managers also have


a hard time getting that high up here in Germany there's a debate about whether the heads of Dax companies earn too much but if you compare it to what people in similar posts in the US the UK or China earn its peanuts so that alone can't ever make you one of the truly ultra rich in them it's a club des vins a new chef creating a list of the wealthiest people


is especially difficult in Germany though not because they aren't enough of them [Music] he somehow managed to amass this huge Empire and fortune in just a few years his wealth is estimated to be around four billion he's the least known super-rich Germans would you let us interview him at home they prefer to fly under the radar to avoid envy the super-rich don't like to melt themselves


so to speak you can normally only get interviews inside the homes it's second-tier once there you'll always get guys who get a kick out of publicity like mr. Marsh - and such getting a good shot of mr. Marsh Mayer is never a problem but the real money keeps itself hidden the super-rich are trying to go unnoticed sometimes they even try to hide there are no photos of


several people on our list you won't find a single picture if you go online and google them there hasn't been a photo of their hymens germany's richest family for decades many stay hidden because they want to live normal lives and think that they won't be able to do so if they're known to be multimillionaires Lemkin for months our interview requests were rejected agreed film shoots were


canceled last minute none of the rich wanted to talk to us about money finally we got lucky in the financial hub Frankfurt here in a prime location tucked behind the Bank towers sits the asset management company for come Foca managers German business family's fortunes its chairman Christian fanbase Todd's home provided some insight as to why rich Germans are so shy


the joy many wealthy Germans are reluctant about stepping out into the public eye because they're afraid they could be seen negatively they ask themselves what do I get from showing myself to the public it doesn't give me anything on the contrary it could lead to some crazy person taking note and breaking into my home or kidnapping one of my children and those fears are not


unwarranted then there's also the fact is that many heirs are inheriting fortunes that are somehow tainted by or related to the Third Reich the Abbott ear of rich Germans avoids publicity as if it were the plague but what are they so afraid of if I asked a number of my friends whether they'd like to be interviewed for this film and each one said no they'd say someone else can do


that better than me I can't do it right and I might come across wrong they think they'd have much more to lose than to gain even calm [Music] after a lot of back and forth with his press team one ultra-rich German did agree to meet us Michele Otto is the chairman of the Supervisory Board of the Auto Group and one of the ten richest Germans we asked him why Germans are reluctant


to show off their wealth any wiring doing so would lead to envy in the u.s. achievement and wealth has seen in a highly positive way but here they carry a bitter aftertaste where does his fortune come from or how did he get his wealth that scares some artists whom with a shrine weren't you I think a lot of people find their own wealth a little nauseating I like it Pindos good dick


Hoffman grew up poor his mother ran a small drugstore in post-war Hanover her son had bigger plans in 1972 the idea of opening the first self-service drugstore in Germany came to him today he's a multi-billionaire why was he happy to step into the spotlight negan unfunctional in the early years it was just about getting the name rossmann out there so when I was invited to a talk show on a small


regional channel I liked going because I thought free publicity for my company but then two or three years ago I started to understand that this slightly flabby balding man whose teeth aren't perfect was hungry for recognition himself back then I thought I was stepping into the limelight to promote the company but everyone rationalizes their motives and I did too y'all stop


adore within 40 years horsemen became the most profitable drug store chain in Europe with stores in six countries in 2018 a total of 56 thousand people were working on the chain Hoffman is active in other business areas as well and he speculates on the stock market that could not take any capitai legal debt so I have a couple of private equity investments the


largest is palliated between 80 and 100 million depending on the spot rate so it's a fair sum there are also shares in different industries introduced a mob will be boring it's off to tear a little I'm fine you're still referred to as an SME because small medium enterprise right why do you think that so I don't really know with 55,000 employees you're not really a medium enterprise anymore


you're in another league we're back at Manager magazine in Hamburg with photos with a special issue are being selected the list of the richest Germans includes a notably high number of company owners from the so called medium-sized businesses it's a German peculiarity and it's not the only one the cover story for this current issue is that last year money rained


down on Germany's super-rich up here we have the chef Lutz the matriarch and her son who now owns 80% of mom has kept only 20 on the chef's loose have been at the top of our list for many many years I'd have to check exactly how much they're worth but around 20 billion give or take a bit down here we have MS Basel toss in the head of the Hinkle class she's the first and so far only woman


chairman of the Supervisory Board of one of the 30 Dax companies Germany's economy is still extremely male-dominated what's interesting is if you compare our list here with a list of the ultra-rich in the US we have a lot of old money old companies that have been around for decades in the US you have all those lads from Facebook Google snapchat and so forth that have bubbled


up to the top of the list we don't have that type of thing here and compared with other countries Germans are very reticent about showing their wealth very few Germans sail around in boats like this usually that's Americans Russians Chinese and so on here you don't really show your money you might have various houses villas and such but there's likely to be a Fulks park and


parked out front it seems the average ultra wealthy German Israeli conspicuous unlike in the US athletes actors and TV personalities rarely make it onto the German list important on guns even though we put a great deal of love in a sweat into estimating guests fortunes they're probably much bigger especially if we're talking about urban real estate prices have exploded over the past 10 years


a lot of people have doubled their property assets so if you started with five billion in that market you're likely to have ten or fifteen billion today money makes money but while rich Germans fortunes have exploded since the financial crisis due to the increase in value of real estate stocks and assets those with average incomes have had to swallow losses even liberal economics


Institute's are concerned about social inequality in Germany how do the super-rich see this disparity the headline in the newspaper reads the rich are getting richer and richer which is true the rich are getting richer yes it's true but it's also false there are 20 million citizens in Germany who have assets worth between 100,000 and 1 million so millions of people are getting richer


now the rich are getting richer even faster because one factor is probability that they can dedicate much more time to increasing their wealth about initiative the Left Party would likely say split up your wealth but my response is I also do things for the world which I live in I don't just take I'm not a socialist though I can only do things because I have things I can y'know Bastogne Posada


Michel Otto is one of the rich whose wealth has increased he successfully transitioned his mail-order business into a digital enterprise over 20 companies now belong to the Auto Group we wanted to know how Otto sees the debates about rich and poor do the rich understand the worries of the poor in the tough one just one good when people talk about those on top who don't


understand those at the bottom I wouldn't say that applies to me because I was not born rich I came to Hamburg as a refugee from West Prussia and my father had to start from scratch that's why I absolutely do understand people living in poverty today on the other hand I'd also say that if Germany is getting more and more millionaires people with small or medium-sized


businesses because generally the millionaires in question did build a business they now run then I think that's great because they are the people creating jobs for me that's what should matter in this debate we should focus on that and not rich versus poor voted is good young ensuring that the wealth of rich families can increase despite current zero interest rates is the mission of


Christian fund based on science company each day he and his employees sent out investment opportunities from banks and other entities given exclusively to his wealthy clients this is a way to present an offer to just a handful of valued clients and focus Oh [Music] is it fair that rich Germans are able to increase their wealth while the rest of the population gets left behind


[Music] I think it's difficult to apply terms like equity and fairness to the distribution of wealth I would say that here in Germany were better off than ever before and people here live in above-average circumstances nonetheless we have to make sure the gap between rich and poor doesn't get too wide because we don't want social conflicts like in the US or Latin America to


happen here for better times company Fulgham is a so-called multi-family office family officers take care of the needs of very wealthy families managing and increasing their assets it only makes sense to use family offices if your wealth is upward of 30 million euros who can afford such a thing well a family office like ours obviously can't discuss its clients we


have well known German business families that's our typical client profile Mikasa she can take someone who is or was a company owner thinks differently to someone who has spent their life as an employee or they are families who have had money for a long time so wealth is so to speak in their genes and then there are families who have just come into their wealth people who are still


pumping with entrepreneurial energy they're usually quite different from heirs option Kaka I see it as a top top in my Stanford [Music] - Allah seems to have plenty of this entrepreneurial energy his company is McFate Europe's largest fitness studio chain its headquarters are in an old baking Factory in Berlin Hina Shula started off small but today his wealth


is valued at 250 million euros [Music] which bring me finish then they're from I was born near long back grew up in a small village normally in villages you do the sports available and when I was around 1516 my role models were Arnold Schwarzenegger and Stallone that's how I came to the fitness world Shahla went from secondary school to complete a Salesman apprenticeship and


became the manager of three supermarkets then he decided to start something new - when I was 25 I decided to switch to the fitness industry my idea was to open a gym where anyone could train the matter kids or her income that was the initial idea and I had big goals I wanted to be number one in Europe but that's all I had I didn't have financing okay good it was 1997 when shala opened his first


fitness studio in verts book close to his home village he made use of some unusual methods on the monetary hoods book was a big step for me I opened my first fitness studio there under the slogan now also inverts book which people saw through as a marketing gag because customers came to me and asked where else we had studios so I gave it some thought and came up with the next marketing gag soon


also in ela anything that did the job and also put me on track to going from verse bog to Erlangen and then it grew from there cuza ten years later Heine Shula reached his goal he's number one in Europe and still expanding McFate now owns ten Fitness companies as well as its own model agency meanwhile Shanna is getting ready to open fitness studios in the US


des moines severan is getting to be number one is much easier than staying number one I think if you want to be successful you need to be a bit of an alpha animal inside the investor would probably pick a brand and say okay that could work I like it although it's probably two steps too far from any but if someone doesn't want to get involved with us because of it so what I'm


convinced that you have an easier time if you fought your way to the top and to success lions feel it and so do partners I think that's our situation which is why I can also imagine that someone who inherits something or takes over or even has to take over a company in the second or third generation will have a much tougher time and Montezuma Muhsin as CSV aha but there are plenty of heirs in


Germany huge fortunes and thousands of companies have been passed from one generation to the next they are heirs who don't want to and others who shouldn't take over their parents businesses succession is hugely important among Germany's richest Michel Otto inherited the mail-order company from his father and successfully managed it from when the kinder is this it will


be harder for my children though because now the Auto Group has a hundred and twenty three companies in over 30 countries I know every single company either because I was involved when it was founded or because I led the takeover negotiation but my children don't yet know that many companies guns wunderkind otto's children have opted against direct succession I think it's important


to give your children the option without pressuring them so you don't force them into a role I think that mistake is made often and I'd say it's bad for both the children and the company when master kundo under Thunder Lim kaput so now we have one point of sale Berg Vader are you a bit tired yes it's for an exhausting couple of days in ten minutes I'll be fine again it was just Bam Bam


Bam Bam dear horsemen also spent a lot of time considering who his successor would be it's now decided how old horsemen the younger of his two sons will take over as manager of the drugstore chain do we actually sell much yarn it's not exactly part of a drugstores range the trend sort of over it was big in 2016 already fading in 2017 and it's been stagnating in 2018


but it's not the worst product we have and it still brings in some revenue rose father had to show him the appeal of being in charge when the boys got more engaged I thought oh now we can't look as if the drugstore business is only about making money so I showed them how we're also active in social issues in Africa and so forth I've always showed my sons that what we do isn't just about


making money it's also about being responsible for others as a child I wanted to become a film director that was always my dream it still is today you might be thinking no I don't want him to be a director I want him in the company so I said well go ahead and become a director but being a director of such a big company is also exciting I didn't manipulate him a beauty goddess


but boy if you could fly off and asked whether I feel competitive toward my father and sure he built up this big company that I'll only take over but it's really difficult to keep something going these days the founding period has its own challenges and just having the idea of founding a self-service drugstore was hugely innovative but the fight to survive has gotten tougher and


that's the one I'm in a coma also now as a thought everyone in my family wants to be good at sports my father and I battle each other in tennis we all compete with one another and that's also shaped our view of life or mine at least as a tumulus miner despite their competitiveness the Osmond family reached a harmonious agreement with regard to succession cling against


Rommel wipes me out I normally prefer playing doubles that's much more appropriate for men my age I'm starting to worry a bit about you you worry about me with your inheritance I wouldn't worry I'd be looking forward to it selecting heirs and successes is usually not quite as amicable as at the Hoffman's christiane fan base time has seen many inheritance disputes in rich families


it's his job to preserve the family's assets and protected from all sorts of dangers maintaining a family fortune over several generations is incredibly difficult because it's under threat from being divvied up through inheritance from wealth disputes from expropriation from wars or simply from stupidity most families will have one or all of these happen to them only a handful of


families have managed to stay more or less afloat over centuries but those right on top have been switched out again and again when based on time speaks from personal experience his own family's history dates back 900 years I have a horribly long name at least on my birth certificate there's my six given names Christian Lawton Ludwig William and Maria as a good Catholic followed by BA horn phone


Malkin Haim Jane and beshte Simon what at work are normally called mr. Phan Bettelheim and at social events baron or lord baron baron from back home is an indirect successor of the figures the richest family in German history when does wealth begin for you if you're asking me at what point I consider someone to be truly right I would say over a hundred son I am definitely not rich I've


escaped nakute but I'm comfortable you and our family is comfortable and I'm certainly not complaining when you have a family history as long as mine your family has seen everything near bankruptcy years overflowing with money inside and years when a lot was lost on the one wall news he takes us to the hunting lodge of the best horse hime family in Turlington it


looked like the Lord had been lost forever during the division of Germany the house was built in 1892 was a hunting lodge for my great-great grand uncle's it's been in the family ever since except for a short interval it was expropriated in 1952 and then restituted in 1992 and since then I've owned it the family's hunting lodge survived expropriation and socialism without much