the supervised machine learning the unsupervised machine learning and the semi supervised machine learning while the supervised machine learning needs the ground tooth that is the label data to build a model the unsupervised machine learning predicts unlabeled data using clustering techniques now the major concern for most data scientists is the labeled data


as we ourselves face the same problem while building a model for wound tissue classification it is very difficult to get an expert to label the whole set of data thus the scarcity of label data is the major con constraint for supervised machine learning now here is where the semi-supervised machine learning plays a major role which takes the advantages of both supervised machine learning and

MixMatch: A Holistic Approach to Semi-Supervised Learning


unsupervised machine learning to give us label data and according to the authors of the paper mixmatch claims to have developed a technique to label the unlabeled data with much better accuracy than the present day there may supervise techniques as per the abstract of the paper mix match unifies the current dominant approaches used in semi-supervised learning


to produce a new algorithm that works by guessing low entropy labels for data augmented unlabeled examples and mixing labeled and unlabeled data using mix up so before going to the actual algorithm let us just have a look at the key terms which will be used to understand the algorithm the consistency regularization it applies uh data augmentation to semi supervised


learning by leveraging the idea that a classifier should output the same class distribution for an unlabeled example even after it has been augmented in other words labels should not change when noise is added the model called pi model is used mixmatch utilizes a form of consistency regularization through the use of standard data augmentation for images such as random horizontal


flips and crops now the entropy minimization it basically means that we need to reduce the randomness in the prediction of the unlabeled data or the classified decision boundary should not pass through the high density region of the margin marginal data distribution this is done by outputting low entropy prediction on the unlabeled data so mix match also implicitly achieves entropy


minimization by adding a loss function and using a sharpening function on the target distribution for unlabeled data the traditional regularization is again a method applied to avoid overfitting of the model we have two types of regularization l one or the lasso regression and l two or the ridge regression the range regression adds squared magnitude of coefficient


as penalty to loss function and mix match uses the squared or the l2 laws on prediction and guest labels and lastly the main teacher to overcome the problem of inconsistency in using the exponential moving average of label prediction on each training set on large data a mean teacher a method that averages model weights instead of label prediction is used mean teacher improves the test accuracy


and enables training with fewer models fewer labels so i will be giving a brief description of the steps involved in mix match as the four points are like the data augmentation label guessing entropy regularization and mix up so in data augmentation it is common approach to compensate the scarcity of the unlabeled data and data augmentation is done by applying transformations on the


input data points such such that the label remains unchanged data augmentation is done both on labeled and unlabeled data the individual augmentations are used for generating a guest label we'll be seeing more about it label guessing for each unlabeled example mix match produces a guess for the examples label using the model's prediction this guess is later used in unsupervised


lost term the entropy regularization to enforce the fact that the classified decision boundary should not pass through the high density region of marginal data distribution the classifier outputs low entropy prediction on unlabeled data this is done by adding loss term which minimizes the entropy for the unlabeled data mix match applies sharpening function to reduce


the entropy of the label distribution and this is done by adjusting the temperature of the categorical distribution now the last is the mix-up a mix-up is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels by doing so mixer regularizes the neural network


to favor simple linear behavior in between training examples mix-up also reduces the memorization of corrupt labels increases the robustness of adversarial example mix match uses mix-up with a slight modification as the final step of its algorithm yes okay brilliant so uh thank you sapna for um giving a brief introduction about the paper before i get into the algorithm i wanted


to just stop for a bit and ask if there are any questions anybody has so far okay i guess not so i start from um talking about more on the algorithm bits where if you see this is an image taken from the paper where it says that the algorithm basically augments the unlabeled image and tries and classifies it based on the number of augmentations and then draws an average to guess


the label of this unlabeled image that it has after it has this average getting predicted or the guess label getting predicted that less label gets sharpened uh what sharpening does is it basically moves the line of prediction or the line where the decisions are made away from the the higher density association of the data points uh because that makes it easier


to give a consistent prediction and also increases the confidence of the predicted labor as you see in label guessing we only guess the label of the unlabeled data however we do augmentations on both labeled and unlabeled data to come to this um label guessing wherein the label is guest based on whatever is in the label data as well as in the unlabeled data but


transformations are mostly done on the unlabeled data as many times as required but according to what the authors have seen and gotten the results uh k that is the number of augmentations uh is two that given a result which did outperform all the standard state-of-the-art methods after this the sharpening formula or the equation that they've used where they do have another tuning


parameter called st which is the temperature required to sharpen the image prediction or the label that is guessed based on the average prediction that is obtained in the previous step they average they take the decision boundary away and make the prediction more confident so that when the loss is calculated on the unlabeled data it it is not affecting the the final prediction


after sharpening what remains is two sets of data one is the augmented label data with the sharpened predictions and the label data which already has its own labels so what is done is a mix-up based on this parameter tuning which is called alpha so what basically makes up does is let's say uh you have an image of a tiger and a cheetah and you want to see how much of one image corresponds to


a tiger and how much one image corresponds to cheetah so what you do is you mix these up together saying 80 20 and have a new label saying that that image is now classified as eighty percent of a tiger and twenty percent of vegeta and that mix up is this alpha value which is again tuned based on your data sets and availability of the number of data or images that are collected i


forgot to mention the sharpening temperature value that they the authors have taken as a constant for their experiments it's 0.5 yeah i think it's 0.5 after the first pattern let me just quickly go back and see um i think it's in the next slide so we can look yeah it's 0.5 so what they they've tuned it uh to sharpen the picture sharpen the the prediction then they tune in the


alpha parameter to have a proper mix up of the labeled and unlabeled data and they form in a big bunch of mixed data set which has unlabeled as well as label data set and i think we've been just talking too much of labeled and unlabeled this is the most important factor of a semi supervised learning method wherein you have your training set that comprises of labeled as well as unlabeled data


the reason being it increases the chances of getting more label data which in turn will be fed to your model to have a larger training set than what was initially available so all this is getting done to ensure that your training set increases in number than what you have earlier the last step in the algorithm is calculating the loss because we have a label set the first


loss that gets calculated is the cross entropy loss for supervised learning and the second one asapna mentioned is the l2 loss for the unlabeled learning i would stop here and ask if there were any questions okay i go forward so this slide explains what i just talked about but in a more um direct way and what all they have tuned in so the key factors to keep in mind for


this algorithm to perform better or worse is tuning these hybrid parameters as the augmentations sorry k i'm sorry then the the sharpening temperature t the the parameters for mix up that's alpha and the weight of the unsupervised consistency loss lambda which is a hundred in this case um the authors performed two sets of experiments one was a normal semi-supervised


experiment set up where they compared it to the state of art uh semi-supervised learning models and um the other one is um the the method in which they cut out all the additional um transformations or sharpening or um any extra effort that they've put in to get into the efficiency or the accuracy level that the model performs to show that which uh bit in the algorithm performed the best


to get in the desired results of the desired accuracy at the end so for their semi supervised experiment set up uh they used a wide resonant model with 28 layers and a growth of factor two the data sets were our standard data sets at the zipper 10 say 400 the svhn which is the street view house numbers and the stls uh the models to compare were the main teacher as satna explained


in her brief introduction about what main teacher does it is one of the semi-supervised learning methods wherein the labels are based on the exponential moving average a virtual adversarial training again a semi-supervised method and the pseudo labeling again a semi-supervised standard model so you see from the results where the model that they used initially was the cipher


10 data set a supervised training of 50 000 examples was trained to to given the said accuracy but as comparison with mismatch it just used in 250 labels to given the desired output similarly they did it with svhn again with 70 3250 examples with no unlabeled data at all and they got in the said with a label of labels of 250 again um it given the said efficiency so what i mean by 250


labels and 2 and 73 257 examples is that the entire data set was used for training purposes for this particular supervised uh experiment out of the 7 73 257 only 250 labels were used in to train the remainder of the unlabeled data set to get in the performance that it shows in the results and this is the abolition study set up where i said that they talked about


adding or removing additional components that they used to come to this level to show which of the exact step in the algorithm performed better than the rest so you see the first one is about the mix and match where it's 100 mix and match and it gave an error rate of 11 for 250 labels and an error rate of six percent which is quite good for 4000 labels they removed the


distribution averaging or the guess labels and said the number of transformations as one it gave in a 17 error rate again no sharpening it given 27 they had an um they do mix up they did it with label data mixer they did it with unlabeled data mix-up and you see in this that the mix up is benefiting the performance of this model um going to the end uh concluding what we have understood


from the paper or the basic purpose of what the authors wanted to showcase using all the components that they have in creating the algorithm it is seen that the hyper parameters that they've used is the augmentations the sharpening the mix and match and the loss that they've calculated it seems that they are have contributed quite well in the performance of the model


but out of that in the as you seen in the previous slide that the mix-up seems to be the most important factor contributing to this a mix-up again is a tuning hyperparameter and if you tune it further you probably might get in better results if you don't tune it as much you might get in less results so these were the positives that i could get or we could see in the paper however


there is a bit of a negative or that that comes out through this algorithm that the time and cost needed to generate all the transformations the mix up and have a neural network set up on it does take a lot of time and it also requires additional gpus to be set in which perhaps could be seen as an additional overhead but it still does perform a lot better also


it leaves needs a lot less data to start to train in terms of having any expert coming in signing up the training set saying that yes a label is an a label b label smb label so you see that this method does showcase that it is one of a better ways to go forward in semi-supervised space for further reading i have listed a couple of uh papers to go ahead and understand


where the thought for building up this model came through um and i think this is the last slide if you have any questions i should be glad to answer them um did you try to use the implementation that they provided on online yes i i did try it i tried on lagoon data set and i could share in my uh code base with you to give it a try so so the the the results were were


really good did it improve uh well we don't have enough of um data for the wounds data sets so i would say it did perform good uh i'm not really sure whether it will perform better uh because the data was quite less but i do see that it will go ahead and do a better job gotcha awesome thanks any further questions i don't have a question but i was wondering if you could sure


um the network that you tried to run it would won't that that's one type you use or was it which category of one did you use all of them it's a semi-supervised learning space so labels are given for all and then i took in half of that as an unlabeled set and not have labels for it okay could you maybe share that on the channel yes appreciate it sure thank you


so yeah i have a question uh so what are the other like i'm not quite like uh quite familiar with semi-supervised vanny algorithms so what is the like what's the state of the art between before this algorithm what what what did they compare their girlfriends to in this paper um so he didn't quite understand your question so so before so when they introduce these


algorithms so they have they should compare it to something that how were things done before this uh mix of algorithms for semi-supervised so they did us uh if you see in here um uh they've used in a supervised method wherein they used all the 50 000 examples that were labeled so they needed labels for 50 000 items or entities and they had to train based on that but with the mix match coming in


they reduced the number of training set or labeled set to 250. if you know what i mean so so let me go back and i'll explain a bit of a semi-supervised learning space so semi-supervised learning space needs both a bit of a label set which is a trained set and a lot of unlabeled data set because what it does it takes a little bit of the label set and a bit of


unlabeled set and forms in a mix of labeled and unlabeled and the reason being that it tries and labels all the unlabeled um examples that it has in this mix so that the end training sample has got more no i i know i know i know like the idea behind but i i i'm not sure what i was just asking what was the state of the art before like before me this mix of algorithms


for semi-supervised it was a simple semi-supervised approach wherein you have iterations where you take in from your unlabeled set perform some basic transformations or have a have some sharpening done for your confidence uh of your label set having say uh prediction probability or maybe have co-training methods that have two classifiers learning on the same view etc and


then give your prediction so mix match as such introduces all of these methods together so they they didn't stop at just using simple uh transformations or simple decision boundary making algorithms or a mixed match they this is a combination of all three together so state of art didn't have this the novelty of this paper is mixing all of these uh semi-supervised learning approaches


together in one to come up with uh their holistic approach so that's why the name okay so basically though like uh those uh pseudo label and v80 and uh was it these are also semi supervised approaches right yes okay uh i have a question so suppose we take three examples right extending what you said cheetahs tigers and we add a third one leopards um and uh so cheetah and tiger are in


the 250 labels that are used and leopards are in the 4000. does that mean all the cheetahs and all the tigers in your data set are labeled or are some of them unlabeled it could be some of them unlabeled or they could be some of them labeled and it can be both it all depends on the percentage of the unlabeled data that you thank you hi i'm curious about one thing you say


that you tested with the wound data set uh did you try perform it against fast ai platform yes yes with first ai yeah and how how your i mean how is their performance with this method compared to plasma eye for the training uh set i did get in about a 64 64 to 69 percent uh accuracy on the training set initially and with this it did bump up to about 70 but i wasn't sure whether it was


uh the right thing that i was doing so that's i'm still playing around with it but uh i know where the where the tuning needs to be done so i should be able to have a better accuracy on that this is on the training set i've not yet gone on to testing the model on the test data they'd probably be higher than that yeah yeah okay thank you me any further questions


should we say that's it then okay thank you very much uh if you have any questions um please get back to me i should be glad to answer them and i will share in the link to my notebook i am still a little not worse than not books i write it in uh prehistoric language as an editor and things like that but i will move on to notebooks soon and i'll post it there


and when you're living by the hormones or stress not a time to create no out of time to open your heart not a time to learn sometimes we do a meditation we start opening our heart and start elevating the body's energy and then those emotions can drive certain thoughts of your future you have to understand that if 95% of who you are is a set of unconscious programs then the

DO THIS Everyday To Unlock The FULL POWER Of Your Mind! | Joe Dispenza


first step is lighting a match in a dark place this will be one of the most powerful videos you watch today right now you're about to discover the three secrets to unlock the power of your mind with dr. Joe Dispenza I hope you enjoy [Music] how do we change our energy and how do we sustain it for extended period of time how long is that time need to be until we really started to see that


sometimes immediate okay so so our research and we've done in the last six years because we were seeing so many incredible incredible things going on in our workshops I mean people stepping out of wheelchairs and all kinds of crazy things my church look kind of like a mega church but hopefully not that based in science yeah but but but isn't it amazing that some of these churches when


people get to believe whether they have science backing it or not they just it's the belief when they step out into the unknown step out of your body and who your size right instantaneously instantaneously and we do see that a lot and some people like yeah that's yeah can that be possible how can it be possible we we've done research now I just assembled a team of scientists


we've done 8,500 brain scans I can tell you I can tell you when a person's about ready to change I can tell you why really don't change I can tell you what it takes to change so what's it take to change well do you change most people keep their attention always their awareness on their body it keeps their attention on everything in their environment with people and things there


the brain is always scanning everything around us to determine what's known as safe and unsafe right and you know we do that all the time so our research shows that the moment you take your attention off your body and you go from somebody to nobody you take your attention off the people in your life and go from who you identify with from someone to no one and


so many people spend their whole life building an identity of being someone take your attention off your cell phone your computer your car and go from something to nothing take your attention off where you're sitting where you need to be someplace you have to go go from somewhere to nowhere and take your attention off time linear thinking about the predictable future they're familiar


past and fall into the generous present moment and go from some time to no time then all you're left with is consciousness and that's the moment are no longer playing by the same rules matter to matter and there's a very elegant moment that takes place in the brain in fact I was just showing my research to a group of researchers and Santa Cruz this past week and they were


blown away and I said now watch this person this person is going to have a transformational moment they said how do you know I buy I've seen enough of these and the next moment the whole brain just lights up that person is switched on they'll never be the same person again they're having a transcendental moment and we could actually predict it and teach it now it's a formula hmm just


like you doing sports if it just becomes a formula and then you change the formula and you add to it right so when you no longer are you're identifying with your body your environment and time that's the moment your pure consciousness now you're just an idea you're an awareness awareness awareness that has nothing to do with local space and time and now if you're no longer yond you're


anything you can go beyond and that's when the brain because the brain doesn't change the brain it takes a long time it takes a long time for the personality to change the personality for the ego to change the ego the programs that change the programs takes forever matter takes a long time to change matter but when you're in this moment you're no longer playing by those rules consciousness is


the phenomenon above matter in fact consciousness is beginning to activate or manipulate circuits in the brain people just think the brain is creating conscious no consciousness is executing the brain right so then if the brain can change then the mind doesn't change the brain mine is the brain in action is consciousness that changes it so when people begin to disengage and


get beyond themselves you are at your absolute best when you get beyond yourself and getting the person to that point how does someone get to that point yeah so we teach them that formula we teach them to that point where all of a sudden they reach that generous present moment where they just feel connected and when they're in that place all the things they thought they wanted they


actually no longer want because they feel like they already have them so then imagine living your life from that place you would be less judgmental you would be less frustrated us impatient reactive and so so the formula then is that it requires a clear intention which is a coherent brain and when you're living stressed out and something goes wrong and you're


threatened or you can't predict an outcome or you have the perception that something's getting worse or you can't control it you switch on that fight-or-flight nervous system they've talked about now here's what happens when that occurs you start shifting your attention from one person to one problem the one thing to another person to another place because your brain is trying to predict the next


moment well every one of those people and things in places has a neurological network in your brain so as you shift your attention from one to the next it's like a lightning storm in the clouds your brain starts firing very incoherent lis when your brain is incoherent you're incoherent and when you're living by the hormones of stress not a time to create no not a time to open your heart not a


time to learn not a time to trust and it's a time to run fight or hide so people spend 70% of their time of their life living in the state Wow so think about it so miserable yes so then when you're under stress if there's if there's a cougar around the corner you're not going to sit down and meditate sit still right but but so I'm a tree you got the survival genes


switched on and nobody is gonna believe in possibility when you're living in survival right yeah so then when you're living in stress what happens is you narrow your focus on the cause you now your focus on matter the object the thing and so people get switched on and all of their attention is on their outer world when the hormones of stress kick on the body gets an arousal now your


attention is on the body and of course when you're under stress you're trying to predict the future based on the past and now you're literally enslaved into three-dimensional reality so then how do you get what you want you gotta try harder you force it more you got a war cardio to fight for it it's matter trying to change matters Austin people just burnout right so then we now know


that when you go from a narrow focus on something and you begin to open your focus you create sense and awareness that the act of opening your focus causes you to stop thinking and if you stop thing can you no longer activate those circuits and you start to slow your brainwaves down hmm as you slow your brainwaves down you start connecting to that autonomic nervous system the thing


that's giving you life and all of a sudden when you get beyond yourself it says he's gone let's step in and just clean up this mess before he gets back really and its job is to create order and balance your body will start to do that for you the innate intelligence will step right in once you've connect you got to connect so you got to know how to change your brainwaves you can't


change your brainwaves you stay in an active state you're basically moving furniture around you're analyzing your life within some disturbing emotion and I can tell you after looking at all those brain stands if you're analyzing your life within some disturbing emotion you're going to make your brain worse in fact you are thinking in the past right so you teach people the formula how to


open their focus change their brainwaves connect to that invisible field and all of a sudden different compartments of the brain start synchronizing the front of the brain starts talking to the back of the brain the right side starts talking to the left side and all of a sudden what sinks in the brain links in the brain all of a sudden you see this person starting to feel more like


themselves and when you see those two hemispheres the brains start lighting up watch out because that person's gonna feel really hope they're gonna start loving life they're gonna feel like they're gonna be in love with life because the union of polarity and duality is wholeness at the exact same time coherent brain when you're resentful when you're judgmental when


you're impatient your heart beats out of rhythm why you're stepping on the gas and you're stepping on the brake at the same time your body and its intelligence living in survival is saying t-rex is back there but you're not running because you're sitting across the table looking at somebody smiling and your body's revved up right so the heart is beating and rhythmically and when that


happens you're you're squandering or you're using all the body's life force and turning it into chemistry right using all that energy to survive as opposed to think beyond right right so you're drawing from your vital life force that invisible field around your body and you're turning into the chemistry you actually are going to shrink your own feel the hormones of


stress caused us to be materialists right we when one of the stress were we're using our senses to determine reality so now you feel more like matter unless like energy more separate from possibility so then to teach a person then how to regulate that heart center and we do this we've done 6000 heart scans why because if I can teach you how to get in that heart state and I can


teach you how to activate that Center and I can teach you how to regulate an elevated emotion the heart starts to create a very coherent signature and when the heart starts beating like a drum like dropping a pebble in water it begins to produce a measurable magnetic field up to 3 meters wide now you're more energy than matter more wave than particle Wow now that field that's being


created is measurable and that's an energy and energy is frequency and all frequency carries information so what is the information when it makes it here that you're sharing in the world it could carry the thought of your healing why because it's consistent with the energy guilt isn't gonna carry the thought of your healing it's a different frequency and all of a sudden now the


person is elevating their emotional state and they're allowing their thought to be carried on their frequency they're broadcasting a whole new energetic signature but thoughts are the language of the brain and feelings of the language of the body and how you think and how you feel creates your state of being so then the question is if you keep practicing creating that state of


being it should become familiar to you yes or no the word meditation literally means to become familiar with so then if you're practicing moving into these elevated states and your heart is coherent and we're measuring and I can say Louis you got it now do it for 30 minutes now do it for 60 minutes and you practice creating that coherence you'll know when you're there and when you're


not yes or no sure and then you would be able to say like a skill like anything else give me a minute I'm gonna step out and you're gonna go back in the heart coherence and bring up that state now we get there in the heart then we practice a formula again rest your attention stop calling up elevated emotions and when you start seeing that that starts happening then you sustain it then you


keep practicing and all of a sudden it gets longer and longer and longer now what's the relevance behind that well we've measured neurotransmitters so when a person actually activates their heart the heart releases a chemical called oxytocin oxytocin is actually love chemical not oxytocin signals nitric oxide nitric oxide signals another chemical called endothelial derived relaxing factor what


does that do causes the vessels in your heart to swell you will literally have energy in your heart you will literally feel like your heart is full now once you have that feeling you're not gonna want to trade that feeling for anyone or anything you're gonna say well why would I judge that person if I judge that person will lose this feeling now all of a sudden you're self-regulating now once


the heart is activated I just was at the research lab this week once the heart is activated it acts as an amplifier and it amplifies energy in the brain so once you start opening that heart and it begins to signal the brain you're going to suppress the survival centers in fact the research shows it will reset your baseline in other words if you're anxious and vigilant and you learn how


to self-regulate you'll actually reset the baseline and you'll say well the trauma was 15 years ago I saw my somebody get murdered or whatever and they will say yeah yeah yeah but the moment the heart not the brain it's the heart that actually resets the amygdala and all of a sudden the person all of a sudden switches down and all something like that I just don't have anxiety we


have thousands of brain scans with anxiety and depression from people from all walks of life they've reset and all of a sudden they don't have that excited they don't have to take medications or do anything is they know how to self-regulate where does all anxiety stem from mmm and anxiety is doing this living in the survival when you're living in survival I'll tell you this when the survival


gene is activated out of the infinite potentials in the quantum field you'll always choose the worst-case scenario why because if you're in survival and you're preparing for the worst there's always better chances of surviving if anything less happens so people are always selecting the worst thing in their mind and they begin to emotionally embrace that future before it happens


thought and emotion you start conditioning so you're conditioning the body to become the mind of fear you keep doing that enough times once the body becomes them it's a subconscious program person as a panic attack try as you made a controller with your conscious mind you can't you programmed it subconsciously now you worry about the next panic attack and as you start


worrying about the next panic attack that's the vigilance that creates the next one well now here's what's happening in our work people who are self-regulating and creating these elevated states we have we have heart scans of them sustaining heart coherence for a whole hour during a meditation then at the end of the day they're still wearing the the monitor it's eight


o'clock at night they're not even in a meditation and for a whole entire hour they're in heart coherence we say to the woman what's going on here she said I have no idea I was just getting ready for bed and all of a sudden my heart just swelled up it was so intense I had to lay on my back and surrender to love instead of surrendering to fear she had a spontaneous love attack


instead of a spontaneous panic attack now I would call that the natural state of being so then if you're living by those elevated states and you know how to feel that emotion of your future before it happens you're less likely to wait for it to happen because you'll feel like it already happened you'll less likely try to control it you'll know that the moment you lose the


feeling you just disconnected and you're gonna make your way back and when you get good at it no person no thing no experience can take it away from you well now you're empowered and if you understand the laws of how creation happens then you're less likely to compete and rush to get what you want you're gonna know that it's gonna come to you and now that's the new model of


how do we create knowing is gonna come to us at the right time what if we want it faster no you just do it again hmmm but remember if you're trying to make it happen faster you're back to the old self right then yourself would never do that the new self would constantly stay there and so then how does it appear it appears in a way that you can't expect because if you can predict


it it's the known it's gonna come in a way that you haven't thought of an unknown and it's and it's got a rock your world it's got to catch you off guard it's got to leave you no doubt that what you've been doing inside of you that produces some effect outside of you and when you correlate what you've been doing inside of you with the effect that you produced outside of you


pay attention to what you did and do it again and the energy uh the joy that you feel when it happens you're gonna use that energy to create again now people say to me well I'm this way because of that person in that thing I would say to them so you mean them that person or that experience out there is controlling your thoughts and feelings that means you're a victim to your environment but


when you start changing your thoughts and feelings and it starts to produce an effect in your environment you're gonna change the belief that you're a victim consciously or subconsciously of your life to becoming more of a creator of your life and now all of a sudden you become more a creator of your life you can't blame anybody you can't say well that person and I think you'd have to


say I got to be greater than that environmental condition who in history can I study that had the same challenges now what was what was what did they do let me just work that into my rehearsal so that I can improve right just like you've done with sports is the same process yeah [Music] was more powerful than our thoughts or our emotions and do our emotions change


our thoughts or do our thoughts change our emotions yeah the answer is yes the answer is both I mean thoughts to me produce an electrical charge in the quantum field and feelings produce a magnetic charge in the quantum field thoughts wait thoughts produce a what an electrical charge okay and feelings produce a magnetic charge and how you think and how you feel broadcasts an


electromagnetic signature that influences every single atom in your life the thought sends the signal out I'll think about this and the feeling draws the event back so you could have the intent that you want wealth you want health you want success that's your intent that's your thought but if you're waiting for the experience to happen to feel it then you're not drawing the


experience to you because you're not feeling the emotion right so then teaching people once again how to balance their thoughts and feelings because you can you can enter that cycle either place sometimes we do a meditation we start opening our heart we start elevating the body's energy and then those emotions can drive certain thoughts of your future other times you


open your awareness you create brain coherence you have the vision of your future you begin to emotionally experience it however you want to jump on that cycle and then sustain it because the longer you're conscious of that energy the more you're drawing your future to you so then most people spend their lives right they we live in this realm called space-time


three-dimensional reality and you move your body through space and three-dimensional reality it takes time yeah so everything all your goals all your dreams all your visions you're gonna have to get your body up and drag it through space every day to pay off that you know that home that's in your future right right when you create from the field instead of from matter when


you're the vibrational match between your energy and some potential and your thoughts and feelings are coherent now you are going to begin to collapse time and space or the experience is going to be drawn to now now your the vortex to your destiny and now you don't have to go anywhere to get it because you're not playing by the rules of three-dimensional reality


you're playing by the rules of energy in the quantum so teaching people how to do this in getting better at it then all of a sudden they're not forcing and controlling outcomes in fact they're trusting and surrendering to outcomes because they don't want to get in the way because the moment you start trying to predict when it's going to happen or how it's gonna happen


you're overlaying a known over a place where there should be an unknown right so teaching people how to do that means we have to lay down the very thing we used our whole life to get what we want for something greater to occur right and so that transcendental moment is something that we're working on the mystifying and and you could be gluten free person you could be a gluten full


person you could drink wine not drink wine you could be rich you could be poor you could be any color any shape any size in fact you can't tell me you're too old to do this work you can't tell me that we got elders in this work that we show you other brain scans and you'd be blown away but they they know how to do it you can't tell me you're too sick to do this work we got people that have


reversed stage four cancer in numerous times and yeah it took a Herculean effort to do it but they love themselves for you can't tell me that you're too out of shape or too overweight or too under weight you can't I've seen it in all shapes and sizes you can't even tell me that you had a brutal past and people that have had very very dismal pasts that are free they're happy people you


can't even tell me you're you never meditated before in fact our research shows that many people have never meditated before have the most profound experiences because they're not trying to make anything happen they're just following the instructions right and and they don't have a habit of doing it so so we don't want to exclude anybody in the process we want to include everybody


so it turns out that our events tend to draw a good portion of men because of the science we have a lot of children now that are you know teenagers that are coming and people in their 20s we have great community of elders we have you know in our events sometimes 63 different cultures well coming to countries coming to our events between 50 and you know 65 so so we want it we


want to make it so inclusive that community becomes the side effects because because with a community of like-minded like like antler similar energy of people everybody understands they get one another you know you you communities tends to be the thing now that in terms of our social medium and the feedback we're getting everybody wants more community because you get a


you get a thousand people in the audience and their energy synchronized now you're talking us something so much bigger we're just gonna measure this I just talked to a researcher yesterday we're going to measure a thousand people when they reach that synchronized moment when they're we can we know that the entire social coherence in the room is orderly then if you're producing a


ambience coherent magnetic field in your heart and you're tuning into a thought or an intent and you got a thousand people doing that and your energy is gonna start interfering and commingling with the person next to you when that Energy starts to synchronize it's gonna produce a bigger wave the higher the amplitude the higher the wave the more energy there is so now you have one mind


in one heart and now when it comes to healing others and we've done the research on this now and we're collecting the data that we're teaching people how to administer a change in energy and the person that's laying there because it's not matter that emits a field that's the wrong way to think about it it's the field that creates matter you change the field you change


matter you know it's not your job to change the tumor the tumor is the illusion it's the pattern in the field that's that that has to be changed so once people start reversing this then you start seeing the tumors disappearing you start seeing blind people seeing deaf people hearing you start seeing people with Parkinson's disease switch on I mean you start seeing stage four


cancer is reversing because now they're you're you're you're swimming upstream you're going to the headwater in making that change so pushing the envelope and them seeing that in a community when a community synchronized towards the second half of a week-long event I mean as I said before we started the show I I'm more surprised than anybody was it's crazy what is the we


talked about I heard you say consciousness a couple of times what's the difference between mindset and consciousness to me consciousness is awareness awareness is paying attention and noticing and so 95% of who we are by the time were 35 years old is a set of unconscious automatic programs that we've just practiced so many times that we're not consciously thinking about


those so in order for you to change to answer the initial question that you asked the first step is you got to become conscious of your unconscious thoughts and you got to you got to start looking at those hardwired thoughts that that you think every day that it's just circuits that have been fired and wired together how do we do that should we take write a list at the end of the day


or one of the most common thoughts we had that day like how does someone become aware you don't have to do that you just have to sit down close your eyes and not move and then you'll get you'll you'll start seeing what am I thinking about right yeah and all you want to do is observe the thought because when you begin to observe that thought you're no longer the program now


you're the consciousness observing the program and you're starting to pull out of the program thinking about the thinking yeah who's doing the thinking of the thinking about the thinking that's who you are when you're not the program that's awareness right you got to become aware of how you speak how you act become so conscious so aware of it that you won't go unconscious and let


that thought or that behavior run you you got to say oh my god this feeling that I've been living by for the last 20 years is actually guilt I didn't know it was guilt because it just feels like me and all of a sudden as you start becoming conscious of it you're beginning to objectify your subjective self here you're pulling you out of those programs and nobody likes to do


that because it's uncomfortable they'd rather turn on their cell phones start texting get on the internet you know watch TV to distract them from that moment and that is what they have to move through in order to get to their to their own personal freedom so the first step is becoming conscious and meditation means to become familiar with to become conscious of to to become so


conscious of your unconscious self that you won't go unconscious to any thought any behavior or emotion and get ready because it takes a tremendous amount of energy to do that and awareness that conscious to stay conscious and so we fall from grace yeah fine you got you got your wake you got another day let's go again how often do you fall oh my gosh I mean how many


times have I done it thousands but I'm not gonna give up because the moments in which I do connect or the moments that I do have that transcendental experience what matters the most after it when I have that transcendental moment I look back at all of those difficult meditations those difficult days and those are the ones you remember you don't remember the good meditations you


remember the ones where you came up against yourself yeah and you want a little further and you say I'm gonna go we'll further and go further or you had a rough day and you just went in and you just you at the end of the day you surrender and you have the classic oh my god moment there's no linear correlation it's just whether you're willing to live in creation instead of living in


survival and so you get better at it you know we just get better at it and for me staying consciousness thing where in staying present is an art because you you know when someone's present with you in your life because they're paying attention to you you know when they're not present with you because they're not paying attention to you so imagine this field of informations is this


intelligence that lives within you and I that's governing everything material in this world it's a self-organizing intelligence you have access to it so you better get present with it as well as you can get present with anything else and just because you can't see it doesn't mean it doesn't exist it that that that realm you can't experience with your senses you can only


experience with your awareness so then people have to take their attend off their bodies and go from uh somebody to a nobody hmm take their attention off the people in their life and go from that they identify with and go from uh someone to a no one take their attention off the things in their life their cell phone their computer the car and go from something to nothing take their


attention off where they sleep where they work where they're sitting and go from somewhere to nowhere take their attention off the predictable future and the familiar past and time and go from some time to no time and now if you're taking all of your attention off of everything material in this three-dimensional reality now there's only one other thing that's left that


means you're an awareness your consciousness and now that is the bridge that is the door to the quantum field and you can't enter the quantum field as a somebody so if someone has spent their whole life working on having the perfect body or so much so they have so much attention on their pain where you place your attention is where you place your energy it's going to take some work for


them to take all of their attention off their body right because they'll go they'll do it and then look go back let's see if the pain so that all the pain still there so it's a little bit of a waltz in the beginning but as people start applying this you start getting better at it as an example we had Bond University University and Australia on the Gold Coast senior researcher took a


large majority of my brain scans and they hadn't she had them analyzed by her graduate students and they statistically looked at everything one of the most startling things for the research team was our community's ability to go to to get to that point where there nobody no one no thing nowhere no time I'm talking for seconds I'm talking five seconds I'm talking nine seconds just like just give


me a second I know how to do this [Music] I feel like we've brainwashed ourselves over the years to believe a story of the past is who we are and who we will always be in the future so how do we brainwash ourselves in times of stress and anxiety in order to become more peaceful loving and successful in the future how do we brainwash ourselves in a different way sure


well that's what I and and Wow I mean the first and most important thing is that you have to understand that if 95% of who you are is a set of unconscious programs then the first step is lighting a match in a dark place if you want to become someone else you got to become aware of who you are yes that means you got to start thinking about what you've been thinking about you got to start


paying attention to how you speak that be conscious being conscious to every thought action most word emotion feeling expression body language everything you need to be aware of and conscious of Selzer is there a way that you help people to track this besides of just like okay I'm aware of this in the moment do they journal the thoughts throughout the day or when a negative


thought comes up today remember the way their body languages throughout the day closed off and guarded or open do they how do they self reflect where they can track it better mmm well that's a big question but I will tell you this that you demystify the word meditation and the word meditation literally means become familiar with yeah when you become so familiar with your thoughts so


aware of your emotions so conscious of your habits that you wouldn't go unconscious to them again now you're no longer the program right so so getting people disentangled from that program we found out as a formula and when we teach people how to do certain things with their focus and opening their awareness when we teach them how to create a very disorderly brain that has been driven by


the hormones of stress into a more orderly coherent plane and teach them how to open their focus and practice that they'll come up against those thoughts and they'll become so familiar with them listen to they won't believe him anymore when they come up any longer Wow and so when they hear them in their day they'll be like that's not gonna stop me from my future


so if they're sitting there and they want to quit just because they're sitting still and they're not quitting then they're developing a will that's greater than those programs and you're breaking out of the shell and you keep doing that you're gonna get up and do the work every day because you did it yesterday and you're gonna want to do more of that because you're getting out


of your past and it feels better and if you keep doing that and you keep feeling better every day the question is why wouldn't you do it every day because you would ultimately just feel better and then the more hole you feel means the more you are connected you should imagine if you felt every single day and this is what I do work on if I could stay connected to the emotions of my


future all day long there's no way I would be looking for when it would be happening how could I look for when it would be happening if I feel like it's happening I wouldn't look anymore which means I wouldn't be separate from it and that's when you start creating the magic right that's when you're in that zone and that's when that reaffirms that personality that you're becoming and now


you don't you know wake up in the morning go I got to create my future jump out of bed excited you're not gonna want that magic to end that's the right so so you teach people how to do this and they start seeing the events in their life they're not gonna want to miss a day in in really just getting beyond their personality and listen it's so cool because it's amazing you see all


these people come in for a success and new careers and new relationships and healing from a disease and mystical experience that come in for all these different kind of reasons healing from a childhood trauma but really they just want wholeness right and so as I start becoming more whole and they start feeling more whole is not coming from anywhere out there it's not coming from


out there nothing out there is making them feel whole when the novelty of the thing wears off you feel empty again they're feeling whole from within oh this is different this is a different game so why wouldn't you want to keep feeling more whole that you no longer want anything now now you're not here now you're not living in separation anymore if someone is so disconnected to their


future their greater future self if they're so negative thoughts suicidal thoughts often hurting themselves potentially often just don't have many close friends don't feel like they identify with themselves in the world don't like anyone understands them no one accepts them no one gets them like this all sounds great in theory but when you're in a place of survival mode and


your thoughts constantly how can someone like that without having to go through the workshop that doesn't have the opportunity to go right now what can they start to do to just give some a little bit of relief and peace in their heart yeah it's simple knowledge experience wisdom philosophy initiate that philosophy master it yep mind body soul thinking doing being


learning with your head applying with your hands knowing it by heart and this is the journey of knowledge because when you learn that information and you really study it you are going to begin to see the world differently because your brain is changing then when you start saying how can I use this how can I apply it how can I personalize that how could I do something initiates this


information what am I going to do how do I get my behaviors to match my intentions now this is the act of trial and error it's so important you don't make it the first time you don't give up you get up and you try to walk again and you start learning how to do this and so as you begin to do it over and over again you start having new experiences well new experiences enrich the circuits


in your brain philosophically you know the brain makes a chemical and now you're feeling more unlimited you're feeling more whole you're teaching your body chemically to understand what your mind intellectually understood and now you are literally literally starting to embody that knowledge yes coming signaling new G's it's new information but but you can't do it one day and


expect your wealth to come you gotta do it over and over again yeah so the repetition of practicing over and over again neuro chemically conditions the mind and body begin to work as one you've done it so many times the he now knows how to do it subconsciously because just like it just like it knew how to subconsciously lean into trauma on victim mode right that kids now the


body's getting new information it's going to adapt and and now you're going to literally become that knowledge you're gonna be coming that's what you're gonna become and so now that's when you no longer have to try it's who you are it's yes yeah you've memorized an internal order that's greater than anything in your outer world that's gonna tell you something else that's key right there


and you become immunity to negative thoughts or negative viruses where if something tries to someone tries to say something to your body just rejects it automatically just as if your physical body would reject some virus coming in is that correct that's absolutely what I said dr. Joe you can't do it you're stupid you're ugly you're not smart enough you're not this


you won't even reverse they won't even just bounce right off you you wouldn't even receive it because your field is so powerful it's pushing all that away no because it's not the truth there you go it's just laughs it'd just be like okay that's just not real but if I kept telling you you know you need this product you need this drug to feel better to look better to be better let's


appeal to your lack and if you buy this it'll make that feeling go away and you try for a while then the feeling doesn't go away anymore you got to try something else and now watch the news and listen to all that information that's telling you you're limited you're you know you're limited it's something out there it's going to get you there's nothing wrong with that this but if you're constantly


saying it's traffic it's the news it's politics it's my ex that's making me feel and think this way then you're subconsciously not consciously affected by your environment and you'll be more affected by your environments not a kind of process but the more you say I'm going to in spite of the fear that we talked about or the anxiety or the frustration or the aggression or the


hostility or whatever it is I'm paying the suffering instead of saying that I can't change that you know what I'm gonna think I'm gonna see if I can where can I find that information where and all of a sudden you find people that are doing it and it makes sense to you and you're like well okay I'm feeling really anxious instead of taking something that's going


to chemically change me let's see if I could chemically change me without something out there let's see if I can make my own pharmacy of any depressants I don't know let's see if I can make my own pain relievers let's see if I can make my own chemicals that cause my immune system to get stronger I'm just curious let's see now now the person is there out of the bleachers and they're


on the field so they'll start believing that they can do it even if they change it a little bit and if they don't get it the first three days but they've seen a testimony of someone who has and you see that person who's totally happy that was abused every day of her life and they don't have the genetic disorder any longer you're gonna say wow that person doesn't look like a movie star that


person doesn't look like they're vegetarian now that person doesn't look young and after whatever it is they're gonna look like a normal person and you're gonna say I identify with her identify her she can do it I'm into it now here's the cool part yeah we've seen then when people do this with the same stand on the stage and tell the story that there's that the person in the


audience with the same genetic condition does it in a shorter amount of time because the evidence becomes a loudest voice why because it's in testimony yes there's truth right in front of you that's it's it's it's right in front of you there's you know we had a guy in Dubai that had he was in a wheelchair with this was this tumor in his spine and the doctor sent him home to die he


came across my book like a week before they got the book becoming supernatural mm-hmm and and then he read the book and he somehow got in there someone gave him a spot he was in a wheelchair stage for cancer or go home and died severe paralysis a limitation crowding the spinal cord the whole bit nothing we can do for you all these pain meds excruciating pain one week after


the week long his tumor reduced by 30% I just saw him in Munich he's he can walk he's walking without his wheelchair I mean he isn't it he's in a new experience he he's believing now in himself and when you believe in yourself you believe in possibility you can't have one without the other do you believe in yourself when you've never believed in yourself and you've


got doubt inside of you all day long knowledge knowledge knowledge keep learning keep studying keep listening to it sooner or later that'll become a louder voice in your head than I can it's too hard I'll never change yeah and you me knowledge of knowledge of new philosophies knowledge of new skills knowledge of new anybody study if you don't want to read get on YouTube and


one of course talk about how you can change your cheese just start looking starts looking at the testimonies and websites I mean we have over 450 testimonies now people heal themselves not small burritos amazing start looking to see what did that person do and when they tell their story that's worse than yours you're gonna start going wow that person really had a tough one and a day


overcame it well jeez why like well I could just forgive my father right now I give you right now I want to let them go and I want to be free but I don't want I don't want to give them my attention because I give him my energy I wanna but I put my attention in my future soon later you're gonna come to it the way you do but if you don't have the knowledge then you believe in it last


see you see there's people that we I saw this just recently and I was looking closely and there are people that do this work do do this transformation work in the meditations that we teach and they're so impatient and they're so entitled and they want an instantaneous change from the lack of feeling right that they never overcome themselves in their meditation they never overcome


themselves in the meditation and when they finish their meditation they believe in this work less hmm then there are people who say I can tell you the moment I made up my mind to change because I had reached the end and I made a decision and that decision to change carried an amplitude of energy that was greater than the hardwired programs in my brain and the emotional conditioning


in my body and my body literally responded to my mind in that moment that the choice that I made became a moment in time I would never forget and they'll tell you and that's the moment I remembered when I was going to change now those people then when they sit down to do the work there that the chemotherapy has worked the injections didn't work the radiation didn't work the surgery


didn't work the diet didn't work the yoga didn't work this is this is now their end they have nothing else to believe in but themselves hmm and they go all in not 50% not 60% they're going all in they have nothing else to believe in now listen when they go a little bit outside the known they've got they went a little further than where they normally would stop they


push themselves to that next limit they started believing in themselves that they could do it a little bit more they they finish the meditation and they get up and believing it's it's it's working more than working less they're the person that's believing in themselves that's why because it's not the work it's your belief in yourself right it's and when you believe in yourself you


believe in possibilities when you believe in possibilities you got to believe in yourself who wasn't gonna believe it so people make these great strides in in the and and their own personal growth is a testament to the living organist and the living organism the species of human beings that's starting to believe well maybe we're not so a slit limited as we've been


programmed to believe maybe we are more unlimited in and I'd rather throw in with that and if you don't think you you're your immune system isn't aware of viruses that could it could handle any virus if I got the right signal and if it was in a state of wholeness and your thymus was activated and your blood flow to that Center was turned on because you decided to turn it on and and you


decided to release those chemicals that suppress the survival centers in your brain because oxytocin does and you wanted to stay there for a period of time and memorize that feeling I guarantee you that thymosin would begin to signal those t-cells and those t-cells would activate their t-cell receptors and those t-cell receptors would be releasing immunoglobulins and


and antibodies that would block those receptors coming off a virus like you may not pass I'm sorry thank you so much for watching this video and if you're looking for more greediest in your life and check out this next video right here all the stories that I'd followed in life were people who went from nothing to something and here was a guy who had something


hi there today we'll look at mix-up beyond empirical risk minimization by a Hong Yi Chang Mustapha sis Yong and dolphin and David Lopes Pass so this paper is actually pretty simple but it introduces a technique that apparently helps with training classifiers and it I have it seen it used in practice so there must be at least something to it it is ultimately very simple

mixup: Beyond Empirical Risk Minimization (Paper Explained)


so usually you input a data point X into your neural network in deep learning so f of X that's your neural network your known neural network has parameters theta you get some output Y hat and along with the X you also have a y:i true label and then you have a loss function that compares what you output with your true label and then you just try to make that loss smaller you want


to adjust your parameters so next time you see data point X its output will be a little closer to the true label Y and we call this empirical miss empirical risk minimization because you don't actually what you think is that your X comes from some distribution from some data distribution D like the space of all natural images or the space of all of language but what you actually have


is you have a data set of a finite amount of data that you can put a that you can sample x and y from and so instead of your minimizing your true risk you minimize your empirical risk the empirical risk minimization right here now what's the problem with that the problem is that you can get overly confident about your data points and nothing else and that will hurt your


generalization so if you have a data point let's say right here and another one right here your network is basically so this this is a this is class 1 this is class 2 your network is going to maybe make decision boundaries like this and like this where it says ok here is class 1 and here is class 2 but it could you know it's very conceivable that here it says ah here is class 4 and


over here is class 7 and right here through is class 9 and by the way here class 4 again so the the empirical risk minimization leaves everything in between the data points open now what this paper proposes is that we should not only train our our classifier on these data points but on all the data points sort of in between the two and this is the mix of data points so this


data point here might be constructed if this is a and this is B from zero point one times B right and plus 0.9 times a because it's mostly a and it's a little bit B and now you think what are the labels here if a belongs to class one and B belongs to class two then of course the label of this data point is zero point one times the class of B which is 2 plus 0.9 times the class of a


which is 1 ultimately because what you do is you input a class like class number two if you want to input this into a machine learning model you just you don't just say it's class number two what you input is a distribution that is basically has zeros everywhere so these small things there zero zero zero one zero and this here is at class number two so this would be class number one


class number two class number three right you input a distribution like this if you want to express class number two now in our sample right here what we would input as a label is simply a mix between class so 0.9 0.9 of class 1 0.1 of class 2 and then zero everywhere else so this would be our label for the data point that we construct right here this will be our


sorry the top one would be our data point formally you take two data points and you mix them using this lambda mixing factor that'll give you a new data point that's in between the other data points and you take the two corresponding labels and you mix them accordingly as well and that will give you the label for that data point and now your model will learn to basically


smoothly interpolate so you will teach your model the thing on the left here is class number one right that's class number one the thing on the right is class number two this here is a half of class one and a half of class two so the model basically learns a smooth interpolation where the situation that's here on top is probably not going to happen anymore but what it would do is


it would sort of create these ISO lines around class two and then around class one where it's sort of smoothly getting less and less sure about the class of the data points but on the way it is always either class 1 or class 2 and they say that can help the generalization performance and it's visible or why right it's just the only thing that's not it it's not clear from


the beginning is that this kind of interpolation actually makes sense because if this means we sort of linearly interpolate between two images so if we have two images we just take half of one and half of the other and that will be not a natural image it will be kind of a blurry thing otherwise you know all our problems would be solved and we could just linearly classify


things but in any case in practice it actually seems to help probably because interpolations of two images linear interpolations are still much more like something like a natural image then any random noise you could come up with so they say it isn't code right here code is pretty simple simply want to mix the two things and the mixing factor this lambda here comes from a beta


distribution and they use a beta I believe of 0.4 or something just want to quickly show you this is the red line here so the red line as you can see mostly most of the time they're going to either sample the thing on the very left or the thing on the very right that means the either sample the first or the second data point but some of the time they actually sample something in the


middle and it's it's fairly uniform in the middle so it appears like a good distribution to sample from if you want to sample these mixing coefficients and by adjusting the the actual number of alpha and beta here you can determine how many times you sample the original data points versus how many times you sample something in the middle okay on this toy data set right here they


showcase what mix up can do so in a classic model you have the orange and the green data points and blue is basically where the classifier believes its class one you see this very hard border here it's quite a hard border now you only have two classes here and so the hard border is sort of a problem in itself because if you think of for example adversarial examples all they


have to do is basically get over that one inch and the classifier is already super duper sure it's the orange class right whereas if you use mix up your border is much much much more fuzzy it's like yeah it's only really sure here and out out here everywhere but in the middle it's sort of like me I don't know and so that's kind of a more desirable situation and of course this here works


particularly in this in this linear to the setting but as we can see the same reasoning applies to sort of higher higher layers and higher dimensionality data points right I have to seem to lost the ability to zoom Oh No that's back okay and that's basically it for this paper this is all they do they propose this method and then they test it they say something interesting here


that mix-up converges to the classical method as off approaches zero so that would push your beta distribution basically in the middle all the way down and you would only sample from the very left or the very right so you can smoothly interpolate between this mixing and the classic method they so their main results are we apply this to classifiers and what I like is since


again is also a classifier so the discriminator is a class fair they also apply it to Ganz and they outperform unstable eyes the classic training on ganz they show that it's more robust towards adversarial attacks because it's not so sure about intermediate things and they generally outperform other methods but also they do this nice investigation here where they measure


the prediction error of in between data and what it means is they say a prediction is counted as a miss if it does not belong to Y I or Y J so you have a sample right here X I in the sample right here X J and you look at what the classifier says in between the two data points so you just interpolate the two data points and just measure what the classifier says and whenever


the classifier either says y I or Y J either either a label of those two data points you count it as correct and you only count it as incorrect if it says something else and you can see here if you train with the classic method erm the these errors happen much more often that's exactly the situation I pointed out at the beginning where in the high dimensions it can you know occur that


all sorts of decision boundary sneak here in between the two data points and by interpolating between them during training you sort of much reduce that you reduce that effect a lot now this they also say that the gradient norm of the gradients of the model with respect to input in between training data it happens the same thing the norm of the gradients in the middle is also


much much lower and this yeah this investigation I find pretty cool I have to say I have seen mixup in practice so it might be useful I've read a paper where they basically say oh it was a big transfer paper yeah where they basically say it is useful if you have for example if you have little data and a big model so you can sort of regularize the model and is also useful to know that they did


test this with drop out so we can they compared it with drop out and the conclusion is basically that this is something else than drop out so it's not doing the same thing drop out of course it means you drop out some of the data points in intermediate activations and that sort of gives you a noisy version of the data point this here can actually be combined with drop out which means


that it gives you an additional benefit you see right here most of the best numbers happen when you use mix up plus drop out so it seems to be just an additional regularization on top of drop out pretty cool pretty cool investigation awesome alright so if you like this I invite you to read the paper if you liked the video please subscribe and like and comment and yeah have a


lecture seven of deep unsupervised learning today we'll be talking about self supervised learning and it is going to be pretty different from the previous lectures you've heard so far so far you've been looking at a lot of generative models how to use various classes of generative models to generate high dimensional data like images audio text and so forth however unsupervised learning is a much

Lecture 7 Self-Supervised Learning -- UC Berkeley Spring 2020 - CS294-158 Deep Unsupervised Learning


broader goal than just being able to generate data and one of the goals of unsupervised learning is to be able to learn rich features from raw unlabeled data such that they can be useful for a lot of downstream tasks and this lecture is going to get at that and recently people have started calling this stuff supervised learning where the data creates its own supervision and so we


are going to look at all the various classes of techniques that are allow us to do stuff supervised learning so so far we've seen density modeling where we've covered Auto regressive models flow models and we also talked about variational inference and we've also looked at implicit generative models implicit density models like jion's and energy based models and both these


classes of techniques allow you to learn generative models which is you are going to be able to generate images and be able to report lac courte scores and so on but other than that we mainly looked at applications of generative models to various modalities of data we haven't actually seen how to use unsupervised learning to learn features so that's the motivation for today's lecture how do we


learn rich and useful features from raw unlabeled Ino such that it can be useful for a wide variety of downstream tasks and we're also going to ask ourselves the question of what are these various pretext or proxy tasks that can be used to learn representations from raw unlabeled data and if we are able to learn good representations how can we leverage that and improve the data efficiency and


performance of downstream tasks with a good pre training model so here is a figure from in Goodfellows deep learning textbook the focus here is how do we learn good representations and here's a simple case study of why representations matter see how a bunch of points two-dimensional and if you visualize this an XY coordinate the Cartesian coordinate the they're clearly two


separate clusters but it's harder to visualize how to linearly separate them but the moment you visualize them in the polar coordinates you can clearly say that there are two different radii and a lot of different angles and so you can have draw linearly separable hyperplane between them so it's clear that representation matters so once you move from the Cartesian representation to


polar coordinate representation things become a lot easier to handle as well you know you can actually use this linear SVM so logistic regression to learn a classifier on this particular polar coordinate representation so what is deep learning doing deep learning is basically using depth and repeated computation to iterating we will find the features as you move thereby they


over there so the bottom was there is using the raw pixels as input and here it's trying to make sense that there is a person in the photograph and you know if it's looking at all the background pixels and understanding that there is a phase so the way it starts it starts with the highest frequency information at the bottom and it refines at the next level to edges and it refines the next


level two corners and contours and then the next load actual object parts and finally it's able to figure out the identity of the objects present and the actual image so just like how we saw representations matter so representations are the higher levels are more semantic representations are the lower levels are more fine-grain and detailed and high frequency so the


deeper on that can be thought of as writing its own representations that every layer every successive there is written on top of the previous layer which is more abstract than the raw input and that allows you to do downstream tasks if you take the topmost layers so here's the Venn diagram that being good for suggests for deeper how to think about deep learning so deep


learning is a subset of representation learning which is a subset of machine learning which which can be considered as sort of AI in general and so the goal of deep learning itself nothing to do with unsupervised learning is to learn good representations of raw data so what is deep unsupervised learning so unsupervised learning is concerned with learning these representations


without labels so it can be considered as another subset of deep learning which is doing deep learning without labels basically so we are gonna get at the goal of representation learning without labels and that's deepens promise learning so it sort of gets at the core goal of the class and recently it's been called a self supervised learning and it's used interchangeably with


unsupervised learning the exact terminology of what itself and what is undoes not matter it's basically concerned with learning representations with our labels whether one self usually refers to the scenario where you can create your own supervision based on the data but at the end of the day it it can be considered as as another way to reprime tries unsupervised learning


so why self supervised learning the expense of producing a new dataset for each task this is really high so there are actually billion dollar startups just doing data annotation for people who can just upload their images say what kind of labels they want and overnight or within an hour or fortnight you can get like high quality labels created by humans who would annotate


this data on the client side so sorry on the server side so so basically you you need to prepare labeling manuals you need to figure out what categories of objects you want you need to have someone else hiring humans you need to hire your humans to annotate data and whoever is doing that job you need to create good graphical user interfaces so that it's the process of annotation is


really fast you also need to create good storage pipelines so that every you know let's say people are annotating usually it's like a lot of mouse clicks per minute or second and every mouse click is automatically recorded and converted to prompt rate data storage formats and stored efficiently into the cloud so there are lots of back-end engineering you need to do not that this is a bad


thing to do it is it is good unlike if we really need to work on better and better pipelines for data creation however good supervision may not be cheap for example annotating what are the objects contained in an image is probably something that you can take for granted now because people have created a lot of datasets but if you move to another domain like medicine or legal


creating another data set may actually be pretty hard so taking advantage of vast amount of unlabeled data on the Internet is something supervisor learning cannot do so no matter how much you can appreciate the success of supervised learning there is still a lot more unlabeled data then there is the amount of label data and it would be nice if we can leverage the


of unlabeled data to further improve the performance of systems that work on label data so it doesn't have to be a dichotomy between hey we just want to do unsupervised learning or we just wanted to supervise for anything but rather we want to figure out how to take advantage of large amounts of unlabeled data order billions of images for lots of text or audio samples or YouTube videos and


learn amazing features and then make the process of doing supervised learning much more cost and compute and time efficient and finally there's this cognitive motivation which is how babies or animals learn in that like when they mostly learn by experimenting and without actually having a labels so a child can just look at other people doing things or its own experience


moving its own hands or looking at other people around in the house or like you know no modern days children grow up with gadgets so they can look at videos and already start learning good features without actually knowing this is it this is a cat this is a cat this is a cat like hundreds of times that's how emission a classifier is learned so that was a really nice code


by Pierre cermony who's one of the leading researchers in the field which is give a robot a label and you feed it for a second but teach a robot a label and you feed it for a lifetime what what he means by this is that if you taught the robot the underlying aspects of various objects in a completely Sal supervised fashion it knows what like a cat or dog is without actually being


taught so that means that it can actually generalize much better so labels are cheap but then it may not be the most optimal way to learn representations so so what exactly is self supervised learning it's it is a version of unsupervised learning where data provides its own supervision so in general the way it could work is you withhold some part of the data and you


task a neural net to predict a bit with health portion from the remaining parts so this could be like you occlude some part of the image you look at the remaining pixels and you try to predict the occluded version or you have a video and you just hide some frames in the video you have the other frames and you try to fill in the blanks of the missing frames or you have a


sentence and you mask out some words and you ask the neural network to fill in those words or you just have to predict the future from the past or the past from the future or present from the past like loss a various different versions depending on the mask so this way the data is creating its own supervision and you can perfect you can ask a neural network we'll learn a lot more than just


predicting labels so the details obviously decide what is a proxy loss or what is the pretext ask you can think about all these withhold and predict classes some kind of pretext ask and you can think of whatever details you use whatever loss functions or whatever tasks you create depending on that the quality of the task could be different and therefore the quality of the


underlying representation study uncover could also be different and so that is basically this whole topic which is how can we create these really good tasks which make the neural network to learn a lot of useful things and therefore be very useful in downstream tasks so the motivation another motivation of why we want to learn good features is one of the biggest reasons for supervised


learning to really take off not just as a research topic but also as an industry of practice is that the you can use a pre trained classifier for a lot of commercial a downstream tasks so a pre trained imagenet state-of-the-art emission a classifier like a rest at 50 can just be taken and the same backbone can be taken and put into a faster or CNN or a mass for CNN or a retina net


and can be used for object detection or instant segmentation or it can also be used in a fully compositional neural net with the backbone as the rest in 54 a semantic segmentation so this way you're able to solve a lot of harder computer vision problems with black collecting label data is much harder and you can actually just retrain a good classifier take those features and start the


underlying downstream tasks with a much better prior and you don't have need so much label data now and you can also converge much faster on these harder problems so that way the recipe is very clear so you just collect a large table data set your trainer model you deploy and as long as you have a lot of good data and you have sufficient data it is basically all you need in terms of


getting some autom automation on on production so most of your industry or usage of computer vision like in video surveillance or in robotics where people have to detect objects or in shopping automated shopping where people you want to detect what people pick or which objects people pick it's basically just object detection and to get a very good object detector all you need is a lot of


labels and a lot of good patron features so what is the goal of self-professed learning the goal is to learn equally good if not even better features without supervision and be able to deploy similar quality systems as what is currently in production without relying on too many labels so what if instead of collecting 10000 labels now you could just collect thousand labels or hundred


labels that makes the process of production much faster much more efficient and you don't have to spend as much and it's also much simpler to maintain and you can keep on bootstrapping more and more data you don't have to rely on high quality expert labeling and you can still uncover the same level features as you have currently with all the effort had to collect labels so it could also


generalize much better potentially because by doing some harder pretext asks than just predicting labels you are expected to learn more about the world and therefore generalizing in the longtail scenario is likely to be better so that is the hope and that's why people want to make self worth learning really work so this has been really you know very nicely put together as a more


inspiring slide by young Nicole and it's often referred to as the lake which we saw in the introduction to the class which is you can think of self supervised learning as the cake if intelligence of the cake you can think of sociable as learning as the cake and you can think of supervised learning as the icing on the cake and you can think of reinforcement learning as the cherry


on the cake and there the argument is that most of the useful bits can come from doing really hard pretext asks just from data and where the machine is predicting some part for missing parts and you get millions of bits that way whereas in supervised learning consider imagenet you have thousand classes so that's ten bits per image and if you have a million images you have basically


a million times ten bit space key that's basically your whole data set and whereas if you're just doing generative modeling you're modeling all possible bits in your data set so that that's that's too huge right so some supervised learning is trying to find a middle ground between these two and it's possible that the bits you get from sabra was learning or more useful that


said there is a caveat that subscrube is learning the bits you get from there or not as high quality bits is the bits you get from supervised tasks when human is telling you that there is a cat here or there is a dog here or like there there is a cat exactly at this coordinate there's a dog there's a bounding box around a human that's very higher quality bit than


saying these two pixels are of the same color or like these this is a flipped version of that image or this image is a 90 degree rotated version of the other image things like that so it's not just a number of bits that matter the quality of the bits is equally important so you should take this slide with not not too seriously it's just for inspiration making the bits argument as a way to


like work on unsupervised learning is fundamentally flawed because the label data bits are much much much more useful much more grounded in the real world you to behave so here is the aleck own suggestion for how to do unsupervised or sub supervised learning which is creating your own proxy tasks and you can think of various different versions of that let's say that there's a video


you could predict the top from the bottom bottom from the top left from the right right from the left it's basically masked some part of your input predict the mass part from the unmask part and obviously depending on the mass the mother's going to learn something trivial or non-trivial so usually like like for instance in a video if you're just masking a couple of


frames in between it may be very easy to fill it up by just using the optical flow information and just using the reference frames and just interpolating between the pixels the model doesn't necessarily have to capture what an object is and whereas if you have a sentence and if you are predicting the missing words or sub words in a sentence it's possible that the model learns a


lot more about the grammar and the syntax and semantics of collective language because it's it's not possible to just copy-paste previous words to fill in the fill in the sentence because language is already syntactic and grammatical and separate like every cent every word is conveying something new whereas pixels are more high-frequency there are more natural signals and so


there is an spatial temporal correlation this is already available naturally so the model may not really learn the actual high-level information that you wanted to learn unless you carefully engineer what are the master you want to use so for the actual technical content we are going to separate it into three parts so the first part is we're going to learn about how to do that we're


going to separate the various cognitive principles basically the first principle is you corrupt your data and you try to predict the actual data from the corrupt aggression and the corruption can just be like you add some noise to your input or it could be like you hide some part if you input an you predict the missing part or it could be like you take your data and you basically do some signal


separation so it could be like hey an image is basically a grayscale in the color so you could predict the color from the grayscale or you have you have the depth image then you have the color image and you could try to predict the depth image from the color image where let's say you're recording everything from your Kinect so it could be source separation and then you try to predict


the separate equation so that is the first principle the second principle is we're going to do something like visual common sense to us where it's more ad hoc and you just trying to create tasks from data in a very creative way and see what kind of features the model can learn and there we are gonna look at three different techniques relative patch prediction jigsaw puzzles and


rotation and finally we are going to look at contrast learning which is really the version of supervised or unsupervised learning that's been taking off very very recently and we're going to look at a foundational work over to back which explains a lot of these foundational ideas like nice contrast loss and then we're going to look at a version that's been used in images


called CPC or contrast to predictive coding and we're also going to look at follow-ups to that that made the CPC pipeline much simpler like instance discrimination and where is state of the art instantiations of that note that in this lecture we are not going to cover the more popular sub supervised techniques like or any anything to do with the latest language retraining pipelines arguably


sub supervised learning has taken off way more language than computer vision but the focus of this lecture is going to be more on computer vision because language pre-training will be covered separately in a guest lecture by Alec Radford and we're also not going to look at how unsupervised learning helps for reinforcement learning that will also be covered separately by Peter in another


lecture ok so now let's go to denoising auto-encoders so do isaac autoencoder the basic idea is add some noise to your input and try to remove the noise and decode the actual image so here you see an amnesty j't and you and you see the noisy input on the on the left and you see the demo image on the right and the end coder takes in the noisy image puts it into a smaller latent representation


which is the features that we care about and the decoder is trying to use these features to get back the original input so you hope that the encoder gets the high level details removes the noise and the decoder can up sample that and get get you back to the actual image so depending on the kind of noise you add you've want to learn more non-trivial things if you don't add any noise you're


just going to learn an identity function that's just an or encoder but if you add some level of noise it's possible that you learn more useful features because you're learning to separate noise from the actual signal right and if you had too much noise then it may actually be a really hard task because the signal-to-noise ratio will you really low so this is the general computation


graph of the denoising auto-encoder where x tilde refers to the noise version and F data basically at the ground truth x and you're trying to figure out how to reconstruct that back and get the Layden's in a very useful way so there are various different versions of noise that you can add to in put in in a denoising auto-encoder so in the original denoising auto-encoder


paper they considered the task of Ensenada they considered three different noises additive isotropic Gaussian noise where you basically just add Gaussian noise to the pixels and another version is the masking noise where you basically some fraction of your input pixels I just chosen at random and you just forced into zero which K you just mash them out and they're going to be a black


and finally there's the salt and pepper noise where some fraction of the elements the chosen at random and you can basically set them either to the minimum possible value or maximum possible value so instead of basically it's it's a version of masking where instead of just assigning masks to be 0 you can randomly assign the mask to be 0 1 so these are three different noises to


consider in the paper you can note that as pixel level noise so you can think of denoising auto-encoders basically learning a tangent hyperplane on your data manifold where around every input x there is a distortion radius created around it based on the noise that you're trying to add so it's very easy to understand in the case of additive Gaussian noise because you can think of


Gaussian is a spherical distortion around your every input and you can think of the decoder as trying to put the distorted version back onto the tangent hyperplane so you can think of the whole denoising auto-encoder pipeline is trying to learn this tangent hyperplane that describes the data manifold so that it's able to put back to distortions around the hyperplane