collecting up these sort of temporal associations to make sense of the world so we have we have experimental of us this might be happening but we don't have really models for it now there have been prior attempts so slow Fourier analysis is a is an attempt to sort of extract those signals that are moving slowly and this has been influential but you know it hasn't sort of unleashed you know a revolution in how we do things but it's an interesting and important set of ideas but what I want to do is look specifically at prediction as an

Predictive Coding Models of Perception


unsupervised learning rule so basically the idea that can we just use the idea of predicting what a future frame is going to look like to help us build better representations and I'm gonna make the argument that brains are particularly adept at prediction I'm gonna use this person who's this person Sarina yes not Venus Sarina you guys are our experts so anyway so she can she can do tennis serve at 207 km/h that's 57 meters per second that means the ball traverses the court in 400 milliseconds about now the latency from the retina to

Predictive Coding Models of Perception


the primary visual cortex is about 60 milliseconds so that means that v1 is operating 3 meters in the past and if we go all the way through the ventral visual hierarchy then we're talking about about a hundred and seventy millisecond latency in a human it's about nine meters so there's a very deep sense in which if you think you saw Serena Williams's tennis serve you couldn't have because your brain you know this year line latency of your brain was way in the past so one suggestion that that's been made is if you're returning that

Predictive Coding Models of Perception


serve a big part of what you're doing is you're looking at the lined up and you're predicting where the ball is going to be and that's where you're putting your racket so you know and this this idea has been sort of you know it can be shown there's a little bit of nuance here but if this is called the flash lag you lose inch so if you look at this dot right here and then watch this sort of clock hand going around you see how there's like a little flashing straight line now is that line behind in front of or lined up with the clock hand

Predictive Coding Models of Perception


behind of course and the rule with all optical illusions is if it looks like it's not lined up it is lined up and if it looks like it is lined up it's not lined up so so this is perfectly collinear and the sort of classical interpretation and there's some nuance here is that basically this line has to go through the full pipeline latency of the visual system but this tracking line is predictable so your visual your perception just sort of puts it where it really is in real time so so this is and then there's another weird things

Predictive Coding Models of Perception


everyone's kind of see that it looks like it's tilted as well I can just hold that thought all right so so so so we wanted to get into this so we were we were you know working in sort of machine learning computer vision and right around the time we were working on this we got interesting this idea of future framed video prediction so basically the idea is that if we see a sequence of images transforming the goal is to basically predict what the next image looks like and this is now that there were there was a little bit of work

Predictive Coding Models of Perception


around the time we were doing this and that's it's a much bigger field now just to acknowledge that there are other people working in this area and we did some very simple things that you would do in in deep learning and again this is sort of fits in the same spirit of what Matt was talking about where so you just want the simplest possible models to start with and just see how much your problem statement sort of you know sort of it makes the problem happen so basically we did is we took an autoencoder type architecture and we

Predictive Coding Models of Perception


just wedged a recurrent neural network in this case an LS TM in the middle and then asked and then we also had a fancier version with gans back then there were only four Gann papers so it seemed exotic and how there's like 4,000 Gann papers but anyway so basically what we discovered is we can build these sort of generative networks that could actually basically sort of renderer faces and this doesn't seem so surprising anymore but at the time I was shocked how well this worked and there's some details about whether you

Predictive Coding Models of Perception


use the Gann or not and whether the ear appears or not but the basic bottom line is this is there a future frame video prediction problem where we see a sequence of frames and then we predict what the next frame might look like it is actually a tractable thing we can do with neural networks now and then the interesting and of course it works on all kinds of different faces including faces you haven't seen before but the interesting thing here was we were really trying to look at what kind of representations do we implicitly induce

Predictive Coding Models of Perception


when we train a network to do this kind of future frame prediction so all we're training we're training the network with back prop but the loss is coming entirely from how well does it reconstruct a future frame that it hasn't seen before and the interesting interesting thing that we find is if we kind of just take that internal representation and just sort of pipe it off and just ask how well can we decode things about other things like the identity of the face what we find is if we do a simple you know 50 Way face

Predictive Coding Models of Perception


recognition task where we need to tell which of 50 people it is and we only get to see 1 2 3 4 so on views of the person what we see is if we start using these predictive networks they're able to do a little bit better with a little bit less data and this is what we're trying to get towards can we get representations where we can do more with less training data and because we're sort of extracting some sort of sort of deep part of you know some sort of deep structure of the image now this was back in in 2015 and then you know we also

Predictive Coding Models of Perception


discovered that you know for instance you could internal within to these the representations that we learned you could also find directions that you can move around in that would do things like make the face more male or more female these are less surprising now that ganzar around and we can do all kinds of fancy things with them but what we really we're driving towards just to get more to the punchline was not this sort of very simple auto encoder with an RNN in the middle but really getting towards this idea of predictive coding which

Predictive Coding Models of Perception


came from neuroscience so the basic idea here this is a paper from 1999 that popularized predictive coding but the idea actually goes back about a decade earlier and the basic idea is this we start with an input we have a feed-forward signal we try and predict away the input and we subtract that off and we only send forward the differences now the original idea here was an efficient coding idea let's try and reduce the number of spikes we have to send because if we subtract away the things we already know then all we need to send forward is the

Predictive Coding Models of Perception


is the the difference from that so what we did was basically to take yeah on what we think the signal is if I'm looking at the outside and I'm trying to submit position of a ball and predicting where the ball is going to be is a different stories and if I'm trying to estimate suppose it's octave all right and then do in how many examples do we actually know what it is that we're trying to predict I mean really for sure right this is only doing that thing right yeah that's that's a great matter question and and I think we can discuss

Predictive Coding Models of Perception


that like what are we estimating in this case for the purposes of this we just took the simplest path I mean again this is sort of the modus operandi is take the simplest possible thing and then see where you get with it so here we're actually gonna generate whole frames so we actually want to confabulate and sort of imagine what the future frames gonna look like so you could imagine in that flash lag illusion example you're percept in that case is actually an imagined thing it's like it you're actually what you're perceiving at a

Predictive Coding Models of Perception


given moment is is a fabricated future State so so we're gonna take it at the pixel level but I agree because if you were interests in different things you might you might use different sort of targets for your prediction but what we did basically was to take the sort of classic idea of predictive coding and a state instantiated in the simplest possible way we could with with deep networks so basically what we do whose inputs come in here we have a running prediction of what the input should look like and then we subtract them off to

Predictive Coding Models of Perception


get an error map and then we send that forward to the next layer and then we have a recurrent layer at each at each stage of the hierarchy that's trying to build up this prediction and it can get feedback just the same as in the brain and it can also get local recurrence and then you basically just subtract off and there's different ways you can do the subtraction doesn't much matter and then you get these errors you only send the errors forward through the network so one nice thing about this is at time one

Predictive Coding Models of Perception


this is a CNN as nothing's come through so the extent that CNN's are a good model for the ventral visual pathway this is the CNN on the first time step and then the other thing that's nice about it is it's also a classic generative model so if we put something in on the top it'll render down into an actual predicted image here so we can see what the network is is seeing or perceiving and I'm given moment and we called we started off by calling these particular works and then we don't maybe deep prediction that's

Predictive Coding Models of Perception


not good name pretty deep no bill net no can't believe it's not Alex net we ended up with pregnant because their rules their rules to how you do this and you have to you know you have to get them right so we're calling these pred nets for better or for worse and what we found is that these networks that had the recurrence at every layer we're able to you know our previous ones were only able to get sort of one degree of freedom and a rotating object now we could actually get as many degrees of freedom basically as we wanted in these

Predictive Coding Models of Perception


synthetic objects so here are examples and basically that just for all these I'm gonna show is this is the actual sequence of incoming images and then what's next but below it is the time-shifted prediction so you can compare the prediction to what actually came and you can see on the first frame you get these weird potato thinks as it doesn't know which way it's going to go but then once it knows which way it's gonna go it locks on and then you start getting pretty good predictions and it can do this with faces it hasn't seen

Predictive Coding Models of Perception


before and all that kind of stuff but again the the reason we're doing this is not for future frame prediction per se but to see if we can actually learn good representations because the idea is in order to predict what's coming next you have to implicitly know lots of things about the structure of the object how light works how shadows work all that kind of stuff and what we find is basically if we try and build decoders again in the same sort of spirit as that face recognition task where we just sort of peel off the

Predictive Coding Models of Perception


representation and just do linear decoding we discover that we can decode lots of different parameters of the image like how fast it's moving what the starting angle is we can also look at things about principal components of the identity so you it sort of fits with this idea again that by virtue of learning good predictions you learn good representations that are useful for lots of things which is kind of what you need to have in an unsupervised or a semi-supervised learning sort of setting we also did a face-recognition version

Predictive Coding Models of Perception


of this again and what we found is that these presents at least at the time were performing on par better than the best semi-supervised learning algorithms that were available called latter networks so you might wonder what happens if we put in things that aren't faces what does it do it actually does something pretty reasonable so this is a network that was trained on faces and we put in this image of a top and you know something came out pretty okay it doesn't always turn out okay here's a here's like a little toy car and you can

Predictive Coding Models of Perception


see it's desperately trying to turn it into some kind of face it's like this like when scream kind trying to happen there but you know and but the bottom line is if you train it on complicated things and enough variety you can get networks that can do pretty good predictions on just about anything you want to do and of course just about anything you want to do is cars right autonomous cars like at least in this year we did this work that's the thing you want to be doing prediction on because that's where all the money is so

Predictive Coding Models of Perception


we took my student bill without any prompting took some some car mounted camera datasets so this we train it on the kiddy data set which is in Germany and then what I'm showing you here is the test set which is the Caltech pedestrian data set and then this is what the predictions look like so they're not perfect they're a little bit blurry if we put a Gann on it we can make it less blurry sure but we're really just focusing on sort of learning what kind of representations we can pull out of this and a couple interesting things

Predictive Coding Models of Perception


here one is it seems to implicitly know a lot about perspective and flow it knows that things need to expand outwards and needs you knows the things that are far away need to you know change less it also knows to infill road here if we look at other examples as well you can see things like you know it knows a little bit about occlusion so this car is going to include this golf cart it knows that that one should go in front it knows you know it knows all kinds of things about you know implicitly about the flow of content in

Predictive Coding Models of Perception


the image and again what we're trying to do here it's interesting to do it to look at feature frame prediction and we can also do farther out future frame predictions as well if we can go out as much as five frames it gets blurrier but it's it's still okay but again the goal here is to say well by virtue of learning how to predict as sort of a surrogate loss can we learn how to decode other things that might be useful and in a car one of the things you might want to learn about is what's the steering angle of the car and what we

Predictive Coding Models of Perception


find again in the same way we can take the representation peel that off put it into a linear decoder and we can discover that without any prior training on these steering angles this can actually outperform a system that was purpose-built to decode steering angles so this is from comma this is a startup that's doing autonomous cars they had a reference CNN on their data set and then basically the president just by learning how to predict the future also implicitly learned what the steering angle of the car was

Predictive Coding Models of Perception


that was kind of what you need to do to do the task yeah not one not just the recurrency but also the fact that you're training it on continuously sample images representation is due to the prediction as opposed to merely the sequence of training images all right so so we have so we've done predictive versions and we've also done sort of a just a autoencoder style and the prediction outperforms the details about are all in the paper how much additional how much of a better representation of you gain by adding the

Predictive Coding Models of Perception


predictive component as opposed to just having from whatever image your sequence of images you have you mean specifically for this the steering angle version so the way the the comma one works is it it takes in frames and it puts them to a CNN and it so it has the temporal component but it doesn't have any notion of prediction it wasn't it wasn't sort of trained to do prediction and what we see here is these were the comma reference CNN's so how well they performed and then this is with different numbers of input frames to the

Predictive Coding Models of Perception


decoder and you can see it can can do quite well with with relatively little data so it seems to be a useful thing but again what we're trying to do is trying to drive towards how might this be a principle that the brain could use to organize itself yeah when we're doing rollouts what do you mean oh into the future so their unique way school what we do is we re inject recurrent recursively the the predictions and then we fine-tune that so so basically if you take the prediction put it in again get the next prediction put it in again put the next

Predictive Coding Models of Perception


prediction and you can get its it starts to get blurrier and blurrier over time but that's basically what we're doing very briefly showed we can do five frames ahead the predictions just get murkier it makes a little bit of a difference it gets murkier because you don't know which way the car is going to go I mean realistic what we should be doing what we are doing now is probabilistically we should actually be putting out a probability distribution of all the possible outcomes and that's that's hard but that's what we're doing

Predictive Coding Models of Perception


and you know there's a lot of interesting sort of wrinkles but even just with this but the straight up let's just does this come up with a mean outcome because there's a problem that if you don't know it's gonna go left or right then you you don't want to split the difference and get blurry what you'd really like to do is but have a system that produces samples that are sort of the correct distribution of actual outputs you could have and again that's something something we're working on something brain publi does - of course

Predictive Coding Models of Perception


then we want to actually since this is a neuroscience talk we want to go back and look at like well does this do something that explains something in neuroscience that we didn't understand before well here's something that everything does so it turns out you get good bores everything you know every neuron that work you train on anything no matter what you do it's like a rule that you'll get Gabor is out and and orientation tuning but more interesting than that is this notion that I sort of flagged earlier which is if you put in a static

Predictive Coding Models of Perception


input the brain doesn't give you a static output it gives you a dynamic output so there's usually a delay there's a burst of activity and then you have this sort of activity that falls off and sometimes you also get off responses so when the stimulus goes away you get a fresh response and it's probably not a great surprise but pride nets do this as well and it's not hard to see why so this is the average of units in the error representation of the pregnant hard to figure out why it's doing this basically what happens is on that first

Predictive Coding Models of Perception


hit it can't predict anything when the first image first flash is on so you get a big bunch of activity but then as it locks on and learns to sort of explain away the data then it goes down and then when you get the image off that's another sort of surprise and then you get another burst of activity yeah if you sprinkled the pixels randomly like you know Toby Delbert's event cameras or something true continue so so it's true we're doing this in discrete time because we're talking about worker L STM's and things like that

Predictive Coding Models of Perception


convalesced hams so we can't accommodate continuous time but I don't think that's I don't think that's a huge huge difference well I mean these aren't like different frames it's like continuous frames so it's a relatively smooth progression through the space of images and when you have nothing and then you put on something that is the kind of discontinuity we're talking about it is in discrete time yeah so what we did is we literally put the image up you know blank screen put the image up and then take the image off and so that's like

Predictive Coding Models of Perception


the standard sort of primate experiment version of this and basically the present that does exactly the same thing has exactly the same dynamics that you would find in in primary visual cortex so this is this is a bit of circumstantial evidence at least that part sort of matches up with expectations originally actually particular coding was it was designed to explain a phenomenon called N stopping which is if you have a bar that's oriented the way a v1 cell likes and then you make the bar longer and longer and longer and longer the response will

Predictive Coding Models of Perception


go up to some point and then once you make it longer actually the response gets suppressed so it's like too much of a good thing you know the longer the bar is that that's in the orientation that likes it actually reduces the response and you know true to form the the present version of predictive coding also has this sort of n stopping response which which is a nice nice thing to have but it goes further than this there's also this notion of surround depression so if you have a stimulus which is a dot and you

Predictive Coding Models of Perception


larger and larger and larger and larger the e1 cells will like it better and better and better and respond more and more and more up to a point but then beyond that they'll start being suppressed as if there's a suppressive surround around the response and what we find is that the the pet net also has this quality so if you look in again the bottom layer the response goes up to a certain size of a dot stimulus and then it's suppressed but interestingly Rick borns lab at Harvard back in 2013 found that if you cool downstream visual areas

Predictive Coding Models of Perception


so if you cool v2 and then record in v1 so in activate feedback connections you actually would find that these surround suppression effects were themselves suppressed they go away if you take away the top-down feedback and this is a way easier experiment to do in in a network because you can just turn off the feedback connections and see what happens and lo and behold you get almost exactly the same pattern of feedback so in the pred net sit extent there surround suppression it's happening as a top-down feedback phenomenon just the

Predictive Coding Models of Perception


same way it seems to be happening in the primate and then you know as we sort of march through these things we also have this interesting sort of sequence learning effects that you see in the end of the ventral visual pathway and lo and behold if you show it pregnant these things so the pred Nets just trained on natural images like our videos and then we're gonna show it these stimuli that are basically the same as what you would find in these experiments in an untrained pred net you get these sort of funny effects where it's you know you

Predictive Coding Models of Perception


flash up an image and it takes a little while to sort of predict it away but then as you train these these sequences so that are expected you get sharper and sharper transitions from between these images and then what happens is basically the same exact result so if B is predictable from a in serial presentations again this is trained on cars and then we just subsequently show it a few of the you know a comparable number hundreds of examples then B will be suppressed if a explains away B but if we show that very same B and you know

Predictive Coding Models of Perception


with something else that's not a that doesn't predict it in front of it we get the exact same result that the Carlos and company got so it's able to track with almost exactly the same number of trials some of these interesting sequence learning facts with no no extra machinery needed to be added and then you might also ask okay that that flash lag illusion so you might remember you know this this clocks going around this bar is flashing it's actually lined up but it looks like it's lagging behind so you might ask

Predictive Coding Models of Perception


oh is that this is the same thing happen with with the pred net and the cool thing about generative neural networks is you can see what they're experiencing because we can just visualize the error layer and so this is what it's seeing now there's a caveat here does anyone know the caveat is yeah yeah so you're getting a double dose so so the network is seeing it and then you're seeing it on top of what the network is seen so if we really want to do this we need to we need to look at freeze frames and we can see exactly what what the network's

Predictive Coding Models of Perception


seeing and there it is there's that weird lagging and there's the tilt too right that was weird there wasn't a good there wasn't a satisfy satisfy explanation for that tilt before and it just sort of falls out of the pregnant yeah just predictable on a longer time scale yeah so it's the same thing with with with the human version of it right like it's predictable but there seems to be some window beyond which it's not it's effectively not predictable that's right there's also weird things like you see these like weird ripoli things happening

Predictive Coding Models of Perception


and I don't know what those are but they're really cool and I like looking at them and we'll figure out what they're for eventually and then I love the scientific community and I love archive so we posted this and a group in Japan picked it up and started using it so they basically wanted to see these illusory motion stimuli they wanted to see what would happen if they showed them to pred nuts so so the way these work hopefully you're experiencing the illusion right now this is a static image but it looks like these are are

Predictive Coding Models of Perception


sort of these ones are sort of are sort of rotating and these ones are not so this is a classic you know sort of illusory motion sort of stimulus and it turns out if you if you train a pret net on these sort of rotating so it's seen real objects rotating in the world or synthetic images of like propellers and things moving in the world and then you compute the flow vectors on the actual predictions if you just let a pregnant look at these static images the pregnant actually produces flow vectors that are consistent with the illusion so when

Predictive Coding Models of Perception


humans see the illusion the pregnant has optic flow that looks like what we see and when humans don't see the illusion there's no optic flow so so this is this is interesting especially because the usual explanation for these things has to do with sort of epiphenomena about the relative Layton sees visual responses and now that might still be true but this at least gives you another possible explanation which is that you have a system that's trained with the predictive loss it's gonna naturally have some of these biases that come from

Predictive Coding Models of Perception


the sort of statistics of the world so with all things neural network you know like we just hide all the like the parameter search that went into it there is indeed so the failure mode in general when it can't when it doesn't learn so if we have like the hyper parameters wrong or whatever is that it predicts the last frame that's usually our benchmark against which to see if it's working so we take the reconstruction lost if you just said it's the same as the last frame and then compare how much we improve the error and then you're

Predictive Coding Models of Perception


right it if it's too herky-jerky then it has trouble learning so so I don't have a good quantitative answer for you but there there is indeed a sweet spot and yeah I mean it depends on how fast you run the framerate we were running these at 10 frame frame rates so that's kind of yeah about about 100 milliseconds it depends on how fast things are moving as well so anyway so I promised that we would get back to a neuroscience experiment so I'm going to tell you about a story about an actual neuroscience experiment that the

Predictive Coding Models of Perception


neuroscience said the wet lab part of my lab did in response to the computational work that was going on in my lab so you might remember that we could read out the steering angle from a self-driving car using the internal representations in the pregnant we also had the idea well you know wouldn't it make sense to actually take the efference copy of the steering wheel all the odometry of the car we could just feed that into the representation surely the network would do a better job of predicting so if I'm trying to predict how the world is going

Predictive Coding Models of Perception


to change if I know the wheels turn this way I can make a better prediction about how the world's going to change I might also be able to do cool things like if I could sort of fictive Li imagine like conditionally generate if I turn the wheel like this what would the world look like so this is something we started doing just because it made sense from a machine learning standpoint and no big surprise when you don't have a friend's copy which is basically the sort of vanilla pret net you get certain convergence over time you know so you

Predictive Coding Models of Perception


get get down to some mean error and with with training epoch but if you include this extra information unsurprisingly the network converges faster and comes up with a better result so that's that's all fine and good but if this were true and this is what was happening in the brain that would imply that in visual cortex we should have signals from motor cortex right they should be there and if they're there we should be able to decode them even in the dark potentially so so my student Greg thought this was a great idea he was doing 24/7 recordings

Predictive Coding Models of Perception


in visual cortex in a rat at the time so he's had boatloads of boatloads of data so we just had that lab meeting well hey well I just put them in a dark box like so there's no light completely light-tight and see if you can actually and you have an accelerometer on the animal's head anyway because you know the minute you're putting electrodes and you might as well put an accelerometer in there too and then so we can record from from many tête roads so we have you know 60 16 tetrode 64 electrodes and we can record local field potentials and

Predictive Coding Models of Perception


then we also have these accelerometer signals so there's a there's a hypothesis that comes from the computational work that says well we should be able to decode you see friends copy signals in you know potentially even in v1 and it turns out that that is exactly what we can do so so these blue bars are in complete darkness and visual cortex so there's no visual stimulus they're not getting any visual optic flow but we can decode a little bit of you know sort of six degree of freedom information about which way the animals

Predictive Coding Models of Perception


nose is pointing and then interestingly if you inject mute samal into area M two which is the putative place where these motor signals you friends copies would be coming from you know this is all still new so you know don't get futzed if this ends up not holding up because we still only get more animals at least in the first animal we did we can actually abolish that signal so if we take away motor cortex signals we actually can't any longer decode the position of the animal's head in 3-space so this is a case where we actually

Predictive Coding Models of Perception


built them up we built a model that was useful in its own right it helped explain some things about neuroscience more or less along the way without having to fit anything and then by looking at it we can make predictions which then led to experiments that we could do to maybe learn something new about how the brain is organized corollary discharged we can't tell and in an electrophysiology we just have some Tet Rhodes recording from some places and we're not exactly sure where we can't reconstruct the image to fully

Predictive Coding Models of Perception


disambiguate that we also have two photon imaging going on in my lab so there is an idea that we could go and be getting you know hundreds of cells that are thousands of cells at a time you know and have a prayer of actually doing those those decoding experiments and that's work that we have that's the song going but but you know right now we're just in the stage where we said can we decode any information and the answer seems to be yes I agree there's a lot of questions that come downstream of that like well you know predictive coding

Predictive Coding Models of Perception


implies certain kinds of correlational structures that we can start to look at we also have the ability to image synapses in the two-photon image of the synapses differently separately from the cell body so we can actually look at feedback synapses specifically and ask how the information that they contained is different than the the cell bodies in layer 2 3 so these are all experiments that are ongoing but I don't have a good answer for you but that's the right kind of question and that's the right kind of question we can ask you know when we

Predictive Coding Models of Perception


have these kinds of models they can guide our experiments ok so this is my pitch so apparently a deep mind it's a it's a circle for me it's just a it's just a ping-pong I don't know I like the circle better maybe I'll change it top away oh yeah that's right so people are picking this up Gooding neuroscientists and we love that if you want to pick it up there it is the codes all free I'm told it's not hard for other people to get working so that's great that's a testament to Bill's hard work to share things so I just want to

Predictive Coding Models of Perception


try and get us back on schedule so this is my lab just went to acknowledge everyone particularly Bill water and then all the people who funded this stuff and then just one shameless plug you know so I've just started as the director of this institute if you're interested in sort of AI and neuroscience Nexus the kinds of stuff that I'm showing you here and you reached in doing that in industry you know find me during one of the breaks I'd be happy to talk right thanks [Applause] yes oh the most of the mud so the motor

Predictive Coding Models of Perception


modulation it's there's a whole cottage industry of like my son trackballs when they run versus when they don't run and and basically there's more information about whether the running or not then there seems to be about the visual system that's sort of like a first approximation but but a lot of the there's been a sort of a dichotomy of people who are either either think you know maybe this is a predictive process and kind of thing versus people who think it's more of like an arousal sort of like when the animals running they're

Predictive Coding Models of Perception


on and when they're not running they're off or something and you can probably imagine which one I think is the more likely scenario and so so so the idea that we can actually get information about the actual direction the nose is pointing I think is interesting and a little bit surprising and and feels like at odds with the idea that it's just sort of an on versus an off kind of transformation now in terms of vestibular signals yeah I mean we're we've talked about you know the you know the experiments aren't pretty but you

Predictive Coding Models of Perception


can imagine what they're like you know it's like I'm just moving I'm moving the rat in the dark Here I am with my rat moving in the dark so you know like sometimes this thought you have to do the science and and it's not it's not glamorous so we say so so we aren't in those experiments I mean it doesn't sort of go against the general theme here like wherever you get the signals from you'd be crazy not to use them I think the fact that we see some reduction in them when we knock out em to suggest a little bit that maybe it's more of a

Predictive Coding Models of Perception


motor you friends copy but it's entirely possible that this tabular system the vestibular inputs are getting through to v1 through so m2 is I mean it's like it's sort of like frontal orienting fields in a monkey it's like an orienting area yeah so as near as we can tell it doesn't seem to so we've done let's go to all the marginal comparisons so you'd expect it to have some effect it turns out you can actually just scoop out all of them fun all the motor cortex by my neighbor at Harvard Ben selves Eskie did this

Predictive Coding Models of Perception


experiment are you basically just carpet bomb all of motor cortex and the animals can still do all kinds of complicated motor tasks so that that's alarming if you study motor cortex but but from our perspective it's it's actually a good things I mean so it's near as we can tell we're not changing with the statistics of the movement one of things were moving towards mister is trying to get more quantitative predictions about what would the actual predictive coatings that are subtractive having a model at least gives you a prayer of

Predictive Coding Models of Perception


being able to sort of ask those kinds of questions I agree it's complicated it's not at the end not the end-all be-all and and also in monkey and primate IT there was there some beautiful studies that were they were kind of lesser-known where they showed that in complete darkness they were psychotic eye movement signals presence he could decode when cicadas happened so so these ideas aren't aren't new it's just for their sort of driving us in new directions [Music] but when you're comparing that to the neural state law of attraction the first

Predictive Coding Models of Perception


example if you're showing if you're comparing that to the N stopping signals or the transient dynamics yeah it turns out if you look at them they almost all look the same which which is disheartening for people who want to disambiguate different populations I mean at least when you marginalize them in this way the the e neurons look like pop and then off response but so did the activation neurons and so there are recurrent neurons that are sort of sitting there and like like in Matt's talk like they're ones that are doing

Predictive Coding Models of Perception


the things you know they have to be doing but there are ones in the in the recurrent layer that also have the sort of transient dynamics as well so so the long story is like if you were sticking electrodes into a monkey brain like I did for five years you know you wouldn't know which ones of these you were getting and they all look surprisingly similar I mean so in the in the cases where we're doing the cutting we're actually decoding from the ours so but I mean you you can decode lots of things from lots of places the arse

Predictive Coding Models of Perception


we thought the arse made a lot of sense because they're the ones who should be holding information about context and representation but I mean there's obviously a lot going on yeah as you get further along though this is a CNN in the static case on the first time step because you basically there's nothing to cancel out this is all zeroes so you just go up through like a CNN so you would expect there to be sort of different levels of representation of higher level and lower level features so it's only long story short it's

Predictive Coding Models of Perception


complicated the the simple fact of training it in this predictive mode is gonna affect all of these weights so it's going to induce representations even in these A's so the bottom line is like you can pretty much decode from anywhere this is bad news for Neuroscience right you can decode from anywhere and all of them qualitatively look very similar in their dynamics so I think about the relationship between what you're doing here and the stuff that I've been working on but honestly really the ideas that were put

Predictive Coding Models of Perception


forth by hoc right originally about supervise and that work would suggest that this system should be able to do something even richer which is to do something like visual system identification so in the phenomena that you've you've talked about you look at the first frame and you can make predictions about what's going to happen because you live in this world but there are situations where I can't predict this observing the dynamics helps me predict the dynamic so if I like let me say it's seeing or someone with a fuse

Predictive Coding Models of Perception


joint or something should be able to do that kind of system identification on the I don't know but I agree and we're I think we're all kind of using the same market fuel right it's like Ellis TMS can do a lot and these are sort of two separate instantiations of like recurrent Nets can can actually do do a lot yeah exactly exactly exactly and and there is a spot that you know they even after you fix things and it's running you know it has state that's that it's accumulating so it can you can do I think system identification so I mean

Predictive Coding Models of Perception


this isn't this isn't fit exactly into your what you're talking about but we also had it doing things like predicting balls bouncing around and things like that and you know and it works you know like the balls bounce off the walls yeah and if we have these like have like electrostatic repulsion or something you could imagine them learning dynamically like well you know this is this is the new rules so yeah and this is the kind of setting I would probably prefer to do it in rather than the other one cuz you can't control the natural

Predictive Coding Models of Perception


world as easily but but I think it's really interesting I mean that there are limits of the capacity of this thing to learn so it you can't you know if you have very complicated movies and things it doesn't do a great job of predicting the next frame so we've far from crack the nut but it just feels like we're kind of moving maybe a little bit in right direction so so much like the project I spent 20 years of my life working I've had so many people tell me that it ultimately got published at nips as unsupervised

Predictive Coding Models of Perception


pixel prediction and it was just trying to predict a moving dot no recognition of anything it was only doing the motion different continuity in time I wrestled with it it was awful it worked kind of okay yeah yeah while I was working as a coder and an algorithm guy and stuff in Silicon Valley what some C++ and a job I worked at Redwood Neuroscience but ultimately what happened was I rely I alternate Lee gave a presentation here that it worked with just using matrices and just the algorithms forget the neurons it was that hard then I throw

Predictive Coding Models of Perception


away even the matrices and finally got a zero parameter model where I just assumed volumetric 3d computational medium intracellular space so if you assume brains just do 3d computations I so I understand I think I think we understand continuity in time and it's just what he said yeah but trying to build it on top of deep learning which was built around essentially n dimensional hyperspace all we have to do with 3d continuous it's not easier oh this is a soft key in fits paper is that the one I think I think I

Predictive Coding Models of Perception


think we I mean I think we I think we saw I think we cited you but the I think I think I think the you know the tools I mean the thing that's nice about deep learning and maybe others have this sort of intuition about it it sort of nice to be able to say here's our like I'm saying like presto magico deep-learning what I'm saying like I think we've gotten to the stage where but that's all abstraction now what we can say is if we just optimize future frame prediction what else obtains and I think that's the right framing for using deep learning to

Predictive Coding Models of Perception


sort of it's not that we're like doing neurons or that they are neurons like this has back prop like clearly you know like there's gonna be a couple suggestions on how we can do it better it's more like if you just optimize this one thing and deep learning lets you very effectively optimize that one thing this is what comes out of it right so that that's kind of the mode we're thinking about it and you know could it be reduced to some other simpler thing sure but we just wanted to have the tools that take us to there to that to

Predictive Coding Models of Perception


that point and I think we're I think we're out of time unfortunate this or that feature of object this is the kind of work you did in your thesis and your adviser certainly you know should not fire at all if it's correctly representing this thing and so I'm trying to picture how the same room can be both representing something and they're not representing something and and the question is maybe the answer is in that your last picture when you said so on the first pass to reader model basically the representational kind of

Predictive Coding Models of Perception


thing is getting passed forward through these error units which are the only things passing anything for which is a pretty strong constraint in their model and so but then later those same units must be passing forward something it means something different which is you know the the non predictive part only those are two very different things at very different at different times and so are you is that the correct correct interpretation which is the sort of there's a time multiplexing where maybe a first pass through the ventral

Predictive Coding Models of Perception


Strait is the representational pass and could lead to successful recognition of an object and then later that thing is somehow shut off because everything is predictable or what have suggested time multiplexing in the responses before so there's a paper from from yesterday's group I could be wrong about that where they're basically claiming that the initial pop of activity had different information than the latter part I would say when when when Jim and I have said in the past that this neuron represents that thing we're taking time like cat

Predictive Coding Models of Perception


spike counts within 100 millisecond window after you know 100 milliseconds after onset of stimulus that that same neuron is categorically not doing that 100 milliseconds after that right like you know there's this weird phenomena that a lot of the neurons are shut down so I think this idea that these neurons the firing of these neurons signal that you know without any further additional context I think that that kind of has to be wrong that's on the level I mean the fact that these things are so dynamic I think the end of the day all the system

Predictive Coding Models of Perception


hello my name is Krishna and welcome to my youtube channel today we are basically going to discuss how to learn data science for free now when I say for free as you know that their whole lot of materials available in the internet with respect to Python programming language with respect to data science with respect to machine learning AI deep learning and whole lot of stuffs so what I will do is that in this particular session I'll show you a systematic way how you can basically complete your data science syllabus within three months so

How To Learn Data Science by Self Study and For Free


that your transition towards data science can be possible just within three months and after that what you can do is that you can also attend interviews by updating your resume now resume part will be discussed later on in the upcoming videos but today we will just try to focus how to learn data science for free I'll show you the systematic way what our YouTube channels you can basically follow you know because there are lot of YouTube channels provides free materials with respect to Python machine learning deep


learning and all apart from that you I'll also be mentioning about various blogs and that finally I'll also be mentioning about the best machine learning book that you can basically use in order to learn data science machine learning very easy now to begin with guys so I have already prepared the word doc over here in my laptop and this particular word doc I will actually upload it in my google drive and share with all of you and that will basically be given in a description box now in this particular video I am basically


going to tell you that how we are going to do how we are going to learn data science with respect to machine learning and deep learning considering Python programming language the reason I'm telling about Python programming language is guys because I have an expert in Python programming language I've referred a lot of materials I have referred a lot of things and I've also done a lot of stuff self-study so that is the reason why I'm actually telling you this for our programming language I need to do a little bit more research


which are the best materials but apart from that all the machine learning deep learning techniques for learning purpose you can basically use this materials use this links that I'm basically giving you but for the practical applications you have to be you have to you should be able to search through various internet resources okay so to begin with first of all as this is for Python programming language the first topic that I will take is basically from where we can basically run Python we can basically learn


sorry so I had two channels in my like most most of the time I referred this channels whenever I wanted to learn about Python whenever I had some queries and the best part of this particular channel this two channels are read in Python as you know if you are learning data science you should just not know Python you should also know object-oriented features in Python apart from that you should also know some other frameworks like flask and Django because this two frameworks are very very important for deployment of machine


learning models and deep learning models okay during the deployment stage you know that you create a flask framework or Django framework and you just uploaded it in in some other servers let it be a platform as-a-service server it may be infrastructure as a service server like ec2 instance of AWS or HID okuu platform and many more platforms are there but initially the web framework is basically created the micro-service framework is basically created with the help of flask or Django so this channel the first channel that I


like to mention is Corrie Schaffer he is a wonderful person he used to work in an IT company before but later on he moved into teaching into YouTube channel itself he has one of the best Python videos guys the link is basically given in the description I mean in the word doc in the description so you can basically refer the YouTube channel of his link and I would suggest you if you have any queries go and see that particular channel with respect to Python and it is started from basic from the basic installation part ok now in


that particular channel you'll also find a playlist on a flask and Django so this was one of the favorite channel that I also refer for learning Python so that is Corey Schaffer the second person is basically sent decks okay send decks is one of the oldest youtuber who uploads videos on machine learning deep learning Python natural language processing so he is also one of my favorite youtuber and he's a very simple guy very very very you know if I see him I really get that motivation motivation because he provides every materials every videos


that he upload he does not have any online tutorial sites he just has some sites where he'll be writing basically about the blogs of whatever he's doing in his YouTube channel so before that saying Dex and again the link will be given in that particular word doc itself now once you learn Python programming language okay now Python programming language I think if you sit and guess guys this this if you're planning to cover this in three months make sure you give three to four hours daily okay give three to four hours and I'm where


I'm saying give three to four hours that should be more productive hours okay now after this now once you finish Python programming language now the next thing is that you move towards machine learning now I know that many of them will ask me a question - Mia asked me a question saying that where is the math part where is the linear algebra part where should we learn it from where should we learn it the differential calculus and many more things right statistics parts and all guys don't go in that particular way we need to


complete the data science syllabus within three months so what you do is that pick up the machine learning algorithm and through reverse engineering understand the mats and try to derive take a use case how to solve it and finally solve that particular use case try to optimize that particular use case try to increase the accuracy so while you're doing this all the steps you will be means you will be learning statistics you will be learning linear algebra will be unlearning differential calculus wherever it is required always


do that reverse-engineer get that knowledge you know now suppose if I want to solve linear regression now when I am learning linear regression I know that there will be an equation of straight-line that will be coming into that particular algorithm like Y is equal to MX plus C then after that I will be saying I'll be just finding out I'll be deep diving how do I find out that coefficient value then over there the gradient descent come into existence then I learn about how this value is basically calculated through


differential calculus I'm just taking as in one one as an example of our linear regulation similarly you have to learn with respect to each and every algorithm so for this I have actually again selected three channel one is you need to understand the maths behind each and every algorithm so you can basically refer or machine learning course by Andrew ng in applied AI so deep learning dot AI sorry not applied air deep learning dot AI channel you can basically and again the link is basically given in the word doc


itself the other thing is that I have also uploaded many videos on machine learning and some of the feedback that I got that the machine learning playlist is not that ordered you know so what what you can do is that whenever you are searching my videos suppose you are learning simple linear regression just search that keyword and just put my name in front of that you will be getting the whole explanation apart from that I have also uploaded videos with respect to practical application ok so you'll be able to do


that now I am trying to order that particular playlist and I'll make and making sure that whatever videos that I upload in the future that will also be ordered so I would like to say one is Andrew ng one is my channel with respect to machine learning if you just want to know the maths about each and every machine learning algorithm go and see Andrew ng from deep learning dai and then you also have sent dex channel again I'm referring send text because he has uploaded videos on Python he's uploaded videos on floss Jango apart


from that he's also uploaded videos with respect to your machine learning algorithms ok and the best part is that he's not uploaded machine learning all he has uploaded deep learning also so I'm going to refer him again in the later links when I'm discussing about deep learning so machine learning three things one is my channel one is sent X and the other one is auntie Ong 4 applied for deep learning dai and um you know Andrew ng has explained the complete maths sometimes what happens is that you will not be able to follow it


but again if you just see again and again you'll be able to follow it because the math is pretty much simpler but what I am making sure that in my channel so I will be uploading a lot of maths thing whenever I am explaining you about some specific algorithms and that will be continuing going on in Andrew ng channel of deep learning dot AI you will not find any practical application so for the practical application you can either refer my videos or you can refer send text videos ok over there send X do not explain you the maths behind any


machine learning algorithm since is more focused towards implementation of machine learning algorithms ok so this was one now I have also seen some people who gets really attracted just not by seeing writings equation they like some animation kind of explanation so if you want some animation kind of explanation there is one channel that I went through is stat quest with Josh stammer ok so this is one of the good channel where they'll show a lot of animation to explain you each and every machine learning algorithm


with you know along with that explanation the theoretical explanation how it is basically done including all statistics linear algebra differential calculation and different kind of maths formula again the link is basically given in the docx file itself okay then after that you have natural language processing natural language processing you can basically go through my playlist because I have uploaded around eight to nine videos with respect to machine learning and I'm planning to upload with respect to deep learning also where I'll


be implemented a lot of NLP things where I'll be implementing with the help of word to work and all the different kind of tools or libraries that are basically present in natural language processing the other channel is basically again sent takes send X has been up has uploaded around twenty to thirty videos with respect to natural language processing and then you can refer that also now we let us go to the deep learning for deep learning I have selected two channels one is Andrew and G again from deep learning day I again


any theoretical components any theoretical things that you need to understand about deep learning can be sick later for that link again just watched the word doc file again in that I mentioned the link also the second channel is my channel because deep learning might be complete deep learning playlist that I have created is completely in order okay to the other one doodle-do tutorial three like that I have have actually created tutorial 22 and I'm including both maths understanding of the algorithms understanding about neural networks and


how to implement that with the help of carols and Python right so I have done that I have ordered it you know first of all I explained you about all the artificial indian or artificial intelligence artificial neural network and I mean artificial neural network I have explained about lot of things like back propagation how to update weights bias all those things and I am also showing it practically how you can basically use for practical implementation with the help of Kira's apart from that how can you basically


optimize your problem statement also now apart from that you can basically refer my channel thrush I'm also going to complete the whole deep learning playlist still they are 15 to 20 videos so total overall display playlist will be having 40 videos okay which will be including LST m RN and CN and everything okay everything in and I am I'm serious about it because I started that playlist still not when you do videos as been completed I have still have plans to upload more and more now after learning all this you'll be having


a lot of idea and this particular hole whatever I have explained is with respect to YouTube channel right now okay the whole lot of materials that are available in YouTube and the next thing is that you refer github links you know github links so suppose if you have a problem with about linear regression go and search linear regression github okay so you'll be getting various abundant materials in the Google in Google search itself and you can basically take one of the problems start solving it now the next thing is that after after learning


all the skills you have to do a lot of practice projects right so I have created one deep learning sorry data science project playlist wherein I have uploaded more than 50 videos 50 different use cases and that is specifically kaggle use cases that I have taken I've solved with the help of Python and machine learning and deep learning so you can basically refer those projects try to solve it again all the code is basically given in the github itself you can refer my github and you can get up all the details ok so


after learning all these things the last part is data science projects you should be able to implement various data science program finally create your resume you know include everything that you have learnt into your resume that's it now first we have discussed about YouTube channels now the second thing is that I'm also going to refer some of the blocks which has all almost all of the problem solutions of every machine learning algorithms or deep learning algorithms one is towards data science blog and the other one is medium with


respect to machine learning and deep learning so all the links is basically given in the word doc itself ok so in short this was there and the last part is basically about books now one of the best machine learning book that I have also read you know the link is basically given in the description about the best machine learning book it is basically written for the O'Reilly publisher if you go and see that particular books book as the best book guys best book on machine learning and deep learning I think every fresher every fresher who


wants to make that transition towards their data science machine learning I they should go and read this particular book and this book is basically you know a boon to all the data scientists because the author his basically or alien gear on sorry if I'm pronouncing it wrong but this particular person has written this book and it is basically the book name is hands on machine learning with sky kid learn and tensorflow okay again the link is basically given in the description and the the publisher name is O'Reilly and I'll tell you this


book is very cheap guys I see there are a lot of free PDFs also available for this book I don't want to share that PDFs I don't know don't hurt want to research and find out the free PDFs over there because this guy has written this author has written this book so nicely and I don't want to disrespect him by just taking the free PDFs and distributing it to you okay I have not even researched and found out whether there is any PDF or not I basically download I basically bought this book the paperback version and I'll just if


you want I can also review this particular book in one of my next videos but awesome book you know it is basically you have Python you have feature engineering you have machine learning you have everything okay and about apart from that guys you can buy this book and it is hardly around 1,500 rupees INR if you consider this in the terms of dollar hardly ten to fifteen dollars fifteen to twenty dollars so I think yes fifteen to twenty dollars you can basically buy this particular book again the link is basically given in the


description go ahead and buy it I know this particular session is all about data science with respect to free but I'd suggest this particular book you know don't search for free PDFs keep this book handy because it will help you for the lifetime okay whenever you want you can basically leave this now the next thing comes that there are small small parts like feature engineering feature selection what you have to do is that I'll be sharing a very good github link about feature engineer and feature selection which I found it through the


internet and I'll be sharing that link in this word doc itself what you have to do is that go inside that link there are many materials present inside just refer each and every notebook file it is clearly written what what is all about feature engineering how feature engineering is basically done similarly they are on 10 to 20 materials notebook file Jupiter notebook files you can just you just have to read it just have to execute that and by that you will be able to understand a lot of thing and similarly for the feature selection so


I'll be sharing those two github link and thanks to the author I'll also be mentioning the author over there who had actually provided those materials and it is available in an Internet completely for free so I'll be providing that two things to you and yes that is all about this particular preparation guys and I think if you are able to give around two to three to four hours I think within three months you'll be able to complete this whole data science syllabus and after three months you'll be also giving the interviews because you have a learnt


a lot of things done data science projects you have practice things in kaggle and make sure you practice a lot of projects after completing all these things through this YouTube channel through blogs through this particular book that I have told you from O'Reilly publisher which is basically hands on machine learning a sky killer intensive flow that was all about this particular video I hope you liked this particular videos share with all your friends subscribe do subscribe this channel if you're not already subscribed I'll see


i have read literally thousands of books on modern psychology metaphysics ancient magic buddhism yogism theosophy christian science unity truth new thought and many other dealings it's what i call mind stuff many of these books were nonsensical others strange and many very profound gradually i discovered that there is a golden thread that runs through all the teachings and makes them work for those who sincerely accept and apply them that thread can be named in a single word belief it is the same element or factor

The Secret Knowledge Of Believing


belief which causes people to be cured through mental healing enables others to climb the ladder of success and gets phenomenal results for all who accept it why belief as a miracle worker is something that cannot be satisfactorily explained but have no doubt about it there's genuine magic in believing the magic of believing became a phrase around which my thoughts steadily revolved i've tried to put down these thoughts as simply and as clearly as i could so that everyone can understand my hope is that anyone who listens will

The Secret Knowledge Of Believing


be helped in reaching their goal in life so you begin with desire if you ever hope to achieve anything or gain more than you have now however as we shall see there is more to it than mere desire it has been said that thought attracts that upon which it is directed thought attracts that upon which it is directed it was job who said for the thing which i greatly feared has come upon me our fearful thoughts are just as creative or just as magnetic and attracting troubles to us as are the constructive and positive ones and

The Secret Knowledge Of Believing


attracting positive results so no matter what the character of the thought it does create after its kind when this sinks into a man's consciousness he gets some inkling of the awe-inspiring power which is his to use i cling to the theory that while thoughts do create an exercise control far beyond any limits yet known to man they create only according to their pitch intensity emotional quality depth of feeling or vibratory plane in other words comparable to the wavelength and wattage of a radio station thoughts have a creative or controlling

The Secret Knowledge Of Believing


force in the exact ratio of their constancy intensity and power let me try to clarify that while many explanations have been offered no one knows whether thought is a form of electrical energy or something else yet to be defined but i have been an experimenter in that branch of electricity known as high frequency pioneered by the great genius nikola tesla and whenever i think of thought and its radiations and vibrations i instinctively link them up with electricity and its phenomena in this manner they become more

The Secret Knowledge Of Believing


understandable to me all persons living in high altitudes have felt and sometimes observed the electric spark resulting from walking across the room then touching some metallic substance that of course is a form of static electricity generated by friction it gives you an idea of how one kind of electricity can be developed through the body sigmund freud the famous austrian psychoanalyst brought the world's attention to the hypothesis that there was a powerful force within us an unilluminated part of the mind

The Secret Knowledge Of Believing


separate from the conscious mind constantly at work molding our thoughts feelings and actions others have called this division of our mental existence the soul some call it the super ego the inner power the super consciousness the unconscious the subconscious and various other names it isn't an organ or so-called physical matter such as we know the brain to be nevertheless it is there and from the beginning of recorded time man has known that it exists the ancients often referred to it as the spirit paracelsus called it the will others

The Secret Knowledge Of Believing


have called it the mind an adjunct to the brain some have referred to it as conscience the creator of the still small voice within still others called it intelligence and have asserted that it is a part of the supreme intelligence to which we are all linked no matter what we call it i prefer the word subconscious it is recognized as the essence of life and the limits of its powers are unknown it never sleeps it comes to our support in times of great trouble it warns us of impending danger often it aids us in what seems impossible

The Secret Knowledge Of Believing


it guides us in many ways and when properly employed performs so-called miracles perhaps the most effective method of bringing the subconscious into practical action is through the process of making mental pictures using the imagination perfecting an image of the thing or situation as you would have it exist in physical form this is usually referred to as visualization however before this visualization can work you must really believe i refer now to deep-seated belief a firm and positive conviction that goes

The Secret Knowledge Of Believing


through every fiber of your being when you believe it heart and soul as the saying goes now call it a phase of emotion a spiritual force a type of electrical vibration anything you please but that's the force that brings outstanding results it sets the law of attraction into operation enables sustained thought to correlate with its object this belief changes the tempo of the mind or thought frequency and like a huge magnet draws the subconscious forces into play changing your whole aura and affecting everything about you

The Secret Knowledge Of Believing


and often people and objects at great distances it brings into your individual sphere of life results that are sometimes startling after studying the various mystical religions and different teachings and systems of mind stuff one is impressed with the fact that they all have the same basic modus operandi and that is through repetition the repeating of certain mantras words formulas or just plain mumble jumbo is common with witch doctors voodoo high priest hexers and many other followers of strange cults they use them to evoke the spirits or

The Secret Knowledge Of Believing


work black magic one finds the same principle at work and chants incantations litanies daily lessons also the frequent praying of the buddhists and muslims alike the affirmation of the theosophists and the followers of unity the absolute truth new thought divine science in fact it is basic to all religions although here it is white magic instead of black magic this brings us to the law of suggestion through which all forces operating within its limits are capable of producing phenomenal results that is it is the power of suggestion

The Secret Knowledge Of Believing


and auto suggestion your own to yourself or hetero suggestion coming to you from outside sources that starts the machinery into operation or causes the subconscious mind to begin its creative work and right here is where the affirmations and repetitions play their part it's the repetition of the same chant the same incantation the same affirmations that lead to belief and once that belief becomes a deep conviction things begin to happen this is the same identical force and the same mechanics that hitler used in building up the

The Secret Knowledge Of Believing


german people to attack the world a reading of mein kampf will verify that dr renee favel a famous french psychologist explained it by saying that hitler had a remarkable understanding of the law of suggestion and its different forms of application it was with uncanny skill and masterly showmanship that he mobilized every instrument of propaganda in his mighty campaign of suggestion hitler openly stated that the psychology of suggestion was a terrible weapon in the hands of anyone who knew how to use it let's see how he worked it to make the

The Secret Knowledge Of Believing


germans believe what he wanted them to and once that belief took hold how they started their campaign of terror slogans huge signs posters masked flags appeared throughout germany hitler's picture was everywhere one reich one folk one leader became the chant it was heard everywhere today we own germany tomorrow the entire world the marching song of the german youths came from thousands of throats daily such slogans as germany has waited long enough stand up you are the aristocrats of the third reich germany is behind hitler to a man and

The Secret Knowledge Of Believing


hundreds others bombarded the people 24 hours a day from billboards sides of buildings the radio and the press every time they move turned around or spoke to one another they got the idea that they were a superior race and under the hypnotic influence of this belief strengthened by repeated suggestion they started out to prove it unfortunately for them there were other nations who also had strong national beliefs that eventually became the means of bringing defeat to the germans i know that it is difficult for the average

The Secret Knowledge Of Believing


person who knows nothing of the subject to accept the idea that all is within but surely the most materialistic person must realize that as far as he himself is concerned nothing exists on the outside plane unless he has knowledge of it or unless it becomes fixed in his consciousness it is the image created in his mind that gives reality to the world outside of him happiness sought by many and found by few therefore is a matter entirely within ourselves our environment and the everyday happenings of life have absolutely no

The Secret Knowledge Of Believing


effect on our happiness except as we permit mental images of the outside to enter our consciousness happiness is wholly independent of position wealth or material possessions it is a state of mind which we ourselves have the power to control and that control lies with our thinking emerson said what is the hardest task in the world to think obviously this is so when one considers that most of us are victims of mass thinking and feed upon suggestions from others we all know that the law of cause and effect is inviolable

The Secret Knowledge Of Believing


yet how many of us ever pause to consider its workings the entire course of a man's life has many times been changed by a single thought which coming to him in a flash became a mighty power that altered the whole current of human events history is replete with the stories of strong-minded resolutely willed individuals who steadfastly holding to their inner convictions have been able to inspire their fellow man and in the face of tremendous and determined opposition have literally created out of nothing great businesses

The Secret Knowledge Of Believing


huge empires and new worlds they had no monopoly of thought power you and every man and woman have it all you have to do is use it you will then become the person you envisage in your imagination know yourself know your power faithfully use the cards in the mirror techniques and you will get results far beyond your fondest expectations just believe that there is a genuine creative magic in believing and magic there will be for belief will supply the power which will enable you to succeed in everything you undertake back your

The Secret Knowledge Of Believing


hello there today we're looking at bootstrap your own latent a new approach to self supervised learning by researchers of deep mind and imperial college so not almost no day goes by where we don't hear some sort of new self supervised algorithm right here this paper on a high level tries to get rid of the necessary negative samples when doing the contrastive loss for self supervised learning and they basically combined momentum contrast and seem clear and then remove the negative samples and that seems to work pretty

BYOL: Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (Paper Explained)


well even though it's magic so yeah if you if you want to see how it's done stick around share the video out if you want other people to see how it's done and leave a comment this this one I really don't get what's going on so if you have ideas put them there I'll I'll read them through it'll be fun alright so they say we introduced bootstrap your own latent or be all a new approach to self supervised image representation learning ok so image representation learning is the simple task of taking an image and then feeding it through a function which


is usually like a neural network let's let's just say this is a neural network and in fact all of these the community has sort of standardized this to be most of the time it's something like a ResNet 50 ok so what you want to do is you want to train a neural network like a ResNet 50 to give you a good representation of the image so this would be like H and H is a vector and H is a representation of this image and the representation should be such that you can then take this representation and solve many tasks with


it which either can be like linear you can put a linear classifier on top of the H or you can fine-tune the entire architecture to solve some other task the idea is if you have a large data set you may use this dataset to train these good representations of these images and then you can transfer learn transfer this to a task where you might be not have as much data and because you don't have as much data it's not enough to completely train an architecture like this but it is enough to take an architecture that's been trained with


the large data set and just adapt it to your small data set and that usually it tends to work pretty well this is called transfer learning this step here is called fine-tuning sometimes and it's sort of the approach that comes from natural language processing from these big transformers like Bert where you first train on a really big data set that might not be the data set that you want in the end but it's really big so you can sort of learn a lot of things from that data set and then the only thing left to do is to fine tune it to


basically adapt it to the nuances of your data set but it will have learned most things already and that's called representation learning so the goal is to learn a good representation now this self supervise here is also important because representation learning can be as easy as if this here is image net the image net data set contains like a million images all with labels you can simply train your ResNet 50 to predict the class this is the this is called supervised pre training or supervised representation learning and that works


pretty well but you need a labelled data set in self supervised learning you do not need labels what you do is you do self supervision and self supervision it has many there are there many ways to do self supervision but what we'll see in this particular paper is that you will take an image and you'll make a different variance of that same image so you'll take the image and you'll make many many variants of it well let's just say two so you have some procedure to sort of change the picture a little bit but it's essentially still the same and you


do that through data augmentation so this could be a random crop or you color jitter or you rotate it or something like this and then you exploit the fact that you know that these two things they should be still sort of the same image so once you send them through your through your encoder the representations of the two images they should be fairly close now let's actually read on right here Buhl relies on two neural networks referred to as online and target networks that interact and learn from each other from an augmented view of an


image we trained the online network to predict the target representation of the same image under a different dog mented view okay that's sort of what we saw so we have we have the same image under a different dog mented view so what does it mean what what I just said you make two versions of the same image one that are slightly different and then their representation should be close now until this point we have always thought that this would degenerate because what if you think of this neural network that does this encoding to the hidden space


this ResNet 50 right here if it wants to if you simply want to make the two representations close what's the best thing it can do you can simply map all the hidden it can simply have the constant function H equals 0 or something like this just a constant function because then this loss here is always going to be 0 like perfect okay so no matter what image comes in if you always map it to the same thing you will always be close in representation space and therefore you always win that doesn't learn a really good


representation right so what people have done is they have included so-called negative samples where you'll say I'll take a different image from you know from this dataset but it's a different image than this image and I also do some maybe some data augmentation with that image and then I send this through the same encoder to also give me an H so this is the H let's call that H original this is H plus because it's the same image but slightly differently augmented and this is H minus which is a different image


and now the task is let's make those two very similar to each other but let's distance them from this other one so we want we want this to be as far away as possible and these two to be close to each other now the network can't simply map everything to a constant function anymore right it needs to actually do something to make these be close together and this be far apart and the combination of this together with the augmentation procedure that goes into augmenting the images has been sort of a good combo to learn good representations


and a lot of papers have alluded to the fact that this is so the negative samples are to do not have these degeneracy right so to not have the simple solutions but the fact that the representation then is actually good like is good for image class image tasks down the line probably comes from the fact of these augmentations right here and there's a lot of evidence of from the fact that depending on which augmentations we choose these representations are going to be better or worse for example random cropping of


an image so the random sub like taking a random crop from the image tends to be very very beneficial because so here this is the same image twice right let's say we take a random crop here and one up here it's sort of maybe there's an overlap here in the middle right so it sort of needs to understand that these random crops sir sort of needs to communicate between these two places in these random crops so the representation has to somehow make sure that the object that is overlapping here is somehow represented but it can't represent it


just as a pixel value because it doesn't know where the crops come from so there's a lot of evidence that these representations are the thing that's responsible for making the representations so good okay now this paper simply says do we really need these negative samples right here let's just get rid of them and with a couple of tricks this seems to work in here this is this is what seems like magic to me because as we go forward think of it nothing nothing keeps this model right here from doing the degenerate solution h equals constant


nothing right now for some reason it doesn't do that and I have the feeling that this is a super delicate balance that you have to do because when you train when you start out it's probably not the constant function right it's probably some some distribution and then simply by the fact that you train it and kind of keep it in the so this is certainly an optimal solution but you might be like in some sort of local minimum once you start training and you simply don't get out of it during training and that's why the network has


an easier time step by step as it updates itself in very small incremental steps it has an easier time actually going for the good representation then it has to see this solution right here and converge to that but yeah it seems delicate so what are they doing they are taking that idea of taking an input image right here and so by the way why is it important that there are no negative samples because now the question is always or where do you get these negative samples from right should they be uniformly sampled


should we keep a buffer should we order them there is this task of hard negative mining where you say Oh any old negative won't do it's actually better if we take negatives that are you know just hard enough there is a curricular curriculum learning problems and so on so it would be best to actually just get rid of these negative things so that's why we want to get rid of them so that's the approach byl bootstrap your own latent there is the input image we take one image at a time and you apply two different random augmentations to it


right so you create two slightly different variants of that image through augmentation and again this can be something like a random crop it can be a horizontal flip randomly you color jitter you Solarize you blur and so on there are all these variants of data augmentation and the fact that down the line that the representation of these two things has to be close to each other I think these random these augmentations here are responsible what the to make the to make the these augmentations are responsible to make the representations


powerful okay the fact that later down the line the network has to sort of learn to ignore these it has to learn that oh you know it doesn't matter where in the image this object is because it's been random cropped for different you know at different locations it doesn't matter where in the image this object is I simply need to have my hidden representation have this particular object in the image and that's what makes it powerful okay I've said that enough now then you have these two slightly different versions and then you map it through


your encoder okay let's go the top path first you see the bottom path has the same encoder but the parameters are different and this is going to be one of the crucial elements right here so this here are your actual parameters that you learn and this here are what are called the target parameters now after each and you can see this for all of these components right here so what happens is that the target parameters are basically a copy of these what's what are called the online parameters okay so after each step you copy over from the online


parameters you copy over to the target parameters you never learn the target parameters you simply copy them after each step now you don't copy them outright what you do is you do an exponential moving average so the target parameters are always going to be sort of a lagging average of your online parameters and that idea comes from the momentum contrast principle where the reasoning sort of behind it is that you need a kind of a stable you kind of need a stable representation as a target but I think it hasn't been fully explored or


explained why exactly that is so helpful but we just know that if if we have the target to be not the same as the the online parameters but actually a kind of a stable version of the past of the online parameters then that tends to work well again it's kind of the same principle as with the augmentations with the augmentations we have two different versions of the same image and now with this procedure here we sort of have two different versions of the same neural network but they're slightly different right and this idea you know has been


around for much longer like the the first Q deep Q networks and so on they had the same principles where they had the the network that they actually learned and then the target network that is copied over every such-and-such episodes and so on so this this seems to work seems to be a fundamental principle that seems to work all right so we take our two slightly different augmented versions of the same image and we run them through our two slightly different encoders to obtain two representations now this thing right


here that's going to be our represent so after this procedure we discard the entire thing right here except that so this here is your whatever your ResNet 50 okay after that follows a projection and the projection is is here to reduce the dimensionality and honestly I'm actually not sure why it is here because you can do it without like technically the algorithm doesn't require this projection so you can imagine the algorithm without the projection but just really quickly the projection simply brings down the representation


which is like 2048 dimensional that comes out of the resin at 50 it has it is a two layer neural network that first pumps this up to like four thousand and ninety two and then compresses it down to 256 dimensions okay so that's the projection Network again there is a part that's learned and then the target projector is simply the exponential moving average of the online projector but again this is why exactly this is here probably simply because it works right but probably because there is no there is no distinction because


you don't have different losses you simply back propagate through everything and then train everything so there is no logical distinction between the projection and the representation other than you have a different dimensionality but maybe that's the point here that you make a different dimensionality even though you could you could do the rest in this 2048 space yeah so for now just this doesn't exist let's just say this doesn't exist and we just work with this representation here let's call this Z z prime okay so what happens is we take


the representation and now we have one neural network the predictor right here that takes the representation of one of the image versions and it simply tries to predict the representation of the other image version so what you want is that Q of Z equals Z prime okay and if we expand that is that Q of F of Z is equal to F target of Z prime and if we expand that even further you can see that Q I'll just write Q and F for now Q of F of a which is an augmentation an augmentation of Z should be one bracket to bracket three bracket should be F of


a of Z sorry not see that's the image X all right so this makes a lot of sense you're simply with Q since these are all different here so f is the target instead of these online parameters a is also different it's a different augmentation that you do but the X is the same okay so the Q simply tries to somehow negate this augmentation and this difference between the target and the online parameters but you don't tell the queue which augmentation was used and you don't tell the Q what are the exact parameters of that network so what


the Q has to do is it has to somehow it's like it's like a it has to take its best guess right so basically the Q is trained to output the expected value of the representation right the expected of the representation f of a of X under all of the different possible image augmentations and that's why it learns to ignore these augmentations so your entire goal with these methods is you learn to ignore these augmentations so you want to learn some method that is independent of the augmentations so by crafting the augmentations in a smart


way we can make these representations contain a lot of semantic information because what we want to do with the augmentation is basically we want to destroy all the non segmenting information sorry non semantic information and random cropping is one of those methods horizontal flipping is one of those methods because we say well whether an image goes left to right or right left most of the time the semantics are the same the pixels are different but the semantics are the same so by putting an augmentation in there


we learn to ignore that augmentation because our representation now needs to be predictable right cuny we learn q to predict the the representation under the expectation of our augmentations and that means it can't be dependent on one particular augmentation okay learns to ignore it so that's basically what's happening here again there is nothing keeping this from simply collapsing it to a trivial solution and it's probably a combination of the of the initialization and the learning procedure itself that it you


know goes on in little little steps one by one that keeps it in the realm of rather having to like it's easier to learn a good representation than it is to collapse to that to that solution okay so again components is image then you ought meant differently then you run it through different encoders but the encoders are similar in the fact that one is the exponential moving average of the other and then you try to predict one from the other and that ultimately makes the representation be independent of the augmentation and that means that


the representation can only include things that are not destroyed by the augmentations and if you construct the augmentations smartly that means you only retain the semantic information that's it so the loss function is pretty simple as you can see right here what you want is and this bar is a normalization what you want is the l2 norm between the this representation be close to the cue of that representation so the cue simply tries to predict the other representation and you do that for both ways so you once stick the image in


here and try to predict the other one and you do it vice versa so you get too lost components each time it's a symmetric loss okay and that's it that's the method and they beat all the other self supervised methods and they get pretty close to the supervised supervised representation learning method as you can see right here as the number of parameters goes up in their model so one of them is resonate 50 but I'm gonna guess this one right here but you can also get to higher architectures and then it appears to work even better


and come even closer to this supervised baseline this could be because you know if you have more parameters technically in a supervised method you would also need more labeled images maybe and therefore it doesn't scale as well I don't I don't know there is a lot of unclarity in this research like all they show is that their numbers are good which is cool right and it's cool that you don't need your you don't need the negative samples anymore and it actually doesn't collapse when you do that kind of stuff but there's a lot of I don't


know there's a lot of things here for example we use a batch size of 4096 split over 512 TP uv3 course with this setup training takes approximately 8 hours for resin at 50 so they train eight hours on 512 TP use just imagine that so that's sort of crazy amount of computation again going into these models and then the second thing here is that you can see that there are some things missing right here and there are these all these annotations which probably means that they take these numbers from those papers now they allude to to the fact


that they try to follow their protocol as closely as possible but I mean that's never that's never given or almost never unless they release like the exact code and even then there are still going to be differences in even like you'd have to replicate the exact thing on the exact same number of TPU cores and whatnot so I I highly like these numbers seem to be I'm not sure especially if you then go and look and at some point they actually do reproduce the same clear baseline so you can see right here that they have a own implementation of


sim clear and they actually compare this to the numbers that they find in the same clear paper and you can see for example here there's like four percentage points that the the their implementation of seeing clear gains above this implementation and if you look at this supervised baseline that's also from that paper and there is a graph further down where they also implement their own version of the their own version of the supervised baseline I forget here so you can see that between the supervised in that paper and the