side is going to be but so it's gonna be the same that you have this alpha beta equals 0.5 as a reasonable prior it's only called irrational a distribution which generalizes this to multinomial variables but the high level I mean there's a lot of symbols here Ohio was the same thing you're just saying I pretend I already saw a few experiments I have pseudo counts for those pretend


experiments and I multiply the probability of those into the likelihood of the actual experiments do the same thing for a Gaussian to make the math work out you want the prior for your mean of your Gaussian they also be a Gaussian because then you have a Gaussian multiplied with a Gaussian and we know that's again a Gaussian and the math will be easy if you said well

Lecture 13 Kalman Smoother, MAP, ML, EM -- CS287-FA19 Advanced Robotics at UC Berkeley


actually I think the prior for the mean is that I know it's guaranteed to be positive can never be negative so a Gaussian is not a right fit because the Gaussian even if up without positive it'll still have something running negative well then it's not going to work out as nicely with your math and your product to make a trade-off you might say you know what it's fine I take


a Gaussian far enough positive a small enough variance there's very low probability mass and a negative and the math will work out cleanly and that's what I'm going to use or you might say no I'm going to use some other kind of prior and now I have to do some numerical optimization to find the maximum likely or maximum a posteriori estimate because it's not closed for him


anymore so if typically we'll make a trade-off between convenience and precision of the prior that you are imposing your problem you can do the same thing for conditional linear gosh you can have priors they're gonna priors over the linear coefficient a but more general kind of priors of the matrix there goes some X to Y or from XT to XT plus one that matrix a here are some examples of


this worked out so the slides can just work through what it looks like when you have a prior so there are some points shown in blue the true relation is shown in green so that's what we were hoping to recover but the data is noisy so the maximum likely this within a small amount of data shown in red is actually pretty far off from green but if it had a prior that in this case thinks the


coefficients are more likely to be small rather than be large it'll kind of regularize that and you'll find the black line which is running closer to horizontal compared to the red line now one thing you also want to do and don't want to forget about is cross-validation so whenever you have some data and you just fit to the data it's possible that you're memorizing data over fitting it


rather than paying attention to the real pattern we saw this in valuation sample based valuation you don't want to just over fit the few samples and make sure that you fit your neural net in a way that it generalizes to other data so people we do is you split your data into train and validation data and then for a range of priors you can compute the maximum posterior e and then C on the


validation data which estimate of your maximum a-posteriori parameter gives the best performance on the validation data and that tells you that the prior you use to estimate that was the better prior that's the same thing in standard neural net learning you'd say I put some like coefficient in front of weights square because I want to keep the weights small the coefficient in front


of that is a choice it's a hyper parameter it's a prior over your weights effectively a Gaussian prior over your weights that you're putting in same thing would be happening here you put in a prior and then in cross-validation find out which prior yielded the best results now what we covered so far assumed in all of the maximum life here the maximum posterior that we observe


all the data we can write out the density or the probability of all that data and there's no unobserved variables what we're going to cover next week Tuesday because I'm Thursday we'll do particle filters when Tuesday will wrap up this part we'll look at what to do when there's variables we did not observe and still want to do maximum likelihood estimation that's it for


hi there today we'll look at big self supervised models are strong semi-supervised learners by ting chen simon Kornbluth kevin sworsky muhammad nur Uzi and Geoffrey Hinton of Google brain so this paper on a high level it's also known as Sinclair v2 demonstrates that if you want to do semi-supervised learning that you're very well served by starting out with self supervised

Big Self-Supervised Models are Strong Semi-Supervised Learners (Paper Explained)


learning and then doing fine-tuning much like NLP models do rather than the kind of semi-supervised approach that image image tasks had so far and they present this same clear v2 which is an improvement over the same clear approach to self supervised pre training and they demonstrate it outperforms a lot of the baselines alright so if you like content like this don't forget to


share it out and leave a like and tell me what you think in the comments so this paper um it sort of is kind of a club together thing of different of different things so they present this new method like simply same clear v2 which is a modification of sim clear and we'll go over that but they also try to make like a and a scientific claim namely that the that somehow bigger


models are better for this pathway of learning and we'll try to untangle all of these things so first of all we're in the semi supprised learning regime right here semi-supervised basically means that you have a data set and you only have labels for a part of that data set so this could be like here at the bottom 10% or so because labels might be expensive to get and so you only have a


few of them but you have much more data that's unlabeled now sometimes this problem is formulated as this here is your data set and then this here is like a different data set but one that's close enough such that you can learn from it and that's usually in in NLP you'll have your data set is like a sentiment classification task but you have all of Wikipedia that is not


labeled but it's just text so you can sort of pre train on it in this case we'll be in a situation where will artificially construct a small data set so this entire thing here is going to be the image net data set and this right here is going to be our labelled portion like we have labels now usually one has labels for image net as well but we artificially restrict ourselves to


simulate a situation where we have lots of data and we only have a fixed budget so we can only because to obtain labels often times you have to ask humans right to label images and let's say we're a company and we've collected this big data set but we only have like maybe 500 bucks on Amazon Mechanical Turk and we only managed to get a very flat aset labeled now we're in the in the regime


of semi-supervised learning ok this is slightly different from what NLP does and as I said in NLP usually assume you have different data sets the large one being the different distribution and in this semi-supervised regime you often assume that it is actually the same data distribution but you only have labels for some of them but there should be a fair bit of overlap between the two


things so oh I've recently made a video about open a eyes image GPT that kind of goes into the into the same direction as this work right here that basically says pre training on unlabeled data like this whole data set without the labels can be a very good pre conditioner for fine-tuning later and this paper says the same thing so basically in in the good old days what you would do is you


would devise a method that somehow takes you know takes in a devise a method that takes in a mini batch and in the mini batch you have your data samples and then some of them would be labeled right here you'd have Y and here it'd have a Y but most of them would be not labeled and you'd have like some sort of loss function that would put special weight on the


ones that are labeled or somehow handle these ones that are unlabeled in a way you might be doing like a some sort of a consistency loss such that if they are very nearest near neighbors to these in the feature space they should have similar labels or things like this so these semi-supervised methods they basically try to solve the problem at once but while taking data that is


labeled and not labeled this paper goes into a different direction this paper says first we should it's actually three stages right here and they have a diagram so I don't need to draw they have a three stage approach three stages the one on the left is unsupervised pre training so they say let's forget about the labels right now even like your unlabeled data so even the data where we


have the labels let's forget about the labels and let's just do unsupervised pre-training now unsupervised pre training in this kind of setting is also known as self supervised pre training and this first stage is done using a contrasted loss and that's very similar to sim clear to this contrastive loss so what you'll do and they describe it very very well here so what you'll do is


given a randomly sampled mini batch of images each image is augmented twice using random crop color distortion and Gaussian blur creating two views of the same example okay so you have an image in your mini batch each image you take and you make two versions of it and each version you crop real random crop somewhere so version one could be random cropped here version cook two could be


random cropped here and then you put some Gaussian blur on it and so on so a little bit of us as you can see random crop color distortion Gaussian blur so what you want is two different versions of the same image each of these versions has been augmented in a different way cropped in a different way blurred in a different way such it's it's two slightly different versions of


the same image and now you want to enforce you want to put this through your network so ultimately as you can see on the right side here what you want to end up is a a network and then okay we'll forget about this right now what you want to Train is this network right here actually including these projection layers we'll get to them later this is the network that you want to Train so


you want to put you take your unlabeled data you take an image you'd make two versions of it and you put those through the network right until the end right here so you'll get Z 1 Z 2 these are the the output of the network for the two images and then what you want to do is you want to take another image that's not this image and also put it through the network maybe also augment it first


and then you have Z 3 so now you have the outputs of 2 things that are supposed to come from the same image and one thing that's supposed to come from a different image and now your loss is simply going to be make those two things close together and push those two things apart or those 3 actually so the loss and this is this is the contrastive loss of self supervised learning as you know


you don't need any labels right here you simply say the things that come from the same image should be close together and the things that come from different images should be far apart and this relies heavily on these data augmentations that you do right here they also employ some other tricks like the momentum encoder from moco from momentum contrast and so on but this is


the main the main part so you can pull a lot of strings here to get like another percent of performance but ultimately they won't see the similarity of Zi and ZJ which are the outputs of the same image to be close together and then this down here they want to be far apart Zi with Z K where K is all the other images okay you can do this in a mini-batch fashion so this is self


supervised learning and the reason why you do this is you don't need labels and it tends we know it tends to give very very good representations so um pass that so what this network here will learn will be very good representation with these self supervised loss with contrastive loss for example gives such good performance there have been papers recently that modify the loss and so on


but it's not super well understood yet but if you do it like this there the network here will give you already very very good representation and we know this because we can take a network like this and then simply train a linear classifier on top of that on a data set and achieve very very good performance and mind you you have trained it with unlabeled data right so


the the network has never been trained to solve like image net classification it has simply been trained to look at the pictures and determine if you know two versions of a picture come from the same picture or from different pictures and now if you simply train a linear classifier on top of these representations you're doing extremely well already so we know these


representations they actually learn something about these images so that's the first part then stage 2 let's cancel all of that stage 2 is you want to do supervised fine tuning now you already see that the arrow here coming out is not this was a task agnostic big CNN the arrow is actually coming out of those those yellow boxes and the yellow boxes are these projection heads so in the


original seem clear paper what they did was they they wanted originally they wanted to train this network right here this is like a ResNet 50 it's pretty standard in these kind of self supervised approaches and so on to train or these few label approaches to train a standardized network and this is like a ResNet 50 so in the original sim clear paper they said we want to make ResNet 50 as strong


as possible but in order to do this loss right here we are going to attach this projection head just to because the dimensionality here I think is like 2048 and we want to do this inner product in a lower dimension of like maybe 256 or so so this these are just multi-layer perceptrons these are just fully connected layers that compress the representation down to that and once


we're done with the unsupervised returning we're going to throw those away right and this ResNet is the thing that we really care about now here they claim okay it actually works better and they have experiments to prove this or to show this if you use one if you actually leave one of these layers here so in the end they I guess they converge on three projection head layers and then


they only throw away the top two and like they make this big deal out of the fact where you know I can just call I can just call this part right here now the encoder and I don't so I don't know exact like I don't see the giant deal here like you've just made your network one layer bigger and now you consider that to be your encoder and the projection head is now two layers and


that will be much easier than calling the projection head three layers but we leave one layer and we train from the middle layers in any case they have this layer additional layer right here compared to the old sim clear and then the representation of that goes into supervised fine-tuning now this is pretty easy this is exactly what it sounds like so now you use only only


the dataset that has labels so the part of the data set that has labels and you do the fine tuning and fine tuning is simply supervised learning you train this network in a supervised fashion on that small fraction of data that has class labels and that already performs pretty well and they show this in experiments but then you can go a step further and do what's known as


distillation or self-training and what's distillation or self-training it's so distillation is when you have a network that you call the teacher Network and that network has been trained to do some classification maybe into three classes pretty pretty well okay but now this is very large and you want maybe a smaller model so you just want like this tiny model because you want to ship it on a


mobile device right but it's also supposed to do this and you know that if you just directly train this which is called the student model it doesn't perform as well as the teacher model there is a better way if you have the teacher model you can sort of transfer the knowledge to the student model you can distill the knowledge and how do you do that you do that by so what would you


do in supervised training in supervised training you would take an image put it in and then put the label that comes along with the image you put it up here and you compare the output to the label and that gives you the loss function right now you do that right here if you distill you put the image into both now the teacher is already trained so its output will be a distribution over


classes it won't be a single label it will be like okay 90% class 1 10% class 2 0 % class 3 something like this and now you take this as like a pseudo label this entire distribution and you put it here and you compare the output of the student to that of the teacher and that's your loss function so this kind of the teacher might have learned to put some nuance into the classification


well I'm pretty sure this is class one but I'm not a hundred percent sure and it can transfer that knowledge to the student and that makes the student better than had you just trained it from the beginning from from with just the labels right so this is distillation and you can do this even what they call self distillation here or self-training so apparently this even helps if the


teacher is if the student model is the same as the teacher model now why does it help in this case and I think it is not exactly the case in this case because they always say their teacher model has this extra projection layer right and then the student model doesn't have that even if they do self-training but why does it help in this case I mean it's it's kind of shocking and I'm


pretty sure it helps in any case but in this particular case it helps because now you're using the unlabeled data again so you have a teacher model and the teacher model is trained first using unsupervised like this is the teacher model right here using unsupervised training then the teacher model is further fine-tuned on the small data right so it is now already pretty good


at the task but how can you get a student model that's even better than the teacher model it's by using again this unlabeled that you have this giant amount of data so what you'll do is you take an image from the unlabeled data and you ask the teacher model teacher model what do you think about that image right and the teacher model will give you a prediction like let's say again


this 90 percent 10% 0% and then you take the student model you input that image and you compare its output to what the teacher said so this combines the teacher model you freeze the teacher model right the teacher model is only trained until here you take it from here the student model is now able to take basically the teacher it takes everything that the teacher model knows


not only about this data but about all the data so it kind of gets to ask the teacher model what do you think about this what do you think about this what do you think about this and it it can incorporate all that knowledge about all of this unlabeled data and that's why the student model here in the end if it's the same size will probably end up even better than the teacher model right


so distillation I think also is still kind of a mystery of why you get a better model or I mean to to make it smaller if you make it a lot smaller usually you don't up end up with a better model but you end up with a pretty good model that you couldn't have gotten by just training the small small model but so that's already pretty cool but why you get a better model with when


they're the same size that's I don't think that's well understood yet so that's the three-stage approach so recap first use all of the data without labels to do unsupervised or self supervised contrastive pre-training second use only the data that has labels to do fine tuning third either distill the learnt classifier to a smaller model or distill it to a model of the same size again


within both cases you would again use the unlabeled all of the unlabeled data okay and that's the three-step approach that's same clear v2 in it's in all of its form alright so they go into fine tuning right here and yeah so they say we elaborate with a three layer projection head so that's the three layer projection head this here is the output of resonate 50 where Sigma is a


Rayleigh non-linearity and we ignore the bias term for gravity power blah blah blah so they contrast this here for fine tuning sim clear uses this right here which is just it's basically just a classifier on top of the ow foot of the ResNet 50 okay yada yada yada yada this is fine tuning from the input layer of the projection head to fine tune from the first layer of the


projection head we have a new encoder function as this which is ResNet followed by fully connected layers and you see they take the resonate 50 output and they ship it through the first projection layer and then there is a task specific classifier now again why I don't even see why they make like this ginormous deal out of it especially especially since the last layer of the


resonate 50 I'm not ok here is I'm not entirely sure but are they taking the low no they're probably not taking the log it's ok but it's yeah I'm it's just weird like is there even a non-linearity at the end right here or is this really just like to matrix multiplications in a row which I'm going to guess there's a big chance that that's the case that the last layer of this encoder is actually


not even followed by non-linearity and therefore you'll just kind of make the dimension different and I don't see why you can't just just incorporate this into the model and have to like say it over and over again that this is a new special thing right again this is equivalent of tuning from a middle layer of the projection head instead of the output layer like ok you just make your


model a bit bigger yeah so the third step is self training or knowledge distillation and they give two variants right here this variant as you can see here this is this is just the cross-entropy but instead of having labels right here Y or you have the T term what the teacher model thinks Y is given X okay that's that's cross-entropy but not with the true labels but with


the output of the teacher model and you can even mix that so you can as you can see right here you can mix this with an actual supervised loss so this would be the supervised loss whatever yeah I guess that I was wrong that wasn't I guess P of Y is always in that case but they don't use this particular kind I think except in one of the ablations so how does this work it


works pretty well and so one of their experiments as you see up here it works pretty well in that if you have 1% of the labels only 1% of imagenet labels which they say is smaller or equal than 13 images per class so there's a thousand classes and you only have 13 labels per class or less if you and they differentiate if your encoder that you train is a resonate 50 then you get and


you can see the dashed line here is a supervised baseline you almost get to the supervised baseline with one percent of the labels and if you actually have a larger ResNet then you get to the supervised performance without without 99 percent of the labels and if you have excuse me ten percent of the labels you pass the supervised baseline so the supervised baseline is on 100% of the


labels mind you and you only have ten percent and this outperforms the supervised baseline now of course you could hear you could have another graphic where you show Oh 100% what if we you know what if we do the whole procedure with 100 percent of the labels so first we don't label the data we do supervised self supervision then we fine-tune on a hundred percent of the


data and then we do this distillation again you would of course be even better and I think they have this somewhere in a table but this is already pretty pretty impressive and another claim they make right here is about the model sizes so and this figure is description this now relates to the title they say bigger models yield larger gains when fine-tuning with


fewer labeled examples so there there are three comparative statement words in one sentence let's unpack this bigger models yield larger gains so the bigger the bigger the model the better the good let's say when fine-tuning with fewer labeled examples let's just look at the graph it's pretty it's really clear so here we have a number of parameters going over so these are the different


models they look at how many parameters they have to do this whole procedure and here is the relative improvement in percent over the top image net one top accuracy so if you do this whole thing with a hundred percent of the labels right I'm gonna guess this here this here is where they start out and you can see as you grow your models you grow the performance and this this is just by


increasing the model size right you have the same data set you have the same amount of labels you have the same number of steps that you train for and so on just by the fact that you make your model bigger you gain in performance okay now you can see that these curves here are above one another and these curves refer to getting small less and less labels okay so if you only


have 10% of the labels your relative gains are a larger this doesn't mean that you perform better with 10% of the labels than with hundred-percent of the labels that would be that would be like ridiculous well I guess in this day and age nothing is ridiculous but for now we're still performing better by having more labels if we do the same procedure right it's not like here so here this


baseline the supervised baseline only does supervised training right so that's why we can outperform it with less of labels but here we do the same procedure this is relative improvement right so this right here the starting point would be if you had 10 percent of labels and a 25 million model parameter model and this right here for example is if you have the same amount


of labels but a 200 million parameter model and this is relative improvement okay but what the graph says is that the relative improvement is larger that the relative improvement is higher the more parameters you have which is the more you go to the right and that effect in itself is higher the fewer labels you have which is the different graphs and you can see that right here so if you


have fewer and fewer labels it becomes more and more important that you have bigger models and that's really counterintuitive right because you would expect that the bigger models they can over fit much more easily to the fewer labels but that doesn't seem the case so this self supervision it really seems to be sort of a counter to this notion of overfitting and if you have larger and


larger models that's what they argue in the paper you might be able to learn more and more features that might be useful for classification so if you have a larger model you might you're gonna learn more kinds of features and then you're going to outperform because you have more chance that these features are going to be useful for classification and I don't think they really make a


statement as to why that happens more with the if you have less labels so let's think about this if I have very few labels very very few labels why does it help me even more if I have a big model well with the same argumentation we could say and maybe they actually say this already so I'm I might be copying them involuntarily maybe with fewer and fewer labels like let's say we have all


the labels that's probably too many right if we can learn a task with some accuracy we probably had too many labels okay it's like weakly like we can't learn a task we know we have too few somewhere there is a border where we have enough but that's like can one number and everything else is too too many technically speaking like learning theoretically speaking so


usually we have too many labels and what does that mean that probably means that there are multiple ways like if we have too many labels there are multiple different features we can pick up tool and there are multiple different paths to learn our goals so if we have imagenet and like that there's this weird task to recognize a three and we get lots and lots and lots of examples


of threes right we can we can decide on a feature we can say oh I all the threes that I see they have this bow down here or all the threes that I see they have this bend here and so on but if I only have very few labels there might only be like a single feature that is even theoretically possible to learn from the labels I'm given and therefore if I have a bigger model in cell in pre-training


because the pre-training happens with the same amount of data right if I have a if I have a bigger model that does the self supervised preaching is going to learn more features and then there's a higher chance that that one feature that I'm that these very few labels that I am able to learn something from is going to be in these features so that's kind of how I make sense of it in combination


what with what they're saying right here okay so this was the main points they do a lot of empirical studies showing the effects of these sizes they've stressed that it's important to have both deep and wide no that works and they also do this additional attention mechanism over the convolution filters I don't want to go into that particularly but they they also do linear evaluation compared to


supervised compared to to fine tuning on with 100% of the labels so they do a very thorough empirical investigation and yeah I I do appreciate that and they kind of show the same things and here they show the number of layers in the projection head so as you increase the number of layers in the projection head and train from the optimal layer in the middle your performance goes up as you


can see but it also this effect is stronger when you have fewer labels right you can see the differences here are greater than the differences here or even here when you have a hundred percent of the labels so the fewer labels the fewer the labels the more benefit you have from the architecture right here and here they show that it's not always optimal to train from the


last projection layer but here the first one so they I guess they converge on three projection layers and you always want to keep the first one around after self supervised training as we mentioned before okay they investigate different different distillation losses and show that it is actually important that you do the distillation loss on labeled and unlabeled sets you can see here if you


only do it if you only train with the labels after fine-tuning you get poor performance if you do the label and distillation loss but only do it on the data set where you have labels then you get more performance if you do label and distillation loss but also include your unlabeled data you get even more performance and then if you do that but you don't do the label loss so before


we've seen you can mix the distillation loss with the label loss if you have lots of labels then you drop in performance again and you can see right here the drop in performance is proportional to how many labeled examples you haven't that's that's natural right if you have the labels you can actually mix that information in with the distillation loss and that make


you better and here they drop point one percent and here they drop less than one percent by leaving away the labeled but their point basically is that it is more important to to distill using also unlabeled data than it is to distill including the label loss and it's much easier to not include the label or so they don't do it I guess alright so I think that was it they they


compared as I said they compare like self distillation where you distill into an equally sized model and down distillation where you distill into a smaller model maybe that's vice-versa and they do a lot of comparison to other methods so this is a very thorough work I feel and yeah if you want more about the exact experiments I invite you to look at the paper and let's just have a final look


at the broader impact statement right here so the broader remember the broader impact statement is supposed to to force you to think about how society might be impacted at large by your work so it says the finding described in this paper can potentially be harnessed to improve accuracy in any application or computer vision where it is more expensive or difficult to label additional data than


to train larger models such applications are clearly beneficial to society for example in medical applications we're acquiring high quality labels requires careful annotation by clinicians better semi-supervised learning approaches can potentially help save lives application of computer vision to agriculture can increase crop yields which may help to improve availability of food however we


also recognize their approach could become a potential component of harmful surveillance systems more over there is an entire industry built around human labeling services and technology that reduces the need for these services could lead to short-term loss of income for some of those currently employed or contracted to provide labels so ask yourself how much of that statement has


to do with the actual novelty this paper and the answer is of course zero right like you can replace like our method in this thing with like machine learning or computer vision in general like oh really sim clear v2 specifically can increase crop yields like that specific invention of this paper will lead to higher crop yields will lead to surveillance systems so I'm yeah you


know I I think like I'm not gonna get too too upset about least I mean this I think it's quite funny but just again I I've I wonder whether the people advocating for these things are happy with these statements because clearly clearly this is just a template that you copy/paste from paper to paper replacing like a few words and if it's computer vision you're like oh my deep fakes and


if it's NLP it's like ah fake news and yeah I I wonder if really anything like particularly is has I wonder whether these people are happy now yeah I just I wonder and if if they are I wonder whether it's really for the reason that they claim that oh now we have a statement here of how it impacts society because like I could have told you that before I could have told you that before


everything we as human beings have created on this planet was essentially first created in our minds all that you see which is human work on this planet first found expression in the mind then it got manifested in the outside world so one thing we need to understand is the wonderful things that we have done on this planet and the horrible things that we have

Law of Attraction simplified by Sadhguru


done on this planet both have come from the human mind so if we are concerned as to what we create in this world it's extremely important that first of all we learn to create the right things in our mind how we keep our minds if we do not have the power to keep our minds the way we want it what we create in the world is also going to be very accidental and haphazard


so learning to create our minds the way we want is the basis of creating the world the way we want there is a wonderful story in the yogic lore on a certain day a man took a walk he went for a long walk accidentally unaware he walked into paradise fortunate isn't he he just took a walk and he landed up in paradise after this long walk he felt a little tired so he thought


oh i'm tired i wish i could rest somewhere he looked around there there was a nice tree underneath which there was very cushiony grass so it was inviting he went and put his head down there and slept after a few hours he woke up well rested and he thought oh i'm well rested but i'm feeling hungry i wish i had something to eat and he thought about all the nice things


that he ever wanted to eat in his life and instantly all those things appeared in front of him you need to understand there are the services like that hungry people don't ask questions food came and he ate stomach became full then he thought oh my stomach is full i wish i had something to drink all the nice things that he ever wanted to drink he thought about it and all of them just


appeared in front of him drinking people also don't ask questions so he drank now with a little bit of alcohol in him you know charles darwin told you all of you were monkeys your tail fell away not me charles darwin told you that you were all monkeys and your tail fell away and then you became human yes definitely the tail fell away but the monkey in yoga we always refer to an


unestablished mind as murkata which means a monkey why we are referring to the mind as a monkey is what are the qualities of a monkey one thing about a monkey is it's unnecessary moment and another thing about the monkey is if i say you're monkeying somebody what does it mean imitation monkey and imitation have become synonymous so these two essential qualities of a


monkey are very much the qualities of an unestablished mind unnecessary moment you don't have to learn it from the monkey you can teach it to the monkey and imitation is full-time job of the mind so when these two qualities are on a mind is referred to as a monkey so this monkey became active within him he just looked around thought what the hell is happening here


i asked for food food came i asked for drink drink came there must be ghosts around here and ghosts came oh the ghosts have come they're going to surround me and torture me he thought immediately the ghost surrounded him and started torturing him then he started screaming in pain and said oh they're going to kill me and he died just now he said he's a fortunate being the problem is he


was sitting under a kalpaviksa or a wishing tree he asked for food food came he asked for drink drink came he asked for ghost ghost came he asked for torture torture came he asked for death death happened now don't go looking for these kalpa rukshas in the forest you can barely find a tree these days a well established mind a mind which is in a state of samyukti


is referred to as a culpa victim if you organize your mind to a certain level of organization it in turn organizes the whole system your body your emotion your energies everything gets organized in the direction once all these four dimensions of you your physical body your mind your emotion and the fundamental life energies are organized in one direction once you are like this anything that you


wish happens without even lifting a little finger actually it would help to assist it with activity but even without doing any activity you can still manifest what you want if you organize these four dimensions in one direction and keep it unwavering in that direction for a certain period of time right now the problem with your mind is every moment it is changing its


direction it is like you want to travel somewhere and every two steps if you keep changing your direction the question of you reaching the destination is very remote unless it happens by chance so organizing our minds and in turn organizing the whole system and these four basic dimensions of who you are right now in one direction if you do this you are a kalpa ruksha yourself


anything that you wish will happen but right now if you look at your lives everything that you have wished for till now if it happens you're finished everything and everybody that you have desired for if all of that lands up in your house today could you live with that so if you want to become empowered it is also important that you become responsible as to what


you ask for and what you don't right now the world situation is just this we are hugely empowered with technology today it doesn't take six six billion people to destroy this planet one man by pressing the wrong button can destroy the whole planet when we are empowered like this it's very important that our physical action emotional action mental action and energy actions are


controlled and properly directed if it is not so we become destructive self-destructive right now that is our problem the technology which is supposed to make our life beautiful and easy has become the source of all the problem that we are destroying the very basis of our life which is the planet so what should have been a boon we are making a curse out of it


what has brought incredible levels of comfort and convenience to us in the last 100 years or so has also become a threat to our life simply because we are not conscious action we are in a compulsive state of action so organizing our minds fundamentally means moving from a compulsive state of activity to a conscious state of activity you might have heard of people for whom


they asked for something and beyond all expectations it came true to true for them generally this happens to people who are in faith now let's say you want to build a house if you start thinking oh i want to build a house to build a house i need 50 lakhs but i have only 50 rupees in my pocket not possible not possible not possible the moment you say not possible


you are also saying i don't want it so on one level you're creating a desire that you want something on another level you are saying i don't want it so in this conflict it may not happen someone who has some faith in a god or in a temple or whatever who is how simple-minded faith works only for those people who are simple-minded thinking people people who are too much


thinking for them it never works a child-like person who has a simple faith in his god or his temple or whatever he goes to the temple and says shiva i want a house i don't know how you must make it for me now in his mind there are no negative thoughts will it happen will it not happen is it possible is it not possible these things are completely removed by the simple act of faith


now he believes shiva will do it for him and it will happen so is shiva going to come and build your house no i want you to understand god will not lift his little finger for you what you refer to as god is a source of creation as a creator he has done a phenomenal job there's no question about it could you think of a better creation than this is it in anybody's imagination to think


anything better than what is there right now so as a creator he has done his job wonderfully well but if you want life to happen the way you want it because right now the very crux of your happiness and your well-being is this if at all if you're unhappy the only and only reason why you're unhappy is life is not happening the way you think it should happen that's all it is so if life is not


happening the way you think it should happen you're unhappy if life happens the way you think it should happen you are happy it's as simple as that so if life has to happen the way you think it should happen first of all how you think with how much focus you think how much stability is there in your thought and how much reverberance is there in the thought process


will determine whether your thought will become a reality or is it just an empty thought or how you do not create any impediments for your thought by creating negative thought process is something possible or not possible is destroying humanity what is possible and not possible is not your business it's nature's business your business is just to strive but what


you want right now you're sitting here if i ask you two simple questions i want you to just look at this and answer this right now from where you're sitting can you just fly off you say no right now from where you're sitting can you get up and walk you'll say yes what is the basis of this why you say no to flying and yes to walking because past experience of life


many times you've gotten up and walked never did you fly off or in other words you're using the past experience of life as a basis for deciding whether something is possible or not possible or in other words you have decided that what has not happened till now cannot happen in your life in future this is a disgrace to humanity and the human spirit what has not happened till


now on this planet can happen tomorrow human beings are capable of making it happen tomorrow so what is possible and what is not possible is not your business that is nature's business nature will decide that you just see what is it that you really want and strive for that and if your thought is created in a powerful way without any negativity without any negative thoughts bringing


down the intensity of the thought process it will definitely manifest the whole existence today modern science is proving is just a reverberation of energy it is a vibration similarly your thought is also a vibration if you generate a powerful thought and let it out it will always manifest itself so generally people are using faith as a means to remove the negative thought


today once you have become thinking human beings your faith is not too deep it doesn't matter how much faith you think you have somewhere doubts always crop up right now the way your minds are made this moment if god appears right here you will not surrender to him you will want an investigation whether he is really god or not with this kind of mind you should not


waste your time on faith so there is an alternative which is commitment if you simply commit yourself to creating what you really care for now once again your thought gets organized in such a way there is no such thing as whether it's possible or not possible there is no hurdle in your thought process your thought flows freely towards what you want once this happens making it happen will


also naturally follow so to create what you really care for first and foremost thing is that what you want must be well manifested in your mind that this is what i want is that what you really want you must look at it because any number of things in your life you have thought this is it the moment you reach there you realize that's not it it's the next one and the


next one and the next one so what is it that one really wants is one thing first of all we must explore once that is clear and we are committed to creating it now there is a continuous process of thought in that direction once you can maintain a steady stream of thought without changing direction definitely this is going to happen in your life or it will definitely manifest


as a reality in your life so either you make this human form into your kalpa riksha or you make it into one big mess which is happening all over one reason why we have not created the kind of world that all of us would want to live in is [Music] too many people are busy looking up too many people are interested in other planets they are not interested in this planet


but every aspect of their life they are checking out with other planets and further too many people are in service of heaven they are not on service of this earth i think right now we need people who are in service of this earth not heaven i believe if heaven is closer to divine than where we are if that is so i believe they are little more organized than us they don't need help from us


this has been a major problem on the planet anything that is valuable for a human being anything that is good the highest aspects of human life have unfortunately been exported to heaven for example if you say love people say god is loving we do not know whether god is loving or not human beings are capable of love it is very very important that people understand


human beings are capable of love human beings are capable of compassion human beings are capable of joy and blissfulness all the good things that are possible for a human being unfortunately have been exported to heaven if we want to create the kind of world that we want to we need to understand whatever you refer to as god the idea of god has entered our mind


only because we have seen creation around us because there is creation we have assumed a creator god is a great creator what you refer to as god is a source of creation that source of creation has not failed us has done a fantastic job but now the question is about management if you want to leave the management in the hands of the creator he will manage it in his own way


according to his agenda but that's not what you want you want life to happen the way you want it now for example let's say all of you here are the india soccer team for the next world cup and i am the coach so these next four years everything that you need to know about football is taught to you everything that i know about football is poured into you in so many ways


now the time to play the match has come you're on the field and the ball comes near your foot but you look at me and it's no good you've seen those coaches sitting there and boiling nothing happens because now once you're on the field it's your job so this is the same thing the creator has done a fantastic job now you're here it is for you and me to see how to


manage this how we want it how to keep this world how in what condition would all of us enjoy best is something that we have to look at so at every stage in our life we tend to think this is it if this one thing happens everything will be fine with my life you reach there and you realize that's not it and you postpone it to something else and something else this is going on


the first and foremost thing is you must be clear what is it that you really want if you do not know what you want the question of creating it doesn't arise if you look at what you really want what every human being wants is he wants to live joyfully he wants to live peacefully in terms of his relationships he wants it to be loving and affectionate or in other words


all that any human being is seeking for is pleasantness within himself pleasantness around him if this pleasantness if it happens in our body we call this health and pleasure if it happens in our mind we call this peace and joy if it happens in our emotion we call this love and compassion if it happens in our energy we call this blissfulness and ecstasy this is all that a human being is


looking for whether he is going to his office to work he wants to make money build a career build a family he sits in the bar sits in the temple he's still looking for the same thing pleasantness within pleasantness around if this is what we want to create i think it's time we addressed it directly and commit ourselves to creating it so you want to create yourself


as a peaceful human being joyful human being loving human being a pleasant human being on all levels and do you also want a world like this a peaceful world a loving world a joyful world no no i want greenery i want food when we say a joyful world that means everything that you want has happened so this is all that you're looking for so all that you need to do is commit


yourself to creating it to create a peaceful joyful and loving world both for yourself and everybody around you every day in the morning if you start your day with this simple thought in your mind that today wherever i go i will create a peaceful loving and joyful world if you fall down 100 times in the day what does it matter for a committed man there is no such


thing as failure if you fall down 100 times 100 lessons to be learned if you commit yourself like this to creating what you really care for now your mind gets organized once your mind gets organized the way you think is the way you feel your emotion will get organized once your thought and emotion is organized your energies will get organized in the same direction


once your thought emotion and energies are organized your very body will get organized once all these four are organized in one direction your ability to create and manifest what you want is phenomenal you are the creator in many ways why i am saying you are the creator is i want you to look at the nature of your life right now if you eat a banana in four hours time this banana becomes a


human being there is something within you a life creating process a process which builds this body the manufacturer of this body is within you give him a banana he makes a human being out of that banana transforming a banana into a human being is not a small thing it is a phenomena it is just that this phenomena is happening within you unconsciously if you could only consciously manifest


this making a banana into a human being you are the creator you are nothing less than that as the theory of evolution goes to make a monkey into a human being it took millions of years over an afternoon you can make a banana into your human being or whatever else a piece of bread that you eat into a human being so the very source of creation is functioning within you


if you organize these four dimensions of mind emotion body and energy in one direction the source of creation is with you you are the creator what you want to create will happen to you effortlessly once you're organized like this now you are not a mess you are a culprit you have the power to create what you want there are tools and technologies as to how to organize the system


in such a way that instead of being a psychological mess you can make yourself into your kalpa ruksha this culture these traditions the whole technology of yoga is just about this transforming yourself from being just a piece of creation to the creator himself this is not in search of god this is in search of becoming a god this is not in search of divine this is


in search of becoming divine because that which you call as divine that which is the source of existence is throbbing within you every moment of your life otherwise a piece of bread cannot become a human being in the course of an afternoon so shifting from being just a piece of flesh and blood to becoming a creator there is a whole science and technology for this


there are tools to make this happen that which is the source of creation is functioning within you every moment of your life it is just that have you kept access to that dimension or not organizing the four basic elements of your life will give you that excess there are tools and technologies to do this the whole signs of yoga the whole technology that we refer to as yoga


is just about this transforming yourself from being just a piece of creation to become a creator for example 100 years ago if i picked up something like this and i start speaking speaking to someone who is across in another part of the world you would think it's some kind of a miracle either i must be a messenger or a son or maybe god himself but today this is just another gadget


that every one of us carry and use today sitting here without using this instrument if i speak to some someone in another part of the world it is still a miracle for you so this instrument happened or because of human mind wanting it to happen 100 years ago nobody thought this was possible but today it is just a common thing similarly many many many things which are not in


our perception yet can be brought into our perception and our ability to create our lives can be greatly enhanced so first and foremost thing is to organize the mind and to organize your emotions body and energy in that line once this happens you're in touch with the fundamental life creating process within you once you're in touch with it once you're in excess of that power you


have the power to create you have the power to create your life and your surroundings the way you want it because we have lost our power to create we are making a mess out of ourselves and the world around us if we operated as the true creator as it is operating within us creating this body for us if we could create our lives with the same sense of organization


[Music] professor we might call this a biography of your curiosity so I want to see some of the origins of your curiosity perhaps in childhood the kinds of books you might have been reading the kind of child you were intellectually um where are we first of all in your childhood let's see I guess I'm about five let's start about five okay and when where is the location I'm on a bus coming home

The Heidelberg Laureate Forum Foundation presents the HLF Portraits: Geoffrey E. Hinton


from school in the countryside okay my English countryside in the English country yeah my mother was a schoolteacher ah she said the headmistress of a little school in the countryside about ten miles outside of Bristol in the UK okay and I went to the set of the school of where she was headmistress Oh which is the nightmare it had some disadvantages occasionally


the kids would surround me in the pay playground and explained to me that I had to persuade my mother to do so it's always the kids that make it in a bit yes but it wasn't too bad but you survived that I survived that so your five on a bus and five on a bus and it the bus has velvet seats where this seats covered with the material that's got a heavy pile and I'm sitting on the


bus and I put a penny and old British pennies were big I put his name on the velvet seat and it moved and it went uphill and this was clearly impossible and I was absolutely fascinated by the fact that you put this penny down it wouldn't always do it but sometimes it would go up hill and it's a bit like those dreams you have I guess I have not everybody has but I


think a lot of people have them when you suddenly discover you can fly and in the dream yes you think this can't be real yes yes but you tried again you can fly and it's amazing and you know it can't quite be real but it's just amazing and it's wonderful that you can fly well this penny moving uphill was the same kind of thing it was sort of weird and wonderful I'm like the first time


you discover magnetism and I couldn't believe what was happening and at the time I just was amazed by it and I was puzzled by it for a long long time and possibly a little experimental you tried it a few times I tried a few times and it didn't work reliably but it did work and I remembered a lot about the circumstances on that bus I don't I remember which side of the bus we were


sitting yes yes I'm wearing the bus we were sitting you for years I just didn't understand how this penny gone uphill and it was a sort of lurking memory that was this weird thing that happened that couldn't possibly have happened wonderful now did you rush to the nearest adult and say I've noticed this can you help me understand why or did you just chemo I guess I was yeah I'm


not that kind of a person I want to figure it out myself okay okay exactly and so I just couldn't figure it out did you make any sense and but it stayed in my mind and and I think might a lot of my research careers like that there's these little niggly things that don't make any sense and they just stay in my mind wonderful and with this when I was a teenager and I guess when I started


physics at school although you wouldn't need to study physics for this I finally understood what must have been handling because the other thing I remember about the bus was it vibrated a lot it had an engine a big engine at the front that was somehow loose that made the whole bus vibrate yes and so clearly what was happening was the flock on the material was at an angle and it was an angle


facing upwards yes so it was slick yes the seat was kind of like this and the flock was like this and so the penny couldn't go down so when it vibrated well when it vibrated this way it will go up and he couldn't vibrate down so that was what was making that making it move up did you actually pursue the problem eventually or it just came to me when it escaped you when I understood


enough about things like vowels I guess I'm going to boringly jump to a more formal part of your education although that was that was the gem of understanding your curiosity but are you getting a good education are you in a well there's the the school your mother is head miss yourself but then the next level are you getting teaching in science are you finding teachers that


interest you yes and no yes so he's I was at my mother's school in the countryside which was mainly children of farmers yes and had a vet they had a very strong Somerset accent yes um there's actually quite a lot of German in the Somerset they use words that sound German to me and I think we must be German in origin and so I had this very strong agricultural accent and I when my mother


got pregnant with my younger sister she had to leave the school so she would have been I'd being about six then yes just turning six heading and I went to a primary school in near where I lived and then my parents wanted me to go to the grammar school which had a it had a sort of genius school I can't on board age but and ice at an exam for the grammar school when I failed it so so then my


parents decided to send me to the a private school our school I never liked it was called Clifton College yeah I'm yes yes I know and they wouldn't let me in I mean they were willing to take me in the junior school but the prep the prep school it was called preparation for the big school but they said I couldn't come until something had been done about my accident your accent my accent I wasn't


allowed into the school because my accent and they said I'd have to stay in the primary school for another year until my accident be fixed may I color than only in England situation maybe I don't know no fair enough all right so you went about improving your accent or your your parents just I I mean at home the accent was fine no but you was the cool accent yeah yeah


and yeah and in the local primary school my accident became more acceptable right and then I went to this prep school and it was a British prep school I didn't like it I was completely out of place and isolated there uh because my father was a Stalinist my mother was a staunch Labour Party supporter yes the Secretary of the local leg up ah ah so you were a classy anyway I'd been taught


that religion was just nonsense yes something I still believe and I mean it's just patently nonsense and this was a sort of I call it muscular Christianity that is it wasn't sort of particularly religious but it was a Christian school and they yes they thought the things that were good for boys were things like playing playing rugger and going to church and so we had


to go to a service every morning and I felt well years later at the service in the morning when I was much older when as a teenager I also had to come in on Sundays because I lived at home most of the kids in the school were boarders who lived in the school but maybe sort of 20% of the school with day boys who lived in Bristol yes and we had to come in on Sundays yeah they were service on


Sunday there was no excusing your absence you couldn't just be at home on Sunday started walking on Sunday service service and they would invite Headmaster's of other schools to come and give sermons and I remember a headmaster for another school giving a sermon about how awful Russia was and how they had forced ideological education in Marxism and people had to


sort of they weren't they couldn't be a fuse they had to sit there and they had to listen to this Marxist nonsense I'm forcibly and I remember sitting there thinking you know I'm sitting here I'm forcibly listening to this religious nonsense and I almost got up and walked out and I somehow wish I had I'd have been thrown out of the school but I wish I'd done that um


few would have had the Kurds to do it I didn't I didn't no one is it's no one noticing your intelligence in this muscular school now um there were some people who some teachers who were very nice to me there was a math teacher called stp Wells and they I have no idea what his first name was Jersey school he was mr. wells um STP wells I remember he was tall and thin and he loved math unlike


many math teachers in junior schools he actually thought math was fun um he had curious ideas about discipline so I remember he did things you would never get away with nowadays so I remember a class where there were two boys talking at the back sitting across the row from each other and they were they were talking softly together well he was trying to explain something


so remember he walked down the row grabbed their two heads by the hair and bang them together that was not accepted practice now even even in such a school it seems he quite muscular to me I remember my my introduction to the school in probably in my first few weeks there there was a teacher who was the art teacher who was fairly sadistic and you won't like to run in the changing


room so you changed for games you hung up your clothes she put on your sports clothes yes you came back you changed again and if you ran in the changing room you explained that was not allowed so well we were changing back into our clothes another boy stole my tie and there was a kind of partition down the middle of the changing room and he and avoid the other


side of the partition would throw the tie from one to the other if I were you I would kind of run around the petitioner try and get it and as I was trying to run around the petition to get my tie this art teacher it's curious I remember that he was Catholic I think this is how prejudice really gets going and I've never forgiven Catholics for him he came into the change room I was


running so he grabbed me he marched me down the corridor to the Headmaster's office and he beat me with a bamboo cane quite severely my trousers were still on but he really he really enjoyed it and I just remember huge sense of grievance but he wouldn't listen to explanations yes I just weren't being enormous ly aggrieved at him and at Catholics but I'm going to


guess and you'll tell me if I'm wrong maybe even against Authority oh yes oh yes I never choose which is the larger our key because of my father but ah so you already had reason I read challenge authority I already had good reason it's gonna show up later in your life that's the only reason i underscore that how are we gonna get this oppressed young man occasionally oppressed by this


school and a status at the end of his time there where he is actually going to be admitted to Cambridge well how am I going to get him either intellectually or morally beyond this this stage of life well before we get there all right please I have one more story I think you would like from that school which is we had to have religious education of course it was kind of Sunday School kind


of religious education about how Jesus was good and yes when God would look after us and I remember several things about it because of course the other kids all believed in God some more than others I had a very intelligent friend there who who believed in God and I remember during religious class the teacher saying all good things come from God and I would have been about eight or


nine then and I remember thinking that there was a problem I don't remember how coherent me I thought it but there's a problem that she was only saying they came from God because they were good and she couldn't use the fact that God you couldn't sort of use this as evidence that God was good because you just assumed they came from God because they were good so it was that's what you


would call a circular argument now I didn't know about circular arguments but I knew there was something deeply suspicious about what she just said and I guess I was in a mood to argue I argued a lot I was very small by the way I was a smallest boy in the school ah the smallest and lightest not an advantage but I made up for it with arguing um so I put up my hand and ice I


tried to explain to her what was wrong with what she just said yes and I think she got frustrated and she said ok Hinton where where do where do you think all good things come from a classic kind of parent move yes um build up build an assumption into the question I leave the kid with the problem of dealing with it um because of my father I um I thought for a moment


and said Russia no this wasn't what was expected in an English private school in the 1950s this would have been kind of the mid-1950s yes and when the Cold War was at its height and that kind of gave you an indication of how much I fitted in with the culture of the school I've already had the indication though can I get you to Cambridge only become at a time you can try I'll try my my


parents tried haha so let's see um when I used to question my parents later on about why on earth that insisted on my staying there even though I was I'm happy yes um they said well the science teaching was good dear and the sons teaching was good was good later on I I enjoyed the science lessons and I didn't enjoy the math much I went in when I was before I got to the senior


school I enjoyed math but then there came a point when I just got confused about math when they introduced functions I didn't understand what functions were I would have been good at arithmetic and good at algebra yeah so neither were these things I didn't understand things like sine X I didn't know what sine was and I was always a very concrete thinker I think in terms of


mechanical analogies yes and I was actually I felt unhappy with functions and not at home with them until I started programming when I was a graduate student yeah as soon as I started programming a function was just a box she gave it one thing and it gave you back something else but until then I had I didn't really I wasn't friends with functions and that made math I got


worse and worse at math um can I generalize and say that if you don't understand its purpose you challenge it or dismiss it maybe I'm reaching to to deepen your childhood for the later right I certainly want to i i'm i i'm not happy with things until i understand them yes and what I mean by understanding them I think it's the same as what Fineman meant you could build


one you know you understand it well enough that you could build one that's that's what I'm I'm wondering it at the X ounds like that was it as later on that's exactly what I felt about psychology in general that you know I want to know what feelings were what sensation was and yes and I felt you never really understand that until you could build one and you as many people


say you couldn't build one but if we make it to your current approach you will not be sentimental about such terms as feelings and emotions but we'll get we'll get there I'm not sure what you mean by not being sentimental I I mean the the hope that they are undefinable not reproducible human as opposed to something that you can analyze you you really I wouldn't call that sentimental


okay I collect just dumb haha fair enough I still haven't gotten you into Cambridge No so you've got a good science together several several good science teacher yes and so he's good at science I was okay at math and because my father had been to a particular College in Cambridge yes he wanted me to go there yes Ferrand ice at the entrance exams and I did quite well in the exam so I got in


did you surprise your school were they assuming you were clever at this point they're not particularly no no they knew I was fairly clever by and then you're as clever but um unruly yes um also might be truth today so going on going on oh you you sit the exams you get in uh-huh I go to Cambridge yes um it's a big shock because at school there were a few kids who were clever and interesting


ideas I had a friend called Inman Harvey who was very clever he got the top math scholarship at Trinity which is the best place in math and Britain yes um but mostly the kids they weren't their primary interest was not ideas weren't they mostly influenced by the kinds of school you went to in fact I mean this is sorry Edie at my at my high school what you call a high


school a North America most of the kids were not any I went to Cambridge everybody was classy I say okay um actually not everybody there was some medics venue but but apart from that yes everybody was clever and that was a bit of a shock it was very nice yes but also quite threatening yes of course so after I've been there a month I left I found it too stressful yes and I went


to London and did various odd jobs [Music] I read a lot of depressing literature like crime and punishment from The Brothers Karamazov remember sitting on the London tube which is a fairly depressing place anyway reading these depressing Russian novels I did various jobs I ended up getting very interested in architecture and I reapplied to Cambridge to do architecture and they


let me back into the same College but before I got there I'd done I've worked in an architect's office over the summer and I discovered what the practice of architecture was like yes I sort of imagined you'd sit there sketching out what wonderful buildings going to be we're going to be like or maybe new ways of constructing buildings actually life in electrics office consists of you know


are we gonna have cheap flooring or cheap door handles because one of them's got to be cheap otherwise we won't meet the budget yes and so after I've done architecture for a day I went to see my tutor and said I want to switch back to doing natural sciences and my teacher said ok which is a bit if I'm right about this one of the advantages of the Oxbridge approach I


mean they they can be flexible they can be very flexible um so my first year I did Natural Sciences and I think I was the only student in that year doing both physics and physiology Oh so I've always been interested in biology uh and I hadn't been allowed to do biology at school because my father wouldn't allow it he said they would teach me genetics and genetics was


nonsense huh what he really meant by that was he was convinced that all sort of macroscopic observable traits would be caused by complex interactions of many genes yes and the sort of standard theory was you know blue eyes is caused by this gene and intelligence is caused by that gene yes and of course that was very much against communist teaching yes and so


for ideological reasons I wasn't allowed to borrow your school even though my father was a biologist he was very good biologist in his way um he got to be a member of the Royal Society yes I actually got to be a member of the world sight at age 49 and I got to be a member of the Royal sight at age 50 which was always irritating Tonica um so I was very competitive with my father anyway


well the Cambridge I did physics and physiology chemistry and physiology was really interesting I'd never done it was it was stuff I didn't know anything about yes and I remember the highlight of the course was going to be the final term where they were going to teach us about the central nervous system yes and I was I was a very very interested in had the brain work because my friend


Inman Harvey at school hood was interested in how the brain worked and we had discussions about it yes and I remember how disappointed I was when they taught us about the central nervous system because the way the central nervous system works is this those neurons and they have axons yes and electrical impulses travel down the axons and cause some chemicals to be


released that get other neurons excited and they taught us a lot about how the electrical impulses travel down the neurons because that was the classic work of hodgkin-huxley and that was how the central nervous system worked and we're being immensely frustrated that they hadn't actually said anything about how it worked I mean they said how the impulses were on propagated but how did


it actually work it was just descriptive basically I mean it was interesting that that's neurons communicate yeah or another but that wasn't what I meant by how it works I wanted to know how how the brain works and how that gives rise to emotions and sensations the song oh why was that if it was such a radical thing to want to know I mean I don't think it was a


radical thing hard to know okay I think lots of you want to know that it's just that in Physiology yes they didn't know okay so after my first year I switched to philosophy because I thought philosophy would teach me more about those the things I was really interested in and that was a big mistake um I remember learning about thick and shiny and getting very depressed because I


couldn't understand it later on I think he was very useful to have learned that stuff but really I think my main the main thing I got from a year of doing philosophy Cambridge was I developed antibodies against philosophy although there was one philosopher there who I got along with extremely well who actually they're two of them there's my tutor someone called Jim Hopkins who was


a very nice guy I'm also interested in how the mind actually worked although he didn't know either and then there was a philosopher called Burnham Williams yes who was a very good philosopher later went to Berkeley I never met him again which was always a regret he used to hold Monday evening a sort of open house where you would go to his room if you're interesting and just discuss philosophy


for an hour - and a number of people would turn up and I got along with him very well he's a great thing yes I know about it he was very eclectic very very fluid in his thinking yes he he wasn't he wasn't dogmatic at all and he was just very interesting to is very interesting to learn from him and he would always take the students ideas seriously and always have something


interesting to say about them he was never dismissive so that was the one for philosopher who I really appreciated then I got fed up with philosophy because it wasn't telling me what I wanted to know and so I switched again to psychology is it fair dude just to finish the philosophy apart is it that they're not asking interesting questions or that they are giving an overly to us


as you're centering questions another philosopher I really like is Daniel Dennett who definitely asked interesting questions okay they just don't have the apparatus to answer them okay my sort of methodological conclusion relates to why it is that philosophers sometimes at a conference will read their paper whereas a scientist very few scientists would do that a scientist will get up and give a


talk yes and make some claims and tell you the evidence maybe philosophers will read their papers and they'll sort of the way they read them is significant so with philosophy it's it's how you say something that's important and the words you say it was that's the content and it's because there is no other content so in in philosophy there's no difference between something sounding


really good and something being good yes there's no empirical test yeah in science something can sign completely loopy like acceleration is gravity yes and yet it can turn out to be true yes and the things that science is established to be true a far crazier than anything any religious connect traumatic could dream up that outside the realm of the crazy things religious fanatics treemap huh


they're always rather boring things like you know there's another world a bit like this one but above the clouds where all the good people go then they're nothing like you know there might be black holes right right so science establishes things that are far crazier than normal people could have a thing called yes yes so this idea that scientists a kind of narrow is just


crazy you know the way you can do it is because it's got something out outside of theorizing that can tell you um where these theories are right or not right and I've heard people I don't understand string theory but I've heard people claiming that um that borderline is being pushed by string theory where it's not quite clear whether it's all just mathematics or whether it really is


being verified by data the whole science has this independent testing philosophy doesn't it's in there sometimes the impossible I may be thinking only at mathematics to speak and search for the elegant solution which always surprises me is terminology because elegance is not necessarily truthful but maybe that is the the core of mathematical elegance it's a very interesting argument about


why why should elegant things be true yes suddenly if you look at physics if you look at particles and things you find kind of fifteen particles which all fit into a nice pattern except there's a missing one yes and the methodology of saying well we should search for that missing one because of all these symmetries it must be there that works and the question is why does and why


does it work so it's not clear that works in biology so when I was a postdoc in San Diego I got to know Francis Crick who who was just become very interested in the brain he was a very impressive thinker but he was of the opinion that the idea that the elegant thing is going to be true did not necessarily apply to biology if philosophy is unsatisfying to the


young undergraduate I would then assume given his future career he would leap to the sciences but in fact he lept at the Social Sciences I don't know allow help me help Meg gain which psychology was a science not a session science and yeah I mean psychology has got these sort of two aspects to it but it was definitely the scientific aspect of Cambridge so we


learned about rats and we learned about section theory so that's you look to the rats well I didn't want to leap to the rats that's what psychology was there right right in fact I actually was very annoyed that it wasn't teaching us anything about people and so I went to see my shooter not my psychology tutor but my sort of general children king's college who was in charge of my welfare


yes and I explained that the psychology course was not telling me anything about real psychology for example there was nothing about psychoanalysis in the psychology course yes and so what I would like to do is I'd like to go to London once a week and get a tutorial from an existential psychoanalyst so I could learn about that because I was really really mad and I would like the


college to pay for it and so my teachers my teacher said well that sounds reasonable so yes the college would pay for that ah that's a real liberal education is oh yeah King's College Cambridge which was loaded not only loaded but I had the capacity to imagine doing that yeah so the audacious undergraduate asked for it the wealthy college agreed yeah and how


important was that actually that so the only thing I really remember about those tutorials and existential psychoanalysis was psychoanalysts had a really beautiful Japanese girlfriend which makes our analyst psychoanalysis seem very good idea but he taught me about his Earl and Heidegger and I don't never really understood any of that didn't understand it couldn't be bothered no I


tried I just didn't didn't really understand yes um I make the assumption that there was something there to be understood but I which now you don't make anymore which I'm not sure I make anymore yeah I'm because we don't have very much time I'm gonna race you through your psychology course but I'll stop at any point that seems critical in your in your intellectual development so


I just felt psychology was totally lacking in any idea of what a proper theory would look like they didn't they didn't sort of have the physics view that our theory should really explain something so he did have the experimental method so it's better than philosophy in that sense but what they were using experiments for was to try and decide between theories that were


sort of hopelessly inadequate and you could just dismiss out of hand without doing experiments because it was just not up to the job yes and so I had to do an experiment and I remember the experiment where you would take you take children between 3 & 5 yes and you try and decide whether they developed during that period in the sense that during that period they started paying more


attention to shape and less attention to things like color and texture so is this model of little kids which is the stimuli like shape and color and texture and they respond to stimuli so this was behaviorist kind of psychology yes I'm just beginning to get a little bit cantar and the experiment is going to be decided to decide if the strength is the response to shape increase in the


strength of a response to color decreased and so you presenting with three objects during training you give them say two triangles and a square all of which are yellow and they learn to pick out the odd one Act which is the square yes and then you give them I better get this right you give them three triangles one of which is red from the other to a yellow and they learnt it


has the red triangle hands down then once they've been trained like that with various different stimulus dimensions you then give them a yellow triangle a red triangle and a yellow square so now they've got a conflict are they going to pick out the old one based on color or are they going to pick out the old one based on shape right and you look to see what they do and the hope is that as as


they get old after the older kids will use shape to pick the old one out and the younger kids will use color I'm not picking up a sense that you were impressed by this experiment well here's what actually happened the experiment was going along and then I got a bright five-year-old yes and the bright five five year old the first time I showed him one of the conflicted ones


yes where there wasn't a clear up one out yes he pointed so will be a yellow triangle the yellow square in a red circle he pointed at the red circle and said you painted that one the wrong color he thought I'd made a mistake right ah because it was the old one out game and clearly I painted that one the wrong color and I thought you know this this this organism that's meant to be


modelled as responding to color or responding to shape yes this organism has just done a piece of reasoning and said you painted that one the wrong color that's just way beyond the scope of any of these theories it's just hugely complicated behavior compared with these seriously or the organism sort of figured out I had intentions and that I made a mistake here it's just utterly out of the realm


I mean if you're trying to sort of model going to the moon by climbing out of a stepladder you know yes that that the the theories they were dealing with stood no more chance of dealing with this kind of behavior than a stepladder would get you to the moon and so that had a big effect on me so you're you're disappointed I'm totally totally disenchanted with psychology because


although it's got an experimental method it's using it in an incredibly naive way to test really dumb theories yeah one man said I have to get you I'm sorry I have to get you out of Cambridge right and into your next step and just the preface for that is of course because of those who might be listening not really quite understanding the state of inquiry in artificial intelligence computers and


so forth how do you then map your next step so then I became a carpenter for a year I quit I could he Mia yes and then I got back into academia by working on a project studying child language development influenced a lot by Chomsky yes um who claimed that claimed on spurious mathematical grounds that almost all language was innate yes in tacky aspects of language were innate


which is complete rubbish and the project was looking at a large section of young children in Bristol to look at their language development empirically by just measuring what happened it happened at the same time with Watergate this project uh-huh and we had little jackets with radio microphones so we put this little jacket on a kid he wear it all day and would be


broadcasting everything that the kid said and inside the house and everything that was said to the kid and inside the house it wasn't regarded as a bad thing back then inside the house we had a recorder that every 20 minutes would take a one-minute sample it would do that by having a cardboard discs that rotated very slowly and a notch in it and a little lever would fall and


connect the tape recorder that's what technology was like then and we would get these little samples of child of children's language and then we would try and analyze them that was the problem but we got very interesting the most interesting utterance we got was when we were looking at tag so a tag is like isn't here aren't they or won't we write and in english tags have a lot of


syntax in them you don't just seen this part or something like that right right you have and so we were looking at tags to see whether children don't express complicated grammatical structures very early on because it's too many phonemes it was because they just don't know these structures and it tanks very good cases very few phonemes but has a lot of grammar in it


yes and we got one kid one parent who said to her child Santa don't give you no toys if you don't talk proper isn't he Wow and I just thought that was a nice example of the kind of data that children got from which they learn to speak good English yes yes but you're not gonna linger you're going to move on so then I started PhD in artificial intelligence because I thought that was


such a program was possible this was 1972 yes and they just set up a couple years earlier a big center of art of intelligence in Edinburgh Edinburgh so the science Research Council had decided to fund one big Centre in artificial intelligence ah and looking back how was it structured this doesn't impress you now in retrospect as to what they they thought they were doing I think it was a


sensible thing to do okay I think was good policy to make one really good Center mmm I think nearly all the people there believed in symbolic err good ol fact what is now called good old-fashioned symbolic AI right I think they were making a huge mistake and the government um a few years later I can't remember the exact time probably in 1974 or something like that got a very eminent


mathematician called Sir James light Hill to do a report on the AI Center yes yes and he produced a very damning report saying that basically saying these guys didn't know what they were talking about there was an interchange with McCarthy who was one of the fathers of AI yes I remember the seeing the interchange years later where McCarthy was saying look anything you can compute you can


compute with symbolic operations so what we doing must be right hon light Hill was saying basically yes but you've no idea how the brain does that which is the only advice we know that can think and you've no idea whether this way of doing it is efficient now retrospect at the time everybody in Britain was outraged that AI was now going to go through hard times because


Sir James like chill didn't believe in it and I think blight chill was entirely correct because both sides were assuming something and both sides were assuming that computation wouldn't get millions of times faster than it was now yes might get thousand time plus we're not millions of times faster yes and under that assumption lecture was completely correct that this symbolic way of doing


things although it theoretically you could do anything that way with the speed of computation we had there was no hope of doing things like perceptional much when I was the report and where are you at the point with undergraduate Street you are a graduate student at the time this pronouncement is made yes and as a result of the pronouncement so it must mean in 74 my advisor who's one of


the three professors who set up the AI yes the school day I remember he leaves head Umbra well he leaves for several reasons personal conflict with one of the other main protagonists was perhaps one of the reasons he left and went to Sussex and I went with him you went with him did you find your fellow graduate students bewildered by this analysis and you you know I didn't or I don't


remember at the time I think I thought light Hill was too severe on a I to ah there have been a lot of paper pressure to believe that yes of course um yeah I don't remember too much with philosophy course but you go you go to Sussex then yes this is before you've completed your degree yes oh yes so I used to have to go back to Edinburgh for one day I've returned to sign a register to say that


I lived in Edinburgh I would get a return ticket from a day return from Brighton to Edinburgh um but some of the in n borough is going to have to approve a dissertation topic course yes who is that who's who's lingering there that there was a more junior guy in a I called Jim Howe who was my official advisor I see but everybody knew it was just an arrangement how did you work out


the topic of the dissertation if that was but so I wanted to work on ural networks and how they learned but I couldn't figure out how they learned from I couldn't buy this I couldn't figure out anything significantly improved over what was called the perceptron convergence theorem which was already known in the early sixties late twenties early 60s neural networks are in the air I mean


ever since touring in a way no not so much when I my advisor had done neural networks until I arrived just before I arrived in Edinburgh he switched his views so he had worked on holographic memories which I was very interested in and about when I arrived as a graduate student he was very impressed by a thesis by Terry Winograd that was using symbolic arrow methods to try and


understand natural language commands like yes put the red block on the green block in the blue box and he was very impressed by that he basically switched his interest from neural Nets just symbolic AI and he'd taken on this great student who was killing it in neural nets and he tried to convince me to switch my interest but I wasn't having any no no you're stubborn I mean I was


very stunned I remembering when I look back on it and having been an adviser of graduates yes yes and having seen the various types of graduate students there are including the extremely stubborn ones I remember him coming into my office when I was a grad student and saying Geoffrey I've had this idea that you might be interested in let me explain it to you so he explained this


idea to me that seemed moderately interesting and at the end of the idea he said so Geoffrey do you think you'd like to work on that and I looked at him in an amazement and said no no I God I didn't my only need to work up um yes so he was very tolerant he was tolerant of having me as a graduate student even though he thought I was doing crazy stuff he's available to say he was


charmed by your audacity or is that No we'd have arguments in the time and I kept agreeing that okay if I hadn't made it work in six months I would switch to doing symbolic AI and then I would keep running on those arguments that's what he's tolerant of you were also unwilling as part of your temperament as we begin to understand it to just go ahead and do the union card thing which is right


whatever it is that the professor says do do it and then get on with your real life you weren't prepared to do that no no I I mean I wasn't gonna work on ideas I didn't believe it yeah you're not cynical yeah I I am quite cynical about a lot of things but like that that's I still want you to to get your degree how do you how does that manage to happen so in the end I managed to do something


that wasn't learning in your all Nets but was inference in your own ads and I made it work and it had a little bit of math that justified it mmm and he was happy with that and so I got a PhD and then I got out of there and I was very disillusioned with it all by then and I took another field oh by the university or the feel academia by academia itself and so again and you've done that before


yes I dropped by frequently yeah and so I took a year off I went to London and taught in a free school that was a very different environment yes free school with a lot of disturbed children yes and then I went back I probably Trebek signal itself then I went back and got a job as a postdoc we're in Sussex in Sussex for a while and finally I saw an investment for a job in San Diego that


seemed like a really nice job in cognitive science where they were gonna recruit six postdocs in various areas who were going to interact with one another and try and understand the mind one of the articles I've read about you not necessarily was accurate but said that the question of monetary support for work was also a factor in your going to america is that not true well because


of the like hill report the were no jobs in AI in vogue were there he killed it basically they killed that there was one job in AI at edinburgh and lots of very good people competing for it so you almost didn't have a choice no I couldn't get it I couldn't get an academic academic job in Britain at all those nations zero shuns I mean I couldn't even get an interview for an


academic job now we're almost at the end and the whole point of this interview is really the origins of your of your thinking so that's not a problem but I I wonder if we can get toward the end a kind of description of the academic environment in the American University and the the sort of strategies that you encountered at that point when you went to San Diego so it's a big contrast that


in Britain the academic establishment in a I was monolithic there was sort of the correct view yes and there wasn't room for multiple camps ah and in America at least you've got two coasts so in linguistics you had a salary of East Coast Chomsky camp and a West Coast film or on lake off camp and it was the same with AI that in San Diego though a group of people particularly David Rama Hart


who came from psychology but was strong mathematically who basically had a completely different view of AI he was interested in how it actually happened in the mind and he thought understanding had happened in the brain would be useful for understanding that Ruth unlike many psychologists and he just had a view that was extremely compatible with what I've been thinking so it's the


first time I've been somewhere because longings had changed his mind just before I arrived yes it's first time I've been somewhere where I was working with someone who really had the same general beliefs about how to go about understanding the mind and what it was like as I did and that was wonderful working life as we're toward the end of this I may I'm gonna