again until we've converged to a local minimum now we can summarize this algorithm which is known as gradient descent because you're taking a gradient and you're descending down down that landscape by starting to initialize our rates wait randomly we compute the gradient DJ with respect to all of our weights then we update our weights in the opposite direction of that gradient and take a small step which we call here ADA of that gradient and this is referred to as the learning rate and we'll talk a little bit more about that


later but ADA is just a scalar number that determines how much of a step you want to take at each iteration how strongly or aggressively do you want to step towards that gradient in code the picture looks very similar so to implement gradient descent is just a few lines of code just like the pseudocode you can initialize your weights randomly in the first line you can compute your loss with respect to those gradients and with respect to those predictions and your data given that gradient you just update your weights in the opposite

MIT 6.S191 (2020): Introduction to Deep Learning


direction of that event of that vector right now the magic line here is actually how do you compute that gradient and that's something I haven't told you and that's something it's not easy at all so the question is given a loss and given all of our weights in our network how do we know which way is good which way is a good place to move given all of this information and I never told you about that but that's a process called back propagation and let's talk about a very simple example of how we can actually derive back propagation using elementary calculus so we'll start with a very


simple network with only one hidden neuron and one output this is probably the simplest neural network that you can create you can't really get smaller than this computing the gradient of our loss with respect to W to here which is that second way between the hidden state and our output can tell us how much a small change in W 2 will impact our loss so that's what the gradient tells us right if we change W 2 in the differential different like a very minor manner how does our loss change does it go up or down how does it change and by how much


really so that's the gradient that we care about the gradient of our loss with respect to W 2 now to evaluate this we can just apply the chain rule in calculus so we can split this up into the gradient of our loss with respect to our output Y multiplied by the gradient of our walk or output Y with respect to W 2 now if we want to repeat this process for a different way in the neural network let's say now W 1 not W 2 now we replace W 1 on both sides we also apply the chain rule but now you're going to notice that the gradient of Y


with respect to W 1 is also not directly computable we have to apply the chain rule again to evaluate this so let's apply the chain rule again we can break that second term up into with respect to now the the state Z ok and using that we can kind of back propagate all of these gradients from the output all the way back to the input that allows our error signal to really propagate from output to input and allows these gradients to be computed in practice now a lot of this is not really important or excuse me it's not as


crucial that you understand the nitty-gritty math here because in a lot of popular deep learning frameworks we have what's called automatic differentiation which does all of this back propagation for you under the hood and you never even see it which is incredible it made training neural networks so much easier you don't have to implement back propagation anymore but it's still important to understand how these work at the foundation which is why we're going through it now ok obviously then you repeat this for every


single way in the network here we showed it for just W 1 and W 2 which is every single way in this network but if you have more you can just repeat it again keep applying the chain rule from output to input to compute this ok and that's the back prop algorithm in theory very simple it's just an application of the chain rule in essence but now let's touch on some of the insights from training and how you can use the back prop algorithm to train these networks in practice optimization of neural networks is incredibly tough in practice


so it's not as simple as the picture I showed you on the colorful one on the previous slide here's an illustration from a paper that came out about two or three years ago now where the authors tried to visualize the landscape of a of a neural network with millions of parameters but they collapsed that down onto just two-dimensional space so that we can visualize it and you can see that the landscape is incredibly complex it's not easy there are many local minima where the gradient descent algorithm could get stuck into and


applying gradient descent in practice in these type of environments which is very standard in neural networks can be a huge challenge now we're called the update equation that we defined previously with gradient descent this is that same equation we're going to update our weights in the direction in the opposite direction of our gradient I didn't talk too much about this parameter ADA I pointed it out this is the learning rate it determines how much of a step we should take in the direction of that gradient and in practice setting this learning rate can


have a huge impact in performance so if you set that learning rate to small that means that you're not really trusting your gradient on each step so if ADA is super tiny that means on each time each step you're only going to move a little bit towards in the opposite direction of your gradient just in little small increments and what can happen then is you can get stuck in these local minima because you're not being as aggressive as you should be to escape them now if you set the learning rate to large you can


actually overshoot completely and diverge which is even more undesirable so setting the learning rate can be very challenging in practice you want to pick a learning rate that's large enough such that you avoid the local minima but small offs such that you still converge in practice now the question that you're all probably asking is how do we set the learning rate then well one option is that you can just try a bunch of learning rates and see what works best another option is to do something a little bit more clever and see if we can


try to have an adaptive learning rate that changes with respect to our lost landscape maybe it changes with respect to how fast the learning is happening or a range of other ideas within the network optimization scheme itself this means that the learning rate is no longer fixed but it can now increase or decrease throughout training so as training progressive your learning rate may speed up you may take more aggressive steps you may take smaller steps as you get closer to the local minima so that you really converge on


that point and there are many options here of how you might want to design this adaptive algorithm and this has been a huge or a widely studied field in optimization theory for machine learning and deep learning and there have been many published papers and implementations within tensor flow on these different types of adaptive learning rate algorithms so SGD is just that vanilla gradient descent that I showed you before that's the first one all of the others are all adaptive learning rates which means that they change their learning rate during training itself so they can increase or


decrease depending on how the optimization is going and during your labs we really encourage you again to try out some of these different optimization schemes see what works what doesn't work a lot of it is problem dependent there are some heuristics that you can you can get but we want you to really gain those heuristics yourselves through the course of the labs it's part of building character okay so let's put this all together from the beginning we can define our model which is defined as this sequential wrapper inside of this


sequential wrapper we have all of our layers all of these layers are composed of perceptrons or single neurons which we saw earlier the second line defines our optimizer which we saw in the previous slide this can be SGD it can also be any of those adaptive learning rates that we saw before now what we want to do is during our training loop it's very it's the same stories again as before nothing's changing here we forward pass all of our inputs through that model we get our predictions using those predictions we can evaluate them and compute our loss our loss tells us how


wrong our network was on that iteration it also tells us how we can compute the gradients and how we can change all of the weights in the network to improve in the future and then the final line there takes those gradients and actually allows our optimizer to update the weights and the trainable variables such that on the next iteration they do a little bit better and over time if you keep looping this will converge and hopefully you should fit your data no now I want to continue to talk about some tips for training these networks in


practice and focus on a very powerful idea of batching your data into mini batches so to do this let's revisit the gradient descent algorithm this gradient is actually very computationally expensive to compute in practice so using the backprop algorithm is a very expensive idea and practice so what we want to do is actually not compute this over all of the data points but actually computed over just a single data point in the data set and most real-life applications it's not actually feasible to compute on your entire data


set at every iteration it's just too much data so instead we pick a single point randomly we compute our gradient with respect to that point and then on the next iteration we pick a different point and we can get a rough estimate of our gradient at each step right so instead of using all of our data now we just pick a single point I we compute our gradient with respect to that single point I and what's a middle ground here so the downside of using a single point is that it's going to be very noisy the downside of using all of the points is


that it's too computationally expensive if there's some middle ground that we can have in between so that middle ground is actually just very simple you instead of taking one point and instead taking all of the points let take a mini batch of points so maybe something on the order of 10 20 30 100 maybe depending on how rough or accurate you want that approximation of your gradient to be and how much you want to trade off speed and computational efficiency now the true gradient is just obtained by averaging the gradient from each of


those B points so B is the size of your batch in this case now since B is normally not that large like I said maybe on the order of tens to a hundreds this is much faster to compute than full gradient descent and much more accurate than stochastic gradient descent because it's using more than one point more than one estimate now this increase in gradient accuracy estimation actually allows us to converge to our target much quicker because it means that our gradients are more accurate in practice it also means that we can increase our


learning rate and trust each update more so if we're very noisy in our gradient estimation we probably want to lower our learning rate a little more so we don't fully step in the wrong direction if we're not totally confident with that gradient if we have a larger batch of gradient of data to they are gradients with we can trust that learning great a little more increase it so that it steps it more aggressively in that direction what this means also is that we can now massively paralyze this computation because we can


split up batches on multiple GPUs or multiple computers even to achieve even more significant speed ups with this training process now the last topic I want to address is that of overfitting and this is also known as the problem of generalization in machine learning and it's actually not unique to just deep learning but it's a fundamental problem of all of machine learning now ideally in machine learning we want a model that will approximate or estimate our data or accurately describes our data let's say like that said differently we want to


build models that can learn representations from our training data that's still generalize to unseen test data now assume that you want to build a line that best describes these points you can see on the on the screen under fitting describes if we if our model does not describe the state of complexity of this problem or if we can't really capture the true complexity of this problem while overfitting on the right starts to memorize certain aspects of our training data and this is also not desirable we want the middle ground


which ideally we end up with a model in the middle that is not too complex to memorize all of our training data but also one that will continue to generalize when it sees new data so to address this problem of regularization in neural network specifically let's talk about a technique of regularization which is another way that we can deal with this and what this is doing is it's trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data


we don't want to learn like very specific pinpoints of the training data that don't generalize well to test data now as we've seen before this is actually crucial for our models to be able to generalize to our test data so this is very important the most popular regularization technique deep learning is this very basic idea of drop out now the idea of drop out is well actually let's start with by revisiting this picture of a neural network that we had introduced previously and drop out during training we randomly set some of these activations of the hidden neurons to


zero with some probability so I'd say our probability is 0.5 we're randomly going to set the activations to 0.5 with probability of 0.5 to some of our hidden neurons to 0 the idea is extremely powerful because it allows the network to lower its capacity it also makes it such that the network can't build these memorization channels through the network where it tries to just remember the data because on every iteration 50% of that data is going to be or 50% of that memorization or memory is going to be wiped out so it's going to be forced to to not only generalize


better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction now we just repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data the second regularization technique that we'll talk about is the notion of early


stopping so what I want to do here is just talk about two lines so during training which is the x-axis here we have two lines the y-axis is our loss curve the first line is our training loss so that's the green line the green line tells us how our training data how well our model is fitting to our training data we expect this to be lower than the second line which is our testing data so usually we expect to be doing better on our training data than our testing data as we train and as this line moves forward into the future both of these lines should kind of decrease go down


because we're optimizing the network we're improving its performance eventually though there becomes a point where the training data starts to diverge from the testing data now what happens is that the training day should always continue to fit or the model should always continue to fit the training data because it's still seeing all of the training data it's not being penalized from that except for maybe if you drop out or other means but the testing data it's not seeing so at some point the network is going to start to


do better on its training data than its testing data and what this means is basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is well we can perform early stopping or we can identify this point this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible and of course if we actually look at on the side of this line if we look at on


the left side that's where a model is under fit so we haven't reached the true capacity of our model yet so we'd want to keep training if we didn't stop yet if we did stop already and on the right side is where we've over fit where we've passed that early stopping point and we need to like basically we've started to memorize some of our training did and that's when we've gone too far I'll conclude this lecture by just summarizing three main points that we've covered so far first we've learned about the fundamentals of neural networks


which is a single neuron or a perceptron we've learned about stacking and composing these perceptrons together to form complex hierarchical representations and how we can mathematically optimize these networks using a technique called back propagation using their loss and finally we address the practical side of training these models such as mini batching regularization and adaptive learning rates as well with that I'll finish up I can take a couple questions and then we'll move on to office lecture on deep sequential modeling I'll take any like maybe a couple questions if


hi there today we're looking at Excel net generalized auto regressive pre-training for language understanding by Jill and yang and other people from Carnegie Mellon University as well as Google brain so this is kind of a the elephant in the room currently as Excel net is the first model to beat Bert which was the previous state of the art and a lot of NLP tasks to be burped at a lot of these same NLP tasks so the chief state of the art result on 18 of of 20 tasks I believe maybe they test one they outperform Bert on 20 the chief state of

XLNet: Generalized Autoregressive Pretraining for Language Understanding


the art on 18 including things asked question answering natural language inference sentiment analysis and so on so those are kind of remarkable results and even more remarkable is that the architecture of the network is actually very very similar to Bert the kind of new introduction is a a pre-training a different free training procedure and we'll look into that so let's actually jump into their main points straight away what they go into is there are two kinds of currently used pre training methods for these NLP test


and both or can be understood as kind of a language modeling one so language modeling for those you don't know is predict the next word in a sequence so if I give you the sequence here unsupervised representation learning has been and then ask you what's next and then you're supposed to say highly right those language modeling in in a nutshell so what they what they differentiate are two kinds of language modeling the first one they say is order aggressive language modeling now what auto regressive language modeling does is exactly what we've


looked at I give you unsupervised learning has been you were supposed to predict highly and then in the next step I give you unsupervised representation learning has been highly and you're supposed to predict success and so on so in the next step I'm gonna give you the entire sentence up until here and you're supposed to do predict in autoregressive because each token can look at the kind of previous ones in the in the sequence so when you sorry you can't see that when you predict when you predict you you can always kind of order


aggressively look at what the previous ones were when including what you've previously predicted of course during training this is a teacher first as I said so you put the actual words there this is auto regressive modeling in contrast to what they call Auto encoding and Auto encoding is what birth does and this is the following so in contrast to that let's say I have the same sequence unsupervised representation learning has been highly successful in the domain of yeah something and then I say okay I give you the sequence but I am going to


delete this and this right and now I ask you to predict these two right so you can see the the task is slightly different as you now have access to all of the sequence basically except the ones that you are asked to predict but you're you kind of asked to predict yet them not in any order but you're asked to predict them at the same time basically so at the same time you're you're asked to predict this word and this word and so the first kind of this Auto regressive language modeling has been used by transformer models until


birth and then basically bert really pushed this auto encoding language model pre-training which made it so successful and now this paper excel net wants to like combine the best of both of them and in order to understand what's the best of both of them so what's good at birth we've already seen it can actually draw information from all of the context of the words it's trying to predict but what is the kind of pitfall of birth and they they actually put this really nicely in an example they give way further down where they say comparison


to but I don't know why that is not like also in the introduction but here they have the sentence New York is a city right New York is a city this one and you're asked to predict these two words and if you now compare birth to what xl9 does if so the context is is a city and you're asked to predict New York what birth does is it simply masks out the two words and says here please fill in these two words now this translates to the kind of objective being separated in the two words such that the prediction of York here is completely independent


of the prediction of new so if you know of any other city that is made of two words for example San Francisco or Los Angeles then these would be as valid and any mixture would be as valid so you might there might end up with laws York is a city and that would be perfectly fine for birth because while it's predicting loss is a perfectly fine prediction for the first word of a two-word City and York is a perfectly fine prediction for the last word of a two-word City right so these are the kind of mistakes that bird can get into


by not being order aggressive by basically predicting all of these tokens at the same time independently of each other whereas x-l net what they will do is we specify an order let's say okay first I will predict the word noon for the first word new something is a city and then when I predict York I will actually take into account the I previously have predicted the word new so um that's the main advantage at that autoregressive training has over Auto encoding now what are the pitfalls the pitfalls or if you have this this


sentence let's look at it I'll write it down New York is a city right if you have this sentence and let's say yeah actually you're not you're not asked to predict you and your crew you're asked to predict the word a here a right you're asked to predict that in order regressive style or a city it's a better example the two words I said in order regressive style if you predict the word a you can only ever look at what comes before hand whereas if Bert were to predict a just the word a it would be able to look at all of it


that's not predict City so you see the kind of auto regressive model is bound to the order of the of the factorization of the sentence that's right it's bound to the order in which it has to predict the tokens so here if it's predicting a you can only look at stuff that comes before it because it needs to do it in order right once it gets to city you can actually look at the entire sentence here but um before that it only ever has partial information about the about the context so actually it wouldn't be much


better if I had said we're trying to predict these two words is and a right and once I predict so so Bert would actually have access to the word City here whereas the auto regressive models only have access to the ones before it I hope that makes it clearer so the main idea in excel net is where did where does this order dependence come in the autoregressive model the order dependence actually comes from the factorization of the sentence of the of the language model so in a language model we're actually trying to assess


the probability distribution of sentences here X is a sentence right and this can be naturally factorized into a product over the words where the probability of each word is only dependent on the words before it so this is a this is an equality is not an approximation this is an equality the probability of a sequence can be decomposed into a product of probabilities like this exactly so this here is exactly what these auto regressive models implement each word is predicted from the words before it right there are other kinds of autoregressive


models that also do the other direction where here they say okay the probability of a sentence is a product and each word is predicted from the words after it but it kind of is the same problem you only ever have access into the one direction basically however you define the order of decoding you only ever have access from a given word to what was before it in the order so the main idea of excel net is they say hey why don't we consider all possible orderings right I mean that that's kind of a that's it's an idea so let's go back to our thing


here they say why don't we consider all possible orderings so basically what we will do is if this sample comes up New York is a city all right what I can do is I can define an ordering let's say I always want to predict two words so the bird typically masks out about 15% of its input to be predicted and here let's say we'll mask out 20% which two words so of this sequence will mask two words and ask the model to predict it that that will be our our pre training objective the first time this sample comes up from the data set


I might specify the order just classically right just one two three four five all right I'll predict the last two words I'll kind of mask them out right I give the model New York is and then I could let it predict a and then in the next step I'll give it New York is a and let it predict City cool so now if the pitfall is the word a here only has access to things before it and not to city itself City has access to everything all right so but then I continue training and the next set time this sample right it's in my data set


New York is the city the next time it comes up I simply go for a different order let's say one two three four five right so now again I'm asked and asking to predict the last two tokens which here our city and York so in the first step I would give it is a and new and I will ask it what's here and I'll ask you to predict city and then in the second step I'll also give it that and I'll ask it okay now what's here given all of that right so new is a city all right you're asked to predict the missing word so that that's pretty


so the first step its new is a hmm and you Resta predicted the second and then the second step is new is the city and the rescue protect you first so now as you can see while predicting City here all of a sudden we didn't no longer in this ordering we don't have access to the world York so we'll have to learn to predict City from the rest of the context now even more even more if we now decide decide on a different ordering again one three four five so now well actually first step is to ask New York City please predict this thing here all right


yeah you might train the model to predict is and then the second step you say New York is City please predict it now we see before before when we are at were asked to predict the word a it only had access to things to the left of it and the very first example but now it actually has access to the entire context so the the the idea is as we sample this data point multiple times and each time we decide on a different ordering duty code for each the prediction of each token token sorry will actually have seen many many


parts many different variants of the context and in expectation will actually have seen all of the context just like Bert but will always having have done it in an order aggressive way so basically you get all the advantages of being order aggressive namely that you are able to decode step by step while always referring to everything in front of you in the ordering so the predictions are not independent but you also get the benefit of Bert that it's able to basically look at all of the rest of the context in expectation in order to make


this prediction so this is this is the main idea of of excel net they formalize this jump up again they formalize it in saying okay what Bert does here is it actually seek it factorized law probability of a sentence into this sum so the product in the law becomes sum into the sum of log probabilities of no sorry this is auto regressive confused ah into the words conditioned on everything in front of them what bird does is it actually approximately factorizes the law of probability into each word and then everything in the


context everything that's not masked in the context and this is only an approximate factorization because you're basically dropping away all these mask tokens and um what they do now is they do the same as the AR as their auto regressive models here they decompose to log probability into a sum of log from abilities over each of the words given all the words before it but now not before it in the sequence but before it in and chosen permutations Z and Z is sampled uniformly from the set of all possible permutations so in expectation


they'll see all of the context so this is the this is the main thing they show this here in a kind of a picture with so here is the neural network this is the input layer then these are the hidden layers as the attention layers go up and up here you're asked to predict the the token so here you're always asked to predict X 3 so there is no there's never going to be any awake here since if you knew X 3 you would be able trivially to predict X 3 all right so in the in the first example the factorization order chosen at random is 3 2 4 1 now you


asked to predict X 3 and we know okay we should only we should only do this with things that are before it in the permutation order well here are since X 3 is the first in the permutation order we actually don't we don't have anything to go on we wait Stickley asked to predict x3 from scratch as if it were the start of a sentence so we'll basically tell the model I have a sentence that goes please predict the third right it's a hard task yeah by the way you're always able to look at this memory thing here don't worry about this for now this is


just this is a an augmentation they do on top of their idea this is not the core idea so okay but now the second time this sample comes up from the training set we decide on a different order so the order here is 2 4 3 1 now again we're asked to predict x3 and we're allowed to look at everything before it so 2 & 4 as you see here there are weights from x2 and x4 into this column that finally is then 8 asked to predict x3 so this is also this is now an easier task right you're allowed to look at the word to the left and to the


right if you have the following permutation order 1 4 2 3 you're actually allowed to look at all of the other words because x3 is at the end of the permutation order in order to produce x-ray so all of these four and the fourth thing is a similar so all of these four things will appear during training and you will learn from them so in expectation C you will basically have seen all variants of different of different versions of the context which which helps a lot apparently right so in the in order to achieve this they had to


make some architectural changes to the the model namely what you want to do is in a single pass through the model here you not only want to predict one token but you want to do many predictions this helps training a lot so vert and naturally always does like the 15 we must add 15% of the tokens or so what was that like 40 50 tokens so it masks them and it predicts them all at the same time now you would like to do this here as well you would like to predict all at the same time the ones that you're asked to predict but of course the problem is


for here if you're asked if in this factorization order 2 4 3 1 if you're asked to predict X 3 you're allowed to look at X 2 and X 4 if you're asked to predict X 1 you're allowed to look at X 2 X 4 and X 3 so if you only have a single pass through the model the question is do you now input X 3 or do you not because the prediction of X 3 is not allowed to look at X 3 while the prediction of X 1 is allowed to look at X 3 so they do an architectural change in order to achieve both things so that you can do have a


single pass through the walk through the model but the prediction of each token only depends on the things in front of it in the permutation order and they do this by having these kind of two stream is masked to stream attention where they basically have not only not one hidden representation like in classic transformers but they have at each step two hidden representations one they call H only called G so the HS are initialized with the embeddings of the tokens and the g's are just initialized randomly and then they get transformed


and the point is the h of the next layer is always able to look at everything in front of it including its own its own H basically it's the one layer down its own position one layer down while the G is only allowed to look at the a it's allowed to look at the ages but the H is from before right so so all the G's here are only ever able to look at the H is from before the current position whereas the H is always allowed here to look at the same but also at the H at the current position and now at the last layer you simply ask


the model to predict the token from just the G and you can easily see that this results in this model only oh yeah only attending to things before it okay the G by the way can also look at the G of the current layer so that's that's also nothing but it cannot look at that at the age so there's never any information flowing from the current from the current word embedding of the token you're trying to predict to the prediction layer so basically that that means the model can't just look like you you're not telling the model the answer


yet you're still able to feed to predict multiple things in a single pass through the model formally this is described here in the attention layer so they divide how they produce the queries and how they produce the keys and values usually the queries and the keys and values are produced from the same hidden representation but here they produce the keys and values from the h's in both cases but to update the G's they produce the queries from the last layers G and do produce HS they produce the queries from the last layer HS and most


importantly when they produce the keys and values the H is they'll look at here to update the G you're only allowed to look at H is before you in the permutation order but to update the H you're allowed to look at everything before including the position you're currently after so that's kind of the that's a it's an engineering solution to the problem introduced by their augmentation I think it's a pretty pretty neat solution pretty cool so the rest of the paper here is incorporating ideas from transformer Excel so transformer Excel is one of


these classic transformers that that is like this AR so this Auto regressive style of transformer but that has a few improvements over the classic vanilla transformer and they incorporate a number of things here namely first of all they incorporate this memory thing so the memory thing allows you to input longer sequences let's say our our transformer input length is maximum of five tokens what the transformer Excel allows you to do is you input five tokens and then you save you do your transformer thing you encode it and they


save something into this memory and then when you input the next five tokens your transformer is then allowed to look at the memory of the last sequence right and also update it so that that that's kind of these this memo oxy Sawyer so you're always allowed to look at these men blocks from last sequence and then the hidden representations here of this sequence they will actually be stored in the member lok for the next sequence this is kind of a trick to carry over information it's not the deep updating the memory part isn't learned with the


objective to make the next prediction better but it's just some information it's a kind of gradient free information to provide to the next step and it apparently helps you can incorporate longer sequences into this transformer excel so they take this over and implement this into excel net they also do relative position encodings relative segment and codings I won't go into this too much more here because it's not the main idea basically so they do experiments and they compared to birth architecture with the


same basically same architecture the same number of parameters and/or the years and they beat Burt in all of these kind of NLP tasks or most of I think they said in 20 they reach new state of the art in 18 NLP tasks so apparently their method works very well so what they do here is a last thing I find important is an ablation study of the effects of their improvements so they wear because kind of my problem is I never know like they have this new idea okay we do these random permutations but then they also say oh and also we


include memory from XL net and we do relative positioning coatings and so on to me these kind of papers of course you reach better numbers you get a new state of the art so it's kind of a landmark paper but to me a paper should more be like a single thing so whatever your idea is this your idea is these or drinks and whatever you need to do to make that work okay fine but then why why the additional transformer Excel things it's it's really then hard to estimate how much of the improvement comes from your ID and how much of the


improvement simply comes from the fact that you already put these other things actually have nothing to do with it so I appreciate these kind of analyses called ablation studies where they kind of try to take away the memory and these things and kind of look at what it's doing to the model and use you see here kind of degrades down here as for example this con degrades as you take stuff away while still being more kind of more successful than Burt so that that I would say also yeah here is more unclear but also kind of seems to degrade a bit


and while being more successful than Bert I appreciate this there's some and of really trying to show that your gains really come from your new idea and not from some other stuff all right so the last thing I want to mention actually is this thing so someone claiming or calculating that it costs two hundred and forty five thousand dollars to train the Excel net model the way they describe it in the paper I'm sure that's gonna be brought down because it was brought down that like the time the train was brought down with Bert as well


but this is just I mean this is crazy this is just training it um it it kind of gives large questions about the state of research and the ability for kind of let's say more academic players to participate in research on the one hand of course we like of course these companies should be able to do this and on the other hand if it seems like currently in some fields just putting more money on the table will get you a better result not this this actually like this paper is actually a cool idea but it's still kind of primitively


[Music] [Music] because the shift from one zodiacal house to another takes more than two millennia scholars wondered how and where Hipparchus could have learned of the precession in the second century BC it is now clear that his source was Sumerian professor Langdon's findings revealed that the Nepean calendar established circa 4400 BC in the Age of Taurus reflects knowledge of the precession and the shift of zodiacal houses that took place 2160 years earlier than that professor Jeremiah's who correlated Mesopotamian astronomical texts with


Hittite astronomical texts was also of the opinion that older astronomical tablets recorded the change from Taurus to Aries and he concluded that the Mesopotamian astronomers predicted and anticipated the shift from Aries to Pisces subscribing to these conclusions professor Willie Hart nur suggested that the Sumerians left behind plentiful pictorial evidence to that effect when the spring equinox was in the zodiac of Taurus the summer solstice occurred in the zodiac of Leo partner drew attention to the recurrent motif of a bull lion


combat appearing in Sumerian depictions from earliest times and suggested that these motifs represented the key positions of the constellations of Taurus bull and Leah lion to an observer at 30 degrees north such as at a circa 4000 BC most scholars consider the Sumerian stress of Taurus as their first constellation as evidence not only of the antiquity of the zodiac dating to circa 4000 BC but also as testifying to the time when Sumerian civilization so suddenly began professor Jeremiah's found evidence showing that the sumerian


zodiacal chronological point 0 stood precisely between the bull and the twins from this and other data he concluded that the zodiac was devised in the age of Gemini the twins that is even before Sumerian civilization began a Sumerian tablet in the Berlin Museum begins the list of zodiacal constellations with that of Leo taking us back to circa 11,000 BC when man had just begun to till the land professor HV Hill Precht went even farther studying thousands of tablets bearing mathematical tabulations he concluded that all the multiplication


and division tables from the temple libraries of nipper and sipar and from the library of Ashurbanipal in Nineveh are based upon the number 12 million 960,000 analyzing this number and its significance he concluded that it could be related only to the phenomenon of the precession and that the Sumerians knew of the great year of twenty five thousand nine hundred and twenty years this is indeed fantastic astronomical sophistication at an impossible time just as it is evident that the Sumerian astronomers possessed knowledge that


they could not possibly have acquired on their own so is there evidence to show that a good deal of their knowledge was of no practical use to them this pertains not only to the very sophisticated astronomical methods that were used who in ancient Sumer really needed to establish a celestial equator for example but also to a variety of elaborate texts that dealt with the measurement of distances between stars one of these texts known as a oh six four seven eight lists the 26 major stars visible along the line we now call


the Tropic of Cancer and gives distances between them as measured in three different ways the text first gives the distances between these stars by a unit called Mohnish oculto measured and weighed it is believed that this was an ingenious device that related the weight of escaping water to the passage of time it made possible the determination of distances between two stars in terms of time the second column of distances was in terms of degrees of the arc of the skies the full-day daylight and night time was


divided into twelve double hours the arc of the heavens comprised a full circle of 360 degrees hence one Barrow or double hour represented 30 degrees of the arc of the heavens by this method passage of time on earth provided a measure of the distances in degrees between the named celestial bodies the third method of measurement was Barrow Enosh ami length in the skies after o dangun pointed out that while the first two methods were relative to other phenomena this third method provided absolute measurements a celestial Barrow


he and others believed was equivalent to ten thousand six hundred ninety-two of our present-day meters eleven thousand six hundred ninety three yards the distance in the skies between the twenty six stars was calculated in the text as adding up to six hundred and fifty-five thousand two hundred Barrow drawn in the skies the availability of three different methods of measuring distances between stars conveys the great importance attached to the matter yet who among the men and women of Sumer needed such knowledge and who among them


could devise the methods and accurately use them the only possible answer is the Nephilim had the knowledge and the need for such accurate measurements capable of space travel arriving on earth from another planet roaming Earth's skies they were the only ones who could and did possess at the dawn of mankind's civilization the astronomical knowledge that required millenia to develop the sophisticated methods and mathematics and concepts for an advanced astronomy and the need to teach human scribes to copy and record


meticulously table upon table of distances in the heavens order of stars and groups of stars heliacal risings and setting a complex Sun Moon Earth calendar and the rest of the remarkable knowledge of both heaven and earth against this background can it still be assumed that the Mesopotamian astronomers guided by the Nephilim were not aware of the planets beyond Saturn that they did not know of Uranus Neptune and Pluto was their knowledge of Earth's own family the solar system less complete than that of distant stars their order and their


distances astronomical information from ancient times contained in hundreds of detailed texts lists celestial bodies neatly arranged by their celestial order or by the gods or the months or the lands or the constellations with which they were associated one such text analyzed by Ernst F Widener has come to be called the great star list it listed in five columns tens of celestial bodies as related to one another to month's countries and deities another text listed correctly the main stars and the zodiacal constellations a text indexed


as B M 8 6 3 7 8 arranged in its unbroken part 71 celestial bodies by their location in the heavens and so on and gone and on in efforts to make sense of this Legion of texts and in particular to identify correctly the planets of our solar system a succession of scholars came up with confusing results as we now know their efforts were doomed to failure because they incorrectly assumed that the Sumerians and their successors were unaware that the solar system was heliocentric that earth was but another planet and that


there were more planets beyond Saturn ignoring the possibility that some names in the star lessons may have applied to earth itself and seeking to apply the great number of other and epithets only to the five planets they believed were known to the Sumerians scholars reached conflicting conclusions some scholars even suggested that the confusion was not theirs but a Chaldean mix-up for some unknown reason they said the Chaldeans had switched around the names of the five known planets the Sumerians referred to all


celestial bodies planets stars or constellations as mule who shine in the heights the Akkadian term kkuk AB was likewise applied by the Babylonians and Assyrians as a general term for any celestial body this practice further frustrated the scholars seeking to unravel the ancient astronomical texts but some mules that were termed Lu bad clearly designated planets of our solar system knowing that the Greek name for the planets was Wanderers the scholars have read Lu bad as wandering sheep deriving from Lu those which are shepherded and bad high


and far but now that we have shown that the Sumerians were fully aware of the true nature of the solar system the other meanings of the term bad the olden the foundation the one where death is assume direct significance these are appropriate epithets for the Sun and it follows that by Lu bad the Sumerians meant not mere wandering sheep but sheep shepherded by the Sun the planets of our Sun the location and relation of the Lu bad to each other and to the Sun were described in many Mesopotamian astronomical texts there were references


to those planets that are above and those that are below and Kugler correctly guessed that the reference point was earth itself but mostly the planets were spoken of in the framework of astronomical texts dealing with mule mule a term that kept the scholars guessing in the absence of a better solution most scholars have agreed that the term mule mule stood for the Pleiades a cluster of stars in the zodiacal constellation of Taurus and the one through which the axis of the spring equinox passed as viewed from Babylon


circa 2200 BC Mesopotamian texts often indicated that the mule mule included seven loom ash seven Wanderers that are familiar and the scholars assumed that these were the brightest members of the Pleiades which can be seen with the naked eye the fact that depending on classification the group has either six or nine such bright stars and not seven posed a problem but it was brushed aside for lack of any better ideas as to the meaning of mule mule Fromm's Kugler reluctantly accepted the Pleiades as the solution but expressed his astonishment


when he found it stated unambiguously in Mesopotamian texts that mule mule included not only Wanderers planets but also the Sun and the moon making it impossible to retain the Pleiades idea he also came upon texts that clearly stated that mule mule rules shul 12 mil mule is a band of 12 of which 10 formed a distinct group we suggest that the term mule mule referred to the solar system using the repetitive mule mule to indicate the group as a whole as the celestial body comprising all celestial bodies Charles viral au transliterated a


Mesopotamian text that describes the members of the mule mule or cacao boo or Kaka boo group the texts last line is explicit kkuk Abu Kakha boo the number of its celestial bodies is 12 the station of its celestial bodies 12 the complete months of the Moon is 12 the texts leave no doubt the mule mule our solar system was made up of twelve members perhaps this should not come as a surprise for the Greek scholar dye odorous explaining the three ways of the Chaldeans and the subsequent listing of 36 celestial bodies stated that of those


celestial gods 12 hold chief authority to each of these the Chaldeans assign a month and a sign of the zodiac ernst Weidner reported that in addition to the way of on you and it's 12 zodiac constellations some texts also referred to the way of the Sun which was also made up of 12 celestial bodies the Sun the moon and 10 others line 20 of the so-called T tablet stated 'no far 12 sheer mesh Halle Chaka Cobb Lucia sinew shamash inna Libby yaku which means all in all 12 members where the Moon and Sun belong where the planets orbit we can


now grasp the significance of the number 12 in the ancient world the great circle of Sumerian gods and of all Olympian gods thereafter comprised exactly twelve younger gods could join this circle only a folder God's retired likewise the vacancy had to be filled to retain the divine number twelve the principal celestial circle the way of the Sun with its twelve members set the pattern according to which each other celestial band was divided into twelve segments or was allocated twelve principal celestial bodies accordingly there were twelve


months in a year twelve double hours in a day each division of Sumer was assigned 12 celestial bodies as a measure of good luck many studies such as the one by s Langdon show that the division of the year and 12 months was from it's very beginnings related to the twelve great gods fritz Hummel and others after him have shown that the 12 months were closely connected with the 12 zodiacs and that both derived from 12 principal celestial bodies Charles F Jean reproduced a Sumerian list 24 celestial bodies that paired 12


zodiacal constellations with 12 members of our solar system in a long text identified by F Thoreau dangun as a temple program for the New Year Festival in Babylon the evidence for the consecration of 12 as the central celestial phenomenon is persuasive the Great Temple the s Aguila had 12 gates the powers of all the celestial gods were vested in Marduk by reciting 12 times the pronouncement my lord is he not my lord the mercy of the God was then invoked 12 times and that of his spouse 12 times the total of 24 was then


matched with the 12 zodiac constellations and twelve members of the solar system the boundary stone carved with the symbols of the celestial bodies by a king of Susa depicts these twenty-four signs the familiar 12 signs of the zodiac and symbols that stand for the twelve members of the solar system these were the twelve astral gods of Mesopotamia as well as of the hurryin hit IDE Greek and all other ancient Pantheon's although our natural counting base is the number 10 the number 12 permeated all matters celestial and


divine long after the Sumerians were gone there were 12 Greek titans 12 tribes of Israel 12 parts to the magical breastplate of the Israelite high priests the power of this celestial 12 carried over to the Twelve Apostles of Jesus and even in our decimal system we count from 1 to 12 and only after 12 do we return to 10 and 3 13 10 and 4 and so on where did this powerful decisive number 12 stem from from the heavens for the solar system the mule mule included in addition to all the planets known to us also the


planet of on you the one whose symbol a radiant celestial body stood in the Sumerian writing for the god on you and for divine the cab of the supreme scepter is one of the sheep in mule mule explained an astronomical text and when Marduk usurped the supremacy and replaced on you as the God associated with this planet the Babylonian said the planet of Marduk within mule mule appears teaching humanity the true nature of earth and of the heavens the Nephilim informed the ancient astronomer priests not only of the planets beyond


Saturn but also of the existence of the most important planet the one from which they came the 12th planet on most of the ancient cylinder seals that have been found symbols that stand for certain celestial bodies members of our solar system appear above the figures of gods or humans an Akkadian seal from the 3rd millennium BC now at the Vorta Asiata shop Tai Lung of the State Museum in East Berlin departs from the usual manner of depicting the celestial bodies it does not show them individually but rather as a group of 11 Globes


encircling a large raid star it is clearly a depiction of the solar system as it was known to the Sumerians a system consisting of twelve celestial bodies the ancient depiction shows a planet unknown to us considerably larger than Earth yet smaller than Jupiter and Saturn which clearly follow it farther on another pair perfectly matches our Uranus and Neptune finally the smallish Pluto is also there but not where we now place it after Neptune instead it appears between Saturn and Uranus treating the moon as a


proper celestial body the Sumerian depiction fully accounts for all of our known planets places them in the correct order with the exception of Pluto and shows them by size the forty five hundred year old depiction however also insists that there was or has been another major planet between Mars and Jupiter it is as we shall show the 12th planet the planet of the Nephilim if this sumerian celestial map had been discovered and studied two centuries ago astronomers would have deemed the Sumerians totally uninformed foolishly


imagining more planets beyond Saturn now however we know that Uranus and Neptune and Pluto are really there did the Sumerians imagine the other discrepancies or were they properly informed by the Nephilim that the moon was a member of the solar system in its own right Pluto was located near Saturn and there was a twelfth planet between Mars and Jupiter the long-held theory that the moon was nothing more than a frozen golf ball was not discarded until the successful conclusion of several US Apollo moon missions the best guesses


were that the moon was a chunk of matter that had separated from earth when earth was still molten and plastic were it not for the impact of millions of meteorites which left craters on the face of the moon it would have been a faceless lifeless history less piece of matter that solidified and forever follows earth observations made by unmanned satellites however began to bring such long-held beliefs into question it was determined that the chemical and mineral makeup of the moon was sufficiently different from that of Earth to


challenge the brakeaway theory the experiments conducted on the moon by the American astronauts and the study and analysis of the soil and rock samples they brought back have established beyond doubt that the moon though presently barren was once a living planet like Earth it is layered which means that it solidified from its own original Moulton's staged like Earth it generated heat but whereas Earth's heat comes from its radioactive materials cooked inside earth under tremendous pressure the moon's heat comes apparently from


layers of radioactive materials lying very near the surface these materials however are too heavy to have floated up what then deposited them near the moon's surface the moon's gravity field appears to be erratic as though huge chunks of heavy matter such as iron had not evenly sunk to its core but were scattered about by what process or force we might ask there is evidence that the ancient rocks of the moon were magnetized but there is also evidence that the magnetic fields were changed or reversed was it by some


unknown internal process or by an undetermined outside influence the Apollo 16 astronauts found on the moon rocks called breccias that result from the scattering of solid rock and its reweld extreme and sudden Heat when and how were these rocks shattered then refused other surface materials on the moon are rich and rare radioactive potassium and phosphorus materials that on earth are deep down inside putting such findings together scientists are now certain that the moon and earth formed of roughly the same elements at


about the same time evolved as separate celestial bodies in the opinion of the scientists of the US National Aeronautics and Space Administration NASA the Moon evolved normally for its first 500 million years then they said the most cataclysmic period came four billion years ago when celestial bodies the size of large cities and small countries came crashing into the moon and formed its huge basins and towering mountains the huge amounts of radioactive materials left by the collisions began heating rock beneath the surface melting massive


amounts of it and forcing seas of lava through cracks in the surface Apollo 15 found a rockslide in the crater collapse key six times greater than any rockslide on earth Apollo 16 discovered that the collision that created the sea of nectar deposited debris as much as 1,000 miles away Apollo 17 landed near a scarp eight times higher than any on earth meaning it was formed by a moonquake eight times more violent than any earthquake in history the convulsions following that cosmic event continued for some 800 million years so that the


moon's makeup and surface finally took on their frozen shape some 3.2 billion years ago the Sumerians then were right to depict the moon as a celestial body in its own right and as we shall soon see they also left us a text that explains and describes the cosmic catastrophe to which the NASA experts refer the planet Pluto has been called the enigma while the orbits around the Sun of the other planets deviate only somewhat from a perfect circle the deviation eccentricity of Pluto is such that it has the most extended and


elliptical orbit around the Sun while the other planets orbit the Sun more or less within the same plane Pluto is out of kilter by a whopping 17 degrees because of these two unusual features of its orbit Pluto is the only planet that cuts across the orbit of another planet Neptune in size pluto is indeed in the satellite class its diameter 3600 miles is not much greater than that of Triton a satellite of Neptune or Titan one of the 10 satellites of Saturn because of its unusual characteristics it has been suggested that this misfit might have


started its celestial life as a satellite that somehow escaped it's master and went into orbit around the Sun on its own this as we shall soon see is indeed what happened according to the Sumerian texts and now we reach the climax of our search for answers to primeval celestial events the existence of the 12th planet astonishing as it may sound our astronomers have been looking for evidence that indeed such a planet once existed between Mars and Jupiter toward the end of the 18th century even before Neptune had been discovered


several astronomers demonstrated that the planets were placed at certain distances from the Sun according to some definite law the suggestion which came to be known as bodes law convinced astronomers that a planet ought to revolve in a place where hitherto no planet had been known to exist that is between the orbits of Mars and Jupiter spurred by these mathematical calculations astronomers began to scan the skies in the indicated zone for the missing planet on the first day of the 19th century the Italian astronomer


Giuseppe Piazzi discovered at the exact indicated distance a very small planet 485 miles across which he named series by 1804 the number of asteroids small planets found there rose to four today nearly three thousand asteroids have been counted orbiting the Sun in what is now called the asteroid belt beyond any doubt this is the debris of a planet that had shattered to pieces Russian astronomers have named it Phaeton chariot while astronomers are certain that such a planet existed they are unable to explain its disappearance


did the planet self explode but then it's pieces would have flown off in all directions and not stayed in a single belt if a collision shattered the missing planet where is the celestial body responsible for the collision did it also shatter but the debris circling the Sun when added up is insufficient to account for even one whole planet to say nothing of to also if the asteroids comprised the debris of two planets they should have retained the axial revolution of two planets but all the asteroids have a single axial rotation indicating they


[Music] this video will present a recent paper from Google AI revisiting self supervised visual representation learning so the headline idea this paper is that the standard architecture designs and convolutional neural network advances that have been working in supervised learning tasks like image net classification don't necessarily translate to these self supervised tasks such as predicting the rotation the permutation of jigsaw puzzles or the exemplar augmentation task so this is the ideas are the neural network

Self-Supervised Learning


architecture designs over fitted to sell to supervised learning tasks such as recognition and detection and should maybe these neural architecture searches be deployed into self supervised learning or into like a pipeline of self supervised learning and then taking the representations into the classification models but then using the hyper parameter neural architecture search and all these heuristic tricks to design this jointly on self supervised tasks forth for the downstream representation learning so self supervised learning is

Self-Supervised Learning


inspired by NLP success and it's just recently it's been tested since like the dawn of alex net but it is definitely gaining traction and so an NLP this is about predicting words from their context so when the sentence predict words from their context words and form would be labelled as positive words and predict one and then tiger an ocean don't appear in the context so they've been labeled as zero or negative so the sole supervised learning has these pretext tasks and so there's other techniques other than context like NLP

Self-Supervised Learning


there's more ideas in visual representation learning and images and computer vision so here the some of the common self supervised visual learning tasks rotation prediction example are classes relative patch location and jigsaw puzzle permutations and many of the studies that have already come out on this use the Aleks net CNN architecture this paper from Google AI is going to use the state-of-the-art ResNet designs like the wide ResNet and then a reversible resume which is a more efficient implementation so rotation

Self-Supervised Learning


task is like this you would take images and then you rotate them either zero degrees 90 degrees 100 to 70 and then the network is basically doing a for class classification task for these different rotations the exemplars used data augmentations to take an image and then change it in like a ton of different ways and then this image corresponds to its own class so it would be like a massive way like maybe like a thousand class and on classification problem so jigsaw puzzle this is an interesting one it's where

Self-Supervised Learning


you take like a patch of the image and then you like you crop the image and then you further refine the crops and then scrambled away in this thing and then try to have you pass each of the crops through the network so each of these each of these square tiles goes into a convolutional Network and then it predicts the where it thinks it might lie in the permutation so I personally don't I'm not really a huge fan of this task I don't really see how it makes a whole lot of sense because how does it know the context really and then

Self-Supervised Learning


relative patch location is the same idea I like this idea more where you just have two patches and you predict how they might relate to one another and then another really interesting study that I'm gonna be making a video on tomorrow so please subscribe if you're interested in this video is a multitask table supervised visual learning where you combine these self supervised learning tasks together and then see what kind of representations are derived from that and they test this and their paper using the ResNet 101 model so

Self-Supervised Learning


representations for image classification what you do is you freeze most of the network and then you take the pre largest layer like the like an intermediate vector representation and then you input that to a logistic regression model and use that as a classifier train with SGD and data augmentation so the vector that they extract feature vector from the sub supervised learning task they vary this from size 2048 4096 6000 144 and 8192 so this is the size of the representation of vector extract from the self supervised learning task

Self-Supervised Learning


so another thing they find is you might ask like how about instead of a logistic regression we do a more complex multi-layer perceptron model so this plot just shows this is the logistic regression in the MLP and they basically perform the same so the logistic regression is has sufficient capacity for this and then another interesting thing is we're in the network do you get the features from so this one thing that's interesting is in the VG g19 network they take the intermediate features from the third block rather than the very end and get

Self-Supervised Learning


better results but with the ResNet architectures they always get better results towards taking it as close to the output as you can get so again this would mean like should you take it from like here or should you take it from down here like towards the output or from the intermediate features so the data sets to the tests our image net which is 1.3 million images in the thousand classes and then the places 205 which is 2.5 million images in 205 classes and these are pretty different data sets and it's used just for a

Self-Supervised Learning


measure of generalization with respect to the data sets so these are the results basically across the rev net ResNet v2 ResNet v1 just showing how amazingly different the results can be for the different architectures so and not only that but inconsistent so even though Greg net kills on rotation it doesn't perform as well on a relative patch location so this is the results in the table so this refers to increasing the widening factor of a resinate so increasing the number of feature maps to each intermediate layer so very

Self-Supervised Learning


interesting trend you see is as they continue to increase the capacity of the model they get better results so the widening factor is highly correlated with success on this task and yeah so as they increase the representation capacity of the model Bay get better results so then also this shows how applying their new ResNet with the winding factor how this compares to the previous papers that have been published on self supervised learning so most interestingly the rotation paper that first came out using an Alex net style

Self-Supervised Learning


architecture achieves 38.7% image net but their model achieves like 20% better and is getting much closer to the fully supervised benchmarks so this is another interesting thing that they present is that the success on this Hope supervised task isn't always correlated with image accuracy like this point right here has like 95 percent on rotation but then only like 20 percent on that image net accuracy so then another very interesting takeaway from the study is that it seems like larger models like increasing the width

Self-Supervised Learning


Okay, now I don't want to alarm anybody in this room, but it's just come to my attention that the person to your right is a liar. (Laughter) Also, the person to your left is a liar. Also the person sitting in your very seats is a liar. We're all liars. What I'm going to do today is I'm going to show you what the research says about why we're all liars, how you can become a liespotter and why you might want to go the extra mile and go from liespotting to truth seeking, and ultimately to trust building. Now, speaking of trust,

How to spot a liar | Pamela Meyer


ever since I wrote this book, "Liespotting," no one wants to meet me in person anymore, no, no, no, no, no. They say, "It's okay, we'll email you." (Laughter) I can't even get a coffee date at Starbucks. My husband's like, "Honey, deception? Maybe you could have focused on cooking. How about French cooking?" So before I get started, what I'm going to do is I'm going to clarify my goal for you, which is not to teach a game of Gotcha. Liespotters aren't those nitpicky kids, those kids in the back of the room that are shouting, "Gotcha! Gotcha!

How to spot a liar | Pamela Meyer


Your eyebrow twitched. You flared your nostril. I watch that TV show 'Lie To Me.' I know you're lying." No, liespotters are armed with scientific knowledge of how to spot deception. They use it to get to the truth, and they do what mature leaders do everyday; they have difficult conversations with difficult people, sometimes during very difficult times. And they start up that path by accepting a core proposition, and that proposition is the following: Lying is a cooperative act. Think about it, a lie has no power whatsoever by its mere utterance.

How to spot a liar | Pamela Meyer


Its power emerges when someone else agrees to believe the lie. So I know it may sound like tough love, but look, if at some point you got lied to, it's because you agreed to get lied to. Truth number one about lying: Lying's a cooperative act. Now not all lies are harmful. Sometimes we're willing participants in deception for the sake of social dignity, maybe to keep a secret that should be kept secret, secret. We say, "Nice song." "Honey, you don't look fat in that, no." Or we say, favorite of the digiratti,

How to spot a liar | Pamela Meyer


"You know, I just fished that email out of my Spam folder. So sorry." But there are times when we are unwilling participants in deception. And that can have dramatic costs for us. Last year saw 997 billion dollars in corporate fraud alone in the United States. That's an eyelash under a trillion dollars. That's seven percent of revenues. Deception can cost billions. Think Enron, Madoff, the mortgage crisis. Or in the case of double agents and traitors, like Robert Hanssen or Aldrich Ames, lies can betray our country,

How to spot a liar | Pamela Meyer


they can compromise our security, they can undermine democracy, they can cause the deaths of those that defend us. Deception is actually serious business. This con man, Henry Oberlander, he was such an effective con man, British authorities say he could have undermined the entire banking system of the Western world. And you can't find this guy on Google; you can't find him anywhere. He was interviewed once, and he said the following. He said, "Look, I've got one rule." And this was Henry's rule, he said, "Look, everyone is willing to give you something.

How to spot a liar | Pamela Meyer


They're ready to give you something for whatever it is they're hungry for." And that's the crux of it. If you don't want to be deceived, you have to know, what is it that you're hungry for? And we all kind of hate to admit it. We wish we were better husbands, better wives, smarter, more powerful, taller, richer -- the list goes on. Lying is an attempt to bridge that gap, to connect our wishes and our fantasies about who we wish we were, how we wish we could be, with what we're really like. And boy are we willing to fill in those gaps in our lives with lies.

How to spot a liar | Pamela Meyer


On a given day, studies show that you may be lied to anywhere from 10 to 200 times. Now granted, many of those are white lies. But in another study, it showed that strangers lied three times within the first 10 minutes of meeting each other. (Laughter) Now when we first hear this data, we recoil. We can't believe how prevalent lying is. We're essentially against lying. But if you look more closely, the plot actually thickens. We lie more to strangers than we lie to coworkers. Extroverts lie more than introverts.

How to spot a liar | Pamela Meyer


Men lie eight times more about themselves than they do other people. Women lie more to protect other people. If you're an average married couple, you're going to lie to your spouse in one out of every 10 interactions. Now, you may think that's bad. If you're unmarried, that number drops to three. Lying's complex. It's woven into the fabric of our daily and our business lives. We're deeply ambivalent about the truth. We parse it out on an as-needed basis, sometimes for very good reasons, other times just because we don't understand the gaps in our lives.

How to spot a liar | Pamela Meyer


That's truth number two about lying. We're against lying, but we're covertly for it in ways that our society has sanctioned for centuries and centuries and centuries. It's as old as breathing. It's part of our culture, it's part of our history. Think Dante, Shakespeare, the Bible, News of the World. (Laughter) Lying has evolutionary value to us as a species. Researchers have long known that the more intelligent the species, the larger the neocortex, the more likely it is to be deceptive. Now you might remember Koko.

How to spot a liar | Pamela Meyer


Does anybody remember Koko the gorilla who was taught sign language? Koko was taught to communicate via sign language. Here's Koko with her kitten. It's her cute little, fluffy pet kitten. Koko once blamed her pet kitten for ripping a sink out of the wall. (Laughter) We're hardwired to become leaders of the pack. It's starts really, really early. How early? Well babies will fake a cry, pause, wait to see who's coming and then go right back to crying. One-year-olds learn concealment. (Laughter) Two-year-olds bluff.

How to spot a liar | Pamela Meyer


Five-year-olds lie outright. They manipulate via flattery. Nine-year-olds, masters of the cover-up. By the time you enter college, you're going to lie to your mom in one out of every five interactions. By the time we enter this work world and we're breadwinners, we enter a world that is just cluttered with Spam, fake digital friends, partisan media, ingenious identity thieves, world-class Ponzi schemers, a deception epidemic -- in short, what one author calls a post-truth society. It's been very confusing for a long time now.

How to spot a liar | Pamela Meyer


What do you do? Well, there are steps we can take to navigate our way through the morass. Trained liespotters get to the truth 90 percent of the time. The rest of us, we're only 54 percent accurate. Why is it so easy to learn? There are good liars and bad liars. There are no real original liars. We all make the same mistakes. We all use the same techniques. So what I'm going to do is I'm going to show you two patterns of deception. And then we're going to look at the hot spots and see if we can find them ourselves.

How to spot a liar | Pamela Meyer


We're going to start with speech. (Video) Bill Clinton: I want you to listen to me. I'm going to say this again. I did not have sexual relations with that woman, Miss Lewinsky. I never told anybody to lie, not a single time, never. And these allegations are false. And I need to go back to work for the American people. Thank you. (Applause) Pamela Meyer: Okay, what were the telltale signs? Well first we heard what's known as a non-contracted denial. Studies show that people who are overdetermined in their denial

How to spot a liar | Pamela Meyer


will resort to formal rather than informal language. We also heard distancing language: "that woman." We know that liars will unconsciously distance themselves from their subject, using language as their tool. Now if Bill Clinton had said, "Well, to tell you the truth ..." or Richard Nixon's favorite, "In all candor ..." he would have been a dead giveaway for any liespotter that knows that qualifying language, as it's called, qualifying language like that, further discredits the subject. Now if he had repeated the question in its entirety,

How to spot a liar | Pamela Meyer


or if he had peppered his account with a little too much detail -- and we're all really glad he didn't do that -- he would have further discredited himself. Freud had it right. Freud said, look, there's much more to it than speech: "No mortal can keep a secret. If his lips are silent, he chatters with his fingertips." And we all do it no matter how powerful you are. We all chatter with our fingertips. I'm going to show you Dominique Strauss-Kahn with Obama who's chattering with his fingertips. (Laughter) Now this brings us to our next pattern, which is body language.

How to spot a liar | Pamela Meyer


With body language, here's what you've got to do. You've really got to just throw your assumptions out the door. Let the science temper your knowledge a little bit. Because we think liars fidget all the time. Well guess what, they're known to freeze their upper bodies when they're lying. We think liars won't look you in the eyes. Well guess what, they look you in the eyes a little too much just to compensate for that myth. We think warmth and smiles convey honesty, sincerity. But a trained liespotter can spot a fake smile a mile away.

How to spot a liar | Pamela Meyer


Can you all spot the fake smile here? You can consciously contract the muscles in your cheeks. But the real smile's in the eyes, the crow's feet of the eyes. They cannot be consciously contracted, especially if you overdid the Botox. Don't overdo the Botox; nobody will think you're honest. Now we're going to look at the hot spots. Can you tell what's happening in a conversation? Can you start to find the hot spots to see the discrepancies between someone's words and someone's actions? Now, I know it seems really obvious,

How to spot a liar | Pamela Meyer


but when you're having a conversation with someone you suspect of deception, attitude is by far the most overlooked but telling of indicators. An honest person is going to be cooperative. They're going to show they're on your side. They're going to be enthusiastic. They're going to be willing and helpful to getting you to the truth. They're going to be willing to brainstorm, name suspects, provide details. They're going to say, "Hey, maybe it was those guys in payroll that forged those checks." They're going to be infuriated if they sense they're wrongly accused

How to spot a liar | Pamela Meyer


throughout the entire course of the interview, not just in flashes; they'll be infuriated throughout the entire course of the interview. And if you ask someone honest what should happen to whomever did forge those checks, an honest person is much more likely to recommend strict rather than lenient punishment. Now let's say you're having that exact same conversation with someone deceptive. That person may be withdrawn, look down, lower their voice, pause, be kind of herky-jerky. Ask a deceptive person to tell their story,

How to spot a liar | Pamela Meyer


they're going to pepper it with way too much detail in all kinds of irrelevant places. And then they're going to tell their story in strict chronological order. And what a trained interrogator does is they come in and in very subtle ways over the course of several hours, they will ask that person to tell that story backwards, and then they'll watch them squirm, and track which questions produce the highest volume of deceptive tells. Why do they do that? Well, we all do the same thing. We rehearse our words, but we rarely rehearse our gestures.

How to spot a liar | Pamela Meyer


We say "yes," we shake our heads "no." We tell very convincing stories, we slightly shrug our shoulders. We commit terrible crimes, and we smile at the delight in getting away with it. Now, that smile is known in the trade as "duping delight." And we're going to see that in several videos moving forward, but we're going to start -- for those of you who don't know him, this is presidential candidate John Edwards who shocked America by fathering a child out of wedlock. We're going to see him talk about getting a paternity test.

How to spot a liar | Pamela Meyer


See now if you can spot him saying, "yes" while shaking his head "no," slightly shrugging his shoulders. (Video) John Edwards: I'd be happy to participate in one. I know that it's not possible that this child could be mine, because of the timing of events. So I know it's not possible. Happy to take a paternity test, and would love to see it happen. Interviewer: Are you going to do that soon? Is there somebody -- JE: Well, I'm only one side. I'm only one side of the test. But I'm happy to participate in one. PM: Okay, those head shakes are much easier to spot

How to spot a liar | Pamela Meyer


once you know to look for them. There are going to be times when someone makes one expression while masking another that just kind of leaks through in a flash. Murderers are known to leak sadness. Your new joint venture partner might shake your hand, celebrate, go out to dinner with you and then leak an expression of anger. And we're not all going to become facial expression experts overnight here, but there's one I can teach you that's very dangerous and it's easy to learn, and that's the expression of contempt.

How to spot a liar | Pamela Meyer


Now with anger, you've got two people on an even playing field. It's still somewhat of a healthy relationship. But when anger turns to contempt, you've been dismissed. It's associated with moral superiority. And for that reason, it's very, very hard to recover from. Here's what it looks like. It's marked by one lip corner pulled up and in. It's the only asymmetrical expression. And in the presence of contempt, whether or not deception follows -- and it doesn't always follow -- look the other way, go the other direction,

How to spot a liar | Pamela Meyer


reconsider the deal, say, "No thank you. I'm not coming up for just one more nightcap. Thank you." Science has surfaced many, many more indicators. We know, for example, we know liars will shift their blink rate, point their feet towards an exit. They will take barrier objects and put them between themselves and the person that is interviewing them. They'll alter their vocal tone, often making their vocal tone much lower. Now here's the deal. These behaviors are just behaviors. They're not proof of deception.

How to spot a liar | Pamela Meyer


They're red flags. We're human beings. We make deceptive flailing gestures all over the place all day long. They don't mean anything in and of themselves. But when you see clusters of them, that's your signal. Look, listen, probe, ask some hard questions, get out of that very comfortable mode of knowing, walk into curiosity mode, ask more questions, have a little dignity, treat the person you're talking to with rapport. Don't try to be like those folks on "Law & Order" and those other TV shows that pummel their subjects into submission.

How to spot a liar | Pamela Meyer


Don't be too aggressive, it doesn't work. Now, we've talked a little bit about how to talk to someone who's lying and how to spot a lie. And as I promised, we're now going to look at what the truth looks like. But I'm going to show you two videos, two mothers -- one is lying, one is telling the truth. And these were surfaced by researcher David Matsumoto in California. And I think they're an excellent example of what the truth looks like. This mother, Diane Downs, shot her kids at close range, drove them to the hospital while they bled all over the car,

How to spot a liar | Pamela Meyer


claimed a scraggy-haired stranger did it. And you'll see when you see the video, she can't even pretend to be an agonizing mother. What you want to look for here is an incredible discrepancy between horrific events that she describes and her very, very cool demeanor. And if you look closely, you'll see duping delight throughout this video. (Video) Diane Downs: At night when I close my eyes, I can see Christie reaching her hand out to me while I'm driving, and the blood just kept coming out of her mouth. And that -- maybe it'll fade too with time --

How to spot a liar | Pamela Meyer


but I don't think so. That bothers me the most. PM: Now I'm going to show you a video of an actual grieving mother, Erin Runnion, confronting her daughter's murderer and torturer in court. Here you're going to see no false emotion, just the authentic expression of a mother's agony. (Video) Erin Runnion: I wrote this statement on the third anniversary of the night you took my baby, and you hurt her, and you crushed her, you terrified her until her heart stopped. And she fought, and I know she fought you. But I know she looked at you with those amazing brown eyes,

How to spot a liar | Pamela Meyer


and you still wanted to kill her. And I don't understand it, and I never will. PM: Okay, there's no doubting the veracity of those emotions. Now the technology around what the truth looks like is progressing on, the science of it. We know, for example, that we now have specialized eye trackers and infrared brain scans, MRI's that can decode the signals that our bodies send out when we're trying to be deceptive. And these technologies are going to be marketed to all of us as panaceas for deceit, and they will prove incredibly useful some day.

How to spot a liar | Pamela Meyer


But you've got to ask yourself in the meantime: Who do you want on your side of the meeting, someone who's trained in getting to the truth or some guy who's going to drag a 400-pound electroencephalogram through the door? Liespotters rely on human tools. They know, as someone once said, "Character's who you are in the dark." And what's kind of interesting is that today, we have so little darkness. Our world is lit up 24 hours a day. It's transparent with blogs and social networks broadcasting the buzz of a whole new generation of people

How to spot a liar | Pamela Meyer


that have made a choice to live their lives in public. It's a much more noisy world. So one challenge we have is to remember, oversharing, that's not honesty. Our manic tweeting and texting can blind us to the fact that the subtleties of human decency -- character integrity -- that's still what matters, that's always what's going to matter. So in this much noisier world, it might make sense for us to be just a little bit more explicit about our moral code. When you combine the science of recognizing deception

How to spot a liar | Pamela Meyer


[Music] so Wolfgang helpfully laid out the dichotomy between industry people and academics and experimentalists and computational people and if you're wondering which one I am the answer is yes so this is I'm going to mostly describe work that happened in my lab at Harvard and work by Bill Lauder who is actually in industry at us doing a start-up because that's what everyone does these days but I recently also took a gig as the director of a new MIT IBM collaboration a quarter-million-dollar AI Institute so if you if you're

Predictive Coding Models of Perception


interested in that come and find me and then I'm going to tell you about some deep learning work we've done but then I'm also going to tell you at the end if I have time about some experimental work that it's inspired in my lab just to kind of reinforce this idea that there's sort of a loop that we can be driving between models and experiments so we all know that deep learning has kind of a maybe kind of connection to biology so we have units and they have synapses and connections between them okay that's that's part of

Predictive Coding Models of Perception


it that's for their artificial neural network part comes from and then they're deep and you know we know that perceptual hierarchies for instance in the and the primate are are sort of deep hierarchical systems and deep CN n sort of capture that but really when you get right down to it you know that's been most of the interplay between these two and I think a lot of us here are sort of trying to think well how can we go back to the brain and get more inspiration but then also how can we use the in you know the deep learning to actually help

Predictive Coding Models of Perception


us understand the brain a little bit better so again this virtuous loop you're gonna hear this I think again and again and again and that's that's wherever we're coming from so it turns out the deep learning so I'm interested in perception so a different part of the stack then the the previous two talks imaged in how we perceive objects and make sense of this sort of terrible complicated flow of information coming in through our senses and it turns out that by accident deep learning systems convolution neural networks turned out

Predictive Coding Models of Perception


to be the best model for doing this so people have built all kinds of computational models of how visual processing worked in the ventral visual pathway and primates but it turns out that when you just started training deep Nets you just took the internal representations compare them to actual neuronal population responses and that's the best model so so here this is a paper from my my former PhD advisor Jim - Carlos lab and basically you see those are sort of model fit quality for different kinds of models that came

Predictive Coding Models of Perception


before including some of my own and then basically once we have deep Nets those explained the data fit the data way better than anything else and you know this has been going on and basically the bottom line is the better the models get on image net the better the fit seems to be between what's going what the representational space looks like in the in the deep net and what the representational space seems to look like in the population so this led to this idea that well maybe maybe visions just tapped out and this is a solved

Predictive Coding Models of Perception


problem and then maybe we don't need to worry about this anymore the answer is deep Nets there's a problem because all the work we've done so far is all looking at static representations and really our visual systems are built for dynamic situations and it's even worse than that because even if you show a static stimulus neuronal populations will not produce static outputs and this is one of the first things you notice when you're a graduate students sticking electrodes into the brain of the monkey which I did for five years if you show a

Predictive Coding Models of Perception


static picture of a monkey face you get these sort of transient responses and in fact the only way to drive a sustained response in an IT neuron at the end of the ventral visual pathway is to show a dynamic stimulus so this is very much at odds with how cnn's work because they have no intrinsic notion of time you put in an input you get an output it's a static thing so something is clearly going on here and there's all kinds of interesting rich dynamics here where sometimes it's sustained sometimes it's not sometimes

Predictive Coding Models of Perception


it looks like it might be oscillating so we don't really have a good picture yet of what that's all about and to that and you know that part which seems really salient isn't actually captured by simple cnn's and it gets even weirder than that too because there's this great experiment by Karl Olsen's lab where he showed successive presentations of image image image image and in some cases the the second image the B image was predicted by the first image so b4 always happened after a4 and you saw this a couple hundred times and b5 was

Predictive Coding Models of Perception


always seen after a5 showed a couple hundred times and then basically you see is you don't get a response to beef-up before if it's preceded by a for does that mean that the cell just doesn't like b4 well no actually it will respond to b4 but only when it's preceded by a different image so there's some kind of higher-order temporal track that's going on but again CNN's just don't capture because they don't have any intrinsic notion of time the other thing too which i think is actually a salient problem for deep learning is

Predictive Coding Models of Perception


that most of the datasets we have to train these things you need tons and tons and tons of data you know if you want to train you know a dog detector you need you know thousands and thousands and thousands of dogs and thousands and thousands of thousands of things that aren't dogs and that's just not the way we learned I don't sit down for my daughter and shower dog dog dog dog dog Cat Cat Cat Cat that's just not the way it works in fact yeah we can just do that experiment right now does anyone know what this is raise your

Predictive Coding Models of Perception


hands if you do okay we have a few electrical engineers in the audience okay even though you've even though you've only seen this for the first time is it present in this image yes how many in this image - how about that one yeah but it's a little weird right so even though you only saw one example you were immediately able to determine what was there you immediately an expert on this kind of object that's called one-shot learning and you know just if you need more evidence that deep nets don't quite work yet is anyone can anyone say what

Predictive Coding Models of Perception


this is it's actually Meret Oppenheim it's called luncheon in fur but everyone agrees it's a cup a saucer in the spoon state of the art deep net says it's a teddy bear and then it gets worse than that even even when things are very clear to the right object so this is a state of the art our CNN detection correctly detects this as a bird but you just put a few objects in the don't belong and it starts you know calls this a cat says that's it you know it's this is a television but so it's calling that a cat so there's something about there's

Predictive Coding Models of Perception


something brittle and adversarial examples we could go on and on and on about ways in which deep nets really aren't quite solving the problem yet and you know the I I and other people think that unsupervised learning is an important piece of this so how can we without labels gather up a lot of structure about the world and build representations that are really good and then that can feed into things like reinforcement learning and all that kind of stuff so this is just a different part of the stack I think we all agree

Predictive Coding Models of Perception


on there's multiple kinds of learning happening here but the kind that I'm really interested in and have been for a long time is this idea of temporal learning if you just look at the environment and let it play out the environment is almost always showing you its structure so you look at a person doing a tennis serve you can see how they're articulated you can see they're put together you can see how the shadows move around on an object that's just all played out in time you don't need supervision you can just

Predictive Coding Models of Perception


observe and learn a lot about the world so that's what we're interested doing it and you know brains from the from the neuroscience literature seemed to be exquisitely tuned for these kinds of temporal statistics so this is one of my favorite studies it's like a physics where basically they took faces and then they rotated them and then had subjects to passively watch these but as a little trick sometimes they would morph the face as it moved and people generally didn't notice this but if you ask the same different later ones that had

Predictive Coding Models of Perception


morphed and they had seen them morph a few times they would incorrectly sort of associate them as being the same same object and then you can even go further this is let me I didn't over my PhD you can even have presented a peripheral object and then have while the subject is a Codding you flip it out for a different object before their eyes land their eyes land on a different object and then you ask them to do same different on peripheral versus fovea and they'll make incorrect associations so it seems like the brain is constantly

Predictive Coding Models of Perception