real world and there are a lot of other applications so let's think back to machine translations so if we want to if I want to translate between Chinese and English or English and Germany that's relatively easy because there are large demand of those language uses and it makes it economical to annotate a lot of data of basically supervised language sentences pairs but then like it's not


economical to do it for the other probably hundreds of languages that exist in the world it just doesn't make sense to annotate that much data and and if we can make this kind of distribution alignment to work without any supervision then it could be used as a lot more it can be used as a way to augment label examples in a kind of semi-supervised way we can be also used

L9 Semi-Supervised Learning and Unsupervised Distribution Alignment -- CS294-158-SP20 UC Berkeley


to do style transfer like some of the things that we have seen that basically had no ground truth in the real world yes yeah well it's just okay so if I have a good translation model between two languages then the value of that model is kind of proportional to the usage that you can get from it let's say to train any pair of languages you need the same amount of


investment that's called fifty million dollars probably on the lower side then like if I throw in this fifty million dollars for between Chinese and English you probably get a ton of usage and you get ads or like what other revenue but then I have like English - like whatever language that probably only a hundred thousand people speak then you get drastically less usage of the model that


means for the same investment you get a lot less out of it so it's not that they are more expensive to label it's just like that value it doesn't make it justified so okay so let's look at this problem again so it would be of course it would be a nice thing to be able to achieve to give me two distributions and then find a way to align them but this is even a feasible problem right so if


we look at the problem statement we are basically told to do two things one thing is we have distributions a we have two random variables a and B two distributions and then we get access to the samples from them like we get a bunch of samples in one domain we also get a bunch of samples from the other domain that's all great but what we crucially don't have is we don't have


any samples from the pairs yet we need to estimate how they are related to each other so this problem form a high level seems pretty hopeless because you have really given too little information to to tacko it so what we would look at next kind of certain so basically the crucial problem now is like where do we even get any training data like if I don't have any supervised pairs like


what do I even train the model on so the the way that people have been doing this is they try to rely on certain invariants that are true for kind of any pairs of distributions and then somehow you could get some meaningful learning signal out of it so the first kind of invariance that we can rely on is something called marginal matching so there's some math here but then the


brief idea is really like if I want to translate one distribution from one distribution to another after the translations the distribution should still look like each other and more precisely what it means is that there was some fundamentally unknown coupling there's some fundamentally unknown relationship between these two random variables a and B that we are trying to


approximate but we don't have access to them so let's call that our approximation Q so given B like what what a is most likely well this is the other this should be so basically we were trying to learn two mappings or two conditional distributions and when you specify this kind of conditional distribution distribution Q of a given B you implicitly induce a marginal


distribution so when I specify Q of B given a I implicitly specifying a marginal distribution on B and the way that you compute it is if I sample a for my one true distribution of a and then you essentially average out this conditional distribution that would be the marginal distribution of p and ideally I warn my Q to be close to the ground truth conditional distribution P


of P given a and that means if I sample a lot of a and then I map it through my conditional distribution the outcome of that should map back to the original ground true of distribution and similarly I can do that for a so I sample from B and then I would from from this B samples I would calculate my approximate conditional distribution and after those transformations they should


be the same as original marginal distribution as a and oftentimes in literature this conditional distribution is just a deterministic mapping so I would say give me any sample from a I would map it to its corresponding sample in domain B so far so good so the question is basically we have stated the question to be in the end we are trying to match this marginal of be approximate


B to the ground truth B which we have access to but in nopon you are saying the new net Q needs to look at a essentially so that is a true statement so if my Q if my Q is so powerful that it could just represent the whole marginal distribution of B so let's say let's call if we have Q of P given a that is equal to Q of B for all a and B pair then your statement would be true


basically you can just like approximate the marginal distribution without even doing any meaningful work so that's why like in practice people would have a fairly restrictive mapping so like that's why like in most of the works that we would look at like it usually takes the form of a deterministic mapping so when when Q of B given a is deterministic then like you don't you don't get to


represent the whole marginal distribution unless P of P itself is only a pawn mass but that is a correct observation but like you said like I suggested like I mean this is a very weak learning signal like if you don't correct you if you don't construct your model in the right way like you you could extract nothing from it so let's see some examples of like how like how


does this how does this in an ideal case work at all right so let's say I have two distribution a and B and they are very simple categorical distributions that only have three possible values a 1 a 2 a 3 I'm just going to draw some frequency here right so and then I'm going to do the same thing for B so this is probability mass function all right so so based on let's say we


have a deterministic mapping that means like each of the a1 has each of the a has to map to some B and each of the B has to map to some a then like based on the marginal meshing like this can seemingly gives us a way to recover the crunch of Correspondence let's say the ground shook of respondents is between a 2 and B 1 and a 3 and B 2 and a 1 and B 3 right so I would argue this would be


the only correspondence that satisfied the marginal matching constraint well it could be not a by direction but then like it might not fulfill the properties until unless some of the unless some of the values have like no probability mass then like you can do whatever right so argue this is the only way that you could make the marginal meshing look the reason is that because each value has a


distinct frequency here so if you match it in the wrong way the marginal distribution of the induced mapping would no longer be measuring the original one but there are still a lot of ambiguity right so if we imagine a distributions that's like the most kind of the most difficult one let's say I have a uniform distributions over two random variables then this is kind of


hopeless because all kind of mapping could work so let's first look at the a to be mapping so a one can map to B 1 a 2 map to beat you but then it's also possible that a 1 can be mapped to beat you a to map to b1 and will still be fine form a marginal matching perspective but then the problem is so well then that means this thing is ambiguous and this is not just hopes


sorry mainly joyed the other way so there are two set of mappings that we are we need to learn here one is G a B which is mapping from A to B and then another set of things is another set of things that we need to learn is GPA and basically you would need to learn the product of these two different possibilities B 1 2 A 1 B 2 2 a 2 B so how many like totally possible


solutions are there to this problem if we just use marginal matching yeah so basically each direction there are two possibilities and then if you multiply them together there are four possible solutions in this problem and they are basically totally ambiguous so this is one of the thing that we are seeing here is you can have your objective function that induce


a really large solution set really in this case is almost all of the solution set and then people realized this can potentially be a problem so they introduced another technique to try to at least restrict the solution set a little bit so this thing is oftentimes referred to as cycle consistency but it has also taken a lot of other names in literature called dual in learning back


translations and really the core idea is that if I if I take my so basically the whole idea is that my apartment mapping should be similar to the ground truth mapping and if those mappings are deterministic then what that means is that if I step through my mapping I should get back my original sample so if we think about the case of P of a be given a given of a would be this would


map aid to his correspondence in B and then if you apply that again from the other direction B to a mapping you should get back a and this should hold you in both directions so this gives you another invariance so if you say that the relationship between these two distributions are indeed deterministic then I would say these things should hold you for all possible


pairs and if we step back to you the example that we just look at so if we impose psycho consistent see what would be the number of possible solutions now why is that no longer four yeah so like we can see that in this case the original total solution set is four but after you impose this cycle consistency constraint you can reduce the solution set to exist like some of


the some of the mapping are no longer valid so let's say if I pick this and then take this this would be no longer valid because an a.1 would get translated into B 1 and then P 1 according to this would get translated into be a 2 so this is G a B this is G B a and this is this no longer satisfy the cycle consistency constraint so that means like I can use this constraint to


reduce the possible solution set in my in my search but still like we can see that it's still fundamentally under defined like we are still left with two possible mappings and we are not sure which one is correct but it at least exponentially shrink the space that is possible so so far we have seen the two core in variances that people have used and these are invariants that's true for


all alignment problems and then we can use them as learning signals and again like obviously like we just look at like even in a extremely low dimension basically categorical examples like it's still not going to work so there are there are definitely problems that this cannot solve but then in practice like people can find problems that this kind of search is amenable to you and then


you can oftentimes ensure that there's no biases in your system by selecting the right architectures loss functions and etc and you can actually get to a certain level of success with this yeah so they expect this this one basically says for an arbitrary data porn in its it's just a generalized version of the cycle consistency thing so for any of the data porn a if I draw samples for my


approximate conditional distribution and then I translate that B back to a using my approximate once the the distribution that induce should be similar to if you do it with the real-world one and what we're saying here is just that like when in the case of both of when we say we know the P and the Q are both deterministic then it reduces to the deterministic mapping example but it


could exist in a more general form so probably the best-known example that uses those learning signals are cycle again so psycho gains laws essentially consists of two parts one part is this marginal matching so meaning after I translate my data from one domain to another the marginal of them still match with each other and you can kind of see it here so essentially my generator no


longer takes a mapping from Z to X instead this thing is trying to translate from X to Y and I want to do it in such a way that it looks like my target image so this is just a standard gain training loss where you're trying to say my mapping from X to Y it should look like just like Y so that's fairly straightforward and so but it's actually instead of looking at frequency you use


again to help you do the marginal matching and the second dimension of this is you can achieve cycle consistency by an l1 loss essentially what there's if we unpack this up this objective function is your sample for data in one of your domain probably your source domain and then you map it to you map it to a cycle then it should look like itself in an l1 sense


so this is I think what they call forward cycle consistency because it's going from X to Y to X and then they also have a backward one where you essentially kind of think of a sample from oil labels and then you map it through this thing again Y dou X dou Y should be similar to yourself in an l1 sense so that's the lost function and then you essentially would combine these


two things together and then train it I think in practice they use at least I think the in practice they use at least square again instead of the original gain objective but probably it doesn't make too much difference so they reported a couple numerical results and the first results that they look at is so in the case of going from photo to semantic mask you can actually calculate


the accuracy so we can get a quantifiable notion of how well the method is doing so the method that we just introduced is called cycle again and it's basically unsupervised so you give it a bunch of distressing images and then a bunch of semantic masks and then you hope them that they somehow align each other and what this shows that they actually do pretty well so a


peg to peg is a fully supervised model so the last rode means you get you actually get pairs of image and their corresponding label so the last rows should be read as basically an upper bound on the performance and then the cycle again by using no labels at all you can actually do pretty fully so like you can roughly say that 60% of the pixels are labeled correctly


with the right with the right class we actually know even with no information on how they should be related to each other know so there's there's no pairs so you you train you train the whole system with just a bunch of unordered images and then a bunch of an order masks and then it's learning to align them good good question I don't think they have that level of analysis but I


think that would be interesting to see like what are the kind of things that are easier for you to align what are the things that are not like a mushrik yeah yeah yeah yeah so the so the question is there was a lot of inductive bias going from one image to another using a conf net and then also using a certain discriminator that operates on a patch basis so you kind of like do a kind of


domain alignment patch like so you can think of it as having its there's really a lot of training signal that is not captured by the loss function at all so unfortunately we don't know the answer to that so I guess what what I know for sure is like if you just scramble the image like like I mean just permute the dimensions in your image tenser then I'm pretty sure you would do full but then


like is does that mean like this is no useful probably not but this still means that we don't fully understand what are the inductive biases they're helping us but that's a good question yes like right so I guess so so the common is around like the a lot of these translation problems like operate in a very local manner like you're kind of like saying I just need to change my


local pixels like when you go from zero to you to two holes like you're just kind of changing a local texture as opposed to something that is global which is presumably much harder I think that's likely the case I have no idea you cannot you can ask Alyosha who will be here soon so I I think this is a long way from supervised learning so I believe supervised learning like


this thing should like I don't know but I I imagine this to be at least 95 percent plus but again like this is not a correct comparison I think the correct comparison is to compare cycle again with takes two pics because they use similar architecture except one is supervised the other is unsupervised all right so they have some Appalachians in terms of loss function or don't know


in terms of architecture the Appalachians basically tell you what we sort of expect that like for one like if you can alone if you just you can along this means you just do marginal matching so which is actually not bad already and then you can see that if you add psycho consistency in there it helps you and there's something that's really puzzling like I'm really confused by what is


happening here it just kills everything I like I have no idea what is happening there so I don't know and and what's also interesting is that it doesn't always help so what we're looking at we're looking at going from photos to labels and then they have another experiment that is going from labels to photos so this is a much higher entropy mapping whereas they still use just


deterministic mapping so you can imagine some there might be something that is playing with that in here that I guess we don't we don't fully understand what's interesting here though is the evaluation map metric is pretty interesting so remember here we are evaluating it from label to photos so basically is give you a semantic mask how well you can generate the scene but


then how do you even evaluate that so they actually have a pretty clever way of evaluating that so what they would do is they would run another pre-trained semantic segmentation now walk fully convolutional network they will run it on the generated image and then they use step to quantify the results so that is a pretty interesting trick to evaluate this mapping kind of like the inception


school except like in this case like we you kind of yeah it's kind of like Inception school but I think it's better than inception in school in this restricted domain these are some of the other codes the first cases where you translate from I guess a schematic annotation of a facade facade going from address to Shu going from shoes to address most of them make sense they


applied this to a wide variety of different problems where you it's just impossible to get label pair it's just like I guess somewhere Yosemite and winter with somebody you can get pears although not exactly the same and translating apples to oranges like just like we said like this is it's it's not like this is not supervised and it's not fully unique they have their set of


failures and in this one what the authors explain the paper is they're like I mean when you train on how movies are success I don't know when you train on horses in imagenet they decide like they're like they would they have not seen a human riding on it and as such like you would just classify or similar texture to be horses and then you just translate that so like this this I guess


goes back to one of one of the question that someone mentioned it's like what are the failure cases so like I think this is one good example of like what it fails on I think this is a good example of what the model is doing is it's trying to find like yellowish pattern and then change that yellowish pattern to stripes so that's apparently what the model is doing so that's that for cycle


gain so essentially it's pretty surprising that it can work on certain domains and when it works I think it's very reasonable so the next thing that we would look at is we look at improving cycle gain in certain dimensions so the crucial dimension that we would look at here is that remember when we talked about the cycle again the cycle again has this deterministic mapping so


you give you an image X which translated into domain Y and but this translation is deterministic but that is fundamentally not correct at least for a lot of the alignment problems that we care about so let's say if I want you go from mask to image semantic mask to image like there was a lot of different ways to satisfy the same semantic mask there are a lot of different ways to


generate that image like semantic mask only tells you there's a car here but what does the car look like what's inside the car what color it is like it's simply specifying none of those so basically there's this high entropy mapping going from semantic mask to image and that is apparently not deterministic so you can say oh one straightforward way to extend cycle


again is to say cycle gain is essentially this right so you take in an image a and then you're trying to map it to image B so one straightforward way to extend that would be to make this mapping taking an additional noise sauce just like in it typical again so you could take in an image a and then you can also take in a noise sauce that probably described like what does the


car look like what is the color other than the contour everything other than the contour and then from that noise source you can map to some image P and then if you sample different Z's hopefully you get different cars is that some motivation make sense so that's all good so and in fact like this has been done concurrent you cycle again there's another paper called do you again where


this is essentially the architecture the mapping would take in both an image from source domain as well as a random noise source however this is not enough to just change your architecture because if you changed if you even if you change your architecture the noise are doomed to be ignored and to see that the reason is essentially our lost function the l1 law the l1 cycle


consistence loss carry to do the following so if my map my a with certain Z this this produce like some kind of B for me and then if I met my be with another Z Prime I should get back to a and we can see that in this whole mapping the choice of z and z prime are essentially ignored so you can choose different Z or Z Prime you still need to satisfy this mapping


so what that means is that the noise source is necessarily ignored when you impose a cycle consistency loss and when you optimize it to is fixed port so that's not good and then there was this augment excite Oakham paper that proposed a way to solve it so you would augment the noise to your architecture but you will also learn so instead of only learning the mapping from A to B


and B to a you will learn an encoder of each of the noise source not the similar to how an encoder is used in a variational method and it's actually pretty interesting so the way that you would go is I have some ground Shu image a and I have then what I'm going to say is that my ground truth image a would comes from a corresponding B and it's corresponding noise sauce za


and then I would have this blue arrow which is a network that infer what's the a it is so basically I'm trying to infer what Zee produce my a and I'm going to infer what P produce my a so instead of only inferring what is my corresponding B I infer that as well as what is the noise source that produced me now with both the noise source and the corresponding B I can use that to map it


to an a prime using this color which is the mapping coming back from B to a and in the end I can say that a and a prime should be similar in l1 in l1 lost sense so now it's okay because I'm choosing a specific Z for each particular data point so if we think about it from an information theoretic sense whatever information that is not captured in B you push it in to Z that allows you to


perfectly reconstruct the original image as well as maintaining the ability to have diversity of mappings because a different egg come from different see yes so the question is how do you prevent the model from putting everything into Z so you could but in my fail the marginal matching criteria right so I guess the statement is like the Amb relationship could could become


decoupled right like from a I would just match an arbitrary B that actually has no corresponding with a from the beginning but then like remember that is always the problem like even with the original cycle again you could still produce an arbitrary mapping that this consistent but it's not the ground truth mapping so this I guess what I'm saying is this doesn't make it worse yeah


so we can say it again so I was saying like basically you can play this multiple steps and then like the evolution of them should still match the original marginal distribution yeah so many ways that you can play with this [Music] yeah you have Ken loss on B and then you would I think you also have Ken loss on Z which like Z you restricted to piece the marginal of that you restrict it to


be some Gaussian or something so in a sense you cannot put infinite information in there so both of them are in a sense information regular lies it's it's really more like an adversarial autoencoder which we didn't cover in a lecture so it's kind of like a VA yi but instead of like a care loss you use again loss so it is more it is but like a it is very much like a Nathan Coe


model that is training with again it could it wouldn't like that applies to everything that we would go over today there are holes in all of them oh you mean why they don't use a Vee I think it's probably the mapping from A to B that he wouldn't do well like you wouldn't do it in like a visually appealing way otherwise I think for Z they could actually use a VAD type of


loss but it would not be a VAD because there's still this thing that is [Laughter] yes no it's not it's not that fair comparison I would say though like if your data set is small like Dan training is usually pretty fast as well anyways but that's off topic so just like operationally what does that mean if we go through like one cycle of that well cycle consistency loss so we get some


image from some from a source domain and then we would randomly sample AZ for my mapping to be because remember that the mapping from A to B is also stochastic so B could take up a lot of different forms and I'm going to generate a noise source that dictate what it is so this is what I'm going to samples and then there will be a set of mappings that go through so the mapping from A to B now


it takes a and annoy sauce for B that gives me B and then from this BN a I can try to guess what was the the Z that has generated my original a so that is my encoder to guess Z of a and then finally I would plug in the B they're generated and the Z they are generated and from these two I get back I get back this a prime which supposedly should be close to my original sample so that's all good


like it's fairly easy to implement like just a small surgery on cycle again how a does it you so the full the first thing that you would want to run is essentially you would want to first give Z as the additional input to the mapping which they call stochastic cycle again I guess so that is without changing the loss function like without introducing the encoder it's what this column that we


are looking at and and then the test here is we simple and add app for my Gwangju of data here and then I will fit it through my cycle again but with different Z terms so this is imagine this is coming from Z 1 Z 2 Z 3 Z 4 so this is surprising right because we ocean aliy we went through this argument of like how just changing cycle games to make you to have to take in Z


wouldn't actually make use of Z like it would just ignore it but what we are seeing here is actually different right so you give it an H mask and it actually generate diverse samples for you so that's interesting like if we look at like this shoe like apparently there are all different colors and even though we do the Augmented one like using the new loss function and the encoder I would


say they look probably about the same the same kind of diversity but it is kind of interesting like why why does why does cycle again walk especially like this is highly contrasting with the analysis that we just went through so if I especially if I take a black shoe and then map it to a semantic mask and then map it back then if I get a Y shoe which is a point to


here is something that could happen if I get a y shoes back I'm going to incur huge l1 loss because black and white are just to end of the spectrum in your color space so so that is somewhat puzzling like what what is this actually doing like if if it can generate this diverse samples then that means it's not optimizing its cycle gain loss well but it is optimizing its cycle gain loss as


well so the very interesting thing here is that cycle again when you go from a high dimension like a high entropy mapping like RGB images to a low entropy one like a semantic mask it can actually hide information in some kind of high frequency pattern so this is what we are this is what we are seeing here so like like going back to the black shoe exam like when you map from a black shoe to


his H pattern like it would give you like seem a plot seemingly plausible H patterns and then as some high-frequency noise to it that they know that this is coming from a black shoes and then you can look at the rough shape of that pattern and then it also reached the high frequency noise that's encoded in there to say oh this should be black and that's how it manages to still do the


cycle consistency right so I consistently get the same color back by hiding imperceptible information in some of my mask on my edge and that's pretty interesting and the way that you can show that it's doing that is by essentially constructing an experiment where you so so this is quite domain a this is domain B you get the B out and then you try to sample different Z that


you try to fix that B and then you try to sample different Z's so if this is coming from a cycle gain loss then you will see that the mask itself even though seemingly it doesn't in cold color information it's implicitly encoding color information so if I take this mask that's coming from my model I will have color information hidden in it in such a way that when I sample


different Z's you always get the same output and that's how is able to still satisfy the cycle consistency constraint and so and then if you do the Augmented cycle gain loss what you will see is that the mask looked basically the same but there are seemingly less information in there so when you sample different random noise or Z you actually get different color of shoes back


I don't remember that's a that's good to check but like I like I guess like us as we have discussed like if Z is very powerful and potentially getting Co too much so I think I think there would be a balance there and this is kind of doing the cycle walk this is somewhat interesting some are similar to what we had discussed so maybe from A to B and then B to a a to be while I cycle


through different noise sauce and then if you do this kind of random walk in an auto augmented cycle again you can see that even though the mask stays relatively the same the over appearances some of the other color texture does change over time whereas if you train it with the original cycle getting lost you will just get the same pairs repeated again and again that's a


somewhat interesting a in my opinion relatively simple and arrogant extension to cycle gain that helped you to deal with stochastic mappings any questions on that before we move on so the next set of questions are can we do better so so far we have covered two learning principles one is marginal matching and then the other is psycho consistency and I guess it's a good


question that whether those are the all of the invariances that we can rely on or are there additional learning signals that we can derive from it it's a good open problem and if we step back and think about this whole problem it's really aligned to distributions we found knowing what's inside was really difficult if we if we think about the categorical distribution that has even


probabilities like it's just impossible to align them because we treat them as pure black paths like all values are the same as each other so one idea that we can move forward from this point is we can look inside a random variable it's we can say this image it's not just a huge a high dimensional random variable to me like I can actually look inside and see what's


in there and then maybe use that to help us and for this kind of high dimensional a and B like they typically have certain structures in them that that could be leveraged and as people have pointed out like when you use a continent and patch phase discriminator in a cycle game they're kind of implicitly employing some of this inductive bias already but I think we there are cases where we can


push this even further so the best example that I could find is in NLP for that so let's say ptomaine a is all English sentences and domain B or French sentences then we can imagine that like I can get a random sentence from or English sentences and a random sentence from all French sentences they might have the same empirical frequency but they might be totally semantically


unrelated which is likely to happen so it's like what consistency wooden also rule out that either this is just basically going back to the problems of when you have distributions to have uniform densities nothing could help but what we do know is that each sentence is made up of words and it's very unlikely that in those two totally semantically unrelated sentences they would have words that


have same kind of statistics so I'm using the term statistics loosely here I'm going to say more about like what we can do with this so basically what the exercise that we have gone through is instead of thinking of it as distribution alignment between sentences in different languages if we are allowed to look inside like what's in between each random variable and look at their


sub components and do some influence on the sub components they can help us circumvent the problem of long enough learning signal so in the case of NLP the sub components are the words and the large and the larger higher dimensional random variables are sentences or paragraphs and so that's interesting so like now what we can do is we can for one we can first of all align the words


like we can think of ways that you can do distribution alignment on words and even more we can think of we can make use of how different words occur together so let's say the word eye is most likely to be followed by M and because these two things Co occur most frequently in within this large random variable sent a sentence so what is the thing that we can make use of this kind


of co-occurrence to the six of sub components well we have learned one of them what you back in two lectures ago probably so just a recap on sip sip gram water back it's really simple so basically always trying to do is is trying to say given one was the world in a sentence I'm going to say that others other words in this sentence is more likely to occur than every other


things in my corpus and in practice you wouldn't sum over all of your dictionary you would do some negative sampling to optimize this but essentially the end we saw this certain vector that described if two vectors are close together in that vector spaced and they're more likely to occur in a sentence and if you train a very very large model on a lot of text data then they capture


how different words are likely to occur together so let's give Graham and what's really interesting is that this kind of woodwork method exhibits really interesting vector calculus so this is again a recap slice what we can look at is that if we look at the direction from a country to his capital the vector is actually relatively similar across a lot of these different pairs and we might we


might be able to ask like so based on this kind of vector if the vector calculus makes sense then does it mean that we can say the vector representation of those words are distributed in a certain manner and more importantly if similar cap vector calculus holds true for all languages meaning I train a word embedding for English let's say on the on the left and


they also training for embedding for Italian if they exhibit the same vector calculus meaning all different in bearings in them placed together in a such way that you could do the kind of country to capital translation then that's a really strong inductive bias for us and if that holds true then we can possibly align words by similarly by just like uncovering some kind of affine


or linear transformation that align these two things together so very surprisingly it's actually true so for the virtual vac that we use let's say in fast text if you train it on one language if your trailer multiple languages and then these embedding space they are only a rotational weight so you're going to learn a rotation matrix that rotate another language into your space and


then the results would be both this this is a graphics that grab from a Facebook blog post that Illustrated really well so you basically get this embedding space that that exhibit similar relative structure and then so what you can do but the absolute location is undefined so what you can do is you can just learn a way to align them together and then after the rotation one point in the


embedding space would be very likely to exhibit the same word but in different languages so this this totally blew my mind that this could work yeah so so initially so it was with the citations here you can basically use a small dictionary play a language to language dictionary you can use a small dictionary to learn the alignment so this it was still supervised but the


search space is much smaller like instead of going from each word map through a new net and then after another word you would be like every word is already presented by some embedding now I'm only learning the rotation that is used across all embeddings but then like so basically a couple data points is enough to to specify that yes I don't know probably two mm I would I would guess


like I've been totally uneducated guess I like I'm not gonna NLP cousin it could be pretty big yeah so that's pretty interesting so that's what happened up until like fari before 2017 ish is people can like you can align these two embedding spy is basically examples like I can just go to you like French English dictionary and then look up a couple Wars and then you


use that to to align these two embeddings that's really cool then you can do that yes what apparently they are scaled the same or less similar enough [Laughter] this recent paper that proposed a way that you can oh actually another thing that I forgot to mention is so no actually this is it so basically that's a supervised way to align this wording batting like so that's that's really


interesting that you can just capture that by a simple rotation well probably not simple but a rotation and what this work has done is to show that you can actually do that with really good performance in an unsupervised way so now I have two embedding space and then you are lying them without any training signal so the way that is done is actually basically just using the


principle of marginal matching so you would similarly apply so basically you your rotation each of each possible rotation basically specify a mapping and then you're going to say after the mapping my marginal distributions to match and then they just train that with ever Cyril training so like again like a loss to spur make sure that the marginal is specified


is the margin always matched and then after they do that but one of the results of possibly do to gain training is usually no very robust and high precision so after they do that they have they have found a rotation that roughly aligned to distributions and then after they have that they would select some top pairs of high-frequency words in those graph alignment and then


they would assume that they are actually gwangju for linemen and then you would use those as like actual pairs to to solve for the exact rotation and apparently this works really well so that's some of the data that you get from there's some additional tricks in terms of embedding nearest neighbor that I didn't go into so but this is the results that they have and they're


comparing that with cost lingo supervision and without any supervision which is their own method and it's really surprisingly they could get you competitive performance with using ground truth data of actual pairs so this again like this is not as complex as translating whole sentences this is only translating words but I still see this as very impressive that this can


work at all and become very competitive with supervised methods so then the next part of this is again another paper from facebook is now you can actually lavish all three of the core principles that we have covered so far you can look at so they use world level alignment meaning they started from what we just look at the end supervised level the unsupervised world


level alignment so this is you're not just looking at sentence level you're looking inside sentence to sub component level statistics and then they also use a monolingual language models to make sure what you translate actually looks like a real sentence so that basically you can see that as marginal matching and then they also have this thing called black translation which is


another variant of cycle consistency so you translate it from English to French and then French back to English you get you should get back the same sentence and this is a paper that essentially utilized all these three methods and then from there I think I think they get say of the art and supervised machine translation results that are I think ten you know I don't remember the precise


results but that were widely beyond previous stay of the arts and these are some of the Appalachian of showing how many training sentences they use to in order to surpass this kind of system so you can see that you would need probably somewhere between half a million data in order to for it to surpass the flat line which is the unsupervised machine translation results so that's that for


that method and so again like these none of the kind of principles that we have talked about so far a bullet proof but I think what's really interesting is you can seemingly extract training signal out of nowhere by just carefully considering the problems of what are the invariants that you can exploit and especially in NLP like really thinking about how thinking about it knock at a random


The power of yet. I heard about a high school in Chicago where students had to pass a certain number of courses to graduate, and if they didn't pass a course, they got the grade "Not Yet." And I thought that was fantastic, because if you get a failing grade, you think, I'm nothing, I'm nowhere. But if you get the grade "Not Yet", you understand that you're on a learning curve.

The power of believing that you can improve | Carol Dweck


It gives you a path into the future. "Not Yet" also gave me insight into a critical event early in my career, a real turning point. I wanted to see how children coped with challenge and difficulty, so I gave 10-year-olds problems that were slightly too hard for them. Some of them reacted in a shockingly positive way. They said things like, "I love a challenge,"


or, "You know, I was hoping this would be informative." They understood that their abilities could be developed. They had what I call a growth mindset. But other students felt it was tragic, catastrophic. From their more fixed mindset perspective, their intelligence had been up for judgment, and they failed. Instead of luxuriating in the power of yet, they were gripped in the tyranny of now.


So what do they do next? I'll tell you what they do next. In one study, they told us they would probably cheat the next time instead of studying more if they failed a test. In another study, after a failure, they looked for someone who did worse than they did so they could feel really good about themselves. And in study after study, they have run from difficulty.


Scientists measured the electrical activity from the brain as students confronted an error. On the left, you see the fixed-mindset students. There's hardly any activity. They run from the error. They don't engage with it. But on the right, you have the students with the growth mindset, the idea that abilities can be developed. They engage deeply. Their brain is on fire with yet.


They engage deeply. They process the error. They learn from it and they correct it. How are we raising our children? Are we raising them for now instead of yet? Are we raising kids who are obsessed with getting As? Are we raising kids who don't know how to dream big dreams? Their biggest goal is getting the next A, or the next test score? And are they carrying this need for constant validation with them


into their future lives? Maybe, because employers are coming to me and saying, "We have already raised a generation of young workers who can't get through the day without an award." So what can we do? How can we build that bridge to yet? Here are some things we can do. First of all, we can praise wisely, not praising intelligence or talent. That has failed. Don't do that anymore.


But praising the process that kids engage in, their effort, their strategies, their focus, their perseverance, their improvement. This process praise creates kids who are hardy and resilient. There are other ways to reward yet. We recently teamed up with game scientists from the University of Washington to create a new online math game that rewarded yet. In this game, students were rewarded for effort, strategy and progress.


The usual math game rewards you for getting answers right, right now, but this game rewarded process. And we got more effort, more strategies, more engagement over longer periods of time, and more perseverance when they hit really, really hard problems. Just the words "yet" or "not yet," we're finding, give kids greater confidence, give them a path into the future that creates greater persistence.


And we can actually change students' mindsets. In one study, we taught them that every time they push out of their comfort zone to learn something new and difficult, the neurons in their brain can form new, stronger connections, and over time, they can get smarter. Look what happened: In this study, students who were not taught this growth mindset continued to show declining grades over this difficult school transition,


but those who were taught this lesson showed a sharp rebound in their grades. We have shown this now, this kind of improvement, with thousands and thousands of kids, especially struggling students. So let's talk about equality. In our country, there are groups of students who chronically underperform, for example, children in inner cities, or children on Native American reservations.


And they've done so poorly for so long that many people think it's inevitable. But when educators create growth mindset classrooms steeped in yet, equality happens. And here are just a few examples. In one year, a kindergarten class in Harlem, New York scored in the 95th percentile on the national achievement test. Many of those kids could not hold a pencil when they arrived at school.


In one year, fourth-grade students in the South Bronx, way behind, became the number one fourth-grade class in the state of New York on the state math test. In a year, to a year and a half, Native American students in a school on a reservation went from the bottom of their district to the top, and that district included affluent sections of Seattle. So the Native kids outdid the Microsoft kids.


This happened because the meaning of effort and difficulty were transformed. Before, effort and difficulty made them feel dumb, made them feel like giving up, but now, effort and difficulty, that's when their neurons are making new connections, stronger connections. That's when they're getting smarter. I received a letter recently from a 13-year-old boy. He said, "Dear Professor Dweck,


I appreciate that your writing is based on solid scientific research, and that's why I decided to put it into practice. I put more effort into my schoolwork, into my relationship with my family, and into my relationship with kids at school, and I experienced great improvement in all of those areas. I now realize I've wasted most of my life." Let's not waste any more lives,


hi there today we're looking at fast reinforcement learning with generalized policy updates by andre barretto shabo ho diana borsa david silver and doina preco so on high level this paper proposes a framework for reinforcement learning where you have many tasks at the same time and they propose a framework where they learn many policies at the same time that

Fast reinforcement learning with generalized policy updates (Paper Explained)


can or cannot correspond to these tasks and then their argument is that if you now have a new task that you haven't seen before you can easily construct a solution to that task from your old policies basically mixing what you learned about your old tasks and it's a pretty general framework and we're going to look at it in my opinion it's it's pretty cool for certain settings


however i think it kind of breaks down the the more general you go which i guess is expected um of such a framework but uh it's as you can see it's kind of math heavy but we'll get into the examples and um what it's potentially useful for all right so that was it on a high level if you like content like this don't hesitate to subscribe to the channel and share it out leave a like and tell


me in the comments what you think i'm still reading all of them uh so i will see it cool let's dive in so they say the combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision making problems that are currently intractable well they're taking they're talking about you know things like um mostly these game playing ais like go


and things like this so we're this combination of deep learning with reinforcement learning has really shined or shun whatever one obstacle to overcome is the amount of data needed by learning systems of this type so again if you look at these systems like alphago they need a simulator and they need to collect enormous amounts of data um even more so with systems like


the dota ai the openai5 dota or starcraft playing alpha star i think it's alpha star they need so many simulations in order to learn about the tasks because they always start from scratch in this article they say we propose to address this issue through a divide and conquer approach we argue that complex decision problems can be naturally decomposed into multiple tasks


that unfold in sequence or in parallel by associating each task with a reward function this problem decomposition can be seamlessly accommodated within the standard reinforcement learning formalism okay so what are they saying right here they are basically saying that if you have a task let's say you want to get whoopsie from here to here and that's very complicated


let's make it complicated super duper complicated you can basically subdivide that task into multiple subtasks right so here it's like left turn right turn go straight left turn go straight right turn and so on and each of these subtasks you can see the two right turns here might share a lot of common information there could also be tasks that are at the same time like you need to go


forward and jump can be decomposed into going forward and to jump now they're saying is if each of these tasks now has its separate reward function in the environment like for some reason the environment tells you this by the way is task task one and you're gonna get a positive reward if you do a right turn and this down here is task two the the left turn task and you're


gonna get a positive reward if for that task so the entire task state can be decomposed into a vector so in our case here we have maybe a vector with three elements okay the three elements correspond to turn right go straight and turn left and now your this this right here is your reward vector so we're no longer talk in this framework we're no longer talking about


just a reward we're talking about a reward vector now each of these tasks is going to give you its own individual reward so let's say you're here and you're actually turning right this is going to give you a reward of one for this task but reward of zero for the other task okay so the environment will somehow tell you which tasks you you get reward for now there is a notion


where you can map this back to a single number and that is the second thing they introduce here so the second thing they introduce here is this thing they call w so w is going to be a mixing vector w is going to be a vector i will call w right here this is the reward vector w is going to be the vector that tells you your final reward so here we're going to do an inner product


so we're going to transpose this and multiply by w and w mixes these rewards and comes up with your final reward right here so this this is maybe the reward vector this is the reward number how we're going to call this reward number so in this case w would have to look something like this let's say this is an example so the task right here would be to only do


right turns now this is not a really nice example we're going to see some nicer examples later on but you can see that now the environment is specified as a vector of rewards and you can create the specific tasks like turning right simply by adjusting how you mix these different things by this vector w and this is going to be the key ingredient here so they discuss


your general reinforcement learning the reinforcement learning lingo and i think we've gone through this a number of times just very very quickly uh in reinforcement learning you're given these transitions you are in a state you take an action and that leads you to get a reward or prime and you get into a state s prime in the next state they say the reward is given by the


reward function so the reward is purely a function of where you are and what you do and where you get to now most reinforcement learning problems you can actually kind of forget about this part right here because well it isn't it is kind of important but you could um most reinforcement learning problems the reward is simply a matter of where you are and what you do


and this can be a random variable there can be randomness but maybe it's easier if you for now think about the reward simply as a function of these two things so what you want to discover is a policy pi where you input you input where you are and the output is going to be what should you do in that situation okay uh that is a policy and associated with each policy is this


thing called a q function so you can see right here the q function of a policy um is going to be a function of where you are and what you do and this is a bit confusing but it basically means that you are in state s so you are here and you have let's say three options action one action two action three to do now the q function tells you the q function this is s and the a's are


the numbers okay so let's say we plug in the state s and for a we plug in number two what it will tell you is if i am in state s and i perform action number two then how valuable is that for me and value is defined by all the reward that i'm going to pick up from now until the end of time or the end of the episode it depends um but let's say until the end of time


well how much how much reward am i going to pick up from now until the end of time is a bit of a vague not a vague question but a difficult question i can tell you how much i could estimate how much reward i'm going to pick up in the next step because i know what action i'm doing i'm performing action number two but what happens after that who knows so that's where this policy right here


comes in this policy right here says so the full definition of the q function is if i'm in state s and i perform action a right now and after that i follow policy pi what is my reward going to be well now it's well defined so right now you do action a and after that you do whatever action the policy tells you in that specific situation okay so that's the q function and you can pretty easily


see that if you have a q function right if you have an accurate q function you can get a good policy by simply always going with the action that gives you the highest q value because um it's because of a recurrence relationship called the the bellman equation uh this thing right here so your q function basically decomposes into the reward in the next step as we


said plus whatever happens after that and whatever happens after that is just by the nature of how the things are defined is going to be the q function of whatever the policy is telling you so you can get a pretty good policy by always doing whatever action your q function tells you is best this step of calculating the q function is called a policy evaluation and this paper here


is going to generalize these notions um sorry so this is a policy evaluation and then the act of selecting an action is going to be a policy improvement these are just names okay but we need to know them because the paper introduces two new things i'm going to where do i highlight policy evaluation i don't know but here they say this is the policy improvement


okay ah here policy evaluation policy improvement these are the two steps so the first step is calculate the queue function the second step is to select an action and you can see how these things interlock namely we can calculate the q function of a given policy and we can improve that policy by selecting whatever action is best for the q function this paper generalizes this and you can


see that there is a little a little r right here so the r is just a specific way to reference the reward function used right here okay and you can see it here as well now usually we have one policy and one reward right and so what we do is we improve the policy and that leads us to better evaluate the q function for a given reward function and that leads us to improve the policy


now this paper is going to transform this into the following we have many policies so we have policy one policy two and so on until policy i don't know p and we also have many reward functions reward 1 reward 2 reward 3 and so on until reward let's call that r so we have many different tasks right here and we have many policies now in essence they don't need to have some anything to


do with each other for the theory of this paper but i can simplify this a bit of how they see the world so let's say you have an agent and the agent has been trained on simply that first task right here and has been trained using classic q learning reinforcement learning what not and that results in this particular policy and then the agent just from scratch you restarted again


you run reinforcement learning just on reward number two and obtained policy number two and so on so you do this for all these rewards individually okay so you give the agent a new task and you ask it to learn a policy for that task now you're in a situation where if you are have a new task so are new the question is do you again need to train a new policy and the answer for this paper is no


because we have all these policies we don't need to train a new we can simply mix and match these policies that we already know to obtain a good solution for the new task so how does the paper do it it does it yeah it does it in the following it defines the successor features okay maybe it since maybe it's better if we first go to an example so the example they give here is the


following otherwise this i guess this might sound just a bit too abstract okay so you have this world here the agent is the thing here in yellow and it can just move so its actions are move left up right down this this is one step okay in the environment there are two different objects one object is a triangle and one object is a square okay so um there are a number of tasks we can


define right now in this thing so we define tasks according to a reward function so the reward let's say the reward one is going to be um one if if it picks up a square sorry the square and zero else just if it picks up a square on any given step we give it a reward of one it we don't care about the blue triangles okay and then reward two is going to be the opposite it's


going to be one not the opposite but one if it picks up a triangle and zero else so you can see the um good policies right here so pi one is a is a good policy for reward one because it just goes and and collects these red things doesn't care about the blue things just goes and collects them pi two it goes and collects the blue things doesn't care about the red things


okay so let's imagine that you have run reinforcement learning twice once for reward one and once for reward two and now you have two policies okay so you have two policies this will lead to pi one this will lead to pi two and now i give you the third task now the third task is a bit special it's one if you pick up a square and it's um it's zero else except it's negative one


if you pick up a blue thing well the order of these is kind of wrong but it just for visual representation okay so now you're asked to um pick up the red things but avoid the blue things okay pick up as many red things as you can avoid the blue things and again as we said the question is do you now have to run reinforcement learning again in this agent with your simulator using


like q learning or something like this from the start or can you come up with a solution just given these two policies that will perform well on the on this new task okay and we're going to see how they do it so what they do is they use successor features so these successor features um i've done a video about successor features and i'll link to that you can look at that but


essentially essentially the successor features are defined like this and for that we need to know what this thing is right here they simply call this a feature function okay it's very it's very um ambiguous term a feature function is a function that takes in a transition so state action next state and maps it to a high dimensional vector note this is almost the same as a reward function


except the reward function simply maps it to a number now this is mapped to a higher dimensional thing again i wanna i kind of wanna leave out the next state right here just to make things easier on you so a feature here can be many many things but the structure of the features is going to be such that the reward function is going to be this feature times this w


vector so it was a bit a bit not correct before when i said the reward is now a vector the reward of a particular task w can be seen as the inner product between the features and the task vector so w specifies the task and the features well they specify the features in our case it can be it can be fairly simple namely yes i was i was definitely wrong at the beginning so the feature


functions right here is which object do you pick up okay so we define the feature function as 1 0 if you pick up a square and we define the feature function as 0 1 if you pick up a triangle and now you can and we define it as we define it as 0 0 if you pick up nothing and now you can fairly easily see that the reward of each task can be simply calculated by mixing the features


accordingly okay so reward one is going to be simply the feature a 1 0 which is the w vector so i can specify a task by giving the appropriate w vector and now you can see that if this is my reward function my agent can go out into the world if it collects a square it is going to be rewarded right here if it collects a triangle even though the features indicate that it collected a triangle it


doesn't care about it because the w is 0 right here if i now want to give it the new tag the same is true for r2 if and i want to give it the new task r3 right and you remember the reward function right there i can achieve that reward function by simply multiplying the same features the exact same feature functions by this vector right here okay remember there is a slight difference between the


reward function and the feature function in this particular example the idea of the paper is that the feature function can be rich in in expressivity and you know tell you all sorts of things about your current state and the reward function is just a number right and then the the reward is specified by simply linearly mixing these features so the structure imposed by the


paper here is that there are such a thing as a feature and any task can be described by mixing these same features okay that's that's the issue right here so the features are going to be constant across tasks whereas the w defines the task all right so the the goal here is that if you have learned many many things um during your tasks what you want to do is you


want to learn this feature representation that is the same across all tasks and then you want to simply have the w specify how to mix these features to get the reward now of course this is a very strict very very definition not not a lot of things will fall into this unless you make the features like exponentially big of course um however they do discuss whenever a task doesn't fall


into that so i hope you're with me so far this is the first kind of restriction we impose on our worlds that we can tackle with this framework namely that all of our worlds have all of our tasks in this world have to be a linear mix of the same features if that's given then our um then we can derive policies for tasks that we have never seen we can derive good policies by


doing zero learning simply by specifying the task we can have a good policy for that task from the policies we've already learned for the other tasks okay so the reward three is now simply this and yeah notice it's not the same as the reward function because the reward function had one if you pick up the square negative one if you pick up the triangle and zero else so the zero we don't have to


specify here because it's not part of our features right so you can see that the reward function is given simply by that and we can now as i said derive a good policy for this reward by looking at the other policies even though none of these policies has ever learned to avoid anything so it makes it defines these successor features right here so the successor features


is much like the q function you can see the signature is almost the same so as a q function tells you um how much reward you're going to get if you do the action a and then follow policy pi the successor features almost the same thing however it doesn't tell you what rewards you're going to get it tells you which features you're going to get and which features by that we mean the


sum of future features now you can see this some this a little bit this uh it of course it comes from the fact of the linearity up here so it's not really an additional restriction but simply to clarify what this means for your environment your environment has to be able to be looked at in terms of these features and these features they need to be cumulative


again that comes from the fact that it's linear but to see so a feature like i want an an even number of steps or something like this would be terrible uh because and they're going into things like this later but it would be terrible because here we have the sum and um as soon as you if you have a feature that is very high if you have an even number of steps then


um or if you have a feature that counts the steps you will never be able to to do well because if you have a feature that counts the steps it simply counts up and up and up and up depending on how many steps you do and your reward can never be specified in terms of a mix of these features and therefore your successor features are going to be useless but in our case where it's where feature


one is pick up is how many of the sorry after rephrase our feature one is whether or not you pick up a square therefore if we sum it up our successor feature one is going to be the number of this is this is a pound sign the number of squares that you pick up okay similarly our feature too is whether or not you pick up a triangle in a particular step so our successor feature number two is


going to be the number of triangles that you pick up over time you can see that the successor features is kind of the analogous of your q function but it is not in terms of a single number the reward it is going to be in terms of these features which is an entire vector okay and because we've constructed this in a linear way you can also pretty clearly see that the


q function is inherently related to the uh to the successor features you can obtain the q function by simply multiplying the successor features by your task vector w now a lot of you might be wondering where does this w come from and in our initial case we're just going to frame everything as being given right so we're we're given this this w we're we're defining everything from


our god-like perspective for now so don't think all of this is learned by now um yeah all right so how can you now derive this this magical new policy okay so we let's say we have this policy one and we have the policy two and they and you have the this features that you've learned constantly over both tasks in fact your it's given right it this pi function we give it


we impose it that the feature one is whether you pick up a red square feature two is whether you pick up a blue square then we know that the reward functions can be achieved by doing the w so this here your w is going to be one zero and your w here is going to be zero one and we now we want a good policy for task three and we know we can achieve this by the one negative one


w how can we derive a good policy and this is this algorithm this general policy evaluation a general policy improvement so it assumes that you as we said you have many many different many different policy so here you can see policy one where's policy two here's policy two and so on it assumes that you have many different features and therefore many different


successor features in fact you have a vector of them right so here you can see feature one feature two and so on and it also assumes that you're in a current state and you have many actions at your disposal right now action one action two and so on okay so this is all the past you've already defined your features you have learned these policies and now you're given a new w w


new in our case it's this one negative one and we want the best action so we are in state s we are given this w we want the best action now here is a method where we can simply calculate the best action in terms by by not reinforcement learning at all in this new task so by structuring things like this here so what does it really say here it this thing says we are going to evaluate


all of these different cells of this tensor right here so we're going to determine what is the successor feature number two for policy pi one um in state s if i right now do a2 this is very abstract so let's say you're here and action action two is actually going to the right okay so you're here oh this was yellow it doesn't matter so this is so this is action one this is action two


so action two is you go to the right okay you can you can see that this will let you pick up um we'll let you pick up a triangle now here that's action three and so on okay so what's this number going to be so we are in state s as we said and we do action two so action two is going to pick up a triangle the triangle the picking up of a triangle means that our pi for this step


or sorry our five for the step is going to be zero one okay so our successor features this is not the features itself this is the successor features the successor features decompose into the next step plus all the next steps that we can follow okay so all the steps that will come so what are these features going to be is it's going to be the sum over that plus everything that follows


and i can take a little bit of a guess here which means that this number so we only care about feature two right here this feature feature two this number is going to be one for the next step because we are going to pick up a triangle if we do action two but then after that we're going to follow policy one and policy one has been trained to pick up the red squares and not care


about triangles so i'm going to guess that every now and then it will kind of step over a triangle but it won't fall we won't you know explicitly go look for them so let's say the episode was 10 more steps but the board has like 100 squares so and it has like three triangles on it so let's say that's like three-tenths um in expectation okay so this is going to be this is going to


be the number that we're looking for we're doing this for every single one of these cells okay this this thing is going to do for every single one of these cells and this is very similar to evaluating q functions except we're evaluating an entire vector right here that's the difference to simply learning many q functions so if you were to evaluate only a q function then you would only


have this first matrix this first block right here okay but you have feature one feature two and so on so you calculate everything in terms of these features and then by linearity you can mix it with that vector so in our case this is going to be the one negative one which will give you the q functions right from what we've seen before you obtain a q function by simply


mixing your successor features with your um with this task vector and if you have a q function you can pretty easily determine uh which action you should take now you have here a q function with respect to every single policy but you can simply take the max right so the max across all of this will determine um will determine so you could take the max across all the


policies which will give you the q function for a particular action over all policies that you consider and then you can simply take the arg max of that and determine the action you should take okay so it's a pretty big evaluation but if you do this that means you don't have to do reinforcement learning on this task it simply determines which action right now is the best given everything


that i know from these old policies about the task and that's not going to be like the optimal policy uh per se but it's going to be one policy that's pretty pretty good and you can actually prove some things across that so they do this right here and you can see that here is what q learning does on this new task of picking up the squares and avoiding the triangles q learning


takes a while to get there however if you do what they are suggesting and you know you give the w you can supply the w almost from the beginning you see right here almost from the beginning it is at a high reward now q learning surpasses it eventually but um it's pretty impressive that without doing any learning you are immediately good right now the caveat here of course is that


they already need these policy pi 1 and pi 2 given to the algorithm and that comes from previous reinforcement learning trials and they say that they give these trials as many steps as q learning uses so they give them this these amounts of steps on these other tasks so the comparison here is a bit shaky if you ask me but the point made is that if you have a new task right now


you can obtain very good solutions uh and you don't have to do anything okay and these solutions can be the basis for new reinforcement learning right you could start q learning off right here and then get here much faster potentially and so on so the next objective right here is that now we have defined the tasks and we had we know what these features are and we know how


to mix these features as imposers of the task so what happens if we only have the reward function we specify the task only in terms of the reward functions but we're kind of looking at the features and we're like agent please figure out yourself how to to apply these features in order to make the reward high and that's what this thing is right here this gp and gpi with regress w so you


don't no longer tell it what the w is um it needs to infer it through reinforcement learning right and it's not really reinforcement learning but what it does where is it yeah it simply it because all of this is linear and this thing here is given so always remember this thing here is given and these are the rewards that you obtain you can simply do a regression


to figure out the w of the task now that's going to take some time but as you can see right here it is going to take um a lot less time than sim than doing q learning from scratch notably because you have good features so this is some this is this gets closer and closer to transfer learning right if you imagine that this right here is your pre-trained neural network and


you simply learn the last layer of it um you you freeze this you do transfer learning fine tune the last layer here we are so um it gets closer and closer and you'll see this trend right here so it's pretty cool what you can do but basically i think it's a lot of math around a framework and the more and more you relax the kind of impositions uh that they need for their framework the more it


gets back to simply well we do reinforcement learning at least in my um estimation so before we look at that this here is a pretty pretty cool experiment where they they look at how the how the different tasks can be achieved if you give different policies so you'll have noticed that we have always given these two two tasks one zero and zero one these were our tasks that we trained on


and then one z negative one is task we evaluated on okay and you might object and say wait a minute these these two tasks you know they're pretty good as let's say pre-training tasks because and it's basically the standard basis right and any other task can be mixed from those so these are orthogonal vectors in this vector space so you're being pretty generous to this


system what happens if we're not as generous so that's what they do here so they have different um policies and they evaluate how much you can learn with these different policies so the way you have to read this diagram is right here is going to be the one zero axis as they well they label it right here and this is going to be the zero one axis and this is evaluation


so every direction on this circle defines a task for example this task right here as you can see is going to define the task of picking up both the squares and the triangles right whatever you pick up you get a reward however the task down here is going to be please pick up the squares but avoid the triangles at all cost okay and now they're going to look what happens if we supply different


policies to choose from remember we're in this situation we're again in this situation where we give everything and we give initial policies we give the task vector and now it's about deriving a good policy just from looking at the old policy so no learning as a baseline you have q learning which into a given direction um tells you basically how how long q learning or takes or how


far q learning gets with a given amount of steps indicated by this one two three four and so on um yeah you see i think this is this is this in how far q learning gets with these amounts of steps is the dotted lines right here so q learning gets this far with 10 to the i don't know 4 and then this far 10 to the 5 and so on so these are comparisons you can see


that on the outside q learning is going to beat this these methods but our hope is going to be that of course if we have this zero shot generalization it's much better than running q learning for really long if we get close to it so the green thing is what we've already seen policies one and two will give you a fairly you know good um fairly good extent right here so what


does it mean it means it can solve it can solve pretty much everything from here here this task this this task this task it kind of falls off once we go down here so once we go to the avoid section it sort of falls off because it has never learned to avoid now still we can of course do the avoidance by simply imposing a negative collection but negative collecting


and avoiding aren't exactly the same thing in these um in these environments right because avoiding can also be going really close to something but not hitting it while collecting it's not the inverse of collecting the inverse of collecting would be like run away as far as as far as possible so we can expect that we've only ever learned to collect we're not going to be super good at


avoiding um then the other extreme is when we give policies three and four and i haven't told you but you can see it right here uh policy three is explicitly to collect one and avoid the other while policy four is the opposite right here avoid the squares collect the triangles and now this policy this policy is should be pretty good on all of the tasks in between as you can


see it has the biggest extent right here and that also makes sense by the way there's nothing down here because the task of avoiding both things doesn't really make sense because you can just stay where you are because there are also these these squares where there's nothing but you can see that the mixture of those is quite potent so already we can see even


though these span a bases in fact an orthogonal basis as much as these because of the nature of the features that we define for the task they are not equivalent in mixing after so we can be more generous we can also be less generous if we only provide policy five and policy five is simply to pick up to pick up both objects then we're going to have a pretty hard


time when it comes to avoiding things so you can see it can do fairly well picking up the various things in a positive manner but as soon as we cross this line into the like this horizontal line into where it's about avoiding a particular object um it's not it's not the the choices of actions we have from policy five aren't going to be super good at that and um they do another they do another


thing right here so that the left thing is where they say it's important which policies we provide and the right thing they want to say something like it's important um so they want to say if we provide more policies that can be advantageous because we basically have more options to choose from okay so now they start off with policy four and policy four is simply


avoid these squares collect the triangle you can see it performs fairly well over here where it's all about avoiding the uh squares and collecting the triangles as soon as you get into you know collecting or even here the opposite directions it's pretty bad right that's the red thing and now they add policy two to policy four so policy two is going to be also


to collect um the the triangles but to just neglect the squares and that will also do a bit better why does it do better because it's better at collecting uh because this policy here also needs to avoid um and this policy here doesn't care so in the regimes where it's better to not care than to avoid adding this policy adding these options is going to be good and you can see that


there's a general expansion here as we add more policies however i want to point out that for example here this black thing which should be technically superior to the blue thing because it contains as you can see here all the policies that the blue thing contains plus another policy um i don't i don't know if my vision but i'm pretty sure here the black thing is inside the blue


thing uh so that means there can also be a disadvantage to adding more policies right here because maybe you got you have too much to choose from and so right here what we say is we add a policy that is all about collecting the squares and it is performing it is actually decreasing the perform the addition of this is decreasing the performance on tasks where you have to


avoid the squares which i'm not sure if if that makes sense again the opposite of collecting isn't avoiding but i'm just pointing this out and this isn't really mentioned in the paper the paper simply says see we add policies and therefore we are getting better i'm not i don't agree with this given these results or maybe it the plotting the plotting is bad all right so


they say okay more policies better which i disagree with they also say ho we can as as much as we can regress the w right we regress w we figure out the task we can even learn the successor features okay we can not the successor features um the pi functions that lead to the succession successor features and you can see if you do it with the true w you're really good at the beginning


if you do it with a regress w we can see that before you can you so this is the small version of this plot right here this is like this uh section i think yeah you know you improve however we can also learn this pi function we can also learn the features where if we're not given the features maybe we can learn the features and they say well we can do this with


but also by regression so here what we can do is we can find the function that minimizes the function and the w along with it that minimizes this error right here okay so you're finding the function and the w that that matches this error and this now really is like learning a neural network i mean you know um so i get i get it you have the i here and the w doesn't depend on the i


and so on um but you're getting more and more back to actually simply learning non-linear functions mixing them linearly right here and i think that's going to be kind of the crux of this method uh the fact that the more complicated your problems are the less you are going to be able to do this kind of stuff and they even go as far as to say well what if like before


we the reward is actually something like whether or not you have collected an even number of triangles or squares then they say well you can simply not have a single w but you can find a function w and now the policy is a function of the function of w and you can do potentially the same regression problem but as you can see it gets so now you um this right here is going to be a


function of state and so you can see that it more and more it simply goes back to basically q learning again the only difference here is that you have this intermediate features but i think you can simply view this let's say as a hidden layer in a neural network i get it some are held constant across sums and so on but you know i i like the method in terms of


um you know in terms of the analysis so if you are given all this stuff it seems pretty cool that you can derive new policies uh it's implication for lifelong learning they say look here um you have a bunch of tasks in your database that you've already learned on your agent is going out into the world it faces a new task it can use this thing you can use this thing to obtain a new


good policy for that task it can then use reinforcement learning rl to refine that policy and then it can simply save that policy into the database so it keeps expanding and expanding this thing so it keeps adding rows and rows and rows right here of new policies that it's learned over the course of its life so once it's facing a new task it can just kind of draw from its experience


and derive a good initial solution however uh the actual analysis only works i feel in quite limited circumstances and if you want to relax these limited circumstances then you need to basically regress and regress and regress away from away from their setup and i'm not sure i'm not sure where this is going to go if this is going to be a general framework for people


it seems like it because it's pretty easy but then also it seems like most of the world doesn't really fall into this category in fact this divide and conquer approach um i'm not sure but from divide and conquer i almost imagine something like you subdivide and subdivide and subdivide until you know you are at some kind of basic task they still only go for you know single tasks like this here


the tasks are somehow in sequence and i'm not i think we should really think about hierarchical rl now this can be a good first step right here but most hierarchical rl even the ones that specify themselves as fully hierarchical like we can do many layers they rarely go above two layers or three like like one one metal layer and one actual layer like this one right here uh they rarely


go further maybe they go two layers but that's about it um i've seen very little in actual hierarchical or dividing conquer reinforcement learning just because it's so hard to train yeah all in all cool paper and if you want to get it into the math a little bit i think it's pretty easy math uh once you kind of set your goals on what it's actually meant to achieve um if you just read


from the beginning all these reinforcement learning papers it seems a bit like why why are we doing this right you just oh we define this we define that we define this and you're a bit like i yeah but why so often it pays in these papers to go at the end to the examples and then uh come back to the theory knowing what they want to achieve all right that was it for me long rant


Thank you. It's a great pleasure to see all of you here tonight. The festival has had conversations about science and religion over the years. Perhaps some of you have come to some of those. And oftentimes, there are two sides represented in that conversation and sometimes the two sides, you know, science and religion, sometimes they're contentious, sometimes they're harmonious,

The Believing Brain: Evolution, Neuroscience, and the Spiritual Instinct


but tonight we're doing something differently. We really only have one side here tonight. So the group of people who are going to come out for this discussion, they're all scientists. They all come from the background of science, but our goal is to see if by walking this one side, this one trajectory of science, we can gain some illumination into the other side.


Into the side of religion side of faith. Before I bring out our esteemed group of panelists, I just want to set some context and to do so, I'm going to begin with something which is presumably familiar to many of you. So this is what a beautiful midnight sky brimming with stars looks like in New York City. Now, I also have a little cabin, Upstate New York in the Catskill Mountains and when I'm


up there and it's a nice, dark night sky, I can look up and see something that looks just like this. Maybe not just like this. This is takes a Hubble Space Telescope, you know? But you get the idea and when you see a wondrous sky like this, you can't help but ask yourself, how does it all work? How did it all come to be? And I have spent part of my professional life trying to advance the scientific understanding


of some of these questions, and because I work on the more mathematical end of physics, when I look up, I tend to see order and harmony in a peculiar language. The language of mathematics, a language of symbols. But, many others, when they look up at a sky like this, it brings to mind other things, right? Ideas of soul, of eternity, of divinity, of God. And for some, that kind of talk, it feels kind of loose or vague.


For some, it's even off-putting. But when you look at the data, you see something utterly remarkable, right here in the 21st century, the modern technological age and we have long since cracked the atom, explored the surface of Mars, detected gravitational waves and so much more. There are still many of us who are believers. So if we look at some of the numbers, about say 2.2 billion of us identify as Christians.


About 1.7 billion is Muslims. Hindus, Buddhists, that gives us another two billion, plus, if we throw in my little tribe, it's about 14 million, right? And then if we add in the atheists, this takes us to one and half billion which is just to say there's a lot of people on this planet who would look to the heavens and think of heaven. So if aliens were able to sweep down toward planet Earth, and let's say they had some


wondrous equipment that allowed them to detect religious belief, to give us a kind of heat map of faith, this is what our planet would look like. You could probably work out the color scheme for yourself. Blue is Protestant, Red is Catholic and so forth. You get the idea. We are a religious planet. Personally, I am not religious in any conventional sense, but I do consider myself spiritual


and I certainly do consider myself curious. One thing that I have certainly gotten ever more curious about is why do we believe? Now, the simplest answer is we have religious belief because what religion tells us is true. That raises a whole lot of challenges that we're all familiar with and perhaps the most relevant for tonight's discussion is there are over 4000 distinct religions practiced


on Earth and if we just take one of them, say we parse Christianity a little more finely, there are over 33,000 distinct denominations. They can't all be right. So the natural supposition is that at most one of them is right, which would mean that if Sarah here, happy in her own beliefs, she denies therefor the beliefs of all others, like Terrik over here who again, happy in his own faith, denies the validity of all


others and that goes true for Pim and for Ofryim and also for Amalyia and it even holds for, say this guy over here, Richard, who not only denies in the validity of all other beliefs, he denies the validity all beliefs. We may be a believing planet, but most of us deny the validity of most beliefs, which means that even if Sarah holds to her religion because it is true, she still needs to explain


why everybody else holds to their own misguided faiths. And that holds true for everybody else. So this takes us to a simple but remarkable conclusion. Normally, the discussion of science and religions, you know, it all comes down to what's right, what's wrong, what's true, what's false. But here we see that even if a given religion is true, it hardly changes the question at


all. We still need to ask why it is that so many of us have a tendency to believe. We have to ask yourself, what is it about the human species that drives us to find order and meaning and, in particular, to find the turn toward the supernatural so utterly natural. 1936, this guy over here, Albert Einstein wrote a letter to a school girl named Phyllis who had asked Einstein about his own religious beliefs.


Everyone who is seriously involved in the pursuit of science becomes convinced that some spirit is manifested in the laws of the universe. One that is vastly superior to that of man. In this way the pursuit of science leads to a religious feeling of a special sort which is surely quite different from the religiosity of someone more naïve. Much has been made about Einstein's use of this phrase, religious feeling, but his later


writings made very clear that he was speaking of an abstract spirituality, not a conventional religion. The word of God is, for me, nothing more than the expression and product of human weaknesses. The bible a collection of honorable, but still primitive legends which are nevertheless pretty childish. No interpretation no matter how subtle can, for me, change this.


Charles Darwin, the Father of Evolution by natural selection, he allowed for the possibility of God. I have never denied the existence of God. I think the theory of evolution is fully compatible with faith in God. I think the greatest argument for the existence in God is the impossibility of demonstrating and understanding that the immense universe, sublime above all measure and man, were the


result of chance. At the same time, Darwin also noted that a religious belief, a religious sensibility could emerge from the interplay between biological and cultural evolution. Nor must we overlook the probability of the constant inculcation in a belief of God on the minds of children, producing so strong and perhaps an inherited effect on their brains, not yet fully developed that it would be as difficult for them to throw off their belief


in God as for a monkey to throw off its instinctive fear and hatred of a snake. The Dalai Lama has his own iconic perspective on these issues. Both Buddhism and modern science shared a deep suspicion of any notion of absolutes, whether conceptualized as a transcendent being, as an eternal unchanging principal such as soul, or as a fundamental substratum of reality.


Both Buddhism and science prefer to account for the evolution and emergence of the cosmos and life in terms of the complex interrelations of the natural laws of cause and effect. From the methodological perspective, both the traditions emphasize the role of empiricism. In the Buddhist investigation of reality, at least in principle, empirical evidence should triumph over scriptural authority, no matter how deeply venerate a scripture


may be. Years ago, I had the pleasure of sharing the stage with the Dalai Lama in an event that took place down in Texas and I had an opportunity to ask him a question. The question is asked was, I said, "Look, there are all these books out there that make the case that what we're doing in modern physics is somehow a recapitulation or a reflection of ideas that ultimately find their origin in eastern religious thought."


So I asked him, "Is this true? Is this your perspective?" And he very forthrightly said, he said, "Look, when it comes to questions of consciousness, that's where we have something to offer science." But he said, "When it comes to understanding the fundamental laws and the particles and all that detail about how the world actually works," he said, "We need to look to science."