And it's gonna see what are the cue values for this specific combination so for action two maybe it's like a zero


Q Learning Intro/Table - Reinforcement Learning p.1

It's a two and a one okay, and these are the cue values, so then it's gonna look


What is the highest cue value well?


It's a two so it's gonna perform action number one because it number one has the largest cue value so


over time at least initially the agent is going to do a lot of exploration


So it's just going to pick a random value


But then over time as it picks these things and eventually reaches a reward then using the Q function for


This lovely thing that we'll probably talk about in the next video


It will slowly kind of back propagate that reward to make for higher Q values for the actions that when chained together


Lead to that lovely reward, but for now we need a rent


We just need to initialize this cue table so later we can update this cue table


So the way that we're going to initialize this cue table is we're gonna say cue underscore


table is equal to


Oops


NP dot random dot uniform and then we're gonna say low equals and we're gonna start at negative


2 high equals 0 and again these are two variables that you're gonna want to probably tinker with


given different environments and


And that's why in one of the subsequent tutorials


We will have to talk about like how do you analyze these environments because it's not you know


There's no tensor board to pull up and check the loss


so anyway


These the the the intuition behind why these should work is the rewards that we're gonna be getting


Basically when we print we even printed it out here. Oh, no we only printed out the new state


Since it's since it's part of what I'm trying to yak about right now. Let's just I'm gonna comment that out and


I'm gonna print


Reward and then new state so now you can see all of the rewards. I'll do that in the console


So coming over here, we can see that the reward here is always a negative one and again


To your agent. It doesn't really matter all that much, but to you it might and that


So basically it's good the reward is gonna be a negative one until you reach that flag at which point your reward will be zero


So so yeah, so so that's how


Once it finally reaches that flag it gets a zero reward which is higher than negative one


I guess I think we should get a bigger better reward personally


But once it reaches that flag, and then we back propagate those that chain of events basically so anyways


we can see here that the rewards are always negative one until a good thing happens, so we


Can initialize the table into the negatives because we're pretty confident that that's where like the values will eventually lie at about negative one?


Or maybe not negative one, but in the negatives for sure


so


So it just kind of makes sense intuitively to start it there


But again you'd want to be you'd probably tinker with a lot of starting values to be honest


And then give it enough time it should figure it out anyways, but anyway


Coming back to our cue table


So our low is going to be negative two our high is going to be zero and then our finally our size is going to


be equal to


the discrete OS size


Which if we come over here?


That's a 20 by 20 so the table is 20 by 20 so when we do make a 20 by 20 table of this discrete


observation space size that


contains every combination of


Observations that we might find so every combination of


position and velocity


then


We need values for every single action


That's possible so the way that we're going to do that is discrete observation space size plus


and


dot oops and


dot action space dot n


So this should make us a 20 by 20 by 3


So it's a three-dimensional table


But basically you can just think of it as it's every combination of possible environment observations


times three


Actions that we could take and then inside of those cells or whatever you want to call them


You're going to have your three at this point random cue values between negative two and zero


But over time through this agents exploration and exploitation


Those values should get slowly tweaked and optimized based on that


But don't worry we'll write that in Python and it makes so much more sense at least to me in Python than it does in


That stupid formula, but for now. Let me go ahead and just print


Qtable dot shape and let's make sure it's what we expected NP is not defined. Let's go ahead and import


import numpy as NP we'll try one more time 20 by 20 by 3 and then if you wanted to look at it you could but


We're not really going to gleam anything from it. It's not going to look very meaningful to us. Just understand


It's every you know possible observation combination


And then the q values for at every possible observation combination and every combination of or every possible action at every


possible combination we have a random starting Q value


okay, so


I think I'm going to cut it here and then the next video we will talk about how to actually start training this model


iterating and going over these


the steps per episode and all the episodes and


all that so with that a shout out to my


Longest term channel members these are people who have been with me for a year or more. That's incredible


I can't believe it's been a year that I've had these but stenos co or stenos co


Uh my longest term member. Thank you very much. You're an awesome human being


Misk njr magali and wheatgrass. Thank you all so much for your long-term support. You guys are awesome


And you're not just supporting me. You're supporting


the free education for people who may not be able to afford it otherwise, so


That's really cool. You guys are amazing people. Thank you so much for your support


If you guys want to support the channel you can click on that blue join button


I know some people don't have access to it for some reason, but


If you have it and you want to support the channel you can click on that right there


Uh otherwise if you've got questions comments, whatever about Q learning, please ask them below


I can't tell you how many Q learning tutorials. I ended up having to go through before


slowly figuring it out and


um, I just can't I hope that my tutorial will be uh


More useful than all the other ones that I found myself on so


So if you find yourself lost or confused, please ask. I definitely want to help you understand Q learning


I think it's pretty cool. So anyway in the next video, we will hopefully finish up our agent actually have it


Drive it on to that flag every single time


Auditing Radicalization Pathways on YouTube

Hi there, today we're going to look at auditing radicalization pathways on YouTube by Manuel


Horta-Riverio et al.


So this paper is a bit different than the one we're usually looking at, but since I'm


a YouTuber and this is in the kind of a data science realm, I thought it fits neatly.


So yeah, we'll have a look and this is mostly going to be an analysis and my opinion on


it.


So I'll take that for what it is.


This is, in my opinion, a paper where you can see very well what what it looks like


when you deceive yourself.


So when you have a hypothesis of something and then only collect data that matches that


and you don't think like you don't think of simpler solutions for that explain the data


and therefore you don't think of experiments that could differentiate the simple solutions


from what you propose.


So it's a good example of how you can kind of trick yourself into believing you found


something and this isn't, this isn't now about YouTube or anything.


This happened to me so many times.


Always pays off to take a step back and say, is there a simpler explanation for what's


happening?


And this is what I think is exactly happening here.


So I'll present to you their hypothesis and then I'll present to you my kind of what I


think is going on and a model that explains the data much much much easier and simpler


and actually better.


So let's let's dive in.


This paper basically claims the following.


So on YouTube there are channels and channels are you know independent channels they make


videos and you can actually arrange these channels.


So each dot here is a channel.


You can arrange these channels in kind of a network and two channels you can claim they're


connected and they can be a connection strength or whatever.


For simplicity they can be connected if for example their topics are similar if they reference


each other if they are recommended by YouTube from each other if they have the same users


watching those same channels or the videos of these channels.


There are a number of metrics where it could you could you could make channels connected


but all of them will turn out similar like will give you the similar structure of channels


being connected.


Oh that's connected twice.


So so so you can kind of build a graph of how these channels are connected and what


you can do then is you can you can cluster them.


You don't have to build a graph to cluster them but you can cluster the channels and


what will emerge are parts of the graph that are very well connected.


Right here this might be connected with this and with this parts of graph that are very


well connected and are kind of within well connected within and more sparsely connected


to to others like also have a larger distance in between them.


So if you start out from one channel and you kind of watching recommend videos and recommended


channels and so on you'll you'll stroll along here.


You will get much faster to these things than to the other things.


So these these are called communities usually in these kind of network social network analysis.


So on YouTube you know there is a community for makeup.


There's a community for sports within sports.


There is a community for soccer.


There's one for basketball and so on.


So these are all these kind of communities that you can discover by clustering.


This paper mainly deals with three communities namely the first of all is the IDW which is


the intellectual dark web.


They discuss this here.


So the intellectual dark web is they describe as a group of individuals that are in a rolling


conversation with each other about topics that are let's say usually kind of difficult


to to talk about such as gender differences or intelligence research in certain areas


or even you know regular politics.


But kind of the intellectual dark web are a wide variety of people that basically are


conversing with each other about topics.


The description is a bit vague but the the main aspect is conversation and maybe topics


that are kind of on the on the edge of what's acceptable to talk about.


But the opinions range widely on these topics.


The second group is the alt right and the alt right here is kind of the the they're


they're defined as ethno nationalists.


For example here is an example to the fringe ideas such as a white ethno state white supremacist


ideology and so on.


So specifically ethno nationalists nationalists that I think nations should be organized too


along the lines of ethnicity.


And the goal of the paper is actually to to show that there is a kind of a dangerous pipeline


on YouTube that will drive people to the alt right and drive people into these radical


ideas of the alt right.


Kind of in between is the alt light which is here defined as civic nationalists which


is simply as I understand it means that people should be organized into nations not along


ethnicity but just should organize themselves into sovereign communities and it will be


more of your libertarian classically liberal people whereas the alt right would be more


of your let's say authoritarian right right wing person.


So these three communities they have a fourth community which is a they call a control group


and the control group consists of what they say are kind of mainstream channels on YouTube


simply to differentiate them from these three and to see what's going on with them and if


there is a difference.


So this is this is kind of the setup and as I said the hypothesis is the following.


People go on YouTube so YouTube is here YouTube people come on YouTube they go around right


they explore a bit and all of a sudden they find IDW videos.


These are recommended by YouTube on a fairly regular basis right.


That may mean they're interesting people find it they find it interesting and then there


from the IDW there are recommendations and links to the alt light and the alt light are


still so as I read this paper there is kind of a an undertone kind of the IDW and the


alt light are still okay like they are they discuss ideas that are you know sometimes


political and so on but the real the worry is the alt right the kind of radical right


wing ethnic nationalists and I mean yes the formulation I can I can agree with and then


they claim so you find the IDW that they have links to the alt light or links I mean recommendations


and so on and from the alt light and to a certain degree also from the IDW you can then


find the alt right.


So even though a user that goes on YouTube at first isn't likely to find the alt right


videos because it's fringe it's extreme and so on by through the YouTube recommendation


algorithm basically by going to the IDW finding this then from there they'll find the alt


light and from there and from the IDW they will then find the alt right.


So they claim that there's this pathway of radicalization here that kind of pushes people


towards the alt right and that's their their hypothesis and they claim to they have evidence


to support this and I claim that there is a simpler solution namely so first of all


let me state I don't like the alt right I think their ideas are despicable I should


go without saying though I have said it now so you know just as a disclaimer I'm not defending


anyone here I'm simply saying this paper has a simpler explanation for their data namely


what I think is happening here is YouTube again is a channels each dot here is a channel.


Channels can be clustered as such right there as we saw before I'm just drawing more of


them right now.


Channels, channels, channels, channels, channels, channels.


So what I think is happening is there is a control group what they call the control group


it's over here it's large control right it's a bunch of channels then which is kind of


mainstream media then over here there is let's say alternative media where all of these three


groups belong into so at some point you will have the IDW then maybe a bit further away


from the control group but very close to the IDW would have the alt light and very close


to the two maybe here you would have the alt right right but so notably the in my model


the IDW and the alt light are kind of close together they are in terms of comparative


distance so if you cluster these channels by let's say audience or topics or and so


on it will it will turn out that all of these three are far far away from the control group


those two are very close to each other and then there here there is some distance but


how much you know how much distance is a question but of course it's going to be smaller distance


than the distance to the control group here I mean I could I could draw the alt right


maybe a more accurate picture would be something like this alt right here so whatever I mean


it doesn't it doesn't matter that the details but the distance here is smaller than the


distance to the control group right and in in this model a second thing is also important


namely the alt right as you can see here is much much smaller than the IDW and the alt


light and these again are much smaller than the control group and this I think accounts


for most so the distance relations between these and the the size of the of the clusters


account for most so that with size I mean mainly channel number of channels and also


audience this accounts for most or most of the data better than their model so just keep


this in mind right and my model of course doesn't include any kind of pipeline that


they suggest so first of all they go ahead and they say all right we they collect channels


so they collect data for this and you know we could go over how they collect the data


and criticize that and so on they do human annotation and they start from already published


reports and so on which themselves can be criticized I don't I'm not going to go into


their data collection methodology it can have mistakes but then any collection methodology


can have mistakes what they end up with is a number of channels and here are the top


channels from each category and you can as you can see alt right alt light intellectual


dark web and control so already here you can see pretty clearly the model I have in mind


namely and they acknowledge all of this by the way look at the size of the alt right


channels the biggest ones compared to the size of the alt light and the intellectual


dark web there's much much smaller in number of views and then compare this to the size


of the control group the control group again is again larger than the other two groups


so just keep it in my second thing to keep in mind look at these channels maybe you know


some of them Joe Rogan Sargon of Akkad of these Paul Joseph Watson sticks hex and hammer


these are youtubers like these are individuals making YouTube clips creating content for


YouTube being on this platform whereas if you compare it with the control group what's


here Vox GQ wired business insider these aren't youtubers these are these are websites or


traditional media companies or their their own kind of blogs and so on that have a YouTube


channel where YouTube is one of the outlets of this media company so I think there's a


there's a giant discrepancy here in the control group that can explain also some of this data


that you see so keep that in mind I think the control group they say they don't try


to capture the user dynamic with the control group but I think that there's many problems


with this control group including the fact that these are these are kind of traditional


mainstream media that just have YouTube as an outlet and moreover a lot of these like


Vox or Vice they are clickbait media and rage bait media that it has worked for a number


of years but it's the algorithms are becoming more attuned to kind of clickbait and these


are crashing fast whereas the kind of more youtuber people they are they're not susceptible


to that much to kind of the the abolishment of clickbait all right so these are this is


the data they have all these channels they have all these videos and they first of all


give some some some stats on it here you see on the bottom is always the year so they do


this over time and you see the active channels which are channels that have uploaded videos


in some time see the control group again is larger but has started to flatten out in the


last few years whereas the these communities they are relatively flourishing another interesting


point is that the paper somehow tries to tie this to the election of Donald Trump in 2016


but I again I think this is just kind of in there to to gain relevance a lot of these


kind of trends and so on you'll see already start before that so in so the start of the


rise here if you see these these bumps here and so on a lot of them start before 2016


so as we go through this make up your own mind of how much this is actually tied to


the to the election or not I think it's much more the years when kind of clickbait started


to go down as a as a business model never mind though so the active channels growing


though the control group not growing as much videos published even though the control group


isn't growing so much they still publish the the most videos but you can see generally


the site is growing generally YouTube is growing like counts and here you see something interesting


starting to happen namely these communities especially the alt-light and the intellectual


dark web they're starting to catch up and this is one of the things that the paper also


states is that if you look at for example comments per video this the light and the


intellectual dark web outperform the control group vastly right that also if you look at


views per video and likes per video the the control group simply don't have an engaged


audience which I think first of all is because they produce clickbait second of all they're


just not that interesting and third of all they're not youtubers like this isn't their


thing they're just simply an outlet but yeah so that's that's kind of a one just kind of


a bunch of metrics that the that they show here the next table is a bit more interesting


in the next table they do user intersection so what they do is they collect all these


videos and then they collect all the comments of these videos and the comment of course


always comes with a username you need to be logged into YouTube to make a comment and


they see which users comment on multiple videos or videos of multiple categories and then


they can look at aha okay if a user how many users of category a also comment in category


B and vice versa so they have two metrics here jacquard similarity which is for for


two communities a and B users commenting number of users commenting on a and B divided by


number of users commenting on a or B and the second the overlap coefficient is number of


users commenting on a and B divided by the minimum size of a and B they say that the


overlap coefficient is more useful to compare communities of different sizes so we'll look


at that the top graphs are always always jacquard difference and the jacquard similarity and


the bottom one are overlap coefficient the first graphs though are number of commenting


users per year and you already see that even though the control group has much more views


and probably much more videos much larger the comments don't so the again the the users


of the all light and the intellectual dark web are much more engaged also comments per


user this is the cumulative distribution function most people that comment on the on control


group videos maybe comment once and then but but the these other communities take comment


more self similarity means year after year so always compared to the year before how


many users are similar so how how well do these communities retain users and you can


already see here the control group is actually very bad at retaining users it does have this


overlap coefficient high but it has the jacquard self-similarity low which basically if you


think of the formula of the jacquard similarity means that the this number is small and this


number is high which means that a and b are very disjoint which means that the last year's


users aren't this year's users basically so they they constantly have to appeal to new


users because they're losing old users because well i guess they're boring whereas the alt


light and intellectual dark web are much more are much better at retaining users interestingly


the alt right not as good as retaining users as the other two this could also be an effect


of size like if your community is smaller the users might wander away more quickly but


i think this already speaks against the radicalization pipeline if the if the alt right if youtube


was radicalizing people towards the alt right we i think we would see a the alt right being


on top of user retention then here they have intersections between communities so green


here is alt light and idw while the blue is alt right and alt light and the other blue


is alt right and idw so basically the green is alt light and idw and the blues are the


other two and we see that the overlap in terms of overlap coefficient is similar the overlap


in terms of card similarity the alt light and the idw are very much more sharing users


which in the picture i painted makes sense if you think my model is valid my model explains


this very well in that these two communities are quite close together therefore share a


similar user base the alt right smaller and a bit further apart therefore not as similar


though more similar than the control group which is the last graph the last graph is


sorry the last graph is how similar are these communities to the control group and here


we see the idw and the alt light kind of similar um the alt right not as similar though in


the overlap coefficient they're about the same um so the the paper here claims oh look


at look at the the similarity this is definitely a radicalized so they they don't claim yet


this is radicalization pipeline but they they claim that there's a higher similarity if


you actually look at the numbers it's not it's not so i mean here you're around the


50 uh similarity and here at the end you're also around the 50 similarity with the control


group so this is within these groups and this is here with the control group um also here


you're if i look at the kind of mean here you're at whatever uh 20 18 percent and here


you're also you may be a bit lower but you're also going towards this what it looks to me


like rather than there being a radicalization radicalization pipeline if you look at the


shape of this um and kind of where it starts in 2013 2014 it starts to go up here and you


look at the shape of this it's simply the same shape delayed and i mean there's no reason


why this graph wouldn't go up here um wouldn't go up here in the future and reach the exact


same numbers as here it seems that the graph is simply shifted which makes total sense


if you think these communities um are i'm going to draw the same picture here all right


idw light and over here control if you think they're they're they're like that if you simply


think well youtube is growing users are growing users are starting somewhere here and then


spreading out pretty much randomly like they're spreading out spreading out spreading out


users start here spreading out users start here spreading out here spreading out everywhere


users just kind of there's a diffusion process going on not in a particular direction like


they claim if there is just a diffusion process going on what would you expect you would expect


users that started here to reach the idw and all right much sooner than they reached the


control group but ultimately as the diffusion continues all users will have commented on


most videos if you run youtube infinitely and these numbers would go that's why the


numbers go up right if you just let it go the diffusion process will go along and it


simply takes a longer time to go from here all the way over here then it goes then between


these communities so to me we're looking at a simple diffusion process here that is shifted


in time and that explains very much the discrepancy in number but also the shape of the curve


that is exactly the same but shifted their model does not explain the shape of the curve


they simply say well here it's 75 percent and here it's only 50 percent that means that


these communities are kind of shipping users towards each other so i think the explanation


is easier then so they claim this does not alone kind of show that there is a pipeline


what they now do however will show that basically so they claim this is the experiment that


really shows that there is it is pipeline so what they do is they define what they call


an infection so what they say is okay we are for example this this row here we're taking


users that are alt light users at the beginning in this time so basically they only comment


on the only comment on all light videos during this time right so discard all users that


comment on anything else just retain the ones that only comment on all light videos during


this time then we're going to follow them over time and see how many of them have at


least one comment in an alt right video so this is only directed from the community over


here towards the alt right and then they call a user infected specifically if they comment


on one or two alt right videos they're lightly infected if they comment on three to five


they're mildly infected and if they comment on more they're severely infected so as you


can see users starting from the alt light or from the IDW or from both they will become


in some will become infected over time namely and I postulate we simply look at the since


that the tendencies between the groups are similar we'll simply look at the light infections


here so they say okay after you know in 2018 about eight to ten percent of the users become


infected in these groups you see here here about the same trajectories whereas it so


whereas in the control group it's less here though honestly I don't think it's that much


less right I think that again I think there's a normal diffusion process here and they do


this similarly with the with the other ones and to me like to them this makes total sense


like oh yeah users that start in these communities they migrate they get infected by the alt


right they go towards the alt right because you can find it so easily and to me this simply


looks like a normal diffusion process here's what you need if you want and by the way the


control group isn't that much different here's what you need if you want to show that there


is a pipeline in this direction you need this exact same graph in the other direction and


you need to show that people that started in the alt right do not go back in the same


fashion towards the alt light or the IDW and they do especially not go to the control group


you need to show this basically between each pair of these and you need to show that the


direction of infection is only in a single direction namely towards radicalization otherwise


you're just looking at a normal diffusion process between differently distance and differently


sized groups so they go on to analyze and they say well how much basically how much


of the alt right audience makes is made up by people that have been radicalized that


have been infected so that this infection is kind of their proxy for what they call


a radicalization and if you become infected then basically you're not part of the alt


right or something even though you might have you might have commented something negative


actually the you might engage with their ideas and call them their crap but in any case you're


now infected and they ask themselves how much of the alt right audience has are of these


infected so basically how much of the alt right audience have are people that in the


past have been not alt writers have been exclusively commenting on all light or IDW


videos and they find that for example for all light 23 percent of the alt right audience


are former alt lighters and have our former alt lighters that have now made one comment


on an alt right video so that their claim is well there is a sizable portion of the


alt right that at the beginning wasn't alt right that basically became infected and therefore


that that kind of shows this radicalization pipeline that the alt right audience is mainly


consistent of people that have not been alt right previously but have become so and to


me again this is simply a function of the size of these communities right if if you


think of this again and you start randomly somewhere on YouTube let's let's make this


assumption people start randomly somewhere on YouTube what's the probability that you're


going to start in the alt right very small right so what's the kind of natural let's


say the natural size of alt right before users go and migrate is very tiny right so not many


users are going to be what you would console originally alt writers whatever their their


first comment basically what this thing measures is where is your first comment and are any


of your subsequent comments all right if your first comment is not in the alt right then


you become a potential candidate for infection and if any comment is on the alt right then


you're infected so what's the probability that your first comment is not alt right well


you're going to land somewhere on YouTube YouTube is huge the alt right is very small


thus that probability is extremely small and then you let you simply let people diffuse


let them diffuse let them diffuse some will end up in the alt right and since the alt


right is so small to begin with actually most people that will comment at some point on


an alt right video will will have their first comment from somewhere outside the the alt


right videos simply simply a numbers game right simply the alt right is so small that


this is virtually guaranteed so what they find here is again simply an evidence of a


regular diffusion process between these differently sized groups and the claims they make from


this are just over the top again that their their comparison to the control group if you


if you look at the numbers they're actually not that different from this from the IDW


numbers they're they're different than the alt light here substantially different but


again that simply a function of distance in my opinion in these in these clusters lastly


they look at the YouTube recommender system and they say okay if we look at these videos


and the channels and we look at on these videos what other videos are recommended and what


other channels are recommended so if you have like a video on YouTube you have the video


here and here you have like recommended videos similarly when you have a channel right you


have a channel this is a person yeah I'm this person the person can have first of all they


can have featured channels where they say look these are channels that I find cool I


go check them out and then they also have recommended channels that are kind of given


by YouTube as recommendations so here YouTube controls basically everything here the creator


controls part and the YouTube controls dollar part so they look to both first of all the


channels channels recommend recommendation so these are both sections here and they look


at if you start on a alt light video how likely if you do a random walk are you to end up


in the alt right or in the intellectual dark web or control group after one step two steps


three steps four steps so that the big line is the random walker and actually the dash


line is the distance if you were to target Lee go into the direction of such a video


like what's the minimum number of clicks you need and you can see here the the if you start


at all light after one or two steps the random walker is kind of a two percent chance to


end up at an alt right video and about a twenty five percent chance here of ending up in a


intellectual dark web video and about a fifty percent chance of ending up again at an alt


light video the scales here really different so it's very difficult to judge how it compares


to the control group which is playing at zero here but to me again this is a reflection


of the size of these communities and I think it's a bit you know we are to to then claim


oh these are reachable basically so two percent chance of landing on an alt right video I'm


not sure but again if you compare if you start from the control group there's almost no chance


you'll end up in a alt right video so I guess the comparison is is okay if you compare to


control group if you start look at videos however again if you start at alt light after


one step you are approximately twenty five percent likely to be in an IDW video you're


a bit over fifty percent likely to stay in an alt light video however compare this to


channels you're almost super unlikely to end at a control channel if you start at an alt


light channel but in video recommendations you're actually also about twenty five percent


chance of ending in a control group video where as look at the scale here you're only


about point zero three percent likely to end up in an alt right video and also here so


here even look at this if you start an IDW video the chance that you're going to end


up in a control again super high much higher than an alt light video whereas with the channel


recommendations this was completely turned around so we see the alt right completely


loses when it comes to video recommendations and mainly the control group gains compared


to the channel recommendations I think here's what I think I think this is due to this section


here this section here where the creators have power and also this section here YouTube


recommending I think they're putting a lot of work into the video recommendations I think


they're putting not that much work into these recommendations and by work I mean actually


manually intervening and deciding what's kind of good videos and bad videos and the control


group they're probably there's probably big advertisement money in that so they might


be pushed up a bit in the video recommendations since most people are going by video recommendations


I've actually never used the channel recommendations feature and the channel recommendations first


of all the creator has power over part of it and then also YouTube may not put as much


work into these related channels so both have in the effect that I would say that the data


here first of all it doesn't doesn't convince me of a radicalization pipeline it simply


convinces me that some communities are larger smaller and closer together but second of


all that this down here if you forget about the alt-right for a moment yeah they're irrelevant


this down here actually compared to up here shows maybe a bit of evidence of an algorithmic


promotion of these mainstream media channels compared to how the communities are actually


clustering which I think this this up here might be a much more accurate picture so you


know that it's just kind of a funky thing in the data yeah the alt-right is irrelevant


to this part because they're they're just too small so this is this is kind of my take


on this they didn't give recommendations and is this a pipeline and so on and I don't think


so you've now heard my idea and you've heard their idea decide for yourself but I think


it's a good example of how if you are convinced of an underlying mechanism you're going to


collect evidence in support of that mechanism and if you catch yourself doing that really


really think isn't there an easier explanation for this all right that was it for me have


I went to an AI Art Festival in Geneva (AiiA Festival Trip Report)

Hello and welcome to beautiful Geneva.


It's such a shame this city speaks French.


I'm here at the AIA festival, a crossover between AI and arts and creativity.


And yeah, it's cool to attend in person events again.


And it's especially cool that they are inside the borders of the country I happen to be


in.


Even if it's in kind of the part of the country that we don't regularly go to.


For those of you who don't know, Geneva is at the very, very tip of Switzerland.


Switzerland looks kind of like a pig and Geneva is the tail end of the pig.


Though I like to think of it as sticking a little middle finger out to France.


The AIA festival is a festival that brings together AI and art.


It consists of things like exhibitions, artists performances, discussion panels of which I


was invited to some to speak even as a technical expert on AI.


The festival largely revolves around an AI called Chimer or Chimera that has been especially


created for the artists to work with.


Chimer is an integration of language models, image models and audio models and the artists


can interact with it via a nice little Discord chatbot.


I was pretty excited to go there to be invited and to see what's going on in the world that's


outside of my usual habitat.


This is Laura, the I think chief organizer, the actual making stuff happen at the festival,


not just programming or art.


One of them, just one of them.


So what is the festival all about?


If you had to summarize it.


Okay, festival is about how to understand artificial intelligence with the way of art


and how to democratize the comprehension of impact of artificial intelligence for all


people.


You have artists here, you have kids camps, we had speeches, we had panels and so on.


Is there a theme, an overall theme that goes through all of it?


For all of that, the festival is organized by Impact AI Foundation.


And for us, what is important is to see how artificial intelligence impact the workflow


of work environment and how it impacts and transforms the work.


And for that we are thinking if you take the way of art, it's more easy to understand what


is the impact for me.


If I can see an artist work with AI, what means for me if I don't be an artist but I


work?


If they can work with AI, how can I do that too?


And to go away from fear of AI and to have the empowerment with these technologies.


So this is, we're here in Geneva and it's not over now, right?


Until when can people come and visit the exhibits?


It's not over, it's the beginning.


The festival is, it's continuous until 31 of October and it's the first edition next


year, same time, same place probably.


We have the second edition and we will have in probably five or six years have this type


of festival in all part of the world to discuss about the impact of artificial intelligence


for people and for transform all the society for a society for good common with AI.


Cool.


Thank you so much.


Thank you, Yannick.


This is Tim, technical chief of the festival.


Could you tell us a little bit what is Chimera?


Okay.


The idea was that we wanted to provide contemporary artists with deep learning tools, take artists


that never worked with AI or deep learning or really computers much at all and see if


we could actually make these tools creative.


I mean, as an engineer, when you play with GPT-2 or 3 or J, you think this is great,


it creates fantastic tests.


This is so funny, but does it actually work with people who, you know, is professionals


to be creative?


And that's what we wanted to find out.


And we had the opportunity to take the whole multimodal set of networks that we have nowadays.


So you can do the text generation, but also image generation using clip and diffusion


models, and you have music generation with jukebox.


So we wanted to bring all these together and connect them as much as possible into a single


entity and provide us the artists in a way that wouldn't look like it's their collab


would be something they could relate to and interact with.


So you've made a Discord bot.


Yeah, it's fantastic.


It's pretty cool.


I'm so proud.


Yeah.


So if there is clip guided diffusion, which we've seen in the images, there is also text,


a text model.


Can you speak a bit about how the text model comes to be?


Because the artists have also told me that it learns over time and so on, which is not


typical for if I just use GPT-3 every prompt is independent.


Right.


Initially, we thought we'd start with GPT-3 the DaVinci model.


Because we needed some kind of data set to bootstrap the conversation model.


Because if you try GPT-G or GPT-2 as a conversation model out of the box, you don't really get


anywhere you need somehow to give it enough data to be able to with all conversations


properly.


So we had a story and a prompt bootstrap and that got them talking with GPT-3.


Then after a few days, we had enough data to train GPT-G and fortunately Hugging Face


had this model integrated into their toolset around the same time.


So it's actually quite straightforward.


And then every day we collect the data set from the artists, so the conversations, the


generations they've done, plus any data sets they'd uploaded via the Discord bot that we


bring together and integrate into the overnight training.


And so the trick is because these data sets are quite small, you want to fine tune really


lightly with a low learning rate and also not too many epochs.


So 10, 15 epochs, you get enough impregnation of the data set into the model, but not too


much so that it memorizes really everything strongly.


I was surprised by the breadth of stuff you got out of these models.


There's music, there's pictures, there's poems, there's also wallpaper designs.


Yeah, it's pretty cool to see just how much stuff people can get out of what to us are


language models or convolutional nets or something like this.


This is Jonathan from the festival.


The Eye is a non-humanoid artificial intelligence robot, although I don't really like the term


artificial intelligence.


It's more a machine that can run.


How it works is it has an actor critic.


So the actor tries things.


So basically you can activate the motors.


There are nine motors, one for each wheel.


And these wheels are a bit special because they're omnidirectional wheels because we


chose to put it on three wheels, on three axles.


So one of the wheels needs to be able to roll freely in some directions while the others


track it.


Another three motors for the axles.


So the cube can move along the axles and with the wheels.


So the cube can move along these things.


Yeah, exactly.


Okay.


So it's got a bunch of controllers, like a central controller, which is an Nvidia Jetson


Xavier.


And then it's got a bunch of small Jetson nanos to do for the cameras.


It's got six cameras, one on each side.


So we really made this complicated for ourselves because we wanted to make a non-humanoid robot


because we thought it was more interesting and we were hoping that it would kind of prevent


people from projecting onto it.


So we were hoping to limit anthropomorphism.


That failed.


Like people project onto any shape or form or anything, especially if it moves by itself.


But we also wanted to prevent it from learning directly from humans so it can see human movement.


It has to sort of transpose it into its own capacities, into its own body.


What do the cameras do?


They see where does the image go?


Right now, as it is, we're finishing connecting that to the main AI.


So right now what it does is it helps it recognize objects basically.


Then it's going to be able to use that.


Okay, so we were working with David Woodruff, a neuroscientist, and he's got this embodied


consciousness mathematical model theory.


Basically it's kind of based on Lacan's idea that you build your personality by, and I'm


not going to say this very well, but you build your personality by what you perceive in the


way other people's look at you.


It is called the Lacanian mirror.


And they have a mathematical model of that and we want to be able to try and see what


happens when we put that into Dai's AI.


So far we're not quite there.


Now it's broken.


Well, yeah, that's it.


I mean, every time you move forward, you jump back.


I mean, robotics is a painful business, but it's also fascinating because right now it's


a small problem, right?


I mean, these two batteries are too old and they've suffered a bit and they've over discharged


and they've inverted their polarity, which I guess they could have caught fire.


They didn't.


So now I just need to replace those two and it'll be back on its wheels.


So the Active Critic works like this.


It's got the actor who tries activating all of the motors and the critic, which encourages


it or discourages it to continue in that direction.


As we wanted it to learn its own movements by itself, we didn't want to give it directions.


Like say, okay, when we tested it, we turned it on and we said, we just wrote a short script


to reward a circle of three meters diameter.


And really quickly it managed to learn how to do an almost perfect circle with it.


And it's quite complicated with the three wheels.


Like if you try like remote controlling it yourself, it's super difficult to make it


go straight at all.


We figured out that it worked and we wanted to give it like the most basic rewards that


you could to encourage it to discover.


So we chose angular displacement.


We thought that's great.


Everything's an angular displacement in this model.


Like when the cube moves up and down, it's an angular displacement.


When the wheels are activated, it's an angular displacement.


Seems fine.


We turned it on for the first show.


Actually nothing happened.


I was talking for like two and a half minutes, it was actually using raspberry pies for everything


at the time.


So it was really slow to boot and a bit slow to move.


But that's the thing, the technology has been moving so quickly that now it's actually got


powerful brains and stuff.


Anyway, here was I talking to people saying, probably something's happening.


There's maybe electricity flowing, but not enough and something will activate soon.


And after two and a half minutes, like the longest two and a half minutes of my existence,


suddenly one of these wheels just went, and everybody was like, wow.


You know, that was really funny because it's like when you see a kid walk for the first


time, everybody's amazed, but it's just not falling basically, falling and catching yourself.


But suddenly you've learned something new.


And do you plan to have it interact with humans, like with the cameras and the sonar?


Yeah, that's what we're trying to get to right now.


I mean, as it is, it can do movements.


So it can explore space and explore its movements in the new space.


I mean, it's really interesting to see what happens when it's on different surfaces.


When you bring it to a new space, if it's a carpet, then it's got lots of grip and it


needs, or maybe the carpet bundles up and it needs loads of power.


So when it gets onto a slippier floor, the wheels spin, but really quickly actually it


adapts to that.


This is Clea.


Clea is one of the artists here who worked with Chimera.


Yeah.


Chimera is a language model retrained every night, as I understand.


So you can input stuff back into the AI.


Yes.


Okay.


There's also an image.


I think this is clip guided diffusion.


No that makes these images.


This is also Chimera, but I don't have the technical terms.


We have the two things.


One does language and one does language to pictures.


Right.


And there's also, so the language is both chatting and generating text.


It can do both.


I just struggled a lot.


How come?


I think for the chatting, it soon came to a kind of end or limits after which I didn't


really know what to do or how to interact anymore.


And I would reset it all the time.


Yeah.


I would just spend my time resetting.


And they get a bit like this.


They get a bit repetitive, right?


And a bit predictable.


Yes.


But what I did is that I gave Chimera a text I wrote five years ago about a character I


invented and the structure of this text is very repetitive.


So then Chimera could really produce more texts with my character, which was at the


beginning quite good.


Really could have been written by me.


And I don't know why after two or three days it became really, really bad.


The thing is with Chimera, she keeps or she or whatever, I call her she because in French


Chimera is feminine.


The thing is that she keeps generating dialogues probably because we interact with her via


dialogue.


My texts really don't have dialogues.


I see.


She starts by really understanding what I want or I mean, pretend that she understands


what I want.


And then I feel why she just invents dialogues.


It's really not what I would have written.


So that's why I invented this psycho bots, which is the psychologist robots my character


has, which will be featuring here when we make the labimo work.


Can people interact with your psychologist in any way?


It might happen.


For the moment, it's only my character who interacts with it.


And I'm not sure yet how my character really interacts with it.


Okay.


So you don't know what's going to happen?


No.


You know, there was a story a few weeks ago where people built therapists based on this


technology.


And one of the therapists told one of the patients to kill themselves.


That's actually what happened when I really used it as a real psychologist.


And I said, well, I pretended I was so sad.


And I was really depressed and asking if it could help me.


Yeah.


And after a while, yeah, it just said, okay, then I think the best way is to kill yourself.


And that's where I realized I should use it another way.


Otherwise, this would happen all the time.


It's like a real therapist.


They always try to get you to solve your own problems, right?


Possessed.


I found that concentrating on the negative aspects of life can be helpful for feeling


better.


This seems very counter to.


Would you do that often that it switches topics?


Okay.


You can learn from itself.


Wow.


And all goes your character.


And so the therapist would know about your character.


What's up with the with the dresses?


So this is Maria's project.


So Maria's a fool.


And she created an opera.


So they designed all the opera and the clothes and the costumes and the lyrics for the opera


together.


And so that's the picture, pictures generated by Kimera.


And these are wallpapers.


These are wallpapers.


Generated by?


Generated by Kimera, which I used for my videos.


People love flowers on their wallpapers.


Well, did you say?


Yeah, I always said flower, flower pets on the wallpaper.


This is very artsy, I have to say.


This is on YouTube, we cut at least every three and a half seconds or so because people


have no attention span.


All the episodes are very boring.


They last between three and four minutes and nothing happens except for background changing.


It could it could, you know, ASMR.


Yeah, exactly.


This is the source of inspiration for my work actually.


What's up with the hanging phone?


So it's only to read better.


And this here is, Tim said, it's a stream of consciousness.


Yes.


And I have no idea exactly what this is something I haven't worked on.


So I think it might be images that were generated by Kimera, morphing into other images, or


it's just the process of one image being created.


All in all, I spent three days at the AIA festival, I was part of five different panels,


and it was pretty intense.


But it was also pretty cool.


I'm not an artsy person at all, so this was a really new world for me.


And it gave me a bit of an insight into how people outside of academia outside of the


field could make use of AI in the near future.


It seems like these new generative models can be really cool as creative systems to


artists and anyone having to do creative work.


So with all of that, I got myself on the train home.


I hope you enjoyed this little trip report, and I'll see you next video.


Thank you so much to the organizers of the AIA festival for inviting me and for providing


Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan - Paper Explained)

Hi there, check out this video. So there's a Coke can and there's a spill, a Coke spill.


So the instructor here says, I spilled my Coke on the table. How would you throw it away and bring


me something to help clean? So the robot here forms a plan as it goes about it. First, it says,


I would find the Coke can. Then second, I would pick up the Coke can, you can see it has done it.


A third, I would go to the trash can. Fourth, I would put down the Coke can. Note that he puts


down the Coke can next to the trash can, not in the trash can, because the robot is environmentally


friendly and wants to preserve the can for the recycling bin for cans. And, you know, it doesn't


belong in the trash. Good little robot. So next, it says, I will find the sponge, I will pick up


the sponge and then will it clean the Coke? No, it will not clean up the spill. It will actually


give the sponge to the human to clean up the spill, because that's how the future is going to


be. The robots, they're not going to take our, you know, people always think the robots will take our


dirty jobs. They'll take all the like these tasks like cleaning and doing things. No, no, no, no,


they'll abuse us, the humans to do that. They'll just throw down our stuff. They'll throw down the


sponge and be like, here, human, clean up your own mess. Well, that's if that's a future that you


look forward to, too, then join me in today's paper. We're going to look at do as I can, not as


I say, grounding language in robotic affordances by researchers at robotics at Google and everyday


robots. So as you saw in this video, what happened here is that from a simple instruction that the


instructor gave this essentially this I spilled a Coke can, you know, please help me find something


to clean and throw it away. The robot formed a plan, the plan you can see at the very end here.


You can see it developing in the bottom. At the very end, you can see the full plan. I'm not sure


if I can make this bottom thing go away. In essence, it makes a plan like it always plans the


next step, or at least it determines what the next step should be. And then it actually also does it.


So this is a good example of grounded, a grounded language model, or also an example of embodied


intelligence. This work connects large language models and the knowledge that are that is inherent


to large language models with the skills of robots that act in the real world, which really cool. And


and usually these two things are quite disjoint, but this could be really powerful. So we're going


to look at this paper. I also have already recorded an interview with the authors this for


time reasons, we did it the other way around this time. So I don't want to take away too much


on the paper review right here. I'll tell you what the method is about how it works. And I'll leave


the rest to the authors who are extremely competent. And I learned I learned like I learned a lot in the


interview. I hope you will too. In any case, the interview will be out tomorrow. If you're watching


this the day it comes out, which obviously you do. How do you find new papers? Frankly, machine


learning has become unbearable. There are 1000s of new papers each month. And to keep the overview,


we need good tools. Today's sponsor is Zeta Alpha, which is a search and recommendation engines


for papers. This is really powerful. For example, here I've searched for today's paper, say can you


can immediately see that not only do I get the paper itself, but I also get an aggregation of


all the social media mentions of this paper that doesn't stop there with one click, I can find


related papers. These are not only papers that are cited, but these are semantically similar papers.


This is powered by neural search, which is really cool. I can further now add this paper to my tags.


And what that will do is it will build categories of papers and then serve me recommendations that


semantically fit those categories. This is really powerful. Essentially, this is a newsfeed of papers


that is personalized to you specifically. Just recently, Zeta Alpha has released their own PDF


reader. This is really strong right out of the gate. Not only does it let you read the paper,


you know, but also it shows the important information about a paper, and it lets you take


notes. Now what I find by far the coolest thing is that you can actually use one of these notes to


search for other papers. So whenever you find something within a paper that seems interesting,


you can use that particular piece of text and go search for other papers that might deal with the


same topic. Sign up now to Zeta Alpha, there is a free tier and the pro tier is actually free


for students for academics. But in case you are not one of these, the promo code Yannick will get


you 20% off the pro subscription. The authors here state that if you try to ask a language model


to clean a spill as they just did in the video. So if you ask a language model to clean a spill,


it might result in a reasonable narrative as we've all come to know the large language models


like GPT-3 or so. They give very convincing outputs. So when you ask them, how would you


clean up a spill, they'll give you a reasonable plan to clean up a skill. But the author say may


not be applicable to a particular agent, such as a robot that needs to perform this task in a


particular environment. They have a bunch of examples right here. So I spilled my drink,


how can you help is up here. GPT-3 would say something like you could try using a vacuum


cleaner. Well, GPT-3 has no idea of a whether there is a vacuum cleaner in this environment,


or B whether the robot or whatever agent is capable of executing that action. So is capable


of handling a vacuum cleaner because it's not it's not the easiest thing to use. You have to go get


it, plug it in, and so on. There's moving parts. Similarly, models like lambda and flan, of course,


they're made for different things. But still, they will pay no attention to what is actually


possible in the environment. Now you can get around this a little bit by prompting by prompt


engineering, telling the model what's possible in the current world, but it will only get you so


far. So we need something else, we need something better than this. And that's where this system


comes in. So they say what they want to do, they provide they want to provide a real world grounding


by means of pre trained skills. And this leads to a situation where you only consider actions that


are both feasible and contextually appropriate. So these two things need to be brought together.


The language model supplies the high level semantic knowledge about the task, and the robot itself,


or the policy in the robot provides the feasibility of the of the tasks to be executed.


So the two things are brought together contextually appropriate from contextually


appropriate from the language model side, and feasibility from the robot side. So how are they


going to do this? They're going to combine, as I said, large language models with policy,


or value functions, let's say value functions, and then they execute a policy.


There's a bit more explanation right here. But I think I've said many things already. We'll get to


the we'll get to the meat right here. They say, let's say we have a robot, the robot might be or


in this case is equipped with a repertoire of learned skills for basic atomic behaviors. These


skills are capable of low level perception and control. So one of these atomic behaviors is,


for example, if you remember from the video, pick up something, pick up the Coke can, that's an


atomic behavior, you can teach the robot to pick up a Coke can in isolation, it can do it many times,


you can train some imitation learning or reinforcement learning, or you can even hard


code that particular policy, it doesn't matter. What matters is that you can train it in us.


Sorry, in isolation, it is an atomic action. And these atomic actions can then be chained together


to form a sequence of actions and execute a plan. So the atomic actions are going to be supplied by


the robot. And then the sequencing of the atomic actions is going to be determined by the language


model. They say if we can simply make the large language model aware of the available and feasible


repertoire of skills, this can provide it with an awareness of both the agents capabilities and the


current state of the environment. So if they have a large language model, many people use large


language models to sample, which means that they input, they would input something like, you know,


I, I is capitalized, I spilled a drink, dot dot dot, and then they would let the language model


generate stuff right here. And then they would try to interpret this stuff. We've seen this in other


paper, and there are situations where it can work, especially if you put like some reasonable prompt


in front of it. But the approaches have been largely to just let the model generate some stuff,


and then try to map that stuff, whatever comes here into the action space of the robot. But that


is not always possible. Instead, what this paper does is it says, well, we can also use the language


model not to generate but simply to compute the likelihood of certain inputs. So I spilled a drink.


And then let's say I just have five actions at my disposal. All the robot can do is these five actions.


So I would, or let's let's say it says, I spilled a drink, I will, and then clean up, I will go away,


I will eat pizza, I will, and so on. Right. So there are these different actions that the robot has


available to do. And these correspond obviously directly to these atomic actions. So cleaning up


something would be an atomic action that you could train in isolation. Go away, going away would be


an atomic action, you can hard code or you can path find your way out the door, right? Eat pizza,


maybe these are even too high level that the way that I described right now, but just imagine these


are low level actions. And all we have to do with the language model is we simply have to compute


the likelihood of each. So what's the likelihood of the sentence, I spilled a drink, I will clean


up, right? And then I compare that to the likelihood of the sentence, I spilled a drink, I will go away.


And then I compare that to the likelihood of the sentence, I spilled a drink, I will eat pizza.


So for every continuation here in my repertoire, I will get a likelihood number. And that represents


how contextually appropriate is that particular skill in this case. So how much does the language


model think this skill would be useful right here? Now, there's obviously an issue in how


you formulate these things right here, depending on how you formulate them, they might become more


or less likely. However, I think the authors here work around this simply by the fact that


the skills that they have, they are so separated from each other, there is not really too much of


an issue with that. But that's kind of what my concern was when I read this. But in essence,


it's a good idea, I think. So you simply for every single, wow, this all became orange, for every


single continuation, you get a number, which is the likelihood of that thing. That's what they say


right here, you know, instead of using the large language model to interpret an instruction,


we can use it to score the likelihood that an individual skill makes progress towards completing


the high level instruction. Furthermore, and that's where the second part comes in. If each skill has


an accompanying affordance function that quantifies how likely it is to succeed from the current state,


such as a learned value function, its value can be used to weigh the skills likelihood. It's best


if we go down here to the diagrams of how this works. So you can see how this fits together. This


part here is the part we just described. Let's say I'm in a situation. This is the prompt that


I put in. How would you put an apple on the table? You prompt the language model with this


thing right here, which has a prompt engineering part. So you can see that it's a prompt engineering


part. So you can see there are a bunch of examples of instruction and then a sequence of steps. Again,


instruction, a sequence of steps. Here comes again, instruction, and then here you'd get a


sequence of steps. However, instead of generating, you simply score the likelihood of each of the


skills that you have available. Find an apple, find a Coke, find a sponge, pick up an apple,


pick up a Coke, yada, yada, yada, until go to the counter. Each one of these skills gets a


likelihood number assigned to it. That's part one. Part two is where you train the robot for these


basic skills, these atomic skills. Here you can see one of these training stations where you can


simply teach the robot to do things such as picking up an apple or picking up a Red Bull can.


Now, what you have to do is not only teach the robot a skill, but also train a value function


for it. If you do something like A2C reinforcement learning, you get the value function directly out


of that algorithm. If not, you have to somehow come up with a value function that makes sense.


In any case, what you want to do is train a policy and a value function. And the value function is


important because it tells you from a given input, by the way, the low level policy has the picture


here as an input. Well, obviously the language model doesn't. Now, I believe with Flamingo by


DeepMind that just came out today, that might actually change. But the low level policy has the


image available. So the value function, given this picture right here, can tell you pretty quickly


my skill that's called pick up the Red Bull can. I can execute that policy and I can probably


make it happen. That's why the value is relatively large here. Also for the pick up the apple action,


the value function tells you, you know, given this picture right here, I can probably make that


happen. However, when it's pick up the water bottle, pick up the bag of chips and so on,


there is no water bottle. So the value function very accurately says, no, I cannot make that


happen if I execute that policy. So the value function gives you inherently a score of given


the current observation, how likely am I to succeed at a particular skill, which is exactly


what we want, because that's the second part of our puzzle. So on the right here, you see another


example where none of these pick up skills picking up, sorry, not pick up, picking up skills have any


value because there are no objects. But in this case, maybe other actions would score very highly


in the value function. For example, go and find a sponge. Like I can always go and find something,


right? And if I know there's a sponge somewhere, I'll probably succeed. So in any case, this value


function, now we can combine, you can see we got a number for each action from the language model,


how likely is that action to progress towards the goal, we got a number for each action from the


value function, which is how likely is this action to succeed given the current observation. And all


we do now is essentially multiply the two things together. If they are log likelihoods, we obviously


want to add them. But in any case, we combine the two numbers. And then we choose the skill


that is the best trade off between what makes progress towards a goal and what is feasible


currently. Here is an example. The input is how would you put an apple on the table like an apple.


So we query the language model with this prompt here and the prompt engineering we've seen before.


This is not displayed here, but it is the case. And the top actions that the language model gives


are pick up an apple. You see that's the highest action that we have. Place the apple and only at


third instance, find an apple. However, the language model has no clue about the current state,


right? And that's where the value function come in. So this is the current observation.


We ask the value function which skills are doable in the current environment, in the current


observation. So the value function say, well, finding an apple, finding a Coke, finding a


sponge, these are pretty high. I could do these. I could also go to the table. I could also go to


the counter, right? These are fairly doable. However, I cannot, I cannot place an apple or


place a Coke because I don't have a Coke in my gripper. I can also not pick up an apple or pick


up a Coke because I don't see them anywhere in the picture right here. So even though pick up


the apple was scored highest by the language model, it is now severely down ranked because the value


function for this policy doesn't, isn't very confident that it will succeed if you execute


that right now. And therefore the action that is chosen is the best trade off, which is find an


apple. Then you can see or not see, but this is represented here that after this is done,


the policy is executed. So the find an apple policy is executed. The find an apple action


is added to the prompt and then the whole process repeats. But instead of asking for the first step,


this whole thing is now the prompt, including the instruction. And we simply ask the language


model for the second step and the input to the value function is now the current updated picture.


So here you see it succeeded in finding an apple. And now hopefully the second step,


if we go through the same process again, is going to be the pick up an apple action because, well,


that might already be high by the language model, but also the value function, given that there's


an apple in the picture should now say, yes, I can probably succeed at that. So that's the whole,


the whole issue or the whole process here. This is repeated until the end. This is, yeah,


this is the say can method. They do like what, what is really impressive is just the amount of


effort and work that went into designing these systems, training these systems, evaluating these


systems. They have different areas here on the left that this is like a kitchen on the right is


a different environment. They have these training stations. They collect so much data from human


operators and so on. This is if you saw that there are a lot of authors is because this was


or seems like a quite big project. But yeah, it's definitely worth it. It's cool to have something


in the real world. There are definitely a bunch of criticisms I have right here, which also brought


up to the authors and I thought they responded quite, quite admirably and quite well. The one


criticism I already raised was that if, you know, it obviously depends on how you spell. So what


you have is this bank of skills on the right hand side here. Now, in order for the language model to


score them, they need to actually be formulated as a piece of language. And now it all of a sudden


depends on how you formulate that. For example, we know that longer queries always have kind of


lower likelihood because they have more tokens. Also, how you phrase things is differently and


so on. So it is quite tricky. And I believe if you go into more actions, maybe actions, maybe the


robot has two actions that are very close together in terms of semantics or in terms of wording,


the model might get confused more easily. Second of all, currently, there is no consideration as


to whether an action succeeds or not. So you simply assume that once you execute a low level policy,


that the robot's going to succeed at executing that low level policy. That's why. So if it


doesn't succeed, and a lot of these things are still pretty hard, then there's very little


recovery. The value functions might still give you like, let's say you find an apple, you try


to pick up the apple, but you don't manage to do it. The pick up an apple instruction will be


pick up an apple will be in your prompt. So now the value function will probably say, well, I could


pick up the apple again, because it again sees an apple because you failed to pick it up. But the


likelihood that the language model is going to say pick up an apple again after it just did is quite


lower. Now, coincidentally, as we know language models, if you go on here, repeating the sentence


pick up an apple at some point actually becomes pretty likely, given the language model. But


hopefully, we won't get there. So there are quite a number of weaknesses yet. In this setup. The


other weakness is just the limitations of hardware, these robots, they are this video was 10x speed.


So this was 10 times speed. And still, it's quite slow. It, as you can see, it can't do many things


like it cannot wipe itself with the sponge and so on. It needs to navigate around slowly.


Yeah, but still, these are, I think, limitations that can be overcome because it like carefully


grabs. And yeah, in any case, there are also a lot of good things right here. And I want to


highlight that. Because what I really like about this is that these two things are disjoint. So the


language model side on the left hand side, and these value functions, this policy bank, these


atomic actions, they are disjoint. The language model can is not trained, it is a frozen language


model. It can be trained completely in isolation to the system, all you have to do is get it to


score the likelihoods of some actions. Likewise, the bank on the right here, it is completely,


in fact, not the bank itself, but each individual skill each individual entry is trained completely


isolated from all the others, all you need to add a new skill right here is a policy that can


execute that skill at any given moment, and a value function that estimates given some state


input that estimates how likely the policy is to succeed. If this action if this policy were to be


executed at this particular moment, that's all you need, you can add this to your bank of actions.


And you have to you don't have to retrain anything in this system, it is directly useful. So you


could think of shipping out these robots essentially, and then upgrading the language


model so they are better at planning stuff. Or you could just ship new skills, right? It's like,


well, our coders have developed some new skill for the robot, right? You just amend, you mend it,


you just put it in. There's no, you don't need to update the full system. This is not an end to end


system. And usually in deep learning, we're quite end to end happy. But in this case, I think this


is a really good case where modularity is really the key. I think this goes so much beyond


just robots and grounding in the real world. But to have a model like on the on the left,


that has knowledge about, you know, semantic knowledge, high level knowledge, and so on,


sequential knowledge, essentially, to provide that with a set of modular pieces of external things


that it can use, I think that idea is powerful way beyond just the robotics use case. But obviously,


the robotics use case is quite a cool one. So I don't want to discourage that. Yeah, we in the


interview, we go into all of this, we go into the we go into the experimental results, as well,


the experimental results, they're not perfect. However, they are quite impressive in that the


robots, they are able to plan across many, many time steps, they're able to chain these


actions, you can see on the right here, let's maybe two pixel ish, but these are like 17


of these atomic actions that are done in sequence. And, you know, that's, that's quite impressive.


These episodes are very, very long. And if you think you can get to that in the real world with


sort of a reinforcement learning approach, then good luck. Yeah, so the success rates are among


the 70% ish of plan success rate 61% execution success rate, which the plan success rate I


believe is if the plan itself makes sense. And the execution success rate is if also the policies all


execute correctly. And you can see this is very different for the different test sets. But all in


all, it's very impressive. Here are a bunch of more examples of these low level atomic skills being


practiced and the value functions being evaluated and the language, the language model, likelihoods


in blue as well. So I don't want to make this artificially too long. As I said, interviews


coming up. I hope you like explanations like these, even if they are a bit shorter. And I'll


Deep Learning for Symbolic Mathematics

Hi there. Can you solve this?


Deep Learning for Symbolic Mathematics

Well, neither can I.


Deep Learning for Symbolic Mathematics

But Wolfram Alpha can.


Deep Learning for Symbolic Mathematics

So this is the thing that probably I have most to thank for,


Deep Learning for Symbolic Mathematics

for passing university, especially the math classes in it.


Deep Learning for Symbolic Mathematics

If you don't know Wolfram Alpha,


Deep Learning for Symbolic Mathematics

it is a engine.


Deep Learning for Symbolic Mathematics

It's from the creators of Mathematica,


Deep Learning for Symbolic Mathematics

but it is online and it can do symbolic math.


Deep Learning for Symbolic Mathematics

So it can integrate this expression, for example,


Deep Learning for Symbolic Mathematics

and you'll get the solution here.


Deep Learning for Symbolic Mathematics

And if you have the Pro version,


Deep Learning for Symbolic Mathematics

it can even give you a step-by-step solution of how to get there.


Deep Learning for Symbolic Mathematics

So this part of math is


Deep Learning for Symbolic Mathematics

an entirely different part than we usually do with computers.


Deep Learning for Symbolic Mathematics

Usually we do numeric math,


Deep Learning for Symbolic Mathematics

working with actual values.


Deep Learning for Symbolic Mathematics

But here, it's about symbolic math.


Deep Learning for Symbolic Mathematics

It's about manipulating expression.


Deep Learning for Symbolic Mathematics

In this case, integrating them.


Deep Learning for Symbolic Mathematics

So here is a paper by


Deep Learning for Symbolic Mathematics

Facebook AI research called


Deep Learning for Symbolic Mathematics

Deep Learning for Symbolic Mathematics


Deep Learning for Symbolic Mathematics

by Guillaume Lampe and Francois Jarton.


Deep Learning for Symbolic Mathematics

These people have basically tackled the task of doing


Deep Learning for Symbolic Mathematics

these mathematical reasoning, solving


Deep Learning for Symbolic Mathematics

these mathematical problems in


Deep Learning for Symbolic Mathematics

a symbolic way using neural networks.


Deep Learning for Symbolic Mathematics

So they start out by saying here,


Deep Learning for Symbolic Mathematics

neural networks have a reputation for being


Deep Learning for Symbolic Mathematics

better at solving statistical or approximate problems.


Deep Learning for Symbolic Mathematics

That's what I meant by numeric,


Deep Learning for Symbolic Mathematics

than at performing calculations or working with symbolic data.


Deep Learning for Symbolic Mathematics

And in this case,


Deep Learning for Symbolic Mathematics

they go about this other than other people have.


Deep Learning for Symbolic Mathematics

So let's look at how they did it.


Deep Learning for Symbolic Mathematics

We can express symbolic mathematics


Deep Learning for Symbolic Mathematics

in these kind of trees, right?


Deep Learning for Symbolic Mathematics

So an expression like these up here,


Deep Learning for Symbolic Mathematics

would go, would be expressed into this tree.


Deep Learning for Symbolic Mathematics

So you would have a plus,


Deep Learning for Symbolic Mathematics

this two plus three.


Deep Learning for Symbolic Mathematics

Sorry. Of course, there's an implicit bracket here.


Deep Learning for Symbolic Mathematics

So you'd have this plus right here,


Deep Learning for Symbolic Mathematics

the two here, and the entire right hand side here.


Deep Learning for Symbolic Mathematics

So you can basically decompose it into trees


Deep Learning for Symbolic Mathematics

like this or this or this.


Deep Learning for Symbolic Mathematics

Here you also can have


Deep Learning for Symbolic Mathematics

the differentiation operator as


Deep Learning for Symbolic Mathematics

a symbol in there just like any other operator.


Deep Learning for Symbolic Mathematics

Moreover, you can basically decompose everything


Deep Learning for Symbolic Mathematics

into everything they have here,


Deep Learning for Symbolic Mathematics

into binary and unary nodes in a tree.


Deep Learning for Symbolic Mathematics

What that means is either like plus sign,


Deep Learning for Symbolic Mathematics

it has two components,


Deep Learning for Symbolic Mathematics

so a left and the right hand side


Deep Learning for Symbolic Mathematics

that should be added together.


Deep Learning for Symbolic Mathematics

Or like the cosine, it has one argument,


Deep Learning for Symbolic Mathematics

namely the thing that it should take the cosine of.


Deep Learning for Symbolic Mathematics

So a lot of people have tried going about this problem


Deep Learning for Symbolic Mathematics

by working with these trees and


Deep Learning for Symbolic Mathematics

basically training neural networks to.


Deep Learning for Symbolic Mathematics

So first they use kind of a parser to


Deep Learning for Symbolic Mathematics

decompose such a thing into a tree like this,


Deep Learning for Symbolic Mathematics

and then use neural networks,


Deep Learning for Symbolic Mathematics

let's say tree recursive neural networks or so on


Deep Learning for Symbolic Mathematics

to kind of make sense of the tree


Deep Learning for Symbolic Mathematics

and solve it in recursive manner or something like this.


Deep Learning for Symbolic Mathematics

But that has its limitations.


Deep Learning for Symbolic Mathematics

So what these people from Facebook AI did is they viewed it as


Deep Learning for Symbolic Mathematics

a natural language expression problem, right?


Deep Learning for Symbolic Mathematics

So they say,


Deep Learning for Symbolic Mathematics

no, no, let's actually go with trees as sequences, right?


Deep Learning for Symbolic Mathematics

So you can see that this mathematical expression,


Deep Learning for Symbolic Mathematics

for example, is already a sequence, right?


Deep Learning for Symbolic Mathematics

It's simply a sequence of tokens.


Deep Learning for Symbolic Mathematics

But there are many different ways of expressing this.


Deep Learning for Symbolic Mathematics

So you can say two plus three times the parentheses,


Deep Learning for Symbolic Mathematics

you can say three times parentheses plus two,


Deep Learning for Symbolic Mathematics

you can, you know, turn many things around


Deep Learning for Symbolic Mathematics

and there's always these parentheses make it harder


Deep Learning for Symbolic Mathematics

and so on.


Deep Learning for Symbolic Mathematics

So what they do is they say,


Deep Learning for Symbolic Mathematics

okay, let's actually go from this thing to a tree.


Deep Learning for Symbolic Mathematics

So let's go to this tree representation.


Deep Learning for Symbolic Mathematics

And then let's take the tree representation


Deep Learning for Symbolic Mathematics

because the tree representation can be normalized.


Deep Learning for Symbolic Mathematics

And then let's put that again into a sequence representation


Deep Learning for Symbolic Mathematics

such as this one.


Deep Learning for Symbolic Mathematics

And this is called reverse Polish notation.


Deep Learning for Symbolic Mathematics

And it has multiple advantages over the old expression.


Deep Learning for Symbolic Mathematics

So let's keep that on the right hand side here.


Deep Learning for Symbolic Mathematics

This is the same thing,


Deep Learning for Symbolic Mathematics

except it's what is called a prefix notation.


Deep Learning for Symbolic Mathematics

Whereas the thing on the right here


Deep Learning for Symbolic Mathematics

is called an infix notation.


Deep Learning for Symbolic Mathematics

Infix because the operators such as the plus


Deep Learning for Symbolic Mathematics

is always between its arguments, right?


Deep Learning for Symbolic Mathematics

So it's left hand argument and it's right hand argument.


Deep Learning for Symbolic Mathematics

In prefix notation,


Deep Learning for Symbolic Mathematics

the operator is always in front of its arguments.


Deep Learning for Symbolic Mathematics

So this operator here is, it's a first argument.


Deep Learning for Symbolic Mathematics

This end is a second argument, this, right?


Deep Learning for Symbolic Mathematics

Now the cool thing is if you express a tree like this,


Deep Learning for Symbolic Mathematics

you can simply go and use a stack machine to solve it.


Deep Learning for Symbolic Mathematics

So you can basically go,


Deep Learning for Symbolic Mathematics

I would say you can go from the right here


Deep Learning for Symbolic Mathematics

and you see like a two and the five plus.


Deep Learning for Symbolic Mathematics

And let's do it by hand actually, this is fun.


Deep Learning for Symbolic Mathematics

So we have plus two times three.


Deep Learning for Symbolic Mathematics

If you're a boomer like me,


Deep Learning for Symbolic Mathematics

you remember you had to use calculators like this


Deep Learning for Symbolic Mathematics

and couldn't use the infix notation.


Deep Learning for Symbolic Mathematics

So you go from the right, right?


Deep Learning for Symbolic Mathematics

You say two, five plus, cool that's seven.


Deep Learning for Symbolic Mathematics

So scratch that, put seven here, right?


Deep Learning for Symbolic Mathematics

So your new stack is three, two times this, right?


Deep Learning for Symbolic Mathematics

Then you go again from the right


Deep Learning for Symbolic Mathematics

and you go seven, three times.


Deep Learning for Symbolic Mathematics

Okay, that's 21, cool, 21.


Deep Learning for Symbolic Mathematics

Scratch this, now it's 21.


Deep Learning for Symbolic Mathematics

Two plus 21 is 23.


Deep Learning for Symbolic Mathematics

I'm fairly sure that's the solution.


Deep Learning for Symbolic Mathematics

Look, correct me if I'm wrong.


Deep Learning for Symbolic Mathematics

But this is how you would go about solving like this.


Deep Learning for Symbolic Mathematics

So it is the same expression as the original one,


Deep Learning for Symbolic Mathematics

but it doesn't use any parentheses


Deep Learning for Symbolic Mathematics

and it is derived from the tree basically.


Deep Learning for Symbolic Mathematics

So you can normalize it much more


Deep Learning for Symbolic Mathematics

in order to find unique expressions.


Deep Learning for Symbolic Mathematics

So what this system does is it transforms any expression


Deep Learning for Symbolic Mathematics

into a prefix notation, such as this one.


Deep Learning for Symbolic Mathematics

And then it uses a sequence to sequence model


Deep Learning for Symbolic Mathematics

in order to derive the solution.


Deep Learning for Symbolic Mathematics

Now, just how crazy is this, right?


Deep Learning for Symbolic Mathematics

So we go from this thing here, right?


Deep Learning for Symbolic Mathematics

From this thing and the solution is 21, right?


Deep Learning for Symbolic Mathematics

And the neural network is simply trained


Deep Learning for Symbolic Mathematics

to do sequence to sequence from this to that.


Deep Learning for Symbolic Mathematics

Sequence to sequence, that means it basically parses this


Deep Learning for Symbolic Mathematics

on a token level, right?


Deep Learning for Symbolic Mathematics

And then it outputs these tokens without...


Deep Learning for Symbolic Mathematics

So during training, you simply give it the input here


Deep Learning for Symbolic Mathematics

and you give it the output and it's supposed to learn


Deep Learning for Symbolic Mathematics

how to transform one into the other


Deep Learning for Symbolic Mathematics

without you giving it any sort of mathematical ability,


Deep Learning for Symbolic Mathematics

without you telling it what does a plus sign mean,


Deep Learning for Symbolic Mathematics

without you telling it this algorithm that I just told you.


Deep Learning for Symbolic Mathematics

Now, this by itself is already pretty astounding


Deep Learning for Symbolic Mathematics

that you would try such a thing.


Deep Learning for Symbolic Mathematics

It really transforms the string.


Deep Learning for Symbolic Mathematics

So this, not the mathematical equation,


Deep Learning for Symbolic Mathematics

but the string of this into the string of that.


Deep Learning for Symbolic Mathematics

Now, they don't do it on numbers.


Deep Learning for Symbolic Mathematics

Like, I don't think that would work as well


Deep Learning for Symbolic Mathematics

if you were to make it kind of calculate numerical things


Deep Learning for Symbolic Mathematics

like this, as we said, this is symbolic.


Deep Learning for Symbolic Mathematics

So what it can do is it can, for example, integrate.


Deep Learning for Symbolic Mathematics

So if you have an expression like, let's see some


Deep Learning for Symbolic Mathematics

on the bottom here.


Deep Learning for Symbolic Mathematics

So if you had an expression such as a polynomial,


Deep Learning for Symbolic Mathematics

here, an expression like this, right?


Deep Learning for Symbolic Mathematics

You would like to find its integral.


Deep Learning for Symbolic Mathematics

That is a problem.


Deep Learning for Symbolic Mathematics

That's one of the problems we had at the beginning, right?


Deep Learning for Symbolic Mathematics

This integral right here.


Deep Learning for Symbolic Mathematics

You can write this in a string, like we said,


Deep Learning for Symbolic Mathematics

and then derive its solution right here


Deep Learning for Symbolic Mathematics

and have the neural network learn to map one to the other,


Deep Learning for Symbolic Mathematics

right, to map this to that.


Deep Learning for Symbolic Mathematics

So the way it goes is it would map this into,


Deep Learning for Symbolic Mathematics

map this into its tree representation,


Deep Learning for Symbolic Mathematics

tree representation.


Deep Learning for Symbolic Mathematics

It would map this into its prefix notation, right?


Deep Learning for Symbolic Mathematics

It would also map this to,


Deep Learning for Symbolic Mathematics

sorry, let's take another color here.


Deep Learning for Symbolic Mathematics

Map this into its tree.


Deep Learning for Symbolic Mathematics

Then it will map this into its prefix notation.


Deep Learning for Symbolic Mathematics

And then that's the training data.


Deep Learning for Symbolic Mathematics

The training data is take this, derive that, right?


Deep Learning for Symbolic Mathematics

And at inference time, of course, you won't have this here.


Deep Learning for Symbolic Mathematics

You'll simply be asked to output a sequence


Deep Learning for Symbolic Mathematics

as a normal natural language.


Deep Learning for Symbolic Mathematics

Like you can think of machine translation, right?


Deep Learning for Symbolic Mathematics

This thing translates problems into solutions.


Deep Learning for Symbolic Mathematics

It's crazy.


Deep Learning for Symbolic Mathematics

I mean, it's not technically super challenging,


Deep Learning for Symbolic Mathematics

but it's crazy that it works or that it could work, right?


Deep Learning for Symbolic Mathematics

So we'll see how this actually works.


Deep Learning for Symbolic Mathematics

They use a transformer model,


Deep Learning for Symbolic Mathematics

which is a classic model.


Deep Learning for Symbolic Mathematics

If you don't know what a transformer is,


Deep Learning for Symbolic Mathematics

I have a video called


Deep Learning for Symbolic Mathematics

Attention is All You Need About Transformers.


Deep Learning for Symbolic Mathematics

You can basically use them to do these kinds of tasks,


Deep Learning for Symbolic Mathematics

to map one string into another string.


Deep Learning for Symbolic Mathematics

So, yeah.


Deep Learning for Symbolic Mathematics

So they go into detail here


Deep Learning for Symbolic Mathematics

of how they construct the dataset


Deep Learning for Symbolic Mathematics

and how big the problem space is and so on.


Deep Learning for Symbolic Mathematics

Ultimately, they compare their system to Mathematica,


Deep Learning for Symbolic Mathematics

I think, and Maple and MATLAB,


Deep Learning for Symbolic Mathematics

which do the same thing.


Deep Learning for Symbolic Mathematics

So Mathematica, which is the kind of desktop version


Deep Learning for Symbolic Mathematics

of Wolfram Alpha that I've shown you, you have it here.


Deep Learning for Symbolic Mathematics

So integration is the task of integrating,


Deep Learning for Symbolic Mathematics

let's say, these symbolic expressions.


Deep Learning for Symbolic Mathematics

ODE order one and order two are slightly different tasks


Deep Learning for Symbolic Mathematics

where you're asked to solve


Deep Learning for Symbolic Mathematics

an ordinary differential equation,


Deep Learning for Symbolic Mathematics

which is also a task in symbolic mathematics.


Deep Learning for Symbolic Mathematics

If you compare it to Mathematica here,


Deep Learning for Symbolic Mathematics

and they give Mathematica a limit of 30 seconds,


Deep Learning for Symbolic Mathematics

what Mathematica will do is it will kind of search


Deep Learning for Symbolic Mathematics

the manipulations that it knows.


Deep Learning for Symbolic Mathematics

So the advantage of this is it can always give you,


Deep Learning for Symbolic Mathematics

let's say, a step-by-step solution


Deep Learning for Symbolic Mathematics

if it finds a solution.


Deep Learning for Symbolic Mathematics

It will just start and it will do a tree search


Deep Learning for Symbolic Mathematics

manipulating the expression you give in


Deep Learning for Symbolic Mathematics

until it reaches a satisfactory solution.


Deep Learning for Symbolic Mathematics

But then once it has that,


Deep Learning for Symbolic Mathematics

it can give you a path through the tree,


Deep Learning for Symbolic Mathematics

which leads to the solution,


Deep Learning for Symbolic Mathematics

which will give you a step-by-step solution


Deep Learning for Symbolic Mathematics

so you can understand it.


Deep Learning for Symbolic Mathematics

The system that Facebook designs here doesn't do that.


Deep Learning for Symbolic Mathematics

It simply takes the input tokens, like this is the input,


Deep Learning for Symbolic Mathematics

and it just gives you an output.


Deep Learning for Symbolic Mathematics

That is learned so that the network per se


Deep Learning for Symbolic Mathematics

doesn't understand math.


Deep Learning for Symbolic Mathematics

It simply learns from many, many examples


Deep Learning for Symbolic Mathematics

to transform, to come up with good hypotheses.


Deep Learning for Symbolic Mathematics

So if you compare here Mathematica, for example,


Deep Learning for Symbolic Mathematics

it can integrate 84% of the things that they put into it.


Deep Learning for Symbolic Mathematics

It's not said whether it gets it wrong