Hey guys welcome to RetaTract your one stop channel for all the data science and machine learning updates. In today's video we look at Twitter recommendation system. Twitter recently open sourced its code base and also published a blog highlighting the main components. We will go through the blog at high level and as well as try to understand how the recommendation system at Twitter is different from other social media that forms intuitively. So let's get started. There are many areas of the app where tweets are recommended like search, explore and add. The recommendation system filters out a small set of tweets, top tweets from approximately 500 million tweets that are posted every day and these are displayed on device for you timeline. So basically you are talking about the personalization that out of 500 million tweets are posted every day. A few tweets which may be more relevant to the users are personalized and displayed in there for you timeline which is the homepage and the recommendation system set up like any other recommended system the Twitter's recommended system also consists of three main stages. One is candidate sourcing which is the retrieval year where the best tweets from different recommendations sources are retrieved then followed by ranking their ML model ranks these tweets but should be the ordering at which they should be displayed to the users in their homepage. And also third stages applying some legal filters like filtering tweets from blogged or muted users and balancing a good mix of tweets so it doesn't happen same type of content is repeating right so it's like different mix or diversity is there in the home feed and also appreciate it. The user has provided some negative feedback that users are like to see this tweet or this kind of tweet or tweet from a particular user so appreciate those feedbacks and so the final tweets are there 500 million tweets then 10-rate generator source a bunch of tweets which are ranked by the ranker and followed by some futuristic and filters it's shown in the homepage finally. Next is the candidate sources so the retrieval year right so candidate sourcing sources 1500 tweets 50% from people you follow it is your in-network and 50% from people you don't follow out of network and this is where the difference between Twitter and other social media platform lies so basically other social media platform like Facebook Instagram is so more content from your friends but Twitter since it's social media have where you try to understand what is happening around the world and topics that you are interested but the influencers are saying on that about that that's why it's important to have more candidate or tweets or source candidate sources from out of network as well so 50% from in-network people you follow it 50% from people you don't follow out of network and now let's understand how in-network sourcing happens right for the getting the in-network tweets the in-network source uses a logistic regression model and a real graph this is a graph algorithm of Twitter to rank tweets from user you follow so it uses a logistic regression in real graph to rank tweets from user you follow the in-network the logistic regression model efficiently ranks tweets based on their relevance while the real graph predicts the likelihood of engagement between two users and help prioritize tweets from those you have stronger connection with so basically the real graph model will try to push up the tweets of the users you follow and you have stronger connection with while logistic regression will finally rank all these tweets and 750 of them will be sent to the anchor because the less 750 will come from the out network now understanding the out of network source tweets Twitter uses two approaches to find relevant tweets outside user social network one is social graph another is embedding spaces social graph estimates relevance by analyzing the engagement of people you follow or those with similar interest using graph which is another graph model of Twitter so basically the people you follow what are they engaging and also the interest you have what are other users with similar interest are engaging with this kind of content which are not directly in your network but people of your network are engaging with or people with similar interest are engaging with are sourced and this is then using a graph graph graph tweeters graph model it generates candidates tweets based on recent engagement recent activities happen that also are taken care of and ranks them using a logistic regression model so this is how the out of network one part of it the social graph works the other part is embedding spaces because embeddings are very hard these days and they are very useful in like once you know the embedded representation of users to each you can do approximate nearest to find relevant tweets and also the next is the embedding space in embedding spaces we need to generate numeric representation of users and tweets in community space so basically users and tweets needs to be represented as a vector on which we can do a approximate nearest number search so embedding space generated numerical representation of users and which in community space now what is this community? Twitter has a algorithm called sync cluster twitter sync cluster algorithm discovers communities anchored by influencer users using a custom nettings of Trison algorithm so basically all the influencers the main celebrities they are clustered into communities for example here you can see this is a community of pop this is a community of news days a community of something else and so on so all these influencers who have like mold followers who people consider as celebrities they are clustered into communities and then all the users and tweets are embedded or the embeddings are generated in this community space which is around 10 to 5 communities is what the block says now that the next point is as I was saying users and tweets are embedded into this community space or community dimension so the idea of this is that users can be there are almost 205 million which is 20.5 crore users that are daily active in twitter but we need to represent them in a condensed way and that condensed way is the community those communities are around 10 to 5 were like communities 0.1 million communities is one of the blocks so these users and tweets are embedded into this smaller space which is the community space and it's very easy to embed them into this space because tweets are embedded into this communities by looking at their current popularity in each community right similarly users can be embedded into communities based on their affinity towards community user can be interested in pop news and maybe if you have see something like that so the user can be interested in those of communities and his value of those dimension representing those communities will be highlighted so in this way tweets and users are embedded into the community dimension or community finally there is the ranking model because the retrieval has happened and as well as from in network out network the retrieval has happened next step is the ranking using ML model 1500 candidate source that is from in network out network are ranked through a 48 million parameter neural network that is continuously trained on quick interactions they mean to account thousands of features to predict the engagement probability that what is the probability of user liking, retweeting or deploying to a tweet so and which are higher probability those will come higher in the home field of user and after candidate source ranking there is a filtering step and blending step in filtering as we have already briefly touched before after ranking hew districts and filters are applied to filter out tweets from blog the newtid users and also a good mix of tweets so that not same kind of content is getting repeated is done and at the last step after filtering the system blends together tweets with other non-quick content like ads or follow recommendation whom to follow and on boarding prompts which are returned to user device to display so finally the rank content is blended with ads follow recommendations and on boarding prompts so the home page 5 blend runs 5 billion thanks per day and completes in under 1.5 seconds on average so this is how the this is how the recommendation system at Twitter works there is a retrieval layer which is candidate source in both roaming network and out network followed by a ranker and then followed by filtering and hew districts and finally blending with ads follow recommended results and other recommendation which is so into the user in their home page hopefully this was helpful to you please like and subscribe and also stay tuned for more such updates why