Deep Reinforcement Learning for News Recommendation. Part 1: Architecture. | by Mikhail Scherbina | Towards Data Science

I will be trying to cover this paper, if you want more details, consider reading it.
.
.
.
At this point, the article is pretty much outdated. It contains a lot of mistakes and inconsistencies. I published another atricle that is 33min read (7k words) that gives an in-depth view of the project. Consider reading it instead. P.s. it is absolutely free of charge: link
towardsdatascience.com
.
.
.
Most of the existent recommendation systems use static approach for news predictions, i.e. they do not understand that your actions (news choices) are sequential. The second problem is that previous methods are often short-sighted, meaning they will not focus on long-term rewards. This problem may easily be solved using higher gamma value, but usually, it is not the case.
Embeddings are a tool that allows us to encode the meaning of some objects in order to find out how similar between each other they are. In the example below, you can see that Tolstoy’s books are quite similar between each other but drastically differ from The Hitchhicker’s. They are expressed using vectors and the similarity is calculated using dot product notation
books = ["War and Peace", "Anna Karenina",           "The Hitchhiker's Guide to the Galaxy"]books_encoded_ideal = [[0.53,  0.85],                       [0.60,  0.80],                       [-0.78, -0.62]]Similarity (dot product) between First and Second = 0.99Similarity (dot product) between Second and Third = -0.94Similarity (dot product) between First and Third = -0.97
A brief introduction to reinforcement learning: we have an agent acting in an environment. The actions are rewarded in form of scores, accordingly. Environment supplies the agent with states and takes an action, taking him to the next step. The whole thing is called the Markov Decision Process (or MDP for short). Basically, there are two types of forecasting: Policy and Q-Learning. Q-Learning trains a neural network trying to estimate the best reward for next and future steps (have in mind that reward might be different for different actions). However, policy learning tries to learn probabilities of the next actions.
States S. A state s is the representation of user’s positive interaction history
Actions A. An action a is a vector of ranking scores. The action is an ideal news piece the user would like to read.
Transitions P. The state is modeled as the representation of a user’s positive interaction history. Hence, once the user’s feedback is collected, the state transition is defined.
Reward R. Given the recommendation based on the action a and the user state s, the user will provide her feedback, i.e., click, not click, or rating, etc. The recommender receives immediate reward R(s, a) according to the user’s feedback.
Discount rate γ. γ ∈ [0, 1] is a factor measuring the present value of long-term rewards.
The problem with Q-Learning is that due to the evaluation on overall actions, the approaches may become very inefficient if the action space is too large. Whereas policies are more convergent on larger action spaces, they cannot understand the sequential features.
That’s enough introduction, let’s see how it looks like:
Suppose, we observed users’ behavior and acquired some news posts they clicked on. It is fed into the Actor Network which decides what we would like to read next. It produces an ideal news embedding. It can be compared with other news embeddings to find similarities. The most matching one will be recommended to the user. The Critic helps to judge the Actor and help it find out what is wrong. It is also helpful to predict future behavior. For example, the recommender suggests the user buy a book. It buys it no problem and receives an immediate reward of 100. But it may also happen that the user will want to return that book in the future, penalizing us by -500. All future actions need to be taken into consideration. So does the Q-Network (Critic), as mentioned before, policy methods are often short-sighted and not capable of forecasting future rewards.
As you can see, the network is comprised of two layers: the Actor and the Critic. Each one resembles different learning types: the Actor learns Policies (probabilities of which action to choose next), whereas the Critic is focused on rewards (Q-Learning). First of all, a bunch of news embeddings are fed into the Actor’s state representation module, where they are encoded. Then a decision (action) is made (in form of a vector). Combined with item embeddings, the action is fed into the Critic Module which aims to estimate how good the reward is going to be.
First goes the state representation module, ‘which explicitly models the complex dynamic user-item interactions to pursuit better recommendation performance’. In simple words, you can think of it as embeddings capturing semantic meanings of the words in a sentence, but with news and clients instead. Embeddings of embeddings, how pathetic. It plays two key roles: for ranking and combined with the action as an input for the Critic network.
For a given user, the network accounts for generating an action based on her state s. The user state, denoted by the embeddings of her n latest positively interacted items is used as the input. They are fed into the State Module to produce a summarized representation of the state for the user. Then the representation is transformed into the action, a vector of size n ranged between (-1, 1).
As you can see, the author doesn’t use softmax for ranking scores:
Note that in our model, an action corresponds to neither recommending an item nor recommending a list of items. Instead, an action is a continuous parameter vector. Taking such an action, the parameter vector is used to determine the ranking scores of all the candidate items, by performing product with item embeddings (multipying vectors).
It is important at the next step:
It is used to estimate how good the reward of the current state and action will be: Qπ (s, a). The input of the Critic network is the user state s generated by the user state representation module and the action generated by the policy network, and the output is the Q-value, which is a vector. According to the Q-value, the parameters of the Actor network are updated in the direction of improving the performance of action a. I am not going to write all the gradients, primarily because I use Pytorch. The loss function is plain MSE since we are going to estimate real-valued rewards that are not typically normalized so it is a regression problem.
DRR-p — utilizes pairwise dependency between items. It computes the pairwise interactions between the n items, by using the element-wise product operator. (The user-item interactions are neglected!)
In DRR-u, we can see that the user embedding is also incorporated. In addition to the local dependency between items, the pairwise interactions of user-item are also taken into account.
When we work with a large long-term batch of news, we don’t expect the positions to matter. But memorizing the positions of items may lead to overfitting if the sequence H is a short-term one. As an average pooling layer is adopted, we call the structure DRR-ave. We can see from Figure 6 that the embeddings of items in H are first transformed by a weighted average pooling layer. Then, the resulting vector is leveraged to model the interactions with the input user. Finally, the embedding of the user, the interaction vector, and the average pooling result of items is concatenated into a vector to denote the state representation.
In the next article, I will try to implement this network in Pytorch using Deep Deterministic Policy Gradient, as it’s described in the original paper. If this gets enough attention, I will explain the algorithm in details. Give it a clap and stay tuned!
Upd: Started a github project (not ideal yet, but the model is learning)
Processed data as it is will be uploaded soon, but now it takes a couple of hours to process. Dm me if you need it. Have some big ideas for the next article which is going to be the last one on this topic, but pretty big and explanatory. Don't forget to star!