Recurrent Exploration Networks for Recommender Systems

Abstract: Recurrent neural networks have proven effective in modeling sequential user feedbacks for recommender systems. However, they usually focus solely on item relevance and fail to effectively explore diverse items for users, therefore harming the system performance in the long run. To address this problem, we propose a new type of recurrent neural networks, dubbed recurrent exploration networks (REN), to jointly perform representation learning and effective exploration in the latent space. REN tries to balance relevance and exploration while taking into account the uncertainty in the representations. Our theoretical analysis shows that REN can preserve the rate-optimal sublinear regret (Chu et al., 2011) even when there exists uncertainty in the learned representations. Our empirical study demonstrates that REN can achieve satisfactory long-term rewards on both synthetic and real-world recommendation datasets, outperforming state-of-the-art models. 

Reviews and author responses:

We thank all reviewers for their constructive comments. They mention that our work is novel/interesting (R1, R3), that our theoretical analysis is sound (R1, R3, R4), and that our experiments show our methods’ effectiveness (R1). They also voice several concerns, which we address one by one below. We also revised the paper, the Appendix, and the supplementary materials accordingly.

Q5: The regret analysis seems to be a trivial application of LinUCB since $\sigma_k = 1 / \sqrt{n_k}$, which in total can be bounded in order of $O(\sqrt{T})$. I would suggest not to sell this as a theoretical contribution.

We would like to reiterate that both (1) the discovery of the key role that item embedding uncertainty plays and (2) the proof of the rate-optimal regret bound are non-trivial. 

Also note that we are the first to consider feature uncertainty in the bandit setting. Additionally, our theory provides guidance on how to select $\lambda_d$ and $\lambda_u$.

The relation between $\sigma_k = 1 / \sqrt{n_k}$ and the assumption in Lemma 4.4 is another interesting finding (note that they are not exactly the same though). It essentially shows that only when the uncertainty decays at the rate of $O(1/\sqrt{t})$ can we preserve the $O(\sqrt{T})$ regret bound.

It is also worth noting that naively combining LinUCB and RNN would not work. As mentioned in the introduction and evaluated in Figure 2, REN-1,2 (which combines LinUCB and RNN) does not work in some cases since the uncertainty of the item embeddings are not taken into account. Including such uncertainty and proving that the rate-optimal regret bound (which is non-trivial) still holds even in face of item embedding uncertainty is one of our key contribution.

Q6: It would be good to see how this result translates to other underlying distributions of item embeddings besides multivariate normal.

This is a good idea. It is indeed possible to extend the theory to handle distributions besides multivariate normal. The key would be to use the variance of such distributions to replace the Gaussian variance in the previous theory and derive a similar confidence bound. As long as the variance of such new distributions also satisfies Lemma 4.4, we can expect the bound to be preserved. Correspondingly, empirical results should be similar in this case too.

We appreciate your constructive comments which we tried our best to respond to as above. We also revised the paper, the Appendix, and the supplementary materials accordingly. If we missed anything, please feel free to let us know and we are happy to provide any more details/results needed.

Thank you for your encouraging comment that our idea is novel and that the effectiveness of our model is demonstrated both theoretically and empirically. Below we addressed the mentioned concerns one by one.

Q1: The reward is lower than other baselines at the beginning. I understand the model is exploring and gather information, but that would more or less hurt users' satisfaction when deploying the model to real-world recommender systems.

You are correct that the initial drop of performance is a necessary sacrifice to explore and gather information. However, we note that one could always pretrain the model so that the model could have a warm start. See Figure 3(middle) for the Trivago experiments where the model is pretrained and therefore achieve satisfactory cold-start performance. We can also see similar results in Figure 3(right) for the Netflix experiments. In both Trivago and Netflix experiments, the initial performance is much higher than RANDOM. We would expect similar results if the model is pretrained for experiments on the toy datasets and MovieLens.

For deployment, part of REN (a version with the first and third terms of Eq. (3)) was recently deployed in the production system, and the full REN is now a production candidate. So far the system is receiving good feedback from customers.

Q2: The datasets the authors used are subsets of movielens and trivago, which are a bit small.

We included additional results for the full dataset of MovieLens-1M in Figure 4 of the Appendix (see the revised paper). Note that the full result of all users is almost the same as Figure 3. This is because (1) All users in the Movielens dataset are evaluated in parallel. (2) The 1000 users are selected randomly from the full dataset with 6040 users. 

In our preliminary experiments we have results for 500 users and 750 users, respectively; the results for 500 users, 750 users, and 1000 users were nearly the same (due to the large sample size) and therefore we expected similar results for the full dataset. This is why we included only 1000 users in the initial version. We refer the reviewer to Figure 4 of the Appendix in the revised paper for more details.

For Trivago, note that this is an extremely large dataset (larger than Netflix with 80 times more items). Even the subset we use contains 57,778 users and 387,348 items with 108,713 interactions. This is already much larger than MovieLens-1M in terms of user space and item space.

Q3: Also how about the cumulative reward? It would be more reasonable to compare their cumulative reward to show the starting exploration is worthy.

This is a good point. In fact, whether the exploration is worthy depends on how long the model can ‘reap’ the benefit. For example, as shown in Figure 1, all non-REN models saturate after some time step while our REN models can achieve the optimal reward. This means REN models will certainly have much higher cumulative rewards than non-REN baselines as long as the time horizon is long enough. This is also why we focus on whether the optimal reward can be achieved rather than cumulative rewards.

Q4: It is said in Section 4 "Note that unlike existing works which primarily consider the randomness from the reward, we take into consideration the uncertainty resulted from the context". Do you mean the reward r is fixed when x is fixed? And the randomness of r is dependent solely on the randomness of x? If so, this formulation is equivalent to the existing works, and why do you choose such a formulation?

Thank you for the clarifying question. We should have been clearer that: 

Existing works assume deterministic $x$ and only assume randomness in the reward. More specifically, they assume that $r = x^T \theta + \epsilon$, and therefore r’s randomness is *independent* of $x$. The problem with this formulation is that they assume $x$ is deterministic and therefore the model only has a point estimate of the item embedding $x$, but does not have uncertainty estimation for such $x$. We find that such uncertainty estimation is crucial for exploration; if the model is uncertain about $x$, it can then explore more on the corresponding item. 

This is why we also assume uncertainty in $x$ for our REM formulation. In REM, the embedding $x$ corresponds to a Gaussian distribution $N(\mu, \sigma)$ rather than being deterministic as in existing works, where $\sigma$ is the uncertainty that helps us choose items to explore (the third term of Equation 3).

We added the clarification above in the revised paper.


Q1: Though previous exploration term of LinUCB (or others) is calculated directly on the input feature, it is not very surprising to use it with respect to the embedding feature especially for highly sparse input, at least heuristically.

In fact, naively combining LinUCB and RNN would not work. As mentioned in the introduction and evaluated in Figure 2, REN-1,2 (which combines LinUCB and RNN) does not work in some cases since the uncertainty of the item embeddings are not taken into account. Including such uncertainty and proving that the rate-optimal regret bound (which is non-trivial) still holds even in face of item embedding uncertainty is one of our key contribution.

Q2: Second, for theoretical results, I think the hardness mainly lies in dealing with calculated by RNN instead of linear form. However, given the strong assumption 4.1, all the theoretical statements and corresponding proof seem to be nearly the same as LinUCB, and I didn't see the role of RNN.

We would like to reiterate that both (1) the discovery of the key role that item embedding uncertainty plays and (2) the proof of the rate-optimal regret bound are non-trivial. 

Also note that we are the first to consider feature uncertainty in the bandit setting. Additionally our theory provides guidance on how to select lambda_d and lambda_u.

Q3: In particular, the $\hat{r}_{t,k}$ notation in equation 4 is the same as Line 9 in Algorithm 2, but their definitions are different, and all the proof seems to rely on notation defined in equation (4) instead of Line 9 in Algorithm 2.

This is a good point and exactly why we included the last paragraph of Section 4.1 for clarification:

RNN to Estimate $\theta_t$. REN uses RNN to approximate $A_t^{-1} b_t$ (useful in the proof of Lemma 4.1) in Equation 4. Note that a linear RNN with tied weights and a single time step is equivalent to linear regression (LR); therefore RNN is a more general model to estimate $\theta_t$. Compared to LR, RNN-based recommenders can naturally incorporate new user history by incrementally updating the hidden states ($\theta_t$ in REN), without the need to solve a linear equation. Interestingly, one can also see RNN's recurrent computation as a simulation (approximation) for solving equations via iterative updating.

Q4: Third, since Algorithm 1 uses RNN through all the time horizon which may be infinite, I am not clear how to do BPTT with respect to RNN. It would be much better if authors could explain it in detail.

Though theoretically Algorithm 1 can support infinite time horizon, in practice the history length of each user is finite and therefore BPTT is doable. It is worth noting that history from different users are not concatenated, therefore the history of each user is a separate sequence fed into the RNN, and BPTT is performed for each user.

Q5: In the first line of section 3.3, there are two "can"

Thanks for the good catch. We have fixed the typo in the revised version.

Q6: authors assume the variance of $x_k$ is a diagonal matrix, I wonder whether results still hold if the covariant matrix is a general PSD matrix

By $x_k$, do you mean $\sigma_k$? This is a good question. In general, our theory can be extended to handle such a full PSD covariance matrix. The trick is to replace the L infinity norm of the vector with some matrix norm, e.g., nuclear norm.

Q7: $A_t$ defined in Algorithm 1 and Algorithm 2 are not used in algorithms, which can be removed

Thanks for the good catch. $A_t$ is meant as a cache to speed up the computation of Line 9 of Algorithm 1 and Line 8 of Algorithm 9. We have revised Algorithm 1 and 2 to make it clearer. 

Q8: $\hat{A}_s$ is undefined in the main content

Thanks for the good catch. $\hat{A}_s$ is defined inside Algorithm 2 (SupREN) of the supplementary materials. We have made this clear in the revised version.

We appreciate your constructive comments which we tried our best to respond to as above. We also revised the paper, the Appendix, and the supplementary materials accordingly. If we missed anything, please feel free to let us know and we are happy to provide any more details/results needed.

Q9: Are you predicting t time steps in the future?

For the toy datasets, Trivago, and Netflix, we predict 1 time step ahead. For MovieLens-1M, we predict 5 time steps ahead. As mentioned in Q8, one time step in Figure 3 represents 10 recommendations to a user, one hour of data, and 100 recommendations to a user for MovieLens-1M, Trivago, and Netflix, respectively.

We appreciate your constructive comments which we tried our best to respond to as above. We also revised the paper, the Appendix, and the supplementary materials accordingly. If we missed anything, please feel free to let us know and we are happy to provide any more details/results needed.

Q1: …The experiment of this paper is weak…

We demonstrated REN’s performance improvement over three state-of-the-art sequential recommender models (GRU4Rec at ICLR 2016, TCN in 2018, and HRNN, which is the best paper at KDD 2020) as baselines. We evaluated all methods on 3 toy datasets and 3 large-scale real-world datasets, with comprehensive ablation studies.

Q2: …for the MovieLens dataset, it only selects 1000 users. Not sure why cannot compare the approach on all the users?

We included additional results for the full user set in Figure 4 of the Appendix (see the revised paper). Note that the full result of all users is almost the same as Figure 3. This is because (1) All users in the Movielens dataset are evaluated in parallel. (2) The 1000 users are selected randomly from the full dataset with 6040 users. 

In our preliminary experiments we have results for 500 users and 750 users, respectively; the results for 500 users, 750 users, and 1000 users were nearly the same (due to the large sample size) and therefore we expected similar results for the full dataset. This is why we included only 1000 users in the initial version. We refer the reviewer to Figure 4 of the Appendix in the revised paper for more details.

Q3: The middle figure of Figure 3 also looks strange. Why the models' performance drops with more time steps. The authors should explain.

Thanks for mentioning this, and we are sorry for being too vague in the last few sentences of the Trivago paragraph. The baseline RNN (e.g., GRU4Rec) models suffer from a drastic drop in rewards because agents are allowed to recommend only 10 items and they choose to focus only on relevance. This will inevitably ignore valuable items and harms the accuracy. In contrast REN variants (e.g., REN-G) can effectively balance relevance and exploration for these 10 recommended items at each time step, achieving higher long-term rewards. We have added the discussion above in the revised version.

Q4: It lacks parameter sensitive analysis to see how different values of lambda_d and lambda_u will impact the result.

We did have some parameter sensitive analysis on lambda_d and lambda_u in the supplementary materials (i.e., the zip file). We essentially observed stable REN performance across a wide range of the hyperparameters. Please refer to Section 3 and Figure 1 of the supplementary materials for more details.

Q5: In section 3.1 "Strictly speaking, in an online setting where the model updates at every time step t, x_{k} also changes over time;", why only when modeling updates in online setting, x_{k} change? Shouldn't already be changing even if it is not online setting?

Yes, you are correct. In general, $x_{k}$ can be changing whether or not it is online setting. Since our paper focuses only on the online setting, the sentence in Section 3.1 is only meant to emphasize the fact that $x_{k}$ changes with $t$, and therefore the notation $x_{t,k}$ is necessary. 

Q6: What is an encoder in an RNN? Did you mean input embedding?

Yes. The encoder takes in the item ID (as a one-hot vector) and output the low-dimensional item embedding.

Q7: What is the time-complexity of the proposed approach? Does it impact the usage in real-time inference?

For training the time complexity is identical to the vanilla RNN models. For inference, the complexity for the vanilla RNN models and REN is $O(Kd)$ and $O(Kd^2)$, respectively. Note that since d is typically much smaller than K (e.g., K = 3900 and d = 256 for the MovieLens-1M experiments), the latency for real-time inference is sufficiently low. 

Part of REN (a version with the first and third terms of Eq. (3)) was recently deployed in the production system, and the full REN is now a production candidate. The system achieved low overhead compared to vanilla RNN-based recommenders. The total latency is below 300 ms per query. The main overhead is due to the matrix inverse in the second term of Eq. (3), where one can use diagonal approximation (with $O(d)$ complexity), Cholesky decomposition, or conjugate gradients to further speed up computation.

Q8: It is not very clear what are the time intervals to divide the data into different settings. What is the time dimension in the result figures?

As mentioned in the caption of Figure 3, one time step in this figure represents 10 recommendations to a user, one hour of data, and 100 recommendations to a user for MovieLens-1M, Trivago, and Netflix, respectively. 

For Figure 1 and Figure 2, one time step represents one interaction step, where in each interaction step the model recommends 3 items to the user and the user interacts with one of them. We have added such description in the caption of Figure 1 for clarity as suggested by the reviewer.


Thank you for your encouraging comment that our approach is interesting, has good theoretical support, and well presented. Below we addressed the mentioned concerns one by one.

Q1: …the approach would have been more interesting/novel if the full equation 3 (or its approximation) was back-propagated to tailor the representations for exploration…

This is a good question. In the preliminary experiments, we did try to back-propagate the full Equation 3 during training and surprising (not too surprisingly in retrospect though) found that it leads to low long-term rewards. The reason might be that the observed feedback is relevance-based and therefore would steer the exploration terms (e.g., the second term) to focus more on relevance, but not exploration. This is why we later decided to exclude the exploration term during training, which leads to much better performance.

Q2: …In the current form the training algorithm is similar to the numerous item selection/sampling schemes that have been proposed…

(1) It is actually an advantage that we can train the RNN recommender as normal, and only need to implement the exploration terms in the inference phase, because this would lead to no overhead during training and only minimal overhead during inference in the production system.

(2) Our REN is fundamentally different from previous item selection/sampling schemes: (a) it provides theoretical guarantees and guidance on setting the hyperparameters; (b) it explores items in a compact latent space, which is more efficient and avoids redundant exploration of similar items; (c) it focuses on modern RNN recommenders and is compatible with any choice of RNN recommender models.

Q3: …Virtually any form of exploration will be beneficial here…

We agree that any form of exploration will be beneficial. However, most of them do not have optimal long-term reward. This is actually why we include the ablation study in Figure 2, which shows that neither REN-1,2 (including only the second term of Equation 3 for exploration) nor REN-1,3 (including only the third term of Equation 3 for exploration) can work perfectly. Only our proposed full REN model, i.e., REN-1,2,3, can work in both large and small item space.

Q4: …I think the difference between properly trained relevance-only models and the proposed approach will be significantly smaller…

This is a good point. This is why we include the Trivago experiment which uses much more data for pretraining; in that experiment, the model is pretrained with as much as 13 hours of data until convergence (over 25% of the evaluation dataset). We can see in Figure 3 (middle) that the model actually performs very well even at time step 0. After step 0, state-of-the-art RNN models’ performance drop dramatically due to lack of exploration while REN can maintain the high performance with proper exploration.

Q5: …a longer time horizon and needs to be properly evaluated…

(1) Thanks for mentioning this. We should have made clearer in the paper that the performance (reward) of all non-REN models in Figure 1 and Figure 3 saturates after some time step. Therefore, training with longer time horizon does not alter their final rewards.

(2) For longer time horizon, please refer to Figure 2 in the supplementary materials where we have results for the Netflix experiments for a much longer horizon of 50.

Q6: …production environment…

Part of REN (a version with the first and third terms of Eq. (3)) was recently deployed in the production system, and the full REN is now a production candidate. The system achieved low overhead compared to vanilla RNN-based recommenders. The total latency is below 300 ms per query. The main overhead is due to the matrix inverse in the second term of Eq. (3), where one can use diagonal approximation (with O(d) complexity), Cholesky decomposition, or conjugate gradients to further speed up computation.

We appreciate your constructive comments which we tried our best to respond to as above. We also revised the paper, the Appendix, and the supplementary materials accordingly. If we missed anything, please feel free to let us know and we are happy to provide any more details/results needed.


The authors propose a recurrent neural network approach to jointly perform representation learning and exploration. At each recurrence step the user-item relevance score is combined with item diversity score. The diversity score compares the item to already recommended items in previous steps to encourage exploration. The trade-off between relevance and exploration is controlled with a hyper-parameter.

I think the proposed approach is interesting, has good theoretical support and is presented well. However, I also have several major concerns. First, the exploration score is not used to update the RNN model or representations. From the second last paragraph in Section 3.3 the authors say that the model is trained using standard relevance objective and variance is fixed. I think the approach would have been more interesting/novel if the full equation 3 (or its approximation) was back-propagated to tailor the representations for exploration. In the current form the training algorithm is similar to the numerous item selection/sampling schemes that have been proposed.

Second, I don't think that the experimental set-up is fair. The models only get to see a small subset of data in [T0, T1), then predict on subsequent intervals and use these predictions for parameter updates. Having only been trained with data in [T0, T1) the models will be quite weak with highly unreliable predictions. This will cause the feed-back loops from subsequent intervals to also have poor signal resulting in very slow learning. Virtually any form of exploration will be beneficial here particularly for the Trivago dataset where there are only 25 alternatives. I don't think this stimulates any production environment since the models are too weak at the beginning to be deployed. This can be seen from the relatively easy MovieLens dataset where precision curves at the beginning are all near 0. I think the difference between properly trained relevance-only models and the proposed approach will be significantly smaller, especially over a longer time horizon and needs to be properly evaluated. Netflix experiments make a step in that direction but the time horizon is very short.

Summary:
The authors study the problem of exploration for recurrent neural networks in the sequential recommendation. This work presents a scoring function that combines relevance, diversity, and uncertainty during exploration.  A sublinear bound on the regret of the model is provided, and experiments show the model achieves higher reward at the final stage. 

Pros:
- This work incorporates diversity into recurrent neural networks with determinantal point processes. Improving diversity for recurrent neural networks is an important task in the sequential recommendation.
- The idea of using the hidden state as the candidate estimate of \theta in linear bandits is novel. It shows an approach to incorporate the exploration idea from bandits.
- The effectiveness is demonstrated both theoretically and empirically. 

Cons:
- The reward is lower than other baselines at the beginning. I understand the model is exploring and gather information, but that would more or less hurt users' satisfaction when deploying the model to real-world recommender systems. 
- The datasets the authors used are subsets of movielens and trivago, which are a bit small.

Comments on the paper:
- The performances are even worse than the RANDOM baseline at the beginning. Such a cold-start performance is not very appealing. Also how about the cumulative reward? It would be more reasonable to compare their cumulative reward to show the starting exploration is worthy.

- It is said in Section 4 "Note that unlike existing works which primarily consider the randomness from the reward, we take
into consideration the uncertainty resulted from the context". Do you mean the reward r is fixed when x is fixed? And the randomness of r is dependent solely on the randomness of x? If so, this formulation is equivalent to the existing works, and why do you choose such a formulation?

- The regret analysis seems to be a trivial application of LinUCB since \sigma_k = 1 / \sqrt{n_k}, which in total can be bounded in order of O(\sqrt{T}). I would suggest not to sell this as a theoretical contribution.

- It would be good to see how this result translates to other underlying distributions of item embeddings besides multivariate normal.

Overall, this paper is well-motivated and clearly written but has several issues that remain to be clarified.

Diversity is undoubtedly an important problem in recommender systems which now relies heavily on neural networks especially RNN. However, since RNN focus on a single output at each time step, it seems challenging to achieve satisfying diversified results in RNN based RS. Based on the similarity between greedy solution of DPP and exploration term in LinUCB, this paper proposes an additional diversity score for each candidate item and then uses the summation of diversity score and original RNN score to select final prediction at each step. Authors prove some theoretical results with regret bound $\sqrt{T}$. Besides, numerical results validate the effects of proposed methods, which may be attractive to practitioners in RS. 

Given the strength of this paper, I still have some concerns. First, the novel contrition of this paper seems limited. Though previous exploration term of LinUCB (or others) is calculated directly on the input feature, it is not very surprising to use it with respect to the embedding feature especially for highly sparse input, at least heuristically. Second, for theoretical results, I think the hardness mainly lies in dealing with $\theta$ calculated by RNN instead of linear form. However, given the strong assumption 4.1, all the theoretical statements and corresponding proof seem to be nearly the same as LinUCB, and I didn't see the role of RNN. In particular, the notation $\hat{r}_{t,k}$  in equation 4 is the same as  Line 9 in Algorithm 2, but their definitions are different, and all the proof seems to rely on notation defined in equation (4) instead of Line 9 in Algorithm 2. Third, since Algorithm 1 uses RNN through all the time horizon which may be infinite, I am not clear how to do BPTT with respect to RNN. It would be much better if authors could explain it in detail. 

Some other comments:
a. In the first line of section 3.3, there are two "can";
b. Om the paper, authors assume the variance of $x_k$ is a diagonal matrix, I wonder whether results still hold if the covariant matrix is a general PSD matrix;
c. $A_t$ defined in Algorithm 1 and Algorithm 2 are not used in algorithms, which can be removed;
d. $\hat{A}_s$ is undefined in the main content;


Summary
This paper proposes a model named recurrent exploration networks (REN) model to address the exploration problem in recommendation system without sacrificing the performance. The proposed approaches include an additional diversity term by using determinantal point processes. The model also adds an additional term to measure the uncertainty. The paper claims that the proposed approach can generate better long-term better performance.

Strengths
1. The diversity in recommendation is an important research topic. Being able to balancing performance vs exploration is important. This paper also identify another important aspect which is the uncertainty.
2. The paper provides theoretical bounds for the proposed approach. 


Weaknesses
1. The experiment of this paper is weak. For example, for the MovieLens dataset, it only selects 1000 users. Not sure why cannot compare the approach on all the users?
2. The middle figure of Figure 3 also looks strange. Why the models' performance drops with more time steps. The authors should explain.
3. It lacks parameter sensitive analysis to see how different values of lambda_d and lambda_u will impact the result.

Minors:
" Essentially items" =>  Essentially, items
"In this section we first" => In this section, we first

Questions

In section 3.1 "Strictly speaking, in an online setting where the model updates at every time step t, x_{k} also changes over time;", why only when modeling updates in online setting, x_{k} change? Shouldn't already be changing even if it is not online setting?
What is an encoder in an RNN? Did you mean input embedding?
What is the time-complexity of the proposed approach? Does it impact the usage in real-time inference?
It is not very clear what are the time intervals to divide the data into different settings.
What is the time dimension in the result figures? Are you predicting t time steps in the future?


Meta-review: This paper overall received borderline negative scores. All the reviewers agree that the paper proposed an interesting approach to exploration for RNN-based recommender systems. However, there are concerns around the experiments as well as the theoretical contribution. Specifically, a few reviewers pointed out that the experimental results are not convincing. Furthermore, some reviewers mentioned that the theoretical analysis is a relatively straightforward extension of LinUCB (to which the authors disagreed, but there wasn't any followup discussion from the reviewers). The bandit analysis is unfortunately outside my expertise. 

I think this paper touches upon two different domains (RNN-based recommender systems and bandit). If the authors consider that their contribution mainly lies in the latter, maybe ICLR is not the most suitable venue. 

Decision: withdrawn-rejected-submissions