On Component Interactions in Two-Stage Recommender Systems

Thanks to their scalability , two-stage recommenders are used by many of today ’ s largest online platforms , including YouTube , LinkedIn , and Pinterest . These systems produce recommendations in two steps : ( i ) multiple nominators—tuned for low prediction latency—preselect a small subset of candidates from the whole item pool ; ( ii ) a slower but more accurate ranker further narrows down the nominated items , and serves to the user . Despite their popularity , the literature on two-stage recommenders is relatively scarce , and the algorithms are often treated as mere sums of their parts . Such treatment presupposes that the two-stage performance is explained by the behavior of the individual components in isolation . This is not the case : using synthetic and real-world data , we demonstrate that interactions between the ranker and the nominators substantially affect the overall performance . Motivated by these findings , we derive a generalization lower bound which shows that independent nominator training can lead to performance on par with uniformly random recommendations . We find that careful design of item pools , each assigned to a different nominator , alleviates these issues . As manual search for a good pool allocation is difficult , we propose to learn one instead using a Mixture-of-Experts based approach . This significantly improves both precision and recall at K . 1 Introduction . Recommender systems play a central role in online ecosystems , affecting what media we consume , which products we buy , or even with whom we interact . A key technical challenge is ensuring these systems can sift through billions of items to deliver a personalized experience to millions of users with low response latency . A widely adopted solution to this problem are two-stage recommender systems [ 12 , 20 , 28 , 98 , 101 ] where ( i ) a set of computationally efficient nominators ( or candidate generators ) preselects a small number of candidates , which are then ( ii ) further narrowed down , reranked , and served to the user by a slower but more statistically accurate ranker . Nominators are often heterogeneous , ranging from associative rules to recurrent neural networks [ 18 ] . A popular choice are matrix factorization [ 56 , 72 ] and two-tower [ 98 ] architectures which model user feedback by the dot product between user and item embeddings . While user embeddings often evolve with the changing context of user interactions , item embeddings can typically be precomputed before deployment . The cost of candidate generation is thus dominated by the ( approximate ) computation of the embedding dot products . In contrast , the ranker often takes both the user and item features as input , making the computational cost linear in the number of items even at deployment [ 20 , 67 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . With few exceptions [ 44 , 52 , 67 ] , two-stage specific literature is sparse compared to that on singlestage systems ( i.e. , recommenders which do not construct an explicit candidate set within a separate nominator stage [ e.g. , 17 , 40 , 46 , 56 , 72 , 78 , 84 ] ) . This is especially concerning given the considerable ethical challenges entailed by the enormous reach of two-stage systems : according to the recent systematic survey by Milano et al . [ 70 ] , recommender systems have been ( partially ) responsible for unfair treatment of disadvantaged groups , privacy leaks , political polarization , spread of misinformation , and ‘ filter bubble ’ or ‘ echo chamber ’ effects . While many of these issues are primarily within the realm of ‘ human–algorithm ’ interactions , the additional layer of ‘ algorithm–algorithm ’ interactions introduced by the two-stage systems poses a further challenge to understanding and alleviating them . The main aim of our work is thus to narrow the knowledge gap between single- and two-stage systems , particularly in the context of score-based algorithms . Our main contributions are : 1 . We show two-stage recommenders are significantly affected by interactions between the ranker and the nominators over a variety of experimental settings ( Section 3.1 ) . 2 . We investigate these interactions theoretically ( Section 3.2 ) , and find that while independent ranker training typically works well ( Proposition 1 ) , the same is not the case for the nominators where two popular training schemes can both result in performance no better than that of a uniformly random recommender ( Proposition 2 ) . 3 . Responding to the highlighted issues with independent training , we identify specialization of nominators to smaller subsets of the item pool as a source of potentially large performance gains . We thus propose a joint Mixture-of-Experts [ 47 , 49 ] style training which treats each nominator as the expert for its own item subset . The ability to learn the item pool division alleviates the issues caused by the typically lower modeling capacity of the nominators , and empirically leads to improved precision and recall at K ( Section 4 ) . 2 Two-stage recommender systems . The goal of recommender systems is to learn a policy π which maps contexts x ∈ X to distributions π ( x ) over a finite set of items ( or actions ) a ∈ A , such that the expected reward Ex Ea∼π ( x ) [ ra |x ] is maximized . The context x represents information about the user and items ( e.g. , interaction history , demographic data ) , and ra is the user feedback associated with item a ( e.g. , rating , clickthrough , watch-time ) . We assume that ra|x has a well-defined fixed mean f ? ( x , a ) : = E [ ra |x ] for all the ( x , a ) pairs . To simplify , we further assume only one item at is to be recommended for each given context xt , where t ∈ [ T ] with [ T ] : = { 1 , . . . , T } for T ∈ N. Two-stage systems differ from the single-stage ones by the two-step strategy of selecting at . First , each nominator n ∈ [ N ] picks a single candidate an , t from its assigned pool An ⊆ A ( An 6= ∅ , ⋃ nAn = A ) . Second , the ranker chooses an item at from the candidate pool Ct : = { a1 , t , . . . , aN , t } , and observes the reward rt = rtat associated with at . Since the goal is expected reward maximization , recommendation quality can be measured by instantaneous regret r ? t − rt where r ? t = rta ? t is the reward associated with an optimal arm a ? t ∈ argmaxa∈A f ? ( xt , a ) . This leads us to an important identity for the ( cumulative ) regret in two-stage systems which is going to be used throughout : R2sT = T∑ t=1 r ? t − rt = T∑ t=1 ( r ? t − rtãt ) ︸ ︷︷ ︸ = : RNT + T∑ t=1 ( rtãt − rt ) ︸ ︷︷ ︸ = : RRT , ( 1 ) with ãt ∈ argmaxa∈Ct f ? ( xt , a ) . In words , RNT is the nominator regret , quantifying the difference between the best action presented to the ranker ãt and the best overall action a ? t , and R R T is the ranker regret which measures the gap between the choice of the ranker at and the best action in the candidate set Ct . The two-stage recommendation process is summarized in Figure 2 ( left ) . While Equation ( 1 ) is typical for the bandit literature ( data collected interactively , only rtat revealed for each t ) , we also consider the supervised learning case ( a dataset with all { rta } ta revealed is given ) . In particular , Section 3 presents an empirical comparison of single- and two-stage systems in the bandit setting , followed by a theoretical analysis with implications for both the learning setups . 3 Comparing single- and two-stage systems . Since the ranker and nominators could each be deployed independently , one may wonder whether the performance of a two-stage system is significantly affected by factors beyond the hypothetical singlestage performance of its components . This question is both theoretically ( new developments needed ? ) and practically interesting ( e.g. , training components independently , as common , assumes targeting single-stage behavior is optimal ) . In Section 3.1 , we empirically show that while factors known from the single-stage literature also affect two-stage systems , there are two-stage specific properties which can be even more important . Section 3.2 then investigates these properties theoretically , revealing a non-trivial interaction between the nominator training objective and the item pool allocations { An } n . 3.1 Empirical observations . Setup We study the effects of item pool size , dimensionality , misspecification , nominator count , and the choice of ranker and nominator algorithms in the bandit setting . We compare single- and two-stage systems where each ( component ) models the expected reward as a linear function ft ( x , a ) = 〈θ̂t , xa〉 ( xa differs for each a ) . Abbreviating xt = xtat , the estimates are converted into a policy via either : 1 . UCB ( U ) [ 6 , 64 ] which computes the ridge-regression estimate with regularizer λ > 0 θ̂t : = Σt t−1∑ i=1 xiri , Σt : = ( λI + t−1∑ i=1 xix > i ) −1 , ( 2 ) and selects actions with exploration bonus α > 0 : at ∈ argmaxa 〈θ̂t , xta〉+ α √ x > taΣtxta . 2 . Greedy ( G ) [ 8 , 53 ] which can be viewed as a special case of UCB with α = 0 . The argmax is restricted toAn ( resp . Ct ) in two-stage systems , with pool allocation { An } n designed to minimize overlaps and approximately equalize the number of items in each pool ( see Appendix B ) . We chose to make the above restrictions of our experimental setup to limit the large number of design choices two-stage recommenders entail ( architecture and hyperparameters of each nominator and the ranker , item pool allocation , number of nominators , etc . ) , and with that isolate the variation in performance to only a few factors of immediate interest . We use one synthetic and one real-world dataset . The synthetic dataset is generated using a linear model rta = 〈θ ? , xta〉 + εta for each time step t ∈ [ T ] and action a ∈ A . The vector θ ? is drawn uniformly from the d-dimensional unit sphere at the beginning and then kept fixed , the contexts xta are sampled independently from N ( 0 , I ) , and εta ∼ N ( 0 , 0.01 ) is independent observation noise . The real-world dataset ‘ AmazonCat–13K ’ contains Amazon reviews and the associated product category labels [ 10 , 69 ] .1 Since ‘ AmazonCat–13K ’ is a multi-label classification dataset , we convert it into a bandit one by assigning a reward of one for correctly predicting any one of the categories to which the product belongs , and zero otherwise . An |A|–armed linear bandit is then created by sampling |A| reviews uniformly from the whole dataset , and treating the associated features as the contexts { xta } a∈A . This method of conversion is standard in the literature [ 27 , 35 , 66 , 67 ] . We use only the raw text features , and convert them to 768-dimensional embeddings using the HuggingFace pretrained model ‘ bert-base-uncased ’ [ 24 , 97 ] ; 2 we further subset to the first d = 400 dimensions of the embeddings , which does not substantially affect the results . Because we are running thousands of different experiment configurations ( counting the varying seeds ) , we further reduce the computational complexity by subsetting from the overall 13K to only 100 categories . Since most of the products belong to 1–3 categories , we take the categories with 3rd to 102nd highest occurrence frequency . This ensures less than 5 % of the data points belong to none of the preserved categories , and overall 10.73 % reward positivity rate with strong power law decay ( Figure 2 , right ) . While the ranker can always access all d features , the usual lower flexibility of the nominators ( misspecification ) is modelled by restricting each to a different random subset of s out of the total d features on both datasets . This is equivalent to forcing the corresponding regression parameters to be zero . Both UCB and Greedy are then run with xt replaced by the s-dimensional xn , t = xn , tat everywhere . The same restriction is applied to the single-stage systems for comparison . In all rounds , each nominator is updated with ( xn , t , at , rt ) , regardless of whether at ∈ An ( this assumption is revisited in Section 3.2 ) . Thirty independent random seeds were used to produce the ( often barely visible ) two-sigma standard error regions in Figures 3 and 4 . More details—including the hyperparameter tuning protocol and additional results—can be found in Appendices B and C. Results Starting with the synthetic results in Figure 3 , we see that the number of arms and the feature dimension d are both correlated with increased regret in single- and two-stage systems . Another similarity between all the algorithms is that misspecification—as measured by d/s—also has a significant effect on performance.3 This is also the case for the Amazon dataset in Figure 4 . 1We did not use ‘ MovieLens ’ [ 38 ] since it contains little useful contextual information as evidenced by its absence even from the state-of-the-art models [ 79 ] . Two-stage recommenders are only used when context matters as otherwise all recommendations could be precomputed and retrieved from a database at deployment . 2Encoded dataset : https : //twostage.s3-us-west-2.amazonaws.com/amazoncat-13k-bert.zip . 3Misspecification error typically translates into a linear regret term T [ 21 , 25 , 31 , 32 , 36 , 58 , 61 ] . We can thus gain some intuition for the concavity of d/s 7→ RT from the L2 error = minθn ( E [ ( ra − 〈θn , xn , a〉 ) 2 ] ) 1/2 where a ∼ Unif ( A ) [ 58 ] . Using xta ∼ N ( 0 , I ) , the minimum is achieved by ( E [ xn , ax > n , a ] ) −1 E [ xn , ara ] = θ ? n , The influence of the number of arms , dimensionality , and misspecification on single-stage systems is well known [ 60 ] . Figures 3 and 4 suggest similar effects also exist for two-stage systems . On the other hand , while the directions of change in regret agree , the magnitudes do not . In particular , two-stage systems perform significantly better than their single-stage counterparts . This is possible because the ranker can exploit its access to all d features to improve upon even the best of the nominators ( recall that nominators and single-stage systems share the same model architecture ) . In other words , the single-stage performance of individual components does not fully explain the two-stage behavior . To develop further intuition about the differences between single- and two-stage systems , we turn our attention to the Amazon experiments in Figure 4 . The top row suggests the performance of two-stage systems improves as the number of nominators grows . Strikingly , the accompanying UCB ranker + nominator plots in the bottom row show the nominator regret RNT dominates when there are few nominators , but gives way to the ranker regret RRT as their number increases . To explain why , first note that the single-stage performance of the ranker can be read off from the bottom left corner of each plot where d = s ( because all the components are identical at initialization , and then updated with the same data ) . Since the size of the candidate set Ct increases with the number of nominators , the two-stage performance in the d > s case eventually approaches that of the single-stage UCB ranker as well , even if the nominators are no better than random guessing . In fact , because ≈ 10 % of the items yield optimal reward , the probability that a set of ten uniformly random nominators with non-overlapping item pools nominates at least one optimal arm is on average 1− ( 910 ) 10 ≈ 0.65 , i.e. , the instantaneous nominator regret would be zero 65 % of the time . To summarize , we have seen evidence that properties known to affect single-stage performance— number of arms , feature dimensionality , misspecification—have similar qualitative effects on twostage systems . However , two-stage recommenders perform significantly better than any of the nominators alone , especially as the nominator count and the size of the candidate pool increase . Complementary to the evidence from offline learning [ 67 ] , and the effect of ranker pretraining [ 44 ] , these observations add to the case that two-stage systems should not be treated as just the sum of their parts . We add theoretical support to this argument in the next section .

Summary: This work compares one-stage and two-stage systems from both of empirical and theoretical observations, which finds that interactions between the ranker and the nominators substantially affect the overall system performance. A Mixture-of-Experts approach is adopted to train the nominators to specialize on different subsets of the item pool. This improves the performance conducted on a Amazon review dataset. Overall, this submission is well presented and easy to follow.