On Component Interactions in Two-Stage Recommender Systems

Thanks to their scalability , two-stage recommenders are used by many of today ’ s largest online platforms , including YouTube , LinkedIn , and Pinterest . These systems produce recommendations in two steps : ( i ) multiple nominators—tuned for low prediction latency—preselect a small subset of candidates from the whole item pool ; ( ii ) a slower but more accurate ranker further narrows down the nominated items , and serves to the user . Despite their popularity , the literature on two-stage recommenders is relatively scarce , and the algorithms are often treated as mere sums of their parts . Such treatment presupposes that the two-stage performance is explained by the behavior of the individual components in isolation . This is not the case : using synthetic and real-world data , we demonstrate that interactions between the ranker and the nominators substantially affect the overall performance . Motivated by these findings , we derive a generalization lower bound which shows that independent nominator training can lead to performance on par with uniformly random recommendations . We find that careful design of item pools , each assigned to a different nominator , alleviates these issues . As manual search for a good pool allocation is difficult , we propose to learn one instead using a Mixture-of-Experts based approach . This significantly improves both precision and recall at K . 1 Introduction . Recommender systems play a central role in online ecosystems , affecting what media we consume , which products we buy , or even with whom we interact . A key technical challenge is ensuring these systems can sift through billions of items to deliver a personalized experience to millions of users with low response latency . A widely adopted solution to this problem are two-stage recommender systems [ 12 , 20 , 28 , 98 , 101 ] where ( i ) a set of computationally efficient nominators ( or candidate generators ) preselects a small number of candidates , which are then ( ii ) further narrowed down , reranked , and served to the user by a slower but more statistically accurate ranker . Nominators are often heterogeneous , ranging from associative rules to recurrent neural networks [ 18 ] . A popular choice are matrix factorization [ 56 , 72 ] and two-tower [ 98 ] architectures which model user feedback by the dot product between user and item embeddings . While user embeddings often evolve with the changing context of user interactions , item embeddings can typically be precomputed before deployment . The cost of candidate generation is thus dominated by the ( approximate ) computation of the embedding dot products . In contrast , the ranker often takes both the user and item features as input , making the computational cost linear in the number of items even at deployment [ 20 , 67 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . With few exceptions [ 44 , 52 , 67 ] , two-stage specific literature is sparse compared to that on singlestage systems ( i.e. , recommenders which do not construct an explicit candidate set within a separate nominator stage [ e.g. , 17 , 40 , 46 , 56 , 72 , 78 , 84 ] ) . This is especially concerning given the considerable ethical challenges entailed by the enormous reach of two-stage systems : according to the recent systematic survey by Milano et al . [ 70 ] , recommender systems have been ( partially ) responsible for unfair treatment of disadvantaged groups , privacy leaks , political polarization , spread of misinformation , and ‘ filter bubble ’ or ‘ echo chamber ’ effects . While many of these issues are primarily within the realm of ‘ human–algorithm ’ interactions , the additional layer of ‘ algorithm–algorithm ’ interactions introduced by the two-stage systems poses a further challenge to understanding and alleviating them . The main aim of our work is thus to narrow the knowledge gap between single- and two-stage systems , particularly in the context of score-based algorithms . Our main contributions are : 1 . We show two-stage recommenders are significantly affected by interactions between the ranker and the nominators over a variety of experimental settings ( Section 3.1 ) . 2 . We investigate these interactions theoretically ( Section 3.2 ) , and find that while independent ranker training typically works well ( Proposition 1 ) , the same is not the case for the nominators where two popular training schemes can both result in performance no better than that of a uniformly random recommender ( Proposition 2 ) . 3 . Responding to the highlighted issues with independent training , we identify specialization of nominators to smaller subsets of the item pool as a source of potentially large performance gains . We thus propose a joint Mixture-of-Experts [ 47 , 49 ] style training which treats each nominator as the expert for its own item subset . The ability to learn the item pool division alleviates the issues caused by the typically lower modeling capacity of the nominators , and empirically leads to improved precision and recall at K ( Section 4 ) . 2 Two-stage recommender systems . The goal of recommender systems is to learn a policy π which maps contexts x ∈ X to distributions π ( x ) over a finite set of items ( or actions ) a ∈ A , such that the expected reward Ex Ea∼π ( x ) [ ra |x ] is maximized . The context x represents information about the user and items ( e.g. , interaction history , demographic data ) , and ra is the user feedback associated with item a ( e.g. , rating , clickthrough , watch-time ) . We assume that ra|x has a well-defined fixed mean f ? ( x , a ) : = E [ ra |x ] for all the ( x , a ) pairs . To simplify , we further assume only one item at is to be recommended for each given context xt , where t ∈ [ T ] with [ T ] : = { 1 , . . . , T } for T ∈ N. Two-stage systems differ from the single-stage ones by the two-step strategy of selecting at . First , each nominator n ∈ [ N ] picks a single candidate an , t from its assigned pool An ⊆ A ( An 6= ∅ , ⋃ nAn = A ) . Second , the ranker chooses an item at from the candidate pool Ct : = { a1 , t , . . . , aN , t } , and observes the reward rt = rtat associated with at . Since the goal is expected reward maximization , recommendation quality can be measured by instantaneous regret r ? t − rt where r ? t = rta ? t is the reward associated with an optimal arm a ? t ∈ argmaxa∈A f ? ( xt , a ) . This leads us to an important identity for the ( cumulative ) regret in two-stage systems which is going to be used throughout : R2sT = T∑ t=1 r ? t − rt = T∑ t=1 ( r ? t − rtãt ) ︸ ︷︷ ︸ = : RNT + T∑ t=1 ( rtãt − rt ) ︸ ︷︷ ︸ = : RRT , ( 1 ) with ãt ∈ argmaxa∈Ct f ? ( xt , a ) . In words , RNT is the nominator regret , quantifying the difference between the best action presented to the ranker ãt and the best overall action a ? t , and R R T is the ranker regret which measures the gap between the choice of the ranker at and the best action in the candidate set Ct . The two-stage recommendation process is summarized in Figure 2 ( left ) . While Equation ( 1 ) is typical for the bandit literature ( data collected interactively , only rtat revealed for each t ) , we also consider the supervised learning case ( a dataset with all { rta } ta revealed is given ) . In particular , Section 3 presents an empirical comparison of single- and two-stage systems in the bandit setting , followed by a theoretical analysis with implications for both the learning setups . 3 Comparing single- and two-stage systems . Since the ranker and nominators could each be deployed independently , one may wonder whether the performance of a two-stage system is significantly affected by factors beyond the hypothetical singlestage performance of its components . This question is both theoretically ( new developments needed ? ) and practically interesting ( e.g. , training components independently , as common , assumes targeting single-stage behavior is optimal ) . In Section 3.1 , we empirically show that while factors known from the single-stage literature also affect two-stage systems , there are two-stage specific properties which can be even more important . Section 3.2 then investigates these properties theoretically , revealing a non-trivial interaction between the nominator training objective and the item pool allocations { An } n . 3.1 Empirical observations . Setup We study the effects of item pool size , dimensionality , misspecification , nominator count , and the choice of ranker and nominator algorithms in the bandit setting . We compare single- and two-stage systems where each ( component ) models the expected reward as a linear function ft ( x , a ) = 〈θ̂t , xa〉 ( xa differs for each a ) . Abbreviating xt = xtat , the estimates are converted into a policy via either : 1 . UCB ( U ) [ 6 , 64 ] which computes the ridge-regression estimate with regularizer λ > 0 θ̂t : = Σt t−1∑ i=1 xiri , Σt : = ( λI + t−1∑ i=1 xix > i ) −1 , ( 2 ) and selects actions with exploration bonus α > 0 : at ∈ argmaxa 〈θ̂t , xta〉+ α √ x > taΣtxta . 2 . Greedy ( G ) [ 8 , 53 ] which can be viewed as a special case of UCB with α = 0 . The argmax is restricted toAn ( resp . Ct ) in two-stage systems , with pool allocation { An } n designed to minimize overlaps and approximately equalize the number of items in each pool ( see Appendix B ) . We chose to make the above restrictions of our experimental setup to limit the large number of design choices two-stage recommenders entail ( architecture and hyperparameters of each nominator and the ranker , item pool allocation , number of nominators , etc . ) , and with that isolate the variation in performance to only a few factors of immediate interest . We use one synthetic and one real-world dataset . The synthetic dataset is generated using a linear model rta = 〈θ ? , xta〉 + εta for each time step t ∈ [ T ] and action a ∈ A . The vector θ ? is drawn uniformly from the d-dimensional unit sphere at the beginning and then kept fixed , the contexts xta are sampled independently from N ( 0 , I ) , and εta ∼ N ( 0 , 0.01 ) is independent observation noise . The real-world dataset ‘ AmazonCat–13K ’ contains Amazon reviews and the associated product category labels [ 10 , 69 ] .1 Since ‘ AmazonCat–13K ’ is a multi-label classification dataset , we convert it into a bandit one by assigning a reward of one for correctly predicting any one of the categories to which the product belongs , and zero otherwise . An |A|–armed linear bandit is then created by sampling |A| reviews uniformly from the whole dataset , and treating the associated features as the contexts { xta } a∈A . This method of conversion is standard in the literature [ 27 , 35 , 66 , 67 ] . We use only the raw text features , and convert them to 768-dimensional embeddings using the HuggingFace pretrained model ‘ bert-base-uncased ’ [ 24 , 97 ] ; 2 we further subset to the first d = 400 dimensions of the embeddings , which does not substantially affect the results . Because we are running thousands of different experiment configurations ( counting the varying seeds ) , we further reduce the computational complexity by subsetting from the overall 13K to only 100 categories . Since most of the products belong to 1–3 categories , we take the categories with 3rd to 102nd highest occurrence frequency . This ensures less than 5 % of the data points belong to none of the preserved categories , and overall 10.73 % reward positivity rate with strong power law decay ( Figure 2 , right ) . While the ranker can always access all d features , the usual lower flexibility of the nominators ( misspecification ) is modelled by restricting each to a different random subset of s out of the total d features on both datasets . This is equivalent to forcing the corresponding regression parameters to be zero . Both UCB and Greedy are then run with xt replaced by the s-dimensional xn , t = xn , tat everywhere . The same restriction is applied to the single-stage systems for comparison . In all rounds , each nominator is updated with ( xn , t , at , rt ) , regardless of whether at ∈ An ( this assumption is revisited in Section 3.2 ) . Thirty independent random seeds were used to produce the ( often barely visible ) two-sigma standard error regions in Figures 3 and 4 . More details—including the hyperparameter tuning protocol and additional results—can be found in Appendices B and C. Results Starting with the synthetic results in Figure 3 , we see that the number of arms and the feature dimension d are both correlated with increased regret in single- and two-stage systems . Another similarity between all the algorithms is that misspecification—as measured by d/s—also has a significant effect on performance.3 This is also the case for the Amazon dataset in Figure 4 . 1We did not use ‘ MovieLens ’ [ 38 ] since it contains little useful contextual information as evidenced by its absence even from the state-of-the-art models [ 79 ] . Two-stage recommenders are only used when context matters as otherwise all recommendations could be precomputed and retrieved from a database at deployment . 2Encoded dataset : https : //twostage.s3-us-west-2.amazonaws.com/amazoncat-13k-bert.zip . 3Misspecification error typically translates into a linear regret term T [ 21 , 25 , 31 , 32 , 36 , 58 , 61 ] . We can thus gain some intuition for the concavity of d/s 7→ RT from the L2 error = minθn ( E [ ( ra − 〈θn , xn , a〉 ) 2 ] ) 1/2 where a ∼ Unif ( A ) [ 58 ] . Using xta ∼ N ( 0 , I ) , the minimum is achieved by ( E [ xn , ax > n , a ] ) −1 E [ xn , ara ] = θ ? n , The influence of the number of arms , dimensionality , and misspecification on single-stage systems is well known [ 60 ] . Figures 3 and 4 suggest similar effects also exist for two-stage systems . On the other hand , while the directions of change in regret agree , the magnitudes do not . In particular , two-stage systems perform significantly better than their single-stage counterparts . This is possible because the ranker can exploit its access to all d features to improve upon even the best of the nominators ( recall that nominators and single-stage systems share the same model architecture ) . In other words , the single-stage performance of individual components does not fully explain the two-stage behavior . To develop further intuition about the differences between single- and two-stage systems , we turn our attention to the Amazon experiments in Figure 4 . The top row suggests the performance of two-stage systems improves as the number of nominators grows . Strikingly , the accompanying UCB ranker + nominator plots in the bottom row show the nominator regret RNT dominates when there are few nominators , but gives way to the ranker regret RRT as their number increases . To explain why , first note that the single-stage performance of the ranker can be read off from the bottom left corner of each plot where d = s ( because all the components are identical at initialization , and then updated with the same data ) . Since the size of the candidate set Ct increases with the number of nominators , the two-stage performance in the d > s case eventually approaches that of the single-stage UCB ranker as well , even if the nominators are no better than random guessing . In fact , because ≈ 10 % of the items yield optimal reward , the probability that a set of ten uniformly random nominators with non-overlapping item pools nominates at least one optimal arm is on average 1− ( 910 ) 10 ≈ 0.65 , i.e. , the instantaneous nominator regret would be zero 65 % of the time . To summarize , we have seen evidence that properties known to affect single-stage performance— number of arms , feature dimensionality , misspecification—have similar qualitative effects on twostage systems . However , two-stage recommenders perform significantly better than any of the nominators alone , especially as the nominator count and the size of the candidate pool increase . Complementary to the evidence from offline learning [ 67 ] , and the effect of ranker pretraining [ 44 ] , these observations add to the case that two-stage systems should not be treated as just the sum of their parts . We add theoretical support to this argument in the next section .

Summary: The work focuses on a very practical problem of designing a good a two-stage recommender systems. The authors shed light on interaction between the two stages (called nominator and ranker) and its effects on an overall performance of the solution. They theoretically prove that the way nominators are trained may significantly affect the end quality and provide an empirical evidence for their findings as well. The authors also provide a recipe for designing a better first stage based on a mixture-of-experts approach that turns out to advantageous over naïve approach in certain cases.