A/B Testing for Recommender Systems in a Two-sided Marketplace
Authors: Preetam Nandy
Divya Venugopalan
Chun Lo
Shaunak Chatterjee
Venue: NeurIPS 2021 Poster (2021)
Keywords: A/B Testing
Two-sided Marketplace
Treatment Effect Estimation
Recommender Systems

Abstract: Two-sided marketplaces are standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buyers or content viewers on one side and producers, sellers or content-creators on the other. Consumer side measurement of the impact of a treatment variant can be done via simple online A/B testing. Producer side measurement is more challenging because the producer experience depends on the treatment assignment of the consumers. Existing approaches for producer side measurement are either based on graph cluster-based randomization or on certain treatment propagation assumptions. The former approach results in low-powered experiments as the producer-consumer network density increases and the latter approach lacks a strict notion of error control. In this paper, we propose (i) a quantification of the quality of a producer side experiment design, and (ii) a new experiment design mechanism that generates high-quality experiments based on this quantification. Our approach, called UniCoRn (Unifying Counterfactual Rankings), provides explicit control over the quality of the experiment and its computation cost. Further, we prove that our experiment design is optimal to the proposed design quality measure. Our approach is agnostic to the density of the producer-consumer network and does not rely on any treatment propagation assumption. Moreover, unlike the existing approaches, we do not need to know the underlying network in advance, making this widely applicable to the industrial setting where the underlying network is unknown and challenging to predict a priori due to its dynamic nature. We use simulations to validate our approach and compare it against existing methods. We also deployed UniCoRn in an edge recommendation application that serves tens of millions of members and billions of edge recommendations daily.

A/B Testing for Recommender Systems in a Two-sided Marketplace
Preetam Nandy, Divya Venugopalan, Chun Lo, Shaunak Chatterjee
LinkedIn Corporation
Mountain View, CA 94083
{pnandy, dvenugopalan, chunlo, shchatterjee) inkedin.comm
Abstract
Two-sided marketplaces standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buy- ers or content viewers the other. Consumer side measurement of the impact of treatment variant can be done via simple online A/B testing. Producer side measurement lenging because the producer experience depends on the treatment assignment of the consumers. Existing approaches for producer side measurement are either based on graph mauster-based randomization or on certain treatment propagation assumptions. The former approach results in low-powered experiments as the crosuuer-consumer network density increases and the latter approach lacks a strict notion of error control. In this paper, we propose (i) a quantification of the quality of a producer side experiment design, and (ii) a new experiment design mechanism that generates high-quality experiments based on this quantification. Our approach, called UniCoRn (Unifying Counterfactual Rankings), provides explicit control over the quality the experiment and computation cost. Further, we prove that our experiment design is optimal to the proposed design quality measure. Our approach is agnostic to the density of the producer-consumer network and does not rely any treatment propagation assumption. Moreover, unlike the existing approaches, we do not need to know the underlying network advance, making this widely applicable to the industrial setting where the underlying network is unknown and challenging to predict a priori due to its dynamic nature. We use simulations to validate our approach and compare it against existing methods. We also deployed UniCoRn an edge recommendation application that serves tens millions of members and billions of edge recommendations daily.
side and producers, sellers
on
chal-
Introduction
Learning via experiments is one of the most powerful and popular ways improve in many domains of life. In the tech industry, experiments are very commonplace to better understand user preferences and how to serve them best. Such experiments, known A/B testing or bucket tests [5] 6 16] [1))
are performed by randomized allocation of a treatment and control variant to some population and measuring the average treatment effect (ATE) [] 4 relative to control. The populations receiving treatment and control are statistically identical since they were randomly selected.
A/B testing is powerful tool because of its design simplicity and ease of setup. It is accurate in applications where the behavior of a measurement unit (e.g., user) is unaffected by the treatment allocated to any other measurement unit. This principle is called "Stable Unit Treatment Value Assumption" or SUTVA MHH. The SUTVA principle is reasonably accurate for experiments in 35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

several viewer side applications of recommender systems (e.g., newsfeed ranking, search), where each viewer acts independently based only on what is shown her.
In marketplace settings, the SUTVA condition is often violated. A bipartite graph is a common abstraction for two-sided marketplaces. Let us consider the example of content recommendation in a newsfeed ranking application with content viewers and producers. While the effect of a ranking change (e.g., showing more visual content) conforms to SUTVA on the viewer side, the effect on the producer side (i.e., the impact on producers who post more/less visual content) does not. A producer's experience is affected by the allocation of treatment to all her potential viewers. For instance, a producer who primarily posts images will get more exposure (which directly affects her behavior) as more viewers are allocated to the new treatment. Sellers and buyers are an identical analogue to producers and consumers. Violations of SUTVA, especially on the producer or seller side experience, are commonplace in many marketplace experiments especially marketplaces gain greater prominence.
g and form an important area of study,
popular approach in experiment design for marketplaces is to partition the graph into near separable clusters [T0 14, [8]. Then each cluster is considered an independent mini-graph, and randomized treatment allocation is done at the cluster level (i.e., all nodes in that cluster are allocated the same treatment). This works well in sparse graphs where many such clusters can be found without ignoring too many edges. A different approach 2 7. relevant especially in the advertising world, "creates" multiple copies of the universe by splitting the limited resources (e.g,, daily budget) of entities on one side of the graph (e.g., advertisers). This works when the ecosystem has periodic reset.
Another recent approach [9] designs an experiment by identifying a modified version of the treatment, which is allocated to a proportion of a node's 's network to mimic the effect on the node that allocating the original treatment to the node's entire network would have had. This works well for denser networks but makes assumptions on how the treatment effect propagates. Many of these approaches require knowing the network structure a priori, and hence do not work for dynamic graphs.
In this work, we propose novel and simple experiment design mechanism to generate high quality producer side experiments, where the quality is defined by a design inaccuracy measure that we introduce (see Definition I). The mechanism also facilitates choosing desired trade off between the experiment quality and the computational cost of running it. Our experiment design is shown to be optimal with respect to the inaccuracy measure. Our solution is applicable to any ranking " system" which forms the viewer side application in most marketplace problems. The key insight that there is unique viewer side ranking corresponding to each producer side treatment when all producers are allocated to that treatment. When producers are allocated to different treatment variants (e.g., to run treatment and control variants simultaneously), there are multiple possibly conflicting "counterfactual" rankings. By unifying the different counterfuctual rankings (hence "UniCoRn" ") based on treatment allocation on the producer side, and careful handling of conflicts, we obtain a high quality experiment. A recent work [3] is specific instance of our generalized design, relying on small producer ramps, which minimize chances of conflict between the counterfactual rankings.
Our solution is designed work at ramps of any size (higher ramps are often necessary for sufficient power). Furthermore, it improves upon the limitations of most prior approaches. It is agnostic to the density of the graph, does not depend on any assumptions on how the treatment effect propagates, and we do not need to know the structure of the graph a priori. One downside is the online computation cost of running an experiment using our design, and we provide a parameter control this cost.
The key contributions of our work are as follows:
An inaceuracy based metric to quantify the quality of an experiment and novel producer side experiment design mechanism that unifies multiple counterfactual rankings.
We prove the optimality of our experiment design, as well as bias and variance bounds.
We show through extensive simulations how the method performs in various synthetic scenarios and against multiple existing approaches 3
A real-world implementation of the proposal in an edge recommendation problem.
1 This paper is focused on two-sided marketplaces. For more than 2 sides, multi-partite graph can be used.
Including single-slot and multi-slot ranking applications

---

The rest of the paper is structured as follows. Section deescrbbes the problem setup in the context of a bipartite graph. The UniCoRn algorithm is presented Section [ along with an example demonstrating the different steps and certain theoretical properties of our method, including its optimality. In Section 4 studies, and we share our experience implementing UniCoRn in an edge recommendation application in one of the biggest social network platforms in the world. Finally, we conclude in Section 5 with a discussion of some extensions of our work and its general implications.
we demonstrate the robustness of our method through detailed simulation
2 Problem setup
Let consider bipartite graph linking two types of entities producers and consumers. A recommender system recommends an ordered set of items generated by the producers to each consumer, where items (e.g., connection recommendations, content recommendations recommendations) are ordered based on their estimated relevance in that consumer sessior We use the terminology "consumer (or producer) side experience" to refer to a measurable quantity associated with a consumer (or producer) that depends on the rank assigned by the recommendation system. One can get an unbiased estimate of the consumer-side impact (with respect to a metric, outcome or response of interest) by randomly exposing two disjoint groups of consumers to the treatment model and the control model respectively and measuring the average difference between the treatment group and the control group.
search
This classical A/B testing strategy does not work for measuring the producer side impact since that depends on the consumers' treatment assignment and should be ideally measured by allocating the same treatment to all the consumers connected to the producer in question. As illustrated in Figure la satisfying this ideal condition simultaneously for all (or many) producers is not possible. For instance, consumeer is connected to producer (in control) and producer 3 (in treatment).
Notation and terminology: We consider an experimental design D with mutually exclusive sets of producers Po. ................................ Pk corresponding to treatments To.... Tk respectively. We refer to To as control model (or recommender system) and all other T' as treatment model(s). The size of Ps is determined by the ramp fraction (i.e., treatment assignment probability) of the corresponding models.
Let Pk, k ..... K. denote the ramp fractions satisfying Ek=o P& = 1. An online experiment typically spans over a time window, in which each consumer can have zero to more than one sessions and each producer can produce zero to more than one items. We denote the set of all sessions and the set of all items by S and I respectively. In each session s, the set of items under consideration denoted by I,, which a subset of I. The counterfactual rank (i, I.) is the rank of item in consumer session with items I, when all items are ranked by treatment TA- We denote the rank of item in the experimental design D by Ri, We use the notation E PA to denote that item i belongs to producer in Pr. We reserve the use of the letters k; and for indexing treatment variant, referring to an item and denoting a session.
Design accuracy and cost: An experimental design to accurately measure the producer side ex- perience should also have reasonable computational cost (hereafter just "cost") of running the experiment, As Section 3 will show, the accuracy and the cost are conflicting characteristics of our experimental design. Thus, having the flexibility to explicitly trade-off accuracy against cost is desirable. To this end, we provide quantification of these characteristics in terms of counterfactual rankings. To define accuracy, we compare the design rankings Ri I,) with the ideal (but typically unrealizable) ranking R. (I.I.) that equals R₁ (i, I.) if E Pk. An example is shown in Figure lb]
Definition 1. The inaccuracy of the experimental design D is given by
Inaccuracy(D)= EE (Ro(i, R² Ii, where R2 (I,
Σ Ru RT ( 4uep)
When treatments are being evaluated online, each i E I, is scored only by the control model To- Section 3 shows that each item might be scored multiple times using different treatment models in an experiment design. We quantify this computational expense the cost of the design.
A session encapsulates the context of the recommendation request such as news feed visit (in case of content recommendations) search query (in the case of search recommendations)
3

---

P
Rg
R,
R
Rg
R,
R
Item
8
Item
4
consumer
producer
Item
v
X
Item
7
Item
consumer
producer 2
Item
v
X
Item
6
3
Item 3
2
3
consumer
producer 3
Item
X
V
Item
4
4
Item
8
8
X
Item
3
7 Item
Item
consumer
producer
Item
X
V
Item
2
2
Item
X
Item
1
Item
5
consumer 5
producer
(i) Problem setup
(i) Unrealizable pair of counterfactual rankings
Realizable pair counterfaclual rankings
consumers
producers
(Ia) Bipartite graph of producers and con- sumers. Due to shared consumers between producers, it is infeasible to ensures all con- sumers connected to producer get the same treatment the producer.
(1b) Counterfactual Rankings. In (i), there are two mutually ex- clusive sets of producers Po and Pt. each of size 4. In (ii), giving each item their ideal position in unified ranking is not possible such conflicts and the ideal unified counterfactual ranking is realizahle.
since there conflicts. In (iii), there
Definition 2.
Let (i, denote the total number of times a scoring function (i.e. one of Tr 's) obtain ranking of the items in I, according D. The cost of an
needs be applied experimental design 'D is given by Cost(D)= == E (i,((i,
Now that we have an inaceuracy measure and a cost metric, we can define an experiment design algorithm that allows us to choose a desired balance between the two.
UniCoRn: Unifying Counterfactual Rankings
typical recommender system comprises of a (possibly composite) model (machine learnt or otherwise) that assigns relevance score to each candidate item. Items are then ranked according to their scores (higher the score, lower the rank; ties broken randomly). We want measure the impact of new recommender system (T1) compared to the control recommender system (To) producers (or sellers) two-sided marketplace via online A/B testing. Many recommender systems in industry have two phases: (i) a candidate generation phase, which considers much larger set of candidates, followed by (ii) a ranking phase using a more sophisticated model with higher computation cost and hence often scoring much fewer items. Minor modifications needed to handle such multi-phase systems are covered Section 4.3 Until then, we focus on single phase ranking systems. We also assume one treatment and one control for and extend multiple treatments in Section 3.2
3.1 The UniCoRu algorithm
For given disjoint producer sets Po and Pi corresponding to To and Ti respectively, we present
class of experimental designs UniCoRa Po. Pl, ax) parametrized by the tuning parameter a E [0.
controlling the cost of the experiment. Recall that (i, denotes ranking of the items in I'
according to Tk in descending order (i.c. (i) 2 (i) implies R& (i,I') ) < Rx(),'') for = 0, 1.
For each consumer session 8, the UniCoRn ( (Po, a)) algorithm provides ranking {&D (i, I.)} of the items in I. such that the rank of item E P& close to Ii, simultaneously for all I, and k 0, 1. Please note that the underlying consumer prrouucer graph is not needed to apply UniCoRni( Po. PI,a) The detailed steps of UniCoRn are provided in Algorithm and Figure 2a
provides a visual walkthrough of UniCoRn using an example. The key components are:
Initial slot allocation (Step 2): Identify positions allocated to all items using To.
Obtain mixing positions (Steps 3 5): Identify the slots C to mix up and accommodate the two counterfactual rankings, and the slots that will not partake in this process.
determines the fraction of Po items and slots that will used in the mixing
All items in P1 and their corresponding slots participate in the mixing
Perform mixing (Steps 6 g Obtain the relative rank of each item using the score according to that item's 's treatment assignment. Use these relative ranks to blend items (that were selected for mixing) from different groups with ties broken randomly (see Figure 2a

---

Algorithm 1 UniCoRnnPPa Pi
Require: producer sets Po. Pl. scoring models To and T3 and tuning parameter os
Ensure: a ranking of items for each session s;
1: for Each session with item set I. do
2: Get a ranking of all items [R,
according to To:
3: Construct Po by randomly selecting producers from Po with probability a;
4: L,,0, I,, and I2 be the sets of items with producers in Po. P1 and P/ respectively;
5: Find the rank positions L = [I. i I,, UI,,0? of the items in I,,1 U I,o:
6: Obtain rankings Ro (i, I,,1 U [ and [Ra(,, UT,,))) according to To and T1;
7:
Compute the following rank-based score
rank score() = I,,1 UI, (iirrg) + (i, UI, (ier.y
8: Rerank I1 U UI,00 in the positions L based on rank_ cank_scoreii) in ascending order (i.e., implies rank(i) I rank(j)) while breaking ties randomly;
rank sock__scrre(i &
In A we guarantee that in the final ranking (i) the ordering among the items in Po respects the To based ranking, (ii) the ordering among the items in P1 respects the Ti based ranking, and (iii) the distribution of the rank of a randomly chosen item in Po the same as the distribution of rank of a randomly chosen item in Pi (i.c., no cannibalization) and the common distribution is Uni form (1, ..., K} if there are slots. It is easy to see that (i) and (ii) hold by design and (iii) follows from the fact that L is uniform sample from f1, K} as Po and P1 are independent of the ranking distributions generated by Tu and Ti (due to randomized treatment allocation).
R,
R
rank-score
R
Rg
R2
R,
Rg
Rg
Ilem
8
Item
Item
Ilem 1
Item
1
Item
Item 2
Item
7
Item
2
Item 2
2
Item 2
3
Item
Itmm
Item 2
4
Item
6
Iteen
3
Item 3
3
Ilem S
5
3
Item
Item
4
Item 4
Item 4
Itom
Item
4
Item
Itemn
8
Item 5
4
Item 5
6
Item
4
Item
8
Itom
Itam
7
Item
3
Item
7
Ihem
Item 6 4
Iter
3
Item
7
Item
Item
5
Item
2
Item
6
Item
Item 7
2
Item
Item
2
2
Item
Item
3
Item
Item
5
Item 6
Item 8
Item
Item
Item 8
(i) A counterfactual mankings
(1) 100=000. 0.5
(li) Hank soore
(iv) Unified ranking in
(v) Final rankinn
(i) Unreelizeble peir of countorfactual ranklinss
(1i)
pr (2.3)
of
(2..566.7.8)
(2a) Algorithm 1 with = 0.5. (i) Complete rankings (2b) The impact of E [0, 1] is algorithm under To and Ti. (ii) Pe sampled as {Ttem 2, Ttem 3). parameter that specifies the amount of flexibility in (iii) Separate ranking of PT using Ro and P1 using Ri. combining the two counter- factual rankings, with (iv) Unified ranking of L = 1% U P; with all ties broken a = 0 being the least flexible and a = 1 being 0 incurs the highest tained by placing items in Pa \ L. in their Ro ranks, then inaccuracy but has the lowest cost, while remaining slots filled with the unified ranking
in favor of the orange items (P1). (v) Final ranking ob- the most. As result,
1 is the most accurate and computationally expensive.
The impact of a: We inserted a in our design to provide an explicit lever to control the balance between accuracy and cost. The more items (i.e., ICI) we include in the mixing, the greater the accuracy. However, the mixing step requires every eligible item in L to be scored by every model, and hence the increased accuracy can come at hefty cost. The implication of different choices of in shown in Figure using the same example. All ties are broken in favor of items in P1. Let Ck denote the computation cost of scoring all items (from all producers, that is) using Tg. where the scoring cost of each item the same. Then the total cost is given by co + (app + P1 )1.
3.2 Handling multiple treatments
Thus far, we have considered one treatment and one control. Simultaneous measurement of multiple treatments (against a control variant) can achieved with a simple extension to the mixing selection step. The effect of each treatment can be observed by independently comparing the corresponding treatment population to the control population. As a quick recap of critical notation, To denotes the
control model. With IC treatments in total and pk denoting the ramp fraction of Tk. Σx Pk- = 1.
Usiing Detinition 2
co Is. This refinement handle variable model complexities.
5

---

This is the trivial extension of Algorithm| I We first fix positions of 1 fraction of items from Po and then mix the remaining Po items with all items from each Ps for k 1.... ,.. This family of designs (by varying a) has higher cost and lower inaccuracy. Hence, it is suitable for offline scoring applications and online applications without striet scoring latency constraints. The total (computation) cost is given by co + (1 (1 a)po) Σx>1 Ck-
Greater mixing:
Limited mixing: An alternative is to select / by picking a fraction of items from each Ps including O. This reduces the cost in the mixing step since L is smaller and fewer items are scored by all models under consideration, but increases inaccuracy (compared to greater mixing) since lesser mixing happens. It is better suited for online applications with stricter latency requirements. The total computational cost is given by co + a Σx>1 + (1 (1-a) a) Σ x21 PlCk.
Next, we analyze some theoretical properties of this design in the two treatment scenario. At a 1, "greater" and "lesser" mixing scenarios are identical and the amount of mixing is the maximum possible. It is not surprising that this is also when the experiment design is provably optimal.
3.3 Theoretical results
We first prove the optimality of UniCoRn((P) Po. P1. 1) with respect to the design inaccuracy mea- sure given in Definition [ Next, in Theorem 2 we provide bias and variance bounds for UniCoRnn(FF Po. Pı, 1) and we show that our bounds are tight in the sense that the equality can be achieved in an adversarial situation. Proofs of all the results are given in the appendix.
Theorem (Optimality of UniCoRn Po. P1, 1)). Let Du be design based on Algorithm) 7 with randomly chosen Po and Pt, and with a 1. Then for any other design
(I) where R is as in Definition | with K. = 1. Equation ( implies the optimality of Dv with respect to the design inaccuracy measure given in Definition [ i.e. Inaccuracyl (Dv. Ti) S Inaccuracy(D. To, Tí) for all To. Ti and all design D. The same results hold for the multiple trealment case described in Section
E (RDD (i,I.) - R I,) I,)² s E (Ro(i,I,) R (1,),)
Theorem (Bias and Variance Bounds). Let Dy be a design based on Algorithm 7 with randomly chosen Po and Pt. and with a = 1. Then, for ls {0, 1}, the conditional bias and the conditional variance of the observed rank RD, (i.Ix) given As,k,,r = {I,, R" (i,I.) = r, E P&} is given by
1. E (RDo (I, (1. Ar...ir)) S (k.P)). and
2. Var (Rou (i.I.) As,...r) S 2min(r- - 1, IZll-) rr) (1- + c(k, P1) (1 c(k, P1)).
where p is the probability of assigning item to the treatment group PI. R is as in Definition 7
with K. = 1, and c(k, (1) = (k(1 P1) + (1 - k)p1)/2. The equality holds in both cases when + I.I+1 and the treatment ranking (Ri (i, [,)] is the reverse of the control ranking (Ro(i,I,) 2
with probability one.
For UniCoRn( Po, PI,a) with a < for all E Po Po. we have Rp(i, = R² (i, implying zero bias and zero variance. For all i & Po P7. it is easy to see that (1, I.) can be written as RD( X (r 1 X) Ri I,, UT;
where X has a Binomial 1, a1 pı)) distribution, and X and RD UI, are conditionally independent given the ordered set of items Do. according to To. Therefore, the bias and bounds can be derived using the results in Theorem We leave detailed computations to the interested reader. Next, we empirically evaluate the impact of a on design inaccuracy and implement UniCoRn to evaluate the producer side impact of a large-scale recommender system.
4 Empirical Evaluation
We analyze various aspects of the design inaccuracy in Section 4.1
followed by an analysis of
the treatment effect estimation error with specific rank to response functions Section 4.2 We conclude this section by sharing our experience of implementing Uni in large-scale edge recommendation application for one of the largest social networks with 750+ million members, demonstrating the scalability of our algorithm.

---

For Sections 4.1 and we create simulated environment with L = 100 positions to generate data for the empinical evaluation of UniCoRn(Po. Pt, a) (in short, UniCoRm(a)). First, we compare the design accuracy and the cost of the variants of UniCoRa(a)) based on a number of values of o. Next, we compare the performances of UniCoRm(a) for a E {0, (0,0,2,1) 1,, the counterfactual ranking method of [3] (we will refer to this as HaT At. and modified version of OASTS [9] for estimating the average treatment effect. To the best of our knowledge, these are the only existing methods that do not require the underlying network to be known priori. We implemented the Algorithms in R.
4.1
Impact of or
For fixed treatment proportion TP -p=110 | + (Pi)), ,, the cost (Definition 2 of UniCoRa(a)) increases with a. We present the cost and inaccuracy results for different values of TP, while taking the average over random choices Po and P1. We also consider four different simulation settings corresponding different levels of correlation p E {-1, -0.4, 0.2, 0.8) between treatment and control scores for comparing the design accuracy. We generated the scores from bivariate Gaussian distribution. More data generation details are Appendix A.2.
proportion: 0:1 error bype: MAE
proportion: emror Type: RMSE
proportion: 0.5 error. error_type: MAE
proportion:
error_ bype: RMSE
inaccuracy
TP
0.1 0.2
ontrr
6
0.5
1000
75
50
75
1000
50
75
100
espacted rank
25 50
75 100 cost
mothod
UniCoPn(()
UniCoRn(D)
0.2
D.B
(3a) Average ranking errors for two measures of inaccuracy (MAE and RMSE) and for two different values of the treatment proportion (0.1 and 0.5) based trade-off at different treat- ment proportions (TP).
(3b) Cost vs. (in)accuracy
No 50000 sessions with L = 100 slots each.
We consider two measures of inaccuracy (see Appendix A.2 for detailed definitions): (i) mean absolute error (MAE) and (ii) root mean squared error (RMSE). Figure 3a shows that the performance of UniCoRn(( (0) and UniCoRm(1 (1) are roughly similar (or slightly better for UniCoRt (0) with respect to MAE, but UniCoRta(1 outperforms U UniCoRa(() with respect to RMSE (validating Theorem D. This because Rp (i. I.) R (i. I.) = for all items Po for (n))o but the errors corresponding the items in P1 are much larger for UniCoRn(() compared to UniCo Note that the slightly better performance of U UnCCoR((O) with respect to MAE does not contradict the optimality result in Theorem 1 which is based
squared errors instead of absolute errors.
Another interesting finding from Figure Balis is that a smaller value of (where -1 is the smallest value) corresponds to more challenging design problem due to the increasing number of conflicts in the counterfactual rankings (cf. the last part of Theorem 2}.
The cost (Definition 2 and inaccuracy (Definition D trade-off for fixed value of p = 0.8 is shown in Figure 33b for different values of the treatment proportion TP. For each value of TP, we obtain the plot by varying E [0, 1]. Since we directly generated the scores from bivariate Gaussian distribution, the cost show in Figure Bb a hypothetical cost according to Definition 2 As we see, designing an experiment with higher 7P is more challenging than with lower TP due the increasing number of conflicts in the counterfactual rankings. Additionally, we see that experiments with lower TP are more sensitive to the choice of a.
4.2 Comparison with existing methods
Note that HtThucEtAl is designed for small ramp experiments. Following the authors' guidelines 3. we will be limiting ourselves to the case where 10% of the population is in control and 10% of the population is in treatment. For UniCoRn(a) and OASIS. we consider two different settings, namely (i) 10% treatment and 90% control and (ii) 50% treatment and 50% control.
CCode is available in the supplementary material.
7

---

OASIS solves constrained optimization problem to match each producer's 's total counterfactual scores and a post-experiment adjustment corrects for mismatches. However, we consider a modification of OASIS which is score-based counterpart of the rank-based UniCoRa(1) algorithm. We assign normalized counterfactual score to each item (i.e., no need for solving an optimization problem or for post-experiment correction). Following Section 5 of [ we define normalized scores
[s, 8) = 2/) EE (x) for k = 0, 1. Then we define the counterfactual scores as p=(x, 2xa0))Exe((o EkE(o.)) Ps (8, (LEP)))
We generate data from a simulated recommendation environment with 100 positions. Note that the computation cost shown in FigureBaais hypothetical (based on Definition 22. as we generated the treatment and the control scores from (correlated) uniform distributions and hence we did not need to apply any scoring function. More data generation details are given in Appendix A.3 We consider the following two rank to response functions:
(nvg_fn) Y; E [
27 and (max__n) fn) Y, max
10 (IoR(1o-R RD (LT.))
10 (10
where the empirical average É and the max are over all items that appeared in a session and belong to producer i. We chose the logarithmic decay function Tog(1044r) to represent the value of position (attention given to item placed at position r)) in ranked list. Then we aggregate (using the average or the max function) the attention received by the items of a producer to define response functions. The treatment effects corresponding to avg_ fn and mar fa are 0.16 and -0.87.
10
treatment proportion: 0.1
freatment Treatment_prpportion. 0.5
150-
Ug
000t
TP 0.1
0.5
50
-1
UniCoRn(O) UniCoRn 2)
UniCoRn(1)
HaThusElAi
OASIS
UniCoRn(O)
UniCoRm(0.2)
UniCoRn(1)
HaThudElAl
OASIS
UnioRRNoo 2)
UniCoRni0 UniCoRm(1)) (on1))
HaThucEiAi
OASIS
(4a) Errors in estimating the average treatment effect for two different rank to response functions and for two different values of the treatment
(4b) Hypothetical cost based on Definition 2 for treatment propor-
proportion based on Ns = 1000 sessions with L = 100 slots each.
tion (TP) equals 0.1 and 0.5.
Each iteration (based on 1000 sessions) including the data generation, reranking based on UniCoRn(a)) for a {0, 0.2, HaThueEtAl and OASIS, and the treatment effect estimation took 36 seconds on average on a Macbook Pro with 2.4 GHz 8-Core Intel Core 19 proces- sor and 32 GB 2667 MHz DDR4 memory, We repeat this 100 times and summarize the results in Figure 4al Both UniCoRn(a) outperform OASIS (even for ment effect estimation error, demonstrating the advantage rank-based methods over score-based methods. UniCorn(1) and U Unicorn(0.2) outperform HaThucEtAl as HoThucEtAl exhibits significantly higher variance due to its limitation to a 10% treatment and 10% control ramp. The performances of the variants of UniCoRM(()) 0.5. whereas UniCoRn(()) is more sensitive to the choice of a at TP = 0.1. This is consistent with the findings in Figure 3b Note that the sensitivity to the choice of is more prominent when the rank to response function is max_fn. This is consistent with Figure Sa since aug_f'n is a sub-linear function of the ranks, but max_ is not. While accounting for the computational cost given in Figure 4b) along with the estimation error in Figure the computationally cheapest method UniCoRn(()) seems to be the best choice at TP 0.5 whereas we need choose the slightly more expensive variant U (n. to ensure an estimation quality as good as UniCoRn((1).
0) in terms of the treat-
roughly equal for treatment proportion (TP)
4.3 Social Network application
Edge recommendations in social media platforms enable members to connect or follow other mem- bers. Edges also bring two sides of a marketplace together, e.g., content producers and consumers where content propagates along existing edges. Thus, edge recommendation products (see Figure
8

---

in the Appendix for a toy example) play vital role in shaping the experience of both producers and consumers. The consumers of edge recommendations are "viewers" and A/B tests can measure the viewer side impact of any ranking change. The candidates (i.e., items) recommended are "viewees", because they are the members that are viewed and receive a connection request. Edge recommenda- tions may have a large viewee impact, with number of viewees impacted often outnumbering viewers. To measure the viewee side effect, we implemented UniColR in an online edge recommender system that serves tens of millions of members, and billions of edge recommendations daily. We chose a = 0 (i.e., UniCoRn(() to minimize the online scoring latency increase. Next, we discuss two experiments conducted that cover candidate generation and scoring stage experiments. Key metrics include (i) Weekly Active Unique (WAU) users, i.c., number of unique users visiting in a week; and (ii) Sessions, i.e., number of user visits.
Candidate generation experiment: Large-scale recommender systems often have a candidate generation phase, which uses simpler algorithm to evaluate a much larger set of items. The best few are then scored in the second ranking phase, which uses more sophisticated and computationally intensive algorithms. The two phases together comprise the ranking mechanism and UniColn handles such scenarios with a simple extension. For any item selected by the control candidate selection model Co (or treatment C1) but not by C (Co). the second phase scoring by treatment Ti (i) (or control (i) is set to - Dc. The extension is detailed in the appendix (Section A.5
In edge recommendation problems, popular candidate generation heuristic is number of shared edges. This heuristic favors candidates with large networks. To neutralize this advantage, we tested variant based normalized version of shared edges (i.e., fraction of the candidate's network that are shared edges with the viewer) and measured the impact using UniCoRm(00 Thus, Co uses number of shared edges and C1 uses the normalized version to generate candidates. The second phase ranking model was unchanged in this comparison, i.e. M- To = Ti.
Metries
Delta % (candidate generation)
Delta % (ranking model)
Weekly Active Unique users
0.51%
+..13%
Sessions
0.57%
+0.11%
Table I: Viewee side impact of new candidate generation model (with the same ranking model as control) and a new ranking model (with the same candidate generation model as control), measured with 40% viewer side traffic. All results are highly significant with p-value < 0.001.
Ranking model experiment: The ranking stage scores all candidates based on the model assignment of the viewers. Ranking models may be composite models optimizing for viewer and/or viewee side outcomes. In one such experiment, the treatment model Ti optimized for viewee side retention, i.e., we boosted viewees likely to visit if they received an edge formation request. Using UniColta (0) and candidate set I,, we obtain the ranking (i, according to To and find the positions L= { (1., I.) : E Iz.1 J. Then, we rescore candidates in positions L according to T₁ to obtain rankings {Ri Ii,) } and rerank them within L to obtain the final list. UniCoRn(0) (0) is less costly because we rescore only the subset of candidates that belong to P,.
UniCoRn(O)'s implementation: To generate the ranked list of viewees for viewer, we first obtain the viewer treatment. If the viewer is not allocated to UniCoRn, we score all items using the allocated model (i.e., viewer treatment). This was also the flow prior to UniCoRn. If the viewer is allocated to UniCoRi, we then obtain the viewee treatment allocations for all viewees. The final ranking is obtained thus: (1) Score all items using a control model, (2) Obtain the viewee side treatment assignment for all viewees (i.e., items), (3) Score each viewee with the necessary treatments and blend using the scores (following Algorithm D). The changes were implemented in Java in our distributed, real-time production serving system with no statistically significant serving latency added by this change.
To Results: Table 1 shows viewee side results using 40% of the viewer with UniCoRn(O) for both the candidate generation and ranking change experiments. For each viewee (dest-member producer), we compute the response Y; defined as the total count of the metric of interest in the experiment window (e.g., the number of visits in the experiment time window). The "Delta %" in Table I the relative percentage difference between the average responses of the treatment and the control Viewee
'We were unable to have 100% viewers in our experiment due to other parallel experiments, resulting in underestimation of the actual treatment effects corresponding 100% viewer side ramps.

---

groups under the Uni design. Both experiments showed positive change in WAUs and sessions as they brought in more viewees onto the platform. Although the exact measurement fidelity could not be validated without the ground truth, we expected to observe a statistically significant positive impact. This is because we observed in a source- side experiment that the viewers tend to send invitations to more viewees under the treatment model than the control model, indicating positive impact of the treatment model on the viewees. Note that these source-side measurements can be accurately obtained from a classical A/B testing setup on the viewer-side.
5 Discussion
A/B testing in social networks and two-sided marketplaces is extremely important to improve the experiences offered to various stakeholders. Our proposed experimentation design mechanism, UniCoRn, allows for high-quality producer side measurement with an explicit parameter to control the cost of the experiment at the expense of accuracy (or quality) loss in the measurement. Our exper- iment design is provably optimal, and our method has significant advantages over prior approaches: (i) It is agnostic to graph density (unlike, e.g., [15). (ii) It makes no assumption on how the treatment effect propagates (unlike, c.g., (9]) or how the response depends on the treatment exposure (unlike, e.g., [10)). (iii) lowers the variance of measurement (unlike, e.g., (3)). and (iii) It does not depend on knowing the graph structure a priori (unlike most existing methods, e.g., 99 TO (5]).
Limitations and Future work: Our experiment design framework focuses capturing the differ- ence in exposure distribution of the producers in the treatment group and the producers in the control group. Hence, the UniCoRR based treatment effect estimates would fail to capture some other types of differences between the treatment and the control. For example, a treatment may have an impact on viewer's attention (e.g.. the amount of time viewer is spending on each session the total number of viewer's ' sessions). This impact would not be captured by the UniCoRn design, where all viewers receive a mix of treatment and control ranking.
The design accuracy measurement framework based on Definition I does not directly translate to the accuracy in the producer side treatment effect estimation without additional assumptions on the ranking to response function. We deliberately refrain from making such assumptions to build a more generally applicable experiment accuracy based framework. In the appendix, we discuss some additional assumptions under which the optimality result given in Theorem I can be extended to the treatment effect estimation problem. An interesting future direction could be to explore other types of loss functions in Definition] and study their connections with treatment effect estimation accuracy.
While UniCoRn is designed to measure the producer side effect, sometimes the two sides of marketplace are the same set of users playing different roles (c.g., a content producer is also a content consumer), In such scenarios, it may be important to measure the combined consumer and producer side effect. Such a measurement can be obtained by having a small set of producers, who are allocated to treatment, have their consumer experience ranked entirely based on that same treatment. This set has to be relatively small since the producer side experience will only be accurate if a large fraction of the consumers are UniCoRR (instead of pure treatment or pure control).
related problem to balance the power of the measurement (via higher producer side ramps) with the risk (which increases with larger consumer side ramps). Also, our proposed methodology can be extended to multi-partite graphs (i.c., marketplaces with more than two sides, such as food delivery platforms that connect users with drivers with restaurants). Such an extension would depend on the dynamics between the different graph partitions (i.e., entity types in the marketplace).
Acknowledgment
We would like to thank Parag Agarwal, Kinjal Basu, Peter Chng, Albert Cui, Weitao Duan, Akashnil Dutta, Aastha Jain, Aastha Nigam, Smriti Ramakrishnan, Ankan Saha, Rose Tan, Ye Tu and Yan Wang for their support and insightful feedback during the development of this system. We would also like to thank the anonymous reviewers for their helpful comments which has significantly improved the paper.
Finally, none of the authors received any third- party funding for this submission and there is no competing interest other than LinkedIn Corporation to which all authors are affiliated.
10

---

References
[1] M. Aronow and A. Middleton. class of unbiased estimators of the average treatment effect in randomized experiments. Journal of Causal Inference, 1(1):135-154) 154, 2013.
[2] G. W. Basse, H. A. Soufiani, and D. Lambert. Randomization and the pernicious effects of limited budgets on auction experiments. In Artificial Intelligence and Statisties, pages 1412- 1420. PMLR, 2016.
[3] V. Ha-Thuc, A. Dutta, R. Mao. M. Wood, and Y. Liu. A counterfactual framework for seller-side A/B testing on marketplaces. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), 2020.
[4] P. W. Holland. Statistics and causal inference. Journal of the American statistical Association, 8 (396):455-960, 1986.
[5] R. Kohavi, A. Deng, R. Longbotham, and Y. Xu. Seven rules of thumb for web site experimenters. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1857-1866. ACM, 2014.
[6] R. Kohavi and R. Longbotham. Online controlled experiments and A/B testing. Encyclopedia of machine learning and data mining, 7(8):922-922 2017.
[7] M. Liu, J. Mao, and K. Kang. Trustworthy online marketplace experimentation with budget-split design. arXiv preprint aiXiv:2012.08724, 2020.
[8] C. Lo, E. De Longueau, A. Saha, and S. Chatterjee. Edge formation in social networks nurture content creators. In Proceedings of The Web Conference 2020, pages 1999-2008, 2020.
[9] Nandy, K. Basu, S. Chatterjee, and Y. Tu. A/B testing in dense large-scale networks: Design and inference. In Proceedings of the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
[10] J. Pouget-Abadie, K. Aydin, W. Schudy, K. Brodersen, and V. Mirrokni. Variance reduction in bipartite experiments through correlation clustering. In Advances in Neural Information Processing Systems 32, pages 13309913319. Curran Associates, Inc., 2019.
[11] D. B. Rubin, Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
[12] D. B. Rubin. Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, pages 34-58, 1978.
[13] D. B. Rubin. Formal mode of statistical inference for causal effects. Journal of statistical planning and inference, 25(3):279-292, 1990.
[14] G. Saint-Jacques, M. Varshney, J. Simpson, and Xu. Using ego-clusters to measure network effects at LinkedIn. arXiv preprint arXiv: 1903.0855... 2019.
[15] G. Saint-Jacques, M. Varshney, J. Simpson, and Y. Xu. Using ego-clusters to measure network effects at LinkedIn. arXiv: 1903.08755, 2019.
[16] D. Tang, A. Agarwal, D. O'Bri,n, and M. Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 17-26, 2010.
[17] Y. Tu, C. Lo, Y. Yuan, and S. Chatterjee. Feedback shaping: A modeling approach to nurture content creation. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages
2019.
[18] Ugander, B. Karrer, L. Backstrom, and J. Kleinberg. Graph cluster randomization: Net- work exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 329-337, 2013.
[19] Y. Xu, N. Chen, A. Fernandez, O. Sinno, and A. Bhasin. From infrastructure to culture: A/b testing challenges in large scale social networks. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2227-2236, 2015.
11

---

7 Checklist
((a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope?
Yes. Please see Sections and 4
1(b) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes
1(c)
Did you discuss any potential negative societal impacts of your work?
N/A
I(d)
Did you describe the limitations of your work?
Yes. Please see section 5
2(a)
Did you state the full set of assumptions of all theoretical results?
Yes.
2(b) Did you include complete proofs of all theoretical results?
Yes. Please see the Appendix.
3(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URI.)?
Yes. All code, data, and instructions needed to reproduce the main experimental results given in the supplementary materials.
3(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?
Yes. The details are given in Section
3(c) Did you report bars (e.g., with respect the random seed after running experiments multiple times)?
Yes. We have shown Box-plots
3(d) Did you include the amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)?
Yes. The details are given in Section
4(a) If your work uses existing assets, did you cite the creators?
N/A.
4(b) Did you mention the license of the assets? N/A.
4(c) Did you include any assets either in the supplemental material URL? N/A
4(d) Did you discuss whether and how consent was obtained from people whose data you 're us- ing/curating?
N/A.
4(e) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation or offensive content?
N/A.
5(a) Did you include the full text of instructions given to participants and screenshots, if applicable? N/A.
5(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
N/.
5(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
N/A.
12


Reviews:
[{"review_id": "zCmFB4SGxb7", "reviewer": "Reviewer_7Bcd", "summary": "In this paper, the author studied the problem of doing A/B testing for \"producer\" in two-side marketplace. Specifically, the author proposed a \n new metric to measure the inaccuracy in measuring the treatment effect for producers. The author also proposed a new experiment design and proves it to be optimal to the new metric. The experiment design was applied to a real world recommendation system.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written. The problem is very realistic and has huge real world impact. The ideas are novel and the reasoning process is also very clear. I would like to see it being accepted.\n\nReasons to accept:\n1. The paper is well written.\n2. The problem has huge real world impact.\n3. The ideas are novel, the author proposed a new metric to measure the in-accuracy of producer-side experiment, and it also proposed a new experiment design which is proved to be optimal to this measure.\n4. The experiment design was applied to real world application.\n\nReasons to reject:\n1. some minor issues about writing. For example, Line 16 \"we prove that our experiment design is optimal\". This seems to be over-claimed. It should be \"we prove that our experiment design is optimal to the proposed in-accuracy measure\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper would lead to large societal impact but I don't think it is a negative one.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "z96LtlCfrLO", "reviewer": "Reviewer_aZdN", "summary": "The authors propose a A/B testing framework that measures producer effectiveness in a two-sided setting.  The authors argue that while there are many ways to measure the consumer-side experience consistently, there is none on the producer-side that measures the effectiveness of a producer consistently, i.e., measure the experience of all consumers that use a particular producer.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors contend that existing approaches to measure producer-side impact such as clustering-based randomizations produce 'low-powered experiments'.  It will greatly help if they can compare the outcomes of the UniCoRn algorithm with the clustering approach.  It is not clear why the set up is considered two-sided since impact of both sides is not measured simultaneously.  \n\nThe paper is slightly hard to read.  How are the counterfactual rankings produced?  There can be a lot of them. In Fig. 2.iii, where is R_2 coming from and how are the ranks assigned?  There is a lot of work on rank aggregation (ranking producers under different user preferences and combining the orderings by minimizing the number of inversions under any individual ordering).  It'll help to compare the techniques used in this work to rank aggregation.\n\n======\n\nI read the authors' rebuttal and thank them for clarifying some of my misunderstanding.  I however feel that without addressing the impact on both sides of the market simultaneously, this is not a two-sided problem.  Also, it's not clear to me why the clustering approach cannot be used for a dynamic graph setting.  Naively, can't it could be done on an instance of the graph in each round of allocation?\n\n======", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "odzf2LU7NVr", "reviewer": "Reviewer_q14B", "summary": "This paper studies the problem of conducting proper producer side A/B testing on 2-sided market/social networks. To be specific, the authors provide a series of heuristic-based methods to solve the ranking conflict which is common in producer side a/b testing because a users recommendation result can be filled with producers exploiting different ranking algorithms. The authors introduce a cost hyper-param to control the trade off between cost and accuracy. The authors provide both offline experiments and online use case study.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a very interesting problem, how to conduct proper producer side A/B testing and resolve potential ranking conflicts. This is interesting because most of A/B testing conducted in recommendation systems are consumer side A/B test and because of the disjoint of consumer-grouping, it would not face a lot of conflicting issues as mentioned in the paper.\n\nBut producer side A/B testing is very important as a lot of recommendation changes impact how producer's items(e.g. news stories, products) are consumed by users and will create a long term impact on recommendation systems. Methods proposed by this paper, though heuristic-based, can be beneficial for others to improve producer-side A/B test design.\n\nThe problem set up is actually quite complicated, but the authors provide a lot of intuition explanation and toy examples which help to make it easier for readers to follow and make the paper itself clearer.\n\nI am also glad to see, in the evaluation part, the authors provide some use-case study on how this evaluation mechanism is deployed in some large scale recommendation applications, which makes the evaluation more convincing.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have some concerns and questions regarding this paper:\n\n1.The method proposed by this paper is still a compromise. Because conflict resolving is needed, the ranking result in the treatment arm will be different compared to when the recommendation algorithm is fully launched to all producers.\n\nA naive way to solve this is to create experiment arms based on dis-joint user cohort and apply different recommendation algorithms to all the producers within each arm. Then observe the impact of different recommendation algorithms on producers by comparing the aggregated producer-side performance from different arms.\n\nWhy is the proposed method better than the naive way? Naive way seems much cleaner.\n\n2. The problem set up section can be further refined to make it easier for others to follow the objective of this paper. Current version is still a little bit difficult to follow.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ViKQmyiN9K", "reviewer": "Reviewer_neLM", "summary": "This paper introduces unicorn, a method to estimate the effect of different recommender systems on the producers in a platform. This is more challenging than consumer-side A/B testing where a simple randomized trial suffices since a producer's content will be recommended across multiple different treatments.\nUnicorn works by ensuring the overall ranking of items presented to a consumer approximates the ranking of all treatments under consideration. The authors first present the algorithm for the case where there exists one control and one treatment, highlighting a parameter that allows them to tradeoff accuracy and cost. They then propose generalizations of unicorn to multiple treatments.\nThe paper goes on to present theoretical results bounding the bias and variance of unicorn (for one control/treatment with alpha set to 1), and showing their experimental design is optimal wrt the ranking error metric they defined.\nFinally, they evaluate unicorn in simulation, comparing it to prior work. They also deploy unicorn in production on a large social media network.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is a well-written paper that addresses an important and novel problem that has seen too little attention from the research community. However, I do have some concerns I hope the authors can address. I have listed them roughly from most important to least important.\n - It seems the paper primarily focuses on minimizing the inaccuracy in Definition 1, both in their experiments and in the theoretical portion of their work. However, this is simply a proxy metric, what we hope to eventually measure is the average treatment effect, as the authors mention in the introduction. As a result, I was surprised that this seemed to not be considered in most of the paper. In the discussion section, the authors mention that their theoretical results extend to treatment effect estimation errors that are monotonic in their defined error metric, however, they show no proof of this, nor is it made clear whether common treatment effect estimates will be monotonic.\n - Following on the previous point, the simulated experiments in section 4.1 only measure MAE/RMSE with respect to the ideal ranking. Yet this would have been the ideal setting to measure the accuracy of various realistic treatment effect estimators under the unicorn framework. This would give the reader significantly more information regarding the reliability of the results they would obtain from running unicorn.\n - The motivation for the choice of rank to response functions defined on line 274 was unclear to me for the experiments in section 4.2.\n - I had a hard time interpreting the results presented in Table 1, how were the deltas computed? The caption mentions the results were highly significant with a very low p-value, however I couldn't find an explanation regarding how that p value was computed.\n - I wasn't sure what the takeaway should be for section 4.3, \n - I think the clarity of the paper could be improved by discussing the intuition behind why the algorithm work in section 3.1 alongside the already very well explained explanation as to how it works.\n\nGiven my concerns, I feel the paper might not yet be publication-ready. However, if the authors address my concerns (especially the top one regarding treatment effects), I am more than happy to raise my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A/B testing and recommender systems are both subfields that have a very large impact on hundreds of millions of users. As such, I feel it is necessary that the authors address the potential societal impacts of their work. As it stands there is no such discussion.\n\nI think the authors did a good job of addressing the technical limitations of their work in the discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this submission, the authors study a very interesting problem, i.e., how to measure from the producer (or seller) side in a two-sided marketplace. This studied task can have great application potential. Due to this, here, I recommend to accept this submission. 

However, I do have the same major concern with reviewer neLM, that is, whether the strong assumptions the authors made can be guaranteed. This can hurt the impact of this submission. 

Further, this submission also can be improved based on the comments from all the reviewers and the discussion between reviewers and authors. Hope they find these useful, and make this submission a better one.

Decision: Accept (Poster)