Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces unicorn, a method to estimate the effect of different recommender systems on the producers in a platform. This is more challenging than consumer-side A/B testing where a simple randomized trial suffices since a producer's content will be recommended across multiple different treatments.
Unicorn works by ensuring the overall ranking of items presented to a consumer approximates the ranking of all treatments under consideration. The authors first present the algorithm for the case where there exists one control and one treatment, highlighting a parameter that allows them to tradeoff accuracy and cost. They then propose generalizations of unicorn to multiple treatments.
The paper goes on to present theoretical results bounding the bias and variance of unicorn (for one control/treatment with alpha set to 1), and showing their experimental design is optimal wrt the ranking error metric they defined.
Finally, they evaluate unicorn in simulation, comparing it to prior work. They also deploy unicorn in production on a large social media network.
main_review: Overall I think this is a well-written paper that addresses an important and novel problem that has seen too little attention from the research community. However, I do have some concerns I hope the authors can address. I have listed them roughly from most important to least important.
 - It seems the paper primarily focuses on minimizing the inaccuracy in Definition 1, both in their experiments and in the theoretical portion of their work. However, this is simply a proxy metric, what we hope to eventually measure is the average treatment effect, as the authors mention in the introduction. As a result, I was surprised that this seemed to not be considered in most of the paper. In the discussion section, the authors mention that their theoretical results extend to treatment effect estimation errors that are monotonic in their defined error metric, however, they show no proof of this, nor is it made clear whether common treatment effect estimates will be monotonic.
 - Following on the previous point, the simulated experiments in section 4.1 only measure MAE/RMSE with respect to the ideal ranking. Yet this would have been the ideal setting to measure the accuracy of various realistic treatment effect estimators under the unicorn framework. This would give the reader significantly more information regarding the reliability of the results they would obtain from running unicorn.
 - The motivation for the choice of rank to response functions defined on line 274 was unclear to me for the experiments in section 4.2.
 - I had a hard time interpreting the results presented in Table 1, how were the deltas computed? The caption mentions the results were highly significant with a very low p-value, however I couldn't find an explanation regarding how that p value was computed.
 - I wasn't sure what the takeaway should be for section 4.3, 
 - I think the clarity of the paper could be improved by discussing the intuition behind why the algorithm work in section 3.1 alongside the already very well explained explanation as to how it works.

Given my concerns, I feel the paper might not yet be publication-ready. However, if the authors address my concerns (especially the top one regarding treatment effects), I am more than happy to raise my score.
limitations_and_societal_impact: A/B testing and recommender systems are both subfields that have a very large impact on hundreds of millions of users. As such, I feel it is necessary that the authors address the potential societal impacts of their work. As it stands there is no such discussion.

I think the authors did a good job of addressing the technical limitations of their work in the discussion section.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a A/B testing framework that measures producer effectiveness in a two-sided setting.  The authors argue that while there are many ways to measure the consumer-side experience consistently, there is none on the producer-side that measures the effectiveness of a producer consistently, i.e., measure the experience of all consumers that use a particular producer.
main_review: The authors contend that existing approaches to measure producer-side impact such as clustering-based randomizations produce 'low-powered experiments'.  It will greatly help if they can compare the outcomes of the UniCoRn algorithm with the clustering approach.  It is not clear why the set up is considered two-sided since impact of both sides is not measured simultaneously.  

The paper is slightly hard to read.  How are the counterfactual rankings produced?  There can be a lot of them. In Fig. 2.iii, where is R_2 coming from and how are the ranks assigned?  There is a lot of work on rank aggregation (ranking producers under different user preferences and combining the orderings by minimizing the number of inversions under any individual ordering).  It'll help to compare the techniques used in this work to rank aggregation.

======

I read the authors' rebuttal and thank them for clarifying some of my misunderstanding.  I however feel that without addressing the impact on both sides of the market simultaneously, this is not a two-sided problem.  Also, it's not clear to me why the clustering approach cannot be used for a dynamic graph setting.  Naively, can't it could be done on an instance of the graph in each round of allocation?

======
limitations_and_societal_impact: None.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the author studied the problem of doing A/B testing for "producer" in two-side marketplace. Specifically, the author proposed a 
 new metric to measure the inaccuracy in measuring the treatment effect for producers. The author also proposed a new experiment design and proves it to be optimal to the new metric. The experiment design was applied to a real world recommendation system.
main_review: Overall, the paper is well written. The problem is very realistic and has huge real world impact. The ideas are novel and the reasoning process is also very clear. I would like to see it being accepted.

Reasons to accept:
1. The paper is well written.
2. The problem has huge real world impact.
3. The ideas are novel, the author proposed a new metric to measure the in-accuracy of producer-side experiment, and it also proposed a new experiment design which is proved to be optimal to this measure.
4. The experiment design was applied to real world application.

Reasons to reject:
1. some minor issues about writing. For example, Line 16 "we prove that our experiment design is optimal". This seems to be over-claimed. It should be "we prove that our experiment design is optimal to the proposed in-accuracy measure"
limitations_and_societal_impact: This paper would lead to large societal impact but I don't think it is a negative one.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of conducting proper producer side A/B testing on 2-sided market/social networks. To be specific, the authors provide a series of heuristic-based methods to solve the ranking conflict which is common in producer side a/b testing because a users recommendation result can be filled with producers exploiting different ranking algorithms. The authors introduce a cost hyper-param to control the trade off between cost and accuracy. The authors provide both offline experiments and online use case study.

main_review: The authors propose a very interesting problem, how to conduct proper producer side A/B testing and resolve potential ranking conflicts. This is interesting because most of A/B testing conducted in recommendation systems are consumer side A/B test and because of the disjoint of consumer-grouping, it would not face a lot of conflicting issues as mentioned in the paper.

But producer side A/B testing is very important as a lot of recommendation changes impact how producer's items(e.g. news stories, products) are consumed by users and will create a long term impact on recommendation systems. Methods proposed by this paper, though heuristic-based, can be beneficial for others to improve producer-side A/B test design.

The problem set up is actually quite complicated, but the authors provide a lot of intuition explanation and toy examples which help to make it easier for readers to follow and make the paper itself clearer.

I am also glad to see, in the evaluation part, the authors provide some use-case study on how this evaluation mechanism is deployed in some large scale recommendation applications, which makes the evaluation more convincing.

limitations_and_societal_impact: I have some concerns and questions regarding this paper:

1.The method proposed by this paper is still a compromise. Because conflict resolving is needed, the ranking result in the treatment arm will be different compared to when the recommendation algorithm is fully launched to all producers.

A naive way to solve this is to create experiment arms based on dis-joint user cohort and apply different recommendation algorithms to all the producers within each arm. Then observe the impact of different recommendation algorithms on producers by comparing the aggregated producer-side performance from different arms.

Why is the proposed method better than the naive way? Naive way seems much cleaner.

2. The problem set up section can be further refined to make it easier for others to follow the objective of this paper. Current version is still a little bit difficult to follow.
review_title: null

Meta-review: In this submission, the authors study a very interesting problem, i.e., how to measure from the producer (or seller) side in a two-sided marketplace. This studied task can have great application potential. Due to this, here, I recommend to accept this submission. 

However, I do have the same major concern with reviewer neLM, that is, whether the strong assumptions the authors made can be guaranteed. This can hurt the impact of this submission. 

Further, this submission also can be improved based on the comments from all the reviewers and the discussion between reviewers and authors. Hope they find these useful, and make this submission a better one.

Decision: Accept (Poster)