Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper uses an optimal-transport approach to ensure that two similar items that differ only on a sensitive feature (e.g., gender) are ranked similarly. In this way, individual items are treated fairly. For example, if two job candidates differ in gender but are otherwise similar, the job search engine should rank them similarly. The authors employ an optimization problem they call SenSTIR where the objective is maximizing expected utility minus a regularization term, and where the fairness properties are encoded in the regularization term. The regularization term is weighted by rho, a parameter than increases fairness as it increases. The regularization term itself encourages similar queries to lead to similar rankings. The authors appeal to duality to derive a way to run stochastic optimization on their problem. They prove one theorem that I don't fully understand. Then they test their method on three data sets: simulated data used in related work, German credit data used in prior work, and Microsoft learning to rank data. They compare to training without fairness, a projection method, Fair-PG-Rank, and random. They try different values of rho, showing the larger rho indeed leads sensibly to more individually fair rankings. They use the NDCG metric for accuracy and Kendall's tau correlation for fairness. They show that they get individual fairness by design and group fairness "for free". In contrast, Fair-PG-Rank design for group fairness does poorly on individual fairness.

The paper is well written and includes an excellent and extensive survey of related work. The authors approach seems reasonable and their analysis and experiments are convincing. I especially like Figure 2 -- this is a nice visualization of individual fairness and was very helpful.

One quibble I had is that the authors said that individual fairness is sufficient for group fairness, or at least "can be sufficient". Usually "sufficient" is a theoretical statement that always holds. Later, it's clear that this is an empirical finding on the particular data sets tried. I'd like the authors to be more careful about using the word "sufficient".

One question I had is how this would generalize to multiple sensitive features? To be practical, that generalization will be valuable.

Can the authors say something about computational complexity? Is the problem 2.4 NP-hard? Is Algorithm 1 an approximation? What is the running time of Algorithm 1 both in theory and in the experiments (seconds). How does the running time of SenSTIR compare with Fair-PG-Rank?

A higher level question is why the authors did not test fair supervised learning algorithms. Yes, as the authors point out, supervised learning is not meant for LTR (and SL doesn't define a fair distance on sets of items and uses different metrics, for example, L2 loss versus NDCG), but a supervised learning method can produce a relevance score. So the experiments could try one of these approaches, and probably should. (I didn't quite understand the point about not defining a distance on sets.)

Minor note: 

References: Some of the references should be cleaned up and made more uniform. For example, the reference "Ke Yang, Vasilis Gkatzelis, and Julia Stoyanovich" repeats the conference title four times.
review_title: solid paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper tackles the problem of providing rankings that are "fair" to individual items being ranked as opposed to groups of items as in previous work on fair ranking.

I have three major issues with the paper:

1- A practical objection to the approach is the use of randomized rankers to get around the important issue that ranking is by its very nature unfair to individuals. To take the example of job applications used in the paper, if 100 people are applying to one job and a ranked list of the candidates is presented to the person making the hiring decision, then well one person will be at the top and one person at the bottom. Now, if there are 100 job openings and the same 100 people apply to all of them, then yes randomizing the results can deal with this issue, but then fairness is not even an issue because everyone gets a job and there won't be any complaints. In other words, fairness is only an issue is there is a lack of resources, which means that randomization doesn't really solve the issue, but just gives one the illusion that it is resolved. I feel like the authors haven't spent too much time thinking about these issues and that they're mostly thinking about the unbounded query scenario. But, even in that setting, there are very few products where the people in charge would be willing to randomize the ranked items because that would result in degradation in user experience (i.e. in this example, the person doing the hiring). There have been some recent attempts to address the planning issue that comes up when dealing with this, so I'd suggest for the authors to have a look at papers like this one: https://arxiv.org/abs/2008.00104

2- A more fundamental issue is that I just can't find a proper metric for individual fairness in the paper. The only proposal is a comparison between the outputs of the ranker before and after altering some features fed into it. This way of evaluating rankers makes no sense to me at all because a really easy way for a ranker to appear fair is by not using the feature being modified (e.g. age or gender) directly, but inferring it from the remaining features. I think the issue is that the authors are confused by the subtle but crucially important distinction between losses and metrics. You're allowed to use whatever information you like when you're designing your loss, but the final evaluation should not rely on the model, but just its output.

3- Finally, on the MSLR dataset, it's important to include LambdaMART as a baseline to report the degradation in NDCG incurred as a result of the fairness constraints. I should point out that there are multiple implementations of LambdaMART out there and it's important to used the best one, which is the one included in the LightGBM package. You can find some results for the MSLR dataset in this paper among others: https://dl.acm.org/doi/pdf/10.1145/3336191.3371844

Given these issues, I don't think the paper is ready to be published and I encourage the authors to think carefully about their problem definition and experimental setup before resubmitting the paper.
review_title: Interesting paper with some major flaws

final_score: rating: 7: Good paper, accept
review_content: The authors extended individual fairness approach to the domain of learning to rank domain. This paper proposes a method for training individually fair learning to rank models by making use of optimal transport-based regularizer.

This work appears to be an extension to Yurochkin et. al. 2020 ICLR SenSR paper in which fair metric was learned from data. While that papers focused on training individually fair ML models by casting the problem as training supervised ML models that are robust to sensitive perturbations, this paper extended the idea to individual fair ranking that are invariant to sensitive perturbations of features of ranked items. 

###############
This paper is well written. The code and the datasets used for the experimentation have been provided. 
I vote to accept this paper. I like the approach presented in this paper for solving this very important issue. 

###############
The importance of the problem of dealing with bias in search results cannot be understated. Example of the issue of fairness in search ranking: how to ensure that minority candidates (or candidates of opposite gender) would be fairly ranked together with other job candidates when having similar qualifications. This paper attempts to find a solution to this and similar type-problems.


###############
In this paper both theoretical and empirical results were presented. The authors showed that using their optimal transport-based regularizer leads to certified individual fair LTR models.
The results were demonstrated on several datasets: synthetic and two publicly available datasets. Results showed that authors’ method exhibited ranking stability with respect to sensitive perturbations when compared with a state-of-the-art fair LTR model.
Importantly, the authors showed empirically that individual fairness implied group fairness by not vice versa. 

###############
Recommendations:
On page 4 section 3 authors reference Theorem 2.3 but in the citation there is no Theorem 2.3.  In the citation there is a relevant equation 2.3 and a note that states this equation comes from another paper. So I believe that this should probably needs to be cited differently (by citing Blanchet & Murthy original paper - assuming they were the first to prove this theorem) 
Additionally, authors keep citing Yurochkin & Sun ArXiv pre-paper even through it has already appeared at ICLR 2020. 

Question:
My only question for the authors is how easy would it be to learn fair metric in a typical application for LTR models? 
review_title: Individually Fair Rankings.  Interesting paper that extends the individual fairness approach to learning to rank domain.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper address the problem of fair ranking. The authors use a notion of individual fairness, meaning that two similar inputs should receive similar outputs. The presented method uses a transport based regularizer to reach fairness.
The authors present a new Algorithm SenSTIR and test it on a synthetic and two real world datasets


The paper is nicely written and readable, however the maths are hard to follow and little intuition is given.
I didn't find an explanation of how the counterfactuals are computed. This is a difficult task that need many assumptions and hypotheses, just "flipping" the gender is hardly enough as gender is correlated to all others attributes.
The legend of figure 3 and 4 are not the same "Kendall's Tau Correlation with Counterfactuals" and "Counterfactual Kendall's Tau Correlation", is it the same metric or something changed ?

In figure 2, Fair-PG-Rank seems to perform very poorly, worse than baseline. As it is the state of the art method, I think some explanation of why it is the case are needed.

Finally most of the paper is based on work done in Yurochkin &Sun 2020 and Yurochkin et al 2020. I think that the authors should explicit the link between the current paper and those two citations. I also think that the main paper should be self content and that Theorem 2.3 of Yurochkin &Sun 2020 should be written down with its hypotheses.

---- After reading answers ------

I'm still not convinced by the computation of counterfactuals, I still disagree with "flipping" the private attribute, because then the naive solution of removing them from the input of the model would appear perfectly fair. This approach seems to forgot about potential proxies, that are not the private attribute, but that a model can learn on and be biases.
Moreover, if I understand correctly, the same computation of counterfactuals is used for the model and the test set. This is unfair as SenSTIR is the only algorithm to use the same kind of counterfactual data than the one used for the evaluation.

I think this is a crucial point, I prefer to lower my rating to 5.
review_title: Weak accept

Meta-review: The paper focuses on individual fair ranking and proposes an approach for that based on optimal transport. The reviewers are in general positive about the paper, however, there are a a couple of concerns that I believe should be addressed before publication.

First, I find the treatment of the term "counterfactual" misleading in the paper.  Counterfactual fairness has been proposed in the literature as a causal notion of individual fairness. However, as far as I can see in the paper, there is not such a causal treatment of counterfactuals in the paper. Thus, I suggest the authors to reconsider their treatment of counterfactuals in the paper, as it may trigger confusion. Second, I also agree with R1 that  it is unfair as SenSTIR is the only algorithm to use the same kind of "counterfactual" data than the one used for the evaluation. 


Decision: Accept (Poster)