Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper formalizes the idea of outcome homogenization, where a specific individual might receive negative outcomes from an array of different (but interconnected) models. The key contribution is to propose formal measures of homogenization and conduct analysis showing how homogenization relates to shared datasets and shared models used to carry out different tasks.
strengths_and_weaknesses: The paper formalizes a very interesting and relevant notion of disadvantage which indeed may be prevalent given the widespread use of shared datasets and shared (foundational) models. While the prior works allude to the same notion (as the paper mentions in Section 3), a formalism and dedicated study has indeed been lacking. Formalizing homogenization opens up the doors for important analysis and insights, for instance the insight in Section 4 that even intersectional fairness metrics may not be able to detect systematic disadvantages. The paper is generally very well-written and easy to follow. Assumptions are stated rather clearly and formalism is followed by appropriate reasoning.

Unfortunately, the definitions and experiments seem quite preliminary. Definitions being preliminary is perhaps OK since the paper is only initiating research into a new area and later papers can study the topic in more detail. But drawing insights from the limited set of experiments presented here seems premature. For these reasons, I think the paper should be improved before it is ready for publication. Please see detailed comments for suggestions / questions:

1. The data sharing insights use just one single model (logistic regression) and dataset (ACS PUMS). How do we expect the insights to vary when we use other datasets or other models? Would the insights change if we use high parametrized models like NNs (where memorization is a concern)? Drawing insights from a single dataset seems insufficient.

2. It would also have been good to see some discussion into where the data sharing ends and model sharing starts. After all, two different versions of a foundational model still have a lot of training data in common (since the underlying model was pretrained on the same data). Given the fact that a large part of the training data is still shared, how do we reconcile the two sets of takeaways from Section 4 and 5?

3. I feel that the definition of homogenization might be too strict. Consider for example Eq. 2. Right now the quantity $\prod_i I^i$ is only 0 or 1 — indicating whether or not *all* systems misclassify the individual in question (even with 1 out of 100 systems going the other way, the takeaway would change completely). Why not use a soft version, for instance, fraction of systems that misclassify the individual?

4. I think the formulation in Section 3.2 can be simplified quite a bit. The weighted average of groups in Eq. 5 seems quite difficult to interpret. Why not simply measure which groups are being disadvantaged at a higher rate than others. In other words, why not measure the gap between the best and worst systematic group failure rates?

5. Line 119 and 120: First the paper mention $x_i^j$ but then the next line mentions $x_j^i$, causing a confusion in the terminology.

6. Looks like Figure 1 is quite sparse in terms of number of points. Given the large change that happens between $10^1$ and $10^3$, it might make sense to add more points in this region. Also, since the results anyway plateau quite quickly, why not consider a non-log scale? Finally, it would be great to add point markers indicating where the observed points are.


questions: Line 89: “not far from reality”. Does a lot of companies using the same provider/software indicate that the software outcomes on a given resume will be the same for all the companies?

Line 232: How are the 5 training runs different from each other? I understood how 5 samples of training data are different (drawn using different seeds), but could not understand the difference with training runs. Is it the model random seed? If yes, that should make no difference since the logistic regression solution is deterministic (owing to convexity).

Line 288: Is the whole transformer trained from scratch? How much time did it take? How did the accuracy compare to the fine tuned models?

Line 309: Do these results also consider different random seeds (there are some error bars in Fig 2 but maybe I missed how these are computed)? This might be quite important given the non-convex nature of these models.

limitations: In general the paper is quite up front about potential limitations. There are still many places where improvements can be made. For suggestions for improvements, please see the main review.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors discuss how model and/or data sharing arising from a monoculture could lead to outcome homogenization, thus giving rise to several individual and group harms at a systemic level. Taking the example of resume screening, if a resume screening model misclassifies the resume of an otherwise well qualified individual and the same model is being utilized at every company, that individual will be experiencing systemic harm as they'll be rejected by every company. Similar case could be made for college admissions. The authors highlight that given recent advances in NLP, with large language models forming the basis of several AI applications, their sharing across applications and across companies could potentially be harmful to users for whom the underlying language model (that is being shared) does not perform accurately. The authors then extend this to group harms (systemic harms that could be faced by minority populations for instance). The research hypothesis that the authors then test is that *if algorithmic systems are increasingly built on the same data or models, then they will increasingly homogenize outcomes*.

The authors conduct experiments on data sharing and model sharing and find supporting evidence for their hypothesis in the data sharing regime, while finding mixed results in model sharing which suggest that the pretraining strategy used to train the underlying model likely plays a role in outcome homogenization.

While overall this is a sound paper, I have several questions. The paper is missing several inferential steps as one has to make some big (and potentially unrealistic) assumptions to arrive at a scenario where the same systems are being used by the same companies and/or are trained on the same data. Furthermore, the results on data sharing are not much different than the scenario where different datasets are used (when you reach closer to 10,000 datapoints). The practical viability of this problem is not established and I hope the authors will take time during the response period to address some of these issues.

-----------------------------------------------------------------------------------
I have updated my score following discussion with the authors but still have some concerns (reflected in my score) about the practical viability of the assumptions made in the experiments in the paper.
strengths_and_weaknesses: ***Strengths***:
- The paper is well situated in prior work on monoculture.
- The experimental setups are well created and the discussion is particularly strong (except the assumptions).
- Limited results on data sharing suggest that there could be outcome homogenization in such scenarios and strikingly clear results on model sharing following certain forms of pretraining.

***Weaknesses***:
- The foundation of this problem is very fragile. It relies on a lot of assumptions being true, where it's unclear how true they might be. Counter example: Chase Bank would likely create its own model on it's own data (that it might merge with the Census data) to predict whether someone should be approved a loan or not. Citi Bank would create it's own. The corporations do not have an incentive to share models or data (or even rely on existing pretrained models when the normal practice in companies is to retrain their models frequently to account for new data) as they are competing against one another. Now let's look at resume screening. Take for example, Companis X and Y. Company X has different needs than Company Y despite both hiring for AI Research Scientists. Would Company X and Y want to use the same resume screening software trained on a generic dataset or have it at least be calibrated to their individual needs?
- Experiments on data sharing are limited to only Census data and it is unclear whether the trends hold beyond that dataset. In absence of additional results, a possible inference from these results could be that the trends you're showing are special to just the Census data.

***Minor comments***:
- The paper often makes broad generalizations when there are important qualifiers that need to be attached. For instance, Line 256 should be saying that data sharing appears to exacerbate outcome homogenization for small datasets...
- When providing examples in introduction (Adam, BERT, Pytorch, etc.), please include citations.
- Typo on Line 366: align aligns ---> align
questions: - Did you try any experiments on data sharing with pretrained architectures? For example, you share the data you're using to finetune models but you're finetuning different pretrained models---that too is data sharing.
- Why did you run logistic regression five times on each sample? Correct me if I'm wrong but shouldn't the variance just be coming from different samples in case of a logistic regression model (this wouldn't have changed your plots as you're only plotting means but still curious).
limitations: I don't think the paper has done a good job at putting its limitations and assumptions up front.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a mathematical framework for characterizing outcome homogenization wherein homogenization is measured as the observed probability of systemic failure normalized by base rates. A conceptual motivation is also provided with the example of a algorithmic hiring. Experiments on US census data demonstrate that data sharing homogenizes outcomes for individuals and racial groups, this is more prominent for small training datasets. 
strengths_and_weaknesses: Strengths:

1. The paper studies an important topic and can be valuable for further research in the context of fairness, accountability, and transparency of ML systems. 

2. The claims made in the paper are backed by reasonable experimentation and exposition of the results

Weakness:

The paper would be much stronger with a more fine grained analysis across use cases and data modalities- see some pointers under questions below. 

questions: 1. How would the type of learning algorithm influence homogenization? For e.g., studies have shown that supervised algorithms reinforce human bias especially in the context of scientific discoveries. 

2. Related to the above question, would the extent of homogenization therefore be a function of the particular use case? 

3.  Would the results be similar to tabular data?

4. How will dataset sparsity and measurement errors influence outcome homogenization?

5. Which of the factors mentioned in Section 2 are the most dominant factors in causing outcome homogenization - is there any intuitive or empirical evidence in support of this?
limitations:  see above
review_title: null

Meta-review: metareview: This paper has the potential to catalyse a new and important line of research within algorithmic fairness. It does that (in a simple, yet interesting way) by identifying that the key to the concerns about algorithmic monoculture lies in homogeneous outcomes and by defining intuitively justified metrics of homogeneous outcomes at individual and group-levels. It then characterises the potential outcome homogenisation in models trained using the same/similar datasets and in models fine-tuned using the same underlying foundational models (a trend that is increasing, of late). To be fair, the reviewers rightly criticised for a somewhat limited set of experiments and the conclusions drawn from them do appear rather preliminary. However, this can be partly justified as the paper is exploring a previously unexplored territory and the potential benefits of accepting the paper and the follow-on work that it will very likely trigger far outweigh the potential risks of not accepting the paper (after all, the experiments are rigorous, even if limited / preliminary and the authors draw appropriate conclusions). 
confidence: Certain
Decision: Accept