Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies two problems related to using neural networks as single winner voting rules. (1) Training networks to mimic well-known existing rules, and (2) training networks to act as novel rules that select utility-maximizing alternatives.

The major novel contribution of the paper is in their choice of network architecture. "Permutation-invariant networks" are used, which are agnostic to the order of input data, thus ensuring that the resultant voting rules are anonymous by default.

Experiments show that the 3 PIN architectures considered are able to learn the studied voting rules with high accuracy after a large amount of training. These results are shown for testing done on each of real and synthetic data. Similarly, when the networks are trained with an oracle to learn the welfare maximizing alternative they perform quite well.
main_review: Originality:
Only a handful of papers have studied the problem of using neural networks as voting rules and have typically focused on relatively basic MLP models. The novel contribution this paper makes is to demonstrate the importance of considering more complex models. This is also (to my knowledge) the first paper to learn voting rules to optimize social welfare.

Quality:
The minor theoretical contribution of the paper seems sound.
The experimental results make a compelling case for the benefit of their contributions. Some experiments could be more thorough or would benefit from more discussion.

Clarity:
The paper has only a few minor typos and is clearly written. Most relevant details are included, though I would be surprised if they were enough to replicate the paper (although the included source code would presumably help with that and has a readme file that appears to have good instructions for running the code)

Significance:
The focus on more complex network architecture is very likely to be built upon by other researchers and appears to be an advance in the state of the art. Demonstrating the learnability of social welfare maximizing candidates, while not a particularly surprising result to me, is novel and likely to be replicated by other researchers in the future.

Minor notes for clarification or consideration:
- explanation of Simpson's rule in 2.2 could be phrased much more clearly
- 2.3 seems to have limited relevance to the remainder of the paper. Consider making the link more clear or cutting it to expand other sections
- My preference would be to see the input-output representation very briefly discussed in the text (footnote 4), it has been an important focus of other papers in this area and would be of interest to some readers
- What is $\alpha$ in section 4? Is there a reason for not giving the exact value? Is it chosen randomly? (If so, how?)
- I can't find details on the distribution of numbers of voters and candidates within a training set. I believe a single set contains examples with varying numbers of voters (distributed how?), I'm not sure if the number of candidates varies. If so, that would be worth emphasizing as other work relies on a fixed number of candidates. 
- Is there a reason for training with 2 candidates? All rules will essentially reduce to plurality at that point.
- Does training with varying numbers of voters help generalization to larger voter populations? ie. Would Table 2 have the same out-of-domain results if the networks were trained on elections with a constant number of voters? The answer might give insight as to whether the training procedure or architecture is responsible for this generalization out of domain.
- footnote 8: Is there a motivating example for why solving batches of elections is important? That doesn't seem obvious
- Table 4 is not referenced in the text. Consider doing that
- Table 4 would be much more compelling if it considered how often all the rules being mimicked found the same winner. Are the networks learning the same thing when mimicking each rule or something different each time?
- there are a number of very relevant papers that have used neural networks to act as voting rules that you should strongly consider referencing, the mains ones are listed below


Typos:
- end of Sec 1: "in detail the proposed the training procedure"
- 3.2.1 line 3: "in the case of [a] fully-connected graph structure[s]" (should add one of [a] or [s])
- Table 3 caption: "and [are] mostly on par with,"
- 4.2, 2nd last paragraph: "The neural network models [are] better at"


References to add:

Xia, Lirong. "Designing social choice mechanisms using machine learning." Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems. 2013.
	
- outlined the general approach most more recent papers have taken to use neural nets as voting rules

Armstrong, Ben, and Kate Larson. "Machine Learning to Strengthen Democracy." NeurIPS Joint Workshop on AI for Social Good. 2019.
	
- describes an approach to generate novel voting rules based on desirable axioms with neural networks

Burka, Dávid, et al. Voting: A machine learning approach. No. 145. KIT Working Paper Series in Economics, 2020.
	
- similar to (Kujawska et al. 2020), describes an approach to mimicking voting rules with neural MLPs

Firebanks-Quevedo, Daniel. "Machine Learning? In MY Election? It's More Likely Than You Think: Voting Rules via Neural Networks." 2020.
	
- similar to (Armstrong and Larson, 2019), generates voting rules automatically based on axioms and other desirable conditions
limitations_and_societal_impact: Yes, the authors very briefly point out the major limitation of this work: there are no clear worst-case performance guarantees so relying on this in practice may be dangerous.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors study the problem of using machine learning techniques to 'automatically' learn voting mechanisms. Specifically, the authors use three permutation-invariant neural network architectures: DeepSet, Set Transformer and Graph Isomorphism Network and show that they can learn voting rules as well as maximize certain social welfare functions.

The first contribution is to show that these networks are universal function approximators of permutation invariant functions. Voting protocols typically fall within this class, as a permutation of votes should not lead to different outcomes: in other words, the identities of the voters do not matter.

The second contribution is to develop a learning procedure and evaluate the learned voting rules produced by these three algorithms (as well as a standard multi-layer perceptron, for reference). The way the training worked is to consider a population of voters and candidates such that the utility each voter has for the candidates is sampled from a distribution. Specifically, if there are $n$ voters and $k$ candidates, the utility of each candidate for voter $i$ is sampled by a Dirichlet distributions with all $k$ parameters set to $\alpha_0$. Different values of $\alpha_0$ correspond to different voters: some may prefer a single candidate or many of them equally. These values are then converted into an ordering, which the voters submit to the learned mechanism. The mechanisms' output is compared with the voting rule we are trying to learn (or the option which maximizes social welfare) a loss is incurred and a gradient descent step follows. On synthetic data, the learned mechanisms perform very well for all voting protocols tested, which are: Plurality, Borda, Copeland, Maximin and Kemeny.

Finally, the same network architectures perform even better on real data sets, such as the Netflix Prize Data.
main_review: The article is very well written, with a clear exposition of the techniques and related work. It is easy to follow and honest: the results claimed in the abstract and introduction are indeed the ones developed later and their analysis is thorough (except possibly for one point).

The main criticism I have (and the reason I cannot fully recommend the paper) is that I am not totally sure of the following two: 1) why would it be difficult to learn voting rules (or maximize welfare) using off-the-shelf ML techniques? 2) Why would anyone try to re-learn any existing voting rule? 

I understand that the answer to the second is that perhaps the network will be able to learn some new voting rules which are not described yet. But how would this scenario appear? We would need some designer of a voting protocol with the capacity to generate hundreds of thousands of votes and outcomes, but without the capacity to formally define their relation. Moreover, the learned voting protocol would be quite difficult to motivate in a real election, since it's internals might be hidden behind the network weights.

Finally, why are the experiments not run for more than 199 voters? I expected the real data sets to run in the hundreds of thousands, but I may have missed something.


limitations_and_societal_impact: The authors adequately explained the limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors study how to learn voting rules, both in terms of mimicking existing voting rules and learning new voting rules to (in particular) maximize different versions of social welfare (e.g., egalitarian or utilitarian social welfare). They show that a particular class of neural network architectures (PINs) are well-suited to this task and evaluate their proposed approach experimentally.

main_review: The authors mainly investigate how to mimic different voting rules. Although most of them are relatively straightforward to compute, they do achieve good results mimicking the Kemeny rule, which is a good sign for PINs. 

Their theoretical result also remains a bit weak. If I understand it correctly, it states that different permutations are mapped to different embeddings, and doesn't say anything about the feasibility of learning these embeddings.

Overall, I appreciate the fact that the authors used more datasets and included results about the Kemeny rule, which is considerably more complex than other rules they mimic. I still do not know if I find their results incredibly compelling -- it seems like the bulk of the paper is about mimicking voting rules, and only section 4.2 talks about learning rules in order to maximize social welfare, which seems like a much more interesting objective overall.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper empirically studies the ability of PIN (Set Transformer, GIN, DeepSets and MLP) applied to predicting the (top-1) winner of voting rules (Plurality, Borda, Copeland, Maximin and Kemeny) and predicting the social-welfare-maximizing (utilitarian and egalitarian) candidates. The PINs are trained on synthetic data, agents whose utility over candidates are generated from Dirichlet distributions. The PINs are tested on synthetic data (Table 1 & 3), synthetic data with more voters than in training time (Table 2) and real world data (Table 4). 
main_review: Pros:
 - The papers is well written and well motivated. The question of "why would anyone want to predict the outcome of voting rule" is addressed out-front on line 38, and the question of "why not compare the full rankings" is acknowledged in Section 6.
 - The result is strong and surprising (that training on synthetic data generalizes to real-world data). 
 - The code is well organized and easy to follow, although I did not attempt to replicate the results. 

Cons / questions:
 - Since the voting rules are traditionally assumed to output a ranking (possibly admitted by some scores), I wonder how the PINs do when evaluated in terms of rankings. Strictly speaking, one can use the binary-predicting PINs for n-1 rounds to predict a full ranking, no? 
 - On the surprising result that training on synthetic data generalizes to real-world data: does this say anything about the real-world data, that it may follow some Dirichlet distributions?

===============

I acknowledge and appreciate the author response.
limitations_and_societal_impact: The authors have adequately addressed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The Authors report a set of experiments with using neural networks with a particular architecture to learn optimal voting rules. They provide a concise theoretical description of the problem and prove that the chosen type of NNs are universal approximators of voting rules. They demonstrate experimentally that NNs can be trained using voter's ranked preferences only.

The experiments are of two kinds:

1. teaching the NN known classical voting rules

2. learning a new rule to maximise a given social welfare function

Their results show that NNs can learn known voting rules very well and that they can discover new voting rules which select optimal candidates more often than classical voting rules.

--- EDIT ---

Increased the score to 7 after the clarifications.
main_review: The Authors tackle an interesting problem and report encouraging results. The paper is clearly written and the theoretical results are explained well. I liked the fact that the Authors made the effort to provide an introduction to the problem domain for lay readers (like me). However, there are some problems which prevented me from giving a higher score:

1. the notation is not always clear:

- is vec{u} a vector of one voters' utilities per candidate, or a vector of utility functions of all voters? Algorithm 1 suggests the 2nd, but the caption under Figure 1 suggests the 1st. This reviewer is a bit confused.

- what exactly is the prediction loss function l optimised by Algorithm 1?

2. the real-world datasets are not described sufficiently even in the Supplementary Material. Since the synthetic datasets used in the paper contain only uniform distributions of preferences (every candidate is equally likely to win on average), it would be interesting to know if the real-world datasets deviate from that. For example, is there a sushi type which is much more popular than all others? Many real-world situations such as presidential elections contain two or three dominant politicians and a string of niche candidates with little hope of winning.

3. the discussion of the limitations of the method is lacking:

- firstly, usually we deal with more than 99 voters. The Authors do not comment at all on the question how well would their method scale to realistic numbers of voters (i.e. up to millions). 

- secondly, since we do not know the actual utility functions of the voters, we probably need voting rules which are robust to perturbations. The Authors did not seem to have tested whether a voting rule learned on one type of utility functions behaves "decently" on other utility functions (e.g. train on "uniform", test on "polarized"). In other words, the network may be overfitting to modelling assumptions.

- thirdly, the Authors do not comment on the question of interpretability of learned voting rules - an important problem if we are to implement such a rule in public life. It would also be fascinating to understand how the learned rules work and how they differ from the classical ones. E.g. how often does a learned rule indicate a different winner than the classical ones?

- finally: as far as I understand, the presented NN architectures do not fulfil the neutrality criterion (as discussed in Appendix C.1 - I think should be moved to the main text). How do we know that, for datasets which have a dominant candidate, the network doesn't simply memorise which candidate should win? Couldn't this be tested by randomly permuting the candidates in the test set? (maybe the Authors have done that? i'm not sure)

4. Figure 1 is hard to understand for me. It seems that the histograms are heavily concentrated in 1, with only the tails differing. Why not also provide the maximised value and compare it with the theoretical maximum? It would be much easier to understand than the histogram. (Especially that the meaning of \vec{u} is not clear.) The same comment applies to similar Figures in Suppl. Mat.

5. Minor: At least a sketch of the proof of Theorem 1 should be present in the main text.

Some of the above comments are probably impossible to address given the page limits, so the Authors should treat them as my wishlist for their further work.
limitations_and_societal_impact: The Authors comment on the societal impact, but the topic needs to be explored further, especially if the learned rules were to be used in a public setting (e.g. political elections).

For limitations, see pt. 3 above.
review_title: null

Meta-review: Majority of the reviewers are excited about the novel applications of ML techniques to social choice and were impressed by the good performance of the solution proposed in this paper. The paper is well-written and the work is solid. Some concerns were raised about the significance and motivation of the actual technical problem solved in this paper (i.e., learning social welfare maximizers) and technical depth in ML.

The response was generally effective and clarified some points raised by the reviewers. After the discussions, reviewers' opinions did not change much (except the score of one reviewer was raised from 6 to 7). The overall sentiment remained positive.

After all, the novelty and potential to stimulate future work and discussions outweigh the cons, which is the main reason behind the recommendation.
Decision: Accept (Poster)