Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper argues that in the wisdom-of-the-crowd paradigm, plurality voting may not necessarily yield the correct answer when the majority makes systematic errors. The paper presents a theoretical framework to elicit thinking hierarchy and demonstrates that their method outperforms plurality voting and also demonstrates certain desirable properties. Apart from presenting the theoretical framework, the paper also conducts crowdsourced user studies to demonstrate the practical effectiveness of their framework. 
strengths_and_weaknesses: Strengths:
1. The paper's motivation is clear and important. If a vast majority displays certain biases, plurality voting would not be very useful. 
2. The paper's writing and organization are good. The examples provided help understanding. 

Weaknesses: 
1. The paper can be strengthened by making the user study more elaborate with a clear description of how are the questions selected. 

questions: 1. How are the questions for the user study selected? 
limitations: No negative societal impact is discussed. I do not see any obvious red flags in terms of negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The crux of this paper is that it provides an empirically validated way to use the wisdom of the crowd rather than the default “popular answer”/”plurality voting” paradigm. In the process, the authors show how to obtain the thinking hierarchy of the crowd for the given set of questions. They claim that knowing this (rich) hierarchy helps in areas like policy making. Using mathematical tools and certain assumptions, the paper demonstrates the superiority of their method especially when obtaining a prior distribution from the crowd is prohibitively expensive.

strengths_and_weaknesses: Originality: The authors distinguish themselves from previous work by discarding the use of a prior distribution. Further, they claim that their thinking hierarchy learning model captures a richer spectrum of answers. Their method reduces bias like previous work does, with the difference that they do not collect a prediction distribution, rather a single prediction answer. They also differentiate themselves from the game theoretic setting. The concept of the thinking hierarchy is sufficiently novel wrt past work.

Quality: The paper provides mathematical justification, although under certain idealistic assumptions, for every statement it makes. The experiment results have been provided at a URL and are easy to grasp. The authors are upfront about the assumptions that are required and also provide alternatives - for e.g. the i.i.d assumption under which respondents make their predictions is contrasted with picking the first prediction that a respondent makes, due to the fact that i.i.d predictions don’t match with reality.

Clarity: The paper is clear with what it aims to achieve and under what conditions it can achieve them. The goal is to: learn the thinking hierarchy among respondents and to do so without collecting prior information. The authors also provide future uses of their work from the ML and scale points of view. The experiments are easy enough for a novice reader to understand, and collect sufficient information.

Significance: The method described in the paper has significance in that it is applicable to cases where collecting prior information from a crowd is difficult, and plurality voting is not sufficient. One can foresee future applications of systematically obtaining a thought hierarchy as more information on a certain topic or policy can only help with decision making.

questions: See above.
limitations: The authors discuss technical limitations such as the i.i.d assumption on user predictions. Negative impacts on society are not particularly discussed, but one could think of situations such as political mercenaries or governments collecting information on thinking patterns of a voting population in pursuit of an agenda.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a mathematical model for thinking hierarchy of users predictions for their answer and other users’ answers. Given the joint distribution of the answers and predictions of other answers, the authors show that the parameters of the model can be derived using a novel matrix factorization. The authors solve an equivalent Frobenius norm minimization problem for the special case when the $W$ matrix (one of the parameters of the model) is semi-orthogonal. And then the authors propose a brute-force search based algorithm to find the ranking of the answers from the thinking hierarchy. Since the joint distribution of answers and predictions may not be available they use an empirical estimate. Naturally, this works if the samples are i.i.d. 
strengths_and_weaknesses: Strengths:

1. The problem of eliciting thinking hierarchy to come up with the correct ranking over various answers is well-motivated, and is an important problem to study.
2. The paper seems to be placed well in the recent literature about using additional information about people’s prediction about other answers to get better accuracy for ranking answers.
3. The theoretical results about the problem are interesting, however somewhat limited.

Weaknesses:

1. The main drawback is that the algorithms proposed in the paper are just brute-force search based algorithms and hence are not efficient for complicated questions.
2. The theoretical guarantees about eliciting thinking hierarchies hold in fairly restricted settings, which may not hold majority of the times in real-world. Especially, the questions and answers very quickly become complicated, in which case, plurality-voting could be a better method? 
3. Basically, for a complete picture, the complexity of the questions and answers needs to be captured in this framework. In what cases does plurality-voting give better ranking than the algorithms in the proposed framework. I don’t understand when is the side information better?
4. The writing can be improved (see questions).
questions: 1. Lines 155 to 161 are confusingly written. Are the people predicting other peoples’ answers or their types?
2. Please write Example 2.3 properly. Since the meaning of $p_t$ and $p_{t \rightarrow t’}$ has already been explained it would be much easier to understand if the authors just use this notation while showing the construction of $\Lambda$.
3. Could the authors comment about what happens if the side information about other people’s predictions is noisy?
limitations: I don’t see any mention of the negative impacts of when the proposed algorithms are bad compared to standard algorithms. This discussion is needed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to aggregate crowdsourcing answers based on a key observation: experts have different expertise levels and experts at a higher level are able to “simulate” the experts at lower levels. The paper aims to find this underlying “thinking hierarchy” —which can be used to find the best answer—by asking people to predict other people’s answers. The paper first proposes a model for this thinking hierarchy. The model assumes that experts have their types, and for each type, there is a thinking oracle that specifies how this type of experts generate their answers. An expert can run the oracles with lower types but never higher types. The paper then develops two algorithms to learn the underlying thinking hierarchy. The algorithms basically generate an answer-prediction matrix based on experts’ answers and predictions and then find a ranking of answers that will reorder the matrix to match the hierarchical structure in the best way. The paper provides theoretical justification for their algorithms. Finally, the paper shows by real-world experiments that their methods outperform plurality voting.

strengths_and_weaknesses: The strengths of the paper are the algorithms and the experiments. The paper proposes novel methods to utilize the underlying thinking hierarchy when aggregating experts’ answers and tests the method through real-world experiments. The algorithms are intuitively reasonable and the experiment results are good. 

The weakness of the paper is the modeling of the thinking hierarchy, which also makes the theoretical justification of the algorithms implausible. The use of thinking oracle seems unreasonable to me. Are these thinking oracles private information or public information? If they are private information, how can the experts of higher types use the lower-type oracles? If they are public information, why can’t the lower-type experts use the higher-type oracles? It is also ungrounded why an expert generates predictions by running thinking oracles of others. The assumption seems crucial for the theoretical analysis, which in my point of view should be more carefully justified. 

questions: 1. Why not just assume there is an underlying ranking of answers and people with higher-type answers can predict the lower-type answers?

2. How do your algorithms compute the best permutation matrix/the matrix W*? Are they polynomial-time algorithms?

limitations: Yes.
review_title: null

Meta-review: metareview: This work proposes the framework to elicit people's "thinking hierarchy" that helps improve the wisdom of the crowd even if the majority is wrong. The reviewers overall appreciate the main idea of the work and believe it makes a nice contribution to the literature.  There have been some questions/concerns raised about the efficiency of the algorithm and the model limitations, to which the authors have provided reasonable responses.  We encourage the authors to incorporate those responses (and other reviewer comments) into the final version of the paper. 
confidence: Less certain
Decision: Accept