Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: - Summary

This paper studies the sample elicitation problem where agents are asked to report samples. The goal is then to evaluate the quality of these reported samples by means of a scoring function S. Following previous related works, the authors use the equivalence between maximizing the expected proper score and minimizing some f-divergence. Their approach relies on the dual expression of the f-divergence which writes as a maximum over a set of functions t. Theoretical guarantees are given for f-scorings obtained (with or without ground truth samples) by first computing the empirical optimal function t, then plugged to estimate the f-divergence. Finally, a deep learning approach is proposed by considering functions f parameterized as sparse deep neural networks.

- Critics

The paper is globally well written but not well motivated and sometimes difficult to understand.
In particular, the notions of "elicitation", "reports" and "score function" should be defined mathematically more clearly.
Moreover, the deep learning aspect of the paper is not well motivated and is introduced in a very arbitrary way. Why not choosing another parametric family of functions? Is there another (broad) family of functions for which the computation of the argmin in Equation (4.3) is more tractable in practice?
A convincing way to motivate this deep learning approach would be to include numerical experiments and to compare to other parametric families.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a sample elicitation framework to tackle the problem of eliciting credible samples from agents for complex distributions. The authors suggest that deep neural frameworks can be applied in this framework for sample elicitation through the derivations. The authors also show the connection between the problem of sample elicitation and f-GAN. However, some problems in the proof on sample elicitation should be clarified or carefully explained.

- In (C.15) in Section C.7, why the upper bound is irrelevant to y or W_L^1 or W_L^2? Since we do not have W_L^1 = W_L^2, how to derive the inequality in (C.15)?

- In the proof of Lemma B.2 in Section B.5, why 1/(4L_0) and (1/\mu_0) are simply removed in the inequality below? 

- The inequalities in the proof of Theorem 3.5 in Section B.1 should be further explained. What is the meaning of “agent believes p = q” and how to apply this to the lower bound? What is the meaning of \bar{S} and how to get these inequalities for the upper bound?

Some minor comments
- The relation of x’ and x in the first paragraph on page 2 should be further clarified, and some formal definition should be given. The readers who are not familiar with this area would be confused about the problem before reading Section 2.1.

review_title: Official Blind Review #2

Meta-review: The primary contribution of this manuscript is a conceptual and theoretical solution to the sample elicitation problem, where agents are asked to report samples. The procedure is implemented using score functions to evaluate the quality of the samples.

The reviewers and AC agree that the problem studied is timely and interesting, as there is limited work on credible sample elicitation in the literature. However, the reviewers were unconvinced about the motivation of the work, and the clarity of the conceptual results. There is also a lack of empirical evaluation. IN the opinion of the AC, this manuscript, while interesting, can be improved by significant revision for clarity and context, and revisions should ideally include some empirical evaluation.
Decision: Reject