reviews: 
Comments: The paper investigates the question of value estimation of information sets in games from a learning perspective, formulating the question as essentially one of supervised learning with noise and hidden information. Through extensive experiments, the paper evaluates the effect of sampling multiple times from the hidden information.
Strengths: The paper addresses a problem of importance to the game theory community, namely, value estimation of information sets in games. The paper is also clearly written and easy to understand.
Weaknesses: The paper's main problem seems to essentially reduce to supervised learning: namely, given a dataset $(x_i, y_i)\_{i=1}^n$, estimate a function $g(x) := \mathbb E_{y|x} y$. From this perspective, as I will discuss below, the results in the paper are, in my opinion, not very illuminating given what is already well understood about supervised learning. Perhaps most critically, the paper does not seem to make any use whatsoever of the hidden information $h$ in any of the proposed techniques, preferring instead to simply sample from the conditional distribution $y|x$ and use those samples to compute some summary statistic (majority vote, or average value) to feed to the learner instead.

This paper is framed in such a way that the main application is supposed to be in games. But this bothers me: in games, the assumption of the distribution $h|x$ being known is very false, since in practice the opponent has control over incomplete information (e.g., in poker, by choosing its range in any given situation). I would argue that this lack of knowledge of the distribution---not any issue of statistical estimation---is the fundamental barrier to good value estimation in games. 

I am also confused about the setup for the MNIST experiment. From what I am understanding, for each training instance $x_i$, you draw $k$ samples $\hat{y}\_i^{(\kappa)}$ ($\kappa=  1, \dots, k$) and then pass the majority vote of these samples, $\hat y_i$, to the learner as the training instance. But this would not optimize the above objective function: indeed, for example, if $p = 0.9$ (say) and $k$ is very large, one would expect $\hat y_i = y_i$ always for every $i$ (simply by LLN) and therefore the classifier would learn to always output the true class. But that would be wrong: when $p = 0.9$, the expected value $g(x) := \mathbb E_{(h, y)|x} y$ is not the true class, due to the high noise. Measuring the resulting classifiers according to their test accuracy also seems wrong, for the same reason (it's not the objective stated by the paper). I think in the MNIST case, like in the game cases, the aggregator should be the average $\mathbb E_\kappa \hat{y}\_i^{(\kappa)}$ (where the $\hat{y}\_i^{(\kappa)}$ are to be interpreted as one-hot vectors), not the majority vote.

It seems also pretty clear to me why the results in Figure 4 are the way they are: with a fixed number of evaluations, having more samples per instance means having fewer unique instances, which could lead to overfitting; on the other hand, with a fixed number of updates, having more samples per instance only means less noise in the training data, which is a good thing.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good