TASK DEFINITION: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
PROBLEM: - Strengths:  - this article puts two fields together: text readability for humans and machine comprehension of texts  - Weaknesses:  - The goal of your paper is not entirely clear. I had to read the paper 4 times and I still do not understand what you are talking about! - The article is highly ambiguous what it talks about - machine comprehension or text readability for humans - you miss important work in the readability field - Section 2.2. has completely unrelated discussion of theoretical topics. - I have the feeling that this paper is trying to answer too many questions in the same time, by this making itself quite weak. Questions such as “does text readability have impact on RC datasets” should be analyzed separately from all these prerequisite skills.  - General Discussion:  - The title is a bit ambiguous, it would be good to clarify that you are referring to machine comprehension of text, and not human reading comprehension, because “reading comprehension” and “readability” usually mean that. - You say that your “dataset analysis suggested that the readability of RC datasets does not directly affect the question difficulty”, but this depends on the method/features used for answer detection, e.g. if you use POS/dependency parse features. - You need to proofread the English of your paper, there are some important omissions, like “the question is easy to solve simply look..” on page 1. - How do you annotate datasets with “metrics”?? - Here you are mixing machine reading comprehension of texts and human reading comprehension of texts, which, although somewhat similar, are also quite different, and also large areas. - “readability of text” is not “difficulty of reading contents”. Check this: DuBay, W.H. 2004. The Principles of Readability. Costa Mesa, CA: Impact information.  - it would be good if you put more pointers distinguishing your work from readability of questions for humans, because this article is highly ambiguous. E.g. on page 1 “These two examples show that the readability of the text does not necessarily correlate with the difficulty of the questions” you should add “for machine comprehension” - Section 3.1. - Again: are you referring to such skills for humans or for machines? If for machines, why are you citing papers for humans, and how sure are you they are referring to machines too? - How many questions the annotators had to annotate? Were the annotators clear they annotate the questions keeping in mind machines and not people?

SOLUTION: en

PROBLEM: The paper describes a method to evaluate generative models such as VAE, GAN and GMMN. This is very much needed in our community where we still eyeball generated images to judge the quality of a model. However, the technical increment over the NIPS 16 paper: “Measuring the reliability of MCMC inference with bidirectional Monte Carlo” is very small, or nonexistent (but please correct me if I am wrong!).  (Grosse et al). The relative contribution of this paper is the application of this method to generative models.  In section 2.3 the authors seem to make a mistake. They write E[p’(x)] <= p(x) but I think they mean: E[log p’(x)] <= log E[p’(x)] = log p(x). Also,  for what value of x? If p(x) is normalized it can’t be true for all values of x. Anyways, I think there are typos here and there and the equations could be more precise. On page 5 top of the page it is said that the AIS procedure can be initialized with q(z|x) instead of p(z). However, it is unclear what value of x is then picked? Is it perhaps Ep(x)[q(z|x)] ? I am confused with the use of the term overfitting (p8 bottom). Does a model A overfit relative to a another model B if the test accuracy of A is higher than that of B even though the gap between train and test accuracy is also higher for B than for A. I think not. Perhaps the last sentence on page 8 should say that VAE-50 underfits less than GMMN-50? The experimental results are interesting in that it exposes the fact that GANs and GMMNs seem to have much lover test accuracy than VAE despite the fact that their samples look great.

SOLUTION: en

PROBLEM: Taking into account the loss in the binarization step through a proximal Newton algorithm is a nice idea. This is at least one approach to bringing in the missing loss in the binarization step, which has recently gone from a two step process of train and binarize to a single step simultaneous train/compress. Performance on a few small tasks show the benefit. It would be nice to see some results on substantial networks and tasks which really need compression on embedded systems (a point made in the introduction). Is it necessary to discuss exploding/vanishing gradients when the RNN experiments are carried out by an LSTM, and handled by the cell error carousel? We see the desire to tie into proposition 2, but not clear that the degradation we see in the binary connect is related. Adam is used in the LSTM optimization, was gradient clipping really needed, or is the degradation of binary connect simply related to capacity? For proposition 3.1, theorem 3.1 and proposition 3.2 put the pointers to proofs in appendix.

SOLUTION:
en