In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
--------
Question: - Strengths: This paper proposes an evaluation metric for automatically evaluating the quality of dialogue responses in non-task-oriented dialogue. The metric operates on continuous vector space representations obtained by using RNNs and it comprises two components: one that compares the context and the given response and the other that compares a reference response and the given response. The comparisons are conducted by means of dot product after projecting the response into corresponding context and reference response spaces. These projection matrices are learned by minimizing the squared error between the model predictions and human annotations.  I think this work gives a remarkable step forward towards the evaluation of non-task-oriented dialogue systems. Different from previous works in this area, where pure semantic similarity was pursued, the authors are going beyond pure semantic similarity in a very elegant manner by learning projection matrices that transform the response vector into both context and reference space representations. I am very curious on how your projection matrices M and N differ from the original identity initialization after training the models. I think the paper will be more valuable if further discussion on this is introduced, rather than focusing so much on resulting correlations.   - Weaknesses:  The paper also leaves lots questions related to the implementation. For instance, it is not clear whether the human scores used to train and evaluate the system were single AMT annotations or the resulting average of few annotations. Also, it is not clear how the dataset was split into train/dev/test and whether n-fold cross validation was conducted or not. Also, it would be nice to better explain why in table 2 correlation for ADEM related scores are presented for the validation and test sets, while for the other scores they are presented for the full dataset and test set. The section on pre-training with VHRED is also very clumsy and confusing, probably it is better to give less technical details but a better high level explanation of the pre-training strategy and its advantages.  - General Discussion:  “There are many obvious cases where these metrics fail, as they are often incapable of considering the semantic similarity between responses (see Figure 1).” Be careful with statements like this one. This is not a problem of semantic similarity! Opposite to it, the problem is that completely different semantic cues might constitute pragmatically valid responses. Then, semantic similarity itself is not enough to evaluate a dialogue system response. Dialogue system response evaluation must go beyond semantics (This is actually what your M and N matrices are helping to do!!!)   “an accurate model that can evaluate dialogue response quality automatically — what could be considered an automatic Turing test —“ The original intention of Turing test was to be a proxy to identify/define intelligent behaviour. It actually proposes a test on intelligence based on an “intelligent” machine capability to imitate human behaviour in such a way that it would be difficult for a common human to distinguish between such a machine responses and actual human responses. It is of course related to dialogue system performance, but I think it is not correct to say that automatically evaluating dialogue response quality is an automatic Turing test. Actually, the title itself “Towards an Automatic Turing Test” is somehow misleading!  “the simplifying assumption that a ‘good’ chatbot is one whose responses are scored highly on appropriateness by human evaluators.” This is certainly the correct angle to introduce the problem of non-task-oriented dialogue systems, rather than “Turing Test”. Regarding this, there has been related work you might like to take a look at, as well as to make reference to, in the WOCHAT workshop series (see the shared task description and corresponding annotation guidelines).  In the discussion session: “and has has been used” -> “and it has been used”

Answer: en


Question: This paper presents a system approach to combine multiple modalities to perform classification in a practical scenario (e-commerce).  In general, I find the proposed approach in the paper sound and solid, but do not see novelty in the paper: feature fusion and decision time fusion are both standard practices in multi-modal analysis, and the rest of the paper offers no surprise in implementing such approaches. This seems to be a better fit for venues that focus more on production systems, and seems to be a bad fit for ICLR where the focus is more on research of novel algorithms and theories.

Answer: en


Question: This paper investigates three simple weight-pruning techniques for NMT, and shows that pruning weights based on magnitude works best, and that retraining after pruning can recover original performance, even with fairly severe pruning.  The main strength of paper is that the technique is very straightforward and the results are good. Itâs also clearly written and does a nice job covering previous work.  A weakness is that the work isnât very novel, being just an application of a known technique to a new kind of neural net and application (namely NMT), with results that arenât very surprising.   Itâs not clear to me what practical significance these results have, since to take advantage of them you would need sparse matrix representations, which are trickier to get working fast on a GPU - and after all, speed is the main problem with NMT, not space. (There may be new work that changes this picture, since the field is evolving fast, but if so you need to describe it, and generally do a better job explaining why we should care about pruning.)  A suggestion for dealing with the above weakness would be to use the pruning results to inform architecture changes. For instance, figure 3 suggests that you might be able to reduce the number of hidden layers to two, and also potentially reduce the dimension of source and target embeddings.  Another suggestion is that you try to make a link between pruning+retraining and dropout (eg âA Theoretically Grounded Application of Dropout in Recurrent Neural Networksâ, Gal, arXiv 2016).  Detailed comments:  Line 111: âsoftmax weightsâ - âoutput embeddingsâ may be a preferable term  S3.2: Itâs misleading to call n the âdimensionâ of the network, and specify all parameter sizes as integer multiples of this number as if this were a logical constraint.  Line 319: You should cite Bahdanau et al here for the attention idea, rather than Luong for their use of it.  S3.3: Class-uniform and class-distribution seem very similar (and naturally get very similar results); consider dropping one or the other.  Figure 3 suggestion that you could hybridize pruning: use class-blind for most classes, but class-uniform for the embeddings.  Figure 4 should show perplexity too.  What pruning is used in section 4.2 & figure 6?  Figure 7: does loss pertain to training or test corpora?  Figure 8: This seems to be missing softmax weights. I found this diagram somewhat hard to interpret; it might be better to give relevant statistics, such as the proportion of each class that is removed by class-blind pruning at various levels.  Line 762: You might want to cite Le et al, âA Simple Way to Initialize Recurrent Networks of Rectified Linear Unitsâ, arXiv 2015.

Answer:
en