TASK DEFINITION: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
PROBLEM: This paper presents several weakly supervised methods for developing NERs. The methods rely on some form of projection from English into another language. The overall approach is not new and the individual methods proposed are improvements of existing methods. For an ACL paper I would have expected more novel approaches.  One of the contributions of the paper is the data selection scheme. The formula used to calculate the quality score is quite straightforward and this is not a bad thing. However, it is unclear how the thresholds were calculated for Table 2. The paper says only that different thresholds were tried. Was this done on a development set? There is no mention of this in the paper. The evaluation results show clearly that data selection is very important, but one may not know how to tune the parameters for a new data set or a new language pair.   Another contribution of the paper is the combination of the outputs of the two systems developed in the paper. I tried hard to understand how it works, but the description provided is not clear.   The paper presents a number of variants for each of the methods proposed. Does it make sense to combine more than two weakly supervised systems? Did the authors try anything in this direction.  It would be good to know a bit more about the types of texts that are in the "in-house" dataset.

SOLUTION: en

PROBLEM: This paper presents an intriguing study of how one can pose architecture search as a meta learning problem. By collecting features from networks trained on various datasets and training a “ranking classifier” (the actual details of the classifier do not seem to be described in detail) one can potentially infer what a good architecture for a new problem could be by simply running the ranker on the extracted features for a new problem setup.  One notable comment from the paper is that the authors fix some important hyper-parameters for all the networks. I am of the opinion that optimizing the learning rate (and its decay schedule) is actually quite important. I hypothesize that a lot of the conclusions of this paper may change quite a bit if the authors did an actual search over the rates instead. I suspect that instead of training 11k nets, one can train 2k nets with 5 learning rates each and get a much better result that is actually compelling.  I am not convinced that the protocol for generating the various architectures is doing a good job at creating a diversity of architecture (simply because of the max depth of 8 layers and 14 components overall). I suspect that most of these generated architectures are actually almost identical performance-wise and that it’s a waste to train so many of them on so many tasks. Unless the authors are already doing this, they should define a pruning mechanism that filters out nets that are too similar to already existing ones.  The batch normalization experiments in Table 2 seem odd and under-explained. It is also well-known that the optimal learning rates when using batch norm vs. not using batch norm can differ by an order of magnitude so given the fixed learning rate throughout all experiments, I take these results with some grain of salt.  I am not sure we got many insights into the kinds of architectures that ended up being at the top. Either visualizations, or trends (or both), would be great.  This work seems to conflate the study of parallel vs. serial architectures with the study of meta learning, which are somewhat distinct issues. I take issue with the table that compares parallel vs. serial performance (table 2) simply because the right way would be to filter the architectures by the same number of parameters / capacity.  Ultimately the conclusion seems to be that when applying deep nets in a new domain, it is difficult to come up with a good architecture in advance. In that sense, it is hard to see the paper as a constructive result, because it’s conclusions are that while the ranker may do a good job often-times, it’s not that reliable. Thus I am not convinced that this particular result will be of practical use to folks who are intending to use deep nets for a new domain.

SOLUTION: en

PROBLEM: This paper is technically sound. It highlights well the strengths and weaknesses of the proposed simplified model.  In terms of impact, its novelty is limited, in the sense that the authors did seemingly the right thing and obtained the expected outcomes. The idea of modeling deep learning computation is not in itself particularly novel. As a companion paper to an open source release of the model, it would meet my bar of acceptance in the same vein as a paper describing a novel dataset, which might not provide groundbreaking insights, yet be generally useful to the community.  In the absence of released code, even if the authors promise to release it soon, I am more ambivalent, since that's where all the value lies. It would also be a different story if the authors had been able to use this framework to make novel architectural decisions that improved training scalability in some way, and incorporated such new insights in the paper.  UPDATED: code is now available. Revised review accordingly.

SOLUTION:
en