launch/ampere huggingface dataset:
This paper examines the effects of RL in an augmented action space which includes sequences of actions (e.g. meta actions) as well as the primitive actions defined by the MDP. The authors extend a GPU-based A3C implementation to include meta actions and show that their algorithm can achieve better sample complexity / and higher performance in most cases. The paper is well written, but fails to mention the relationship between meta-actions and the Options framework (Sutton et al 99). In particular, it seems that meta-actions can just be viewed as a set of predefined options given to the agent. Much prior work has studied how to combine options with Deep RL. To name a few: Multi-Level Discovery of Deep Options (Fox et al 17), Classifying Options for Deep RL (Arulkumaran 16), and Deep Exploration via Bootstrapped DQN (Osband et al). The former even learns the options rather than pre-defining them. This connection needs to be made explicit. I have concerns regarding the results in this paper: •	 Why on Qbert is the switching agent able to do so much better than both IU and DU? I suspect the curves may not be averaged over enough trials and results may be noisy, as it seems this shouldn’t be possible. Results curves should show the standard deviation or variance of the 3 runs. •	I am concerned this approach will not scale to games that have more actions than the 4 games explored. The concern is that A4C exponentially increases the size of the action space as a function of k. Cartpole as well as the explored Atari games all have relatively small action spaces, so I think it is critical to show that it scales to games with larger spaces as well. My concern is that in larger actions spaces, the gains A4C gets from meta-actions will be outweighed by the difficulty of having to learn with so many different actions. •	What is the value for k used in Atari experiments? Overall, I was very excited about this paper after reading the introduction. I really like the idea of allowing the network to decide on sequences of actions, and I think many games do have opportunity to identify and re-use combos of primitive actions (e.g. to stay between lanes in Beam Rider). However, I don’t think the architecture, algorithm, and results live up to this motivation. Simply augmenting the action space with all possible sequences of actions begs for a better solution. Pros:•	The authors show that in certain domains, exploration can be aided by pre-defined meta actions. •	The authors introduce an algorithm to squeeze more gradient updates out of a meta-action (DU-A4C). This is related to the insight that meta actions can be thought of as sequences of primitive actions. Cons:•	Relationship to Options is not identified. •	Results are only given for games with small action spaces. Unclear how the method scales to larger action spaces. •	Method for augmenting action space is not particularly interesting. •	Heuristic switching is somewhat undesirable. It would be nice to understand why DU stops working well and how to improve it.
This paper proposed to jointly train a multilingual skip-gram model and a cross-lingual sentence similarity model to construct sentence embeddings.  They used cross-lingual classification tasks for evaluation.  This idea is fairly simple but interesting.  Their results on some language pairs showed that the joint training is effective  (results on table 1 showed that sent-LSTM worked best with dim=128).  The downside of this paper is that their results could not outperform state-of-the-art results.  Some detailed comments: -	The authors should weaken some of the statements, e.g. ‘since our multilingual skip-gram and cross-lingual sentence similarity models are trained jointly, they can inform each other through the shared word embedding layer and promote the compositionality of learned word embeddings at training time’.  Actually, there are no experimental results and evidences in this paper supporting this statement. -	I don’t  see that ‘Amenable to Multi-task modeling’ is a contribution of this paper.  The authors should report additional experimental results to prove this statement.
Summary: The paper proposes a learnable skimming mechanism for RNN. The model decides whether to send the word to a larger heavy-weight RNN or a light-weight RNN. The heavy-weight and the light-weight RNN each controls a portion of the hidden state. The paper finds that with the proposed skimming method, they achieve a significant reduction in terms of FLOPS. Although it doesn’t contribute to much speedup on modern GPU hardware, there is a good speedup on CPU, and it is more power efficient. Contribution: - The paper proposes to use a small RNN to read unimportant text. Unlike (Yu et al., 2017), which skips the text, here the model decides between small and large RNN. Pros: - Models that dynamically decide the amount of computation make intuitive sense and are of general interests. - The paper presents solid experimentation on various text classification and question answering datasets. - The proposed method has shown reasonable reduction in FLOPS and CPU speedup with no significant accuracy degradation (increase in accuracy in some tasks). - The paper is well written, and the presentation is good. Cons: - Each model component is not novel. The authors propose to use Gumbel softmax, but does compare other gradient estimators. It would be good to use REINFORCE to do a fair comparison with (Yu et al., 2017 ) to see the benefit of using small RNN. - The authors report that training from scratch results in unstable skim rate, while Half pretrain seems to always work better than fully pretrained ones. This makes the success of training a bit adhoc, as one need to actively tune the number of pretraining steps. - Although there is difference from (Yu et al., 2017), the contribution of this paper is still incremental. Questions: - Although it is out of the scope for this paper to achieve GPU level speedup, I am curious to know some numbers on GPU speedup. - One recommended task would probably be text summarization, in which the attended text can contribute to the output of the summary. Conclusion: - Based on the comments above, I recommend Accept
The paper proposes a technique for training quantized neural networks, where the precision (number of bits) varies per layer and is learned in an end-to-end fashion. The idea is to add two terms to the loss, one representing quantization error, and the other representing the number of discrete values the quantization can support (or alternatively the number of bits used). Updates are made to the parameter representing the # of bits via the sign of its gradient. Experiments are conducted using a LeNet-inspired architecture on MNIST and CIFAR10. Overall, the idea is interesting, as providing an end-to-end trainable technique for distributing the precision across layers of a network would indeed be quite useful. I have a few concerns: First, I find the discussion around the training methodology insufficient. Inherently, the objective is discontinuous since # of bits is a discrete parameter. This is worked around by updating the parameter using the sign of its gradient. This is assuming the local linear approximation given by the derivative is accurate enough one integer away; this may or may not be true, but it's not clear and there is little discussion of whether this is reasonable to assume. It's also difficult for me to understand how this interacts with the other terms in the objective (quantization error and loss). We'd like the number of bits parameter to trade off between accuracy (at least in terms of quantization error, and ideally overall loss as well) and precision. But it's not at all clear that the gradient of either the loss or the quantization error w.r.t. the number of bits will in general suggest increasing the number of bit (thus requiring the bit regularization term). This will clearly not be the case when the continuous weights coincide with the quantized values for the current bit setting. More generally, the direction of the gradient will be highly dependent on the specific setting of the current weights. It's unclear to me how effectively accuracy and precision are balanced by this training strategy, and there isn't any discussion of this point either. I would be less concerned about the above points if I found the experiments compelling. Unfortunately, although I am quite sympathetic to the argument that state of the art results or architectures aren't necessary for a paper of this kind, the results on MNIST and CIFAR10 are so poor that they give me some concern about how the training was performed and whether the results are meaningful. Performance on MNIST in the 7-11% test error range is comparable to a simple linear logistic regression model; for a CNN that is extremely bad. Similarly, 40% error on CIFAR10 is worse than what some very simple fully connected models can achieve. Overall, while I like the and think the goal is good, I think the motivation and discussion for the training methodology is insufficient, and the empirical work is concerning. I can't recommend acceptance.
Monte-Carlo Tree Search is a reasonable and promising approach to hyperparameter optimization or algorithm configuration in search spaces that involve conditional structure. This paper must acknowledge more explicitly that it is not the first to take a graph-search approach.  The cited work related to SMAC and Hyperopt / TPE addresses this problem similarly.  The technique of separating a description language from the optimization algorithm is also used in both of these projects / lines of research.  The [mis-cited] paper titled “Making a science of model search …” is about using TPE to configure 1, 2, and 3 layer convnets for several datasets, including CIFAR-10.  SMAC and Hyperopt have been used to search large search spaces involving pre-processing and classification algorithms (e.g. auto-sklearn, autoweka, hyperopt-sklearn).  There have been near-annual workshops on AutoML and Bayesian optimization at NIPS and ICML (see e.g. automl.org). There is a benchmark suite of hyperparameter optimization problems that would be a better way to evaluate MCTS as a hyperparameter optimization algorithm: http://www.ml4aad.org/automl/hpolib/
(Score before author revision: 4)\n (Score after author revision: 7)\n\n I think the authors have taken both the feedback of reviewers as well as anonymous commenters thoroughly into account, running several ablations as well as reporting nice results on an entirely new dataset (MultiNLI) where they show how their multi level fusion mechanism improves a baseline significantly. I think this is nice since it shows how their mechanism helps on two different tasks (question answering and natural language inference).\n\n Therefore I would now support accepting this paper.\n\n ------------(Original review below) -----------------------\n\n The authors present an enhancement to the attention mechanism called \"multi-level fusion\" that they then incorporate into a reading comprehension system. It basically takes into account a richer context of the word at different levels in the neural net to compute various attention scores.\n\n i.e. the authors form a vector \"HoW\" (called history of the word), that is defined as a concatenation of several vectors:\n\n HoW_i = [g_i, c_i, h_i^l, h_i^h]\n\nwhere g_i = glove embeddings, c_i = COVE embeddings <b> there is no predicate put with 15 </b> (McCann et al. 2017),  and h_i^l and h_i^h are different LSTM states for that word.\n\n The attention score is then a function of these concatenated vectors i.e. \\alpha_{ij} = \\exp(S(HoW_i^C, HoW_j^Q))\n\n <b> it cannot stay by itself merge with 21 </b> Results on SQuAD show a small gain in accuracy (75.7->76.0 Exact Match). The gains on the adversarial set are larger but that is because some of the higher performing, more recent baselines don't seem to have adversarial numbers.\n\n The authors also compare various attention functions (Table 5) showing a particular one (Symmetric + ReLU) works the best.\n\n Comments:\n\n-I feel overall the contribution is not very novel. The general neural architecture that the authors propose in Section 3 is generally quite similar to the large number of neural architectures developed for this dataset  (e.g. some combination of attention between question/context and LSTMs over question/context). The only novelty is these \"HoW\" inputs to the extra attention mechanism that takes a richer word representation into account.\n\n -I feel the model is seems overly complicated for the small gain (i.e. 75.7->76.0 Exact Match), especially on a relatively exhausted dataset (SQuAD) that is known to have lots of pecularities (see anonymous comment below). It is possible the gains just come from having more parameters.\n\n -The authors (on page 6) claim that that by running attention multiple times with different parameters but different inputs (i.e. \\alpha_{ij}^l, \\alpha_{ij}^h, \\alpha_{ij}^u) it will learn to attend to \"different regions for different level\". However, there is nothing enforcing this and the gains just probably come from having more parameters/complexity.
In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent ('factored'). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments. - The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the 'marginalized' baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments). - The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines (two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector $a_{-i}$). It's not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing $a_{-i}$ and the location of the hole to the baseline? (side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit 'iffy' ; the agent requires information to train, but is not provided the information to act. Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a 'fair' training procedure?) - Minor: in equation 2- is the correct exponent not t'? Also since $\rho_\pi$ is define with a scaling $(1-\gamma)$ (to make it an actual distribution), I believe the definition of $\eta$ should also be multiplied by $(1-\gamma)$ (as well as equation 2).
The paper proposes a new evaluation measure for evaluating GANs. Specifically, the paper proposes generating synthetic images using GAN, training a classifier (for an auxiliary task, not the real vs fake discriminator) and measuring the performance of this classifier on held out real data. While the idea of using a downstream classification task to evaluate the quality of generative models has been explored before  (e.g. semi-supervised learning), I think that this is the first paper to evaluate GANs using such an evaluation metric. I'm not super convinced that this is an useful evaluation metric as the absolute number is somewhat to interpret and dependent on the details of the classifier used. The results in Table 1 change quite a bit depending on the classifier. It would be useful to add a discussion of the failure modes of the proposed metric. It seems like a generator which generates samples close to the classification boundary (but drops examples far away from the boundary) could still achieve a high score under this metric. In the experiments, were different architectures used for different GAN variants? I think the mode-collapse evaluation metrics in MR-GAN are worth discussing in Section 2.1 Mode Regularized Generative Adversarial Networks https://arxiv.org/abs/1612.02136
The paper proposes to improve the kernel approximation of random features by using quadratures, in particular, stochastic spherical-radial rules. The quadrature rules have smaller variance given the same number of random features, and experiments show its reconstruction error and classification accuracies are better than existing algorithms. It is an interesting paper, but it seems the authors are not aware of some existing works [1, 2] on quadrature for random features. Given these previous works, the contribution and novelty of the paper is limited. [1] Francis Bach. On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions. JMLR, 2017. [2] Tri Dao, Christopher De Sa, Christopher Ré. Gaussian Quadrature for Kernel Features. NIPS 2017
The authors consider new attacks for generating adversarial samples against neural networks. In particular, they are interested in approximating gradient-based white-box attacks such as FGSM in a black-box setting by estimating gradients from queries to the classifier. They assume that the attacker is able to query, for any example x, the vector of probabilities p(x) corresponding to each class. Given such query access it’s trivial to estimate the gradients of p using finite differences. As a consequence one can implement FGSM using these estimates assuming cross-entropy loss, as well as a logit-based loss. They consider both iterative and single-step FGSM attacks in the targeted (i.e. the adversary’s goal is to switch the example’s label to a specific alternative label) and un-targeted settings (any mislabelling is a success). They compare themselves to transfer black-box attacks, where the adversary trains a proxy model and generates the adversarial sample by running a white-box attack on that model. For a number of target classifiers on both MNIST and CIFAR-10, they show that these attacks outperform the transfer-based attacks, and are comparable to white-box attacks, while maintaining low distortion on the attack samples. One drawback of estimating gradients using finite differences is that the number of queries required scales with the dimensionality of the examples, which can be prohibitive in the case of images. They therefore describe two practical approaches for query reduction — one based on random feature grouping, and the other on PCA (which requires access to training data). They once again demonstrate the effectiveness of these methods across a number of models and datasets, including models deploying adversarially trained defenses. Finally, they demonstrate compelling real-world deployment against Clarifai classification models designed to flag “Not Safe for Work” content. Overall, the paper provides a very thorough experimental examination of a practical black-box attack that can be deployed against real-world systems. While there are some similarities with Chen et al. with respect to utilizing finite-differences to estimate gradients, I believe the work is still valuable for its very thorough experimental verification, as well as the practicality of their methods. The authors may want to be more explicit about their claim in the Related Work section that the running time of their attack is “40x” less than that of Chen et al. While this is believable, there is no running time comparison in the body of the paper.
This paper presents a framework to recover a set of independent mechanisms.  In order to do so it uses a set of experts each one made out of a GAN. \n\nMy main concern with this work is that I don't see any mechanism in the framework that prevents an expert  (or few of them) to win all examples except its own learning capacities.  p7 authors have also noticed that several experts fail to specialize  and I bet that is the reason why. \nThus, authors should analyze how well we can have all/most experts specialize in a pool vs expert capacity/architecture. \nIt would also be great to integrate a direct regularization mechanism in the cost  in order to do so.  Like for example a penalty in how many examples a expert has catched. \n\nMoreover, the discrimator D  (which is trained to discriminate between real or fake examples) seems to be directly used to tell if an example is throw from the targeted distribution.  It is not the same task.  How D will handle an example far from fake or real ones ?  Why will D answer negatively (or positively) on this example ?
This paper introduces a simple correlation-based metric to measure whether filters in neural networks are being used effectively, as a proxy for effective capacity. The authors then introduce a greedy algorithm that expands the different layers in a neural network until the metric indicates that additional features will end up not being used effectively. The application of this algorithm is shown to lead to architectures that differ substantially from hand-designed models with the same number of layers: most of the parameters end up in intermediate layers, with fewer parameters in earlier and later layers. This indicates that common heuristics to divide capacity over the layers of a network are suboptimal, as they tend to put most parameters in later layers. It's also nice that simpler tasks yield smaller models (e.g. MNIST vs. CIFAR in figure 3). The experimental section is comprehensive and the results are convincing. I especially appreciate the detailed analysis of the results (figure 3 is great). Although most experiments were conducted on the classic benchmark datasets of MNIST, CIFAR-10 and CIFAR-100, the paper also includes some promising preliminary results on ImageNet, which nicely demonstrates that the technique scales to more practical problems as well. That said, it would be nice to demonstrate that the algorithm also works for other tasks than image classification. I also like the alternative perspective compared to pruning approaches, which most research seems to have been focused on in the past. The observation that the cross-correlation of a weight vector with its initial values is a good measure for effective filter use seems obvious in retrospect, but hindsight is 20/20 and the fact is that apparently this hasn't been tried before. It is definitely surprising that a simple method like this ends up working this well. The fact that all parameters are reinitialised whenever any layer width changes seems odd at first, but I think it is sufficiently justified. It would be nice to see some comparison experiments as well though, as the intuitive thing to do would be to just keep the existing weights as they are. Other remarks: Formula (2) seems needlessly complicated because of all the additional indices. Maybe removing some of those would make things easier to parse. It would also help to mention that it is basically just a normalised cross-correlation. This is mentioned two paragraphs down, but should probably be mentioned right before the formula is given instead. page 6, section 3.1: "it requires convergent training of a huge architecture with lots of regularization before complexity can be introduced", I guess this should be "reduced" instead of "introduced".