launch/ampere huggingface dataset:
Adversarial example is studied on one synthetic data. A neural networks classifier is trained on this synthetic data.  Average distances and norms of errorneous perturbations are computed.  It is observed that small perturbation (chosen in a right direction) is sufficient to cause misclassification.  CONS: The writing is bad and hard to follow, with typos:  for example what is a period just before section 3.1 for?  Another example is "Red lines indicate the range of needed for perfect classification", which does not make sense.  Yet another example is the period at the end of Proposition 4.1.   Another example is "One counter-intuitive property of adversarial examples is it that nearly ".  It looks as if the paper was written in a hurry, and it shows in the writing.    At the beginning of Section 3, Figure 1 is discussed.  It points out that there exists adversarial directions that are very bad.  But I don't see how it is relevant to adversarial examples.  If one was interested in studying adversarial examples, then one would have done the following.  Under the setting of Figure 1, pick a test data randomly from the distribution (and one of the classes), and find an adversarial direction I do not see how Section 3.1 fits in with other parts of the paper.  Is it related to any experiment?  Why it defining a manifold attack? Putting a "conjecture" on a paper has to be accompanied by the depth of the insight that brought the conjecture.  Having an unjustified conjecture 5.1 would poison the field of adversarial examples,  and it must be removed. This paper is a list of experiments and observations, that are not coherent and does not give much insight into the topics of "adversarial examples".  The only main messages are that on ONE synthetic dataset, random perturbation does not cause misclassification and targeted classification can cause misclassification.  And, expected loss is good while worst-case loss is bad.  This, in my opinion, is not enough to be published at a conference.
The paper basically propose keep using the typical data-augmentation transformations done during training also in evaluation time, to prevent adversarial attacks. In the paper they analyze only 2 random resizing and random padding, but I suppose others like random contrast, random relighting, random colorization, ... could be applicable. \n\nSome of the pros of the proposed tricks is that it doesn't require re-training existing models, although as the authors pointed out re-training for adversarial images is necessary to obtain good results. \n\n\nTypically images have different sizes , however in the Dataset are described as having 299x299x3 size, are all the test images resized before hand? How would this method work with variable size images? \n\nThe proposed defense requires increasing the size of the input images, have you analyzed the impact in performance? Also it would be good to know how robust is the method for smaller sizes. \n\nSection 4.6.2 seems to indicate that 1 pixel padding or just resizing 1 pixel is enough to get most of the benefit, please provide an analysis of how results improve as the padding or size increase. \n\nIn section 5 for the challenge authors used a lot more evaluations per image, could you provide how much extra computation is needed for that model?
This submission proposes a new seq2sel solution by adopting two new techniques, a sequence-to-set model and column attention mechanism.  They show performance improve over existing studies on WikiSQL dataset. While the paper is written clearly,  the contributions of the work heavily depends on the WikiSQL dataset.  It is not sure if the approach is generally applicable to other sequence-to-sql workloads.  Detailed comments are listed below: 1. WikiSQL dataset contains only a small class of SQL queries, with aggregation over single table and various filtering conditions.  It does not involve any complex operator in relational database system, e.g., join and groupby.  Due to its simple structure, the problem of sequence-to-sql translation over WikiSQL is actually simplified as a parameter selection problem for a fixed template.  This greatly limits the generalization of approaches only applicable to WikiSQL.  The authors are encouraged to explore other datasets available in the literature. 2. The "order-matters" motivation is not very convincing.  It is straightforward to employ a global ordering approach to rank the columns and filtering conditions based on certain rules, e.g., alphabetical order.  That could ensure the orders in the SQL results are always consistent. 3. The experiments do not fully verify how the approaches bring performance improvements.  In the current version, the authors only report superficial accuracy results on final outcomes, without any deep investigation into why and how their approach works.  For instance, they could verify how much accuracy improvement is due to the insensitivity to order in filtering expressions. 4. They do not compare against state-of-the-art solution on column and expression selection.  While their attention mechanism over the columns could bring performance improvement,  they should have included experiments over existing solutions designed for similar purpose.  In (Yin, et al., IJCAI 2016), for example, representations over the columns are learned to generate better column selection. As a conclusion, I find the submission contains certain interesting ideas but lacks serious research investigations.  The quality of the paper could be much enhanced, if the authors deepen their studies on this direction.
In this work, discrete-weight NNs are trained using the variational Bayesian framework, achieving similar results to other state-of-the-art models. Weights use 3 bits on the first layer and are ternary on the remaining layers. - Pros: The paper is well-written and connections with the literature properly established. The approach to training discrete-weights NNs, which is variational inference, is more principled than previous works (but see below). - Cons: The authors depart from the original motivation when the central limit theorem is invoked. Once we approximate the activations with Gaussians, do we have any guarantee that the new approximate lower bound is actually a lower bound? This is not discussed. If it is not a lower bound, what is the rationale behind maximizing it? This seems to place this work very close to previous works, and not in the "more principled" regime the authors claim to seek. The likelihood weighting seems hacky. The authors claim "there are usually many more NN weights than there are data samples". If that is the case, then it seems that the prior dominating is indeed the desired outcome. A different, more flat prior (or parameter sharing), can be used, but the described reweighting seems to be actually breaking a good property of Bayesian inference, which is defecting to the prior when evidence is lacking. In terms of performance (Table 1), the proposed method seems to be on par with existing ones. It is unclear then what the advantage of this proposal is. Sparsity figures are provided for the current approach, but those are not contrasted with existing approaches. Speedup is claimed with respect to an NN with real weights, but not with respect existing NNs with binary weights, which is the appropriate baseline. - Minor comments: Page 3: Subscript t and variable t is used for the targets, but I can't find where it is defined. Only the names of the datasets used in the experiments are given, but they are not described, or even better, shown in pictures (maybe in a supplementary). The title of the paper says "discrete-valued NNs". The weights are discrete, but the activations and outputs are continuous, so I find it confusing. As a contrast, I would be less surprised to hear a sigmoid belief network called a "discrete-valued NN", even though its weights are continuous.
This paper explores learning dynamic filters for CNNs.  The filters are generated by using the features of an autoencoder on the input image, and linearly combining a set of base filters for each layer.  This addresses an interesting problem which has been looked at a lot before, but with some small new parts.  There is a lot of prior work in this area  that should be cited in the area of dynamic filters and steerable filters.  There are also parallels to ladder networks that should be highlighted.  The results indicate improvement over baselines,  however baselines are not strong baselines.  A key question is what happens when this method is combined with VGG11 which the authors train as a baseline?  What is the effect of the reconstruction loss?  Can it be removed?  There should be some ablation study here. Figure 5 is unclear what is being displayed,  there are no labels. Overall I would advise the authors to address these questions and suggest this as a paper suitable for a workshop submission.
This paper proposes a simple modification to the standard alternating stochastic gradient method for GAN training, which stabilizes training, by adding a prediction step. This is a clever and useful idea,  and the paper is very well written.  The proposed method is very clearly motivated, both intuitively and mathematically,  and the authors also provide theoretical guarantees on its convergence behavior.  I particularly liked the analogy with the damped harmonic oscillator. The experiments are well designed and provide clear evidence in favor of the usefulness of the proposed technique.  I believe that the method proposed in this paper will have a significant impact in the area of GAN training. I have only one minor question: in the prediction step, why not use a step size, say $\bar{u}_k+1 = u_{k+1} + \gamma_k (u_{k+1} − u_k)$, such that the "amount of predition" may be adjusted?
Authors provide an interesting loss function approach for clustering using a deep neural network.  They optimize Kuiper-based nonparametric loss and apply the approach on a large social network data-set.   However, the details of the deep learning approach are not well described.  Some specific comments are given below. 1.Further details on use of 10-fold cross validation need to be discussed including over-fitting aspect. 2. Details on deep learning, number of hidden layers, number of hidden units, activation functions, weight adjustment details on each learning methods should be included. 3. Conclusion section is very brief  and can be expanded by including a discussion on results comparison and  over fitting aspects in cross validation.  Use of Kuiper-based nonparametric loss should also be justified as there are other loss functions can be used under these settings.
This paper investigates meta-learning strategy for automated architecture search in the context of RNN.  To constraint the architecture search space, authors propose a DSL that specifies the RNN recurrent operations.  This DSL allows to explore RNN architectures using either random search or a reinforcement-learning strategy.  Candidate architectures are ranked using a TreeLSTM that tries to predict the architecture performances.  The top-k architectures are then evaluated by fully training them on a given task. Authors evaluate their approach on  PTB/Wikitext 2 language modeling and Multi30k/IWSLT'16  machine translation.  In both experiments, authors show that their approach obtains competitive results and can sometime outperforms RNN cells such as GRU/LSTM.  In the PTB experiment, their architecture however underperforms other LSTM variant in the literatures. - Quality/Clarity The paper is overall well written and pleasant to read. Few details can be clarified.  In particular how did you initialize the weight and bias for both the LSTM/GRU baselines and the found architectures?  Is there other works leveraging RNN that report results on the Multi30k/IWSLT datasets? You state in paragraph 3.2 that human experts can inject the previous best known architecture when training the ranking networks.  Did you use this in the experiments?  If yes, what was the impact of this online learning strategy on the final results?  - Originality The idea of using DSL + ranking for architecture search seems novel. - Significance Automated architecture search is a promising way to design new networks.  However, it is not clear why the proposed approach is not able to outperforms other LSTM-based architectures on the PTB task.  Could the problem arise from the DSL that constraint too much the search space ?  It would be nice to have other tasks that are commonly used as benchmark for RNN to see where this approach stand. In addition, authors propose both a DSL, a random and RL generator and a ranking function.  It would be nice to disentangle the contributions of the different components.  In particular, did the authors compare the random search vs the RL based generator or the performances of the RL-based generator when the ranking network is not used? Although authors do show that they outperform NAScell in one setting,  it would be nice to have an extended evaluation (using character level PTB for instance).
This writeup describes an application of recurrent autoencoder to analysis of multidimensional time series.  The quality of writing, experimentation and scholarship is clearly below than what is expected from a scientific article.  The method is explained in a very unclear way,  there is no mention of any related work.  I would encourage the authors to take a look at other ICLR submissions and see how rigorously written they are, how they position the reported research among comparable works.
The paper proposes to use a regularizer for tensor completion problems that can be written in a similar fashion as the variational factorization formulation of the trace norm aka nuclear norm for matrices. The paper introduces the regularizer with the nice argument that the gradient of the L3 norm to the power of 3rd will be easy to compute, but if we were raising the L2 norm to the 3rd power it would not be the case. They mention that their argument can generalize from D=3 to higher order tensors. Authors mention the paper by Friedland and Lim that introduces this norm and provides first theoretical results on it. Authors develop on the tensor equivalent of the matrix max norm which is built with the motivation of bringing robustness to heavy nodes in the graph (very popular content). This is again straightforward on the technical side. Empirical results are fine but do not show huge improvements compared to baselines so I do not think this is a strong argument for accepting the paper. On the scalability, authors do not show that their approach is better suited than baselines.
Quality: Although the research problem is an interesting direction  the quality of the work is not of a high standard.  My main conservation is that the idea of perturbation in semantic latent space has not been described in an explicit way.  How different it will be compared to a perturbation in an input space?  Clarity: The use of the term "adversarial" is not quite clear in the context  as in many of those example classification problems the perturbation completely changes the class label (e.g. from "church" to "tower" or vice-versa) Originality: The generation of adversarial examples in black-box classifiers has been looked in GAN literature as well and gradient based perturbations are studied too.  What is the main benefit of the proposed mechanism compared to the existing ones? Significance: The research problem is indeed a significant one  as it is very important to understand the robustness of the modern machine learning methods by exposing them to adversarial scenarios where they might fail. pros: (a) An interesting problem to evaluate the robustness of black-box classifier systems (b) generating adversarial examples for image classification as well as text analysis. (c) exploiting the recent developments in GAN literature to build the framework forge generating adversarial examples. cons:(a) The proposed search algorithm in the semantic latent space could be computationally intensive.  any remedy for this problem? (b) Searching in the latent space z could be strongly dependent on the matching inverter $I_\gamma(.)$.  any comment on this? (c) The application of the search algorithm in case of imbalanced classes could be something that require further investigation.
This work attempts to improve the global consistency of samples generated by generative adversarial networks by replacing the discriminator with an autoregressive model in an encoded feature space.  The log likelihood of the classification model is then replaced with the log likelihood of the feature space autoregressive model.  It's not clear what can be said with respect to the convergence properties of this class of models,  and this is not discussed. The method is quite similar in spirit to Denoising Feature Matching of Warde-Farley & Bengio (2017),  as both estimate a density model in feature space -- this method via a constrained autoregressive model and DFM via an estimator of the score function,  although DFM was used in conjunction with the standard criterion whereas this method replaces it.  This is certainly worth mentioning and discussing.  In particular the section in Warde-Farley & Bengio regarding the feature space transformation of the data density seems quite relevant in this work. Unfortunately the only quantitative measurements reporter are Inception scores,  which is known to be a poor measure  (and the scores presented are not particularly high, either);  Frechet Inception distance or log likelihood estimates via AIS on some dataset would be more convincing.  On the plus side, the authors report an average over Inception scores for multiple runs.  On the other hand, it sounds as though the stopping criterion was still qualitative.
This work introduces a particular parametrization of a stochastic policy (a uniform mixture of deterministic policies). They find this parametrization, when trained with stochastic value gradient outperforms DDPG on several OpenAI gym benchmarks. This paper unfortunately misses many significant pieces of prior work training stochastic policies. The most relevant is [1] which should definitely be cited. The algorithm here can be seen as SVG(0) with a particular parametrization of the policy. However, numerous other works have examined stochastic policies including [2] (A3C which also used the Torcs environment) and [3]. The wide use of stochastic policies in prior work makes the introductory explanation of the potential benefits for stochastic policies distracting, instead the focus should be on the particular choice and benefits of the particular stochastic parametrization chosen here and the choice of stochastic value gradient as a training method (as opposed to many on-policy methods). The empirical comparison is also hampered by only comparing with DDPG, there are numerous stochastic policy algorithms that have been compared on these environments. Additionally, the DDPG performance here is lower for several environments than the results reported in Henderson et al. 2017 (cited in the paper, table 2 here, table 3 Henderson) which should be explained. While this particular parametrization may provide some benefits, the lack of engagement with relevant prior work and other stochastic baselines significant limits the impact of this work and makes assessing its significance difficult. This work would benefit from careful copyediting. [1] Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., & Tassa, Y. (2015). Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (pp. 2944-2952). [2] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., ... & Kavukcuoglu, K. (2016, June). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (pp. 1928-1937). [3] Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.
The paper proposes a novel way of compressing gradient updates for distributed SGD, in order to speed up overall execution. While the technique is novel as far as I know (eq. (1) in particular), many details in the paper are poorly explained (I am unable to understand) and experimental results do not demonstrate that the problem targeted is actually alleviated. More detailed remarks: 1: Motivating with ImageNet taking over a week to train seems misplaced when we have papers claiming to train ImageNet in 1 hour, 24 mins, 15 mins... 4.1: Lemma 4.1 seems like you want B > 1, or clarify definition of V_B. 4.2: This section is not fully comprehensible to me. - It seems you are confusingly overloading the term gradient and words derived (also in other parts or the paper). What is "maximum value of gradients in a matrix"? Make sure to use something else, when talking about individual elements of a vector (which is constructed as an average of gradients), etc. - Rounding: do you use deterministic or random rounding? Do you then again store the inaccuracy? - I don't understand definition of d. It seems you subtract logarithm of a gradient from a scalar. - In total, I really don't know what is the object that actually gets communicated, and consequently when you remark that this can be combined with QSGD and the more below it, I don't understand it. This section has to be thoroughly explained, perhaps with some illustrative examples. 4.3: allgatherv remark: does that mean that this approach would not scale well to higher number of workers? 4.4: Remarks about quantization and mantissa manipulation are not clear to me again, or what is the point in doing so. Possible because the problems above. 5: I think this section is not too useful unless you can accompany it with actual efficient implementation and contrast the practical performance. 6: Given that I don't understand how you compress the information being communicated, it is hard to believe the utility of the method. The objective was to speed up training time because communication is bottleneck. If you provide 12,000x compression, is it any more practically useful than providing 120x compression? What would be the difference in runtime? Such questions are never discussed. Further, if in the implementation you discuss masking mantissa, I have serious concern about whether the compression protocol is feasible to implement efficiently, without writing some extremely low-level code. I think the soundness of work addressing this particular problem is damaged if not implemented properly (compared to other kinds of works in current ML related research). Therefore I highly recommend including proper time comparison with a baseline in the future. Further, I don't understand 2 things about the Tables. a) how do you combine the proposed method with Momentum in SGD? This is not discussed as far as I can see. b) What is "QSGD, 2bit" If I remember QSGD protocol correctly, there's no natural mapping of 2bit to its parameters.