launch/ampere huggingface dataset:
The paper proposes a technique for quantizing the weights of a neural network, with bit-depth/precision varying on a per-parameter basis.  The main idea is to minimize the number of bits used in the quantization while constraining the loss to remain below a specified upper bound.  This is achieved by formulating an upper bound on the number of bits used via a set of "tolerances";  this upper bound is then minimized while estimating any increase in loss using a first order Taylor approximation. I have a number of questions and concerns about the proposed approach.  First, at a high level, there are many details that aren't clear from the text.  Quantization has some bookkeeping associated with it: In a per-parameter quantization setup it will be necessary to store not just the quantized parameter, but also the number of bits used in the quantization (takes e.g. 4-5 extra bits), and there will be some metadata necessary to encode how the quantized value should be converted back to floating point (e.g., for 8-bit quantization of a layer of weights, usually the min and max are stored).  From Algorithm 1 it appears the quantization assumes parameters in the range [0, 1].  Don't negative values require another bit?  What happens to values larger than 1?  How are even bit depths and associated asymmetries w.r.t. 0 handled  (e.g., three bits can represent -1, 0, and 1, but 4 must choose to either not represent 0 or drop e.g. -1)?  None of these details are clearly discussed in the paper,  and it's not at all clear that the estimates of compression are correct if these bookkeeping matters aren't taken into account properly. Additionally the paper implies that this style of quantization has benefits for compute in addition to memory savings.  This is highly dubious,  since the method will require converting all parameters to a standard bit-depth on the fly (probably back to floating point, since some parameters may have been quantized with bit depth up to 32).  Alternatively custom GEMM/conv routines would be required which are impossible to make efficient for weights with varying bit depths.  So there are likely not runtime compute or memory savings from such an approach. I have a few other specific questions: Are the gradients used to compute \mu computed on the whole dataset or minibatches?  How would this scale to larger datasets?  I am confused by the equality in Equation 8: What happens for values shared by many different quantization bit depths  (e.g., representing 0 presumably requires 1 bit, but may be associated with a much finer tolerance)?  Should "minimization in equation 4" refer to equation 3? In the end, while do like the general idea of utilizing the gradient to identify how sensitive the model might be to quantization of various parameters,  there are significant clarity issues in the paper,  I am a bit uneasy about some of the compression results claimed without clearer description of the bookkeeping,  and I don't believe an approach of this kind has any significant practical relevance for saving runtime memory or compute resources.
Recently some interesting work on a role of prior in deep generative models has been presented. The choice of prior may have an impact on the expressiveness of the model [Hoffman and Johnson, 2016]. A few existing work presents methods for learning priors from data for variational autoencoders [Goyal et al., 2017] [Tomczak and Welling, 2017]. The work, "VAE with a VampPrior," [Tomczak and Welling, 2017] is missing in references. The current work focuses on adversarial autoencoder (AAE) and introduces a code generator network to transform a simple prior into one that together with the generator can better fit the data distribution. Adversarial loss is used to train the code generator network, allowing the output of the network could be any distribution. I think the method is quite simple but interesting approach to improve AAEs without hurting the reconstruction. The paper is well written and is easy to read. The method is well described. However, what is missing in this paper is an analysis of learned priors, which help us to better understand its behavior. The model is evaluated qualitatively only. What about quantitative evaluation?
The paper proposes an interesting alternative to recent approaches to learning from logged bandit feedback, and validates their contribution in a reasonable experimental comparison. The clarity of writing can be improved (several typos in the manuscript, notation used before defining, missing words, poorly formatted citations, etc.). Implementing the approach using recent f-GANs is an interesting contribution and may spur follow-up work. There are several lingering concerns about the approach (detailed below) that detract from the quality of their contributions. [Major] In Lemma 1, L(z) is used before defining it. Crucially, additional assumptions on L(z) are necessary (e.g. |L(z)| <= 1 for all z. If not, a trivial counter-example is: set L(z) >> 1 for all z and Lemma 1 is violated). It is unclear how crucially this additional assumption is required in practice (their expts with Hamming losses clearly do not satisfy such an assumption). [Minor] Typo: Section 3.2, first equation; the integral equals D_f(...) + 1 (not -1). [Crucial!] Eqn10: Expected some justification on why it is fruitful to *lower-bound* the divergence term, which contributes to an *upper-bound* on the true risk. [Crucial!] Algorithm1: How is the condition of the while loop checked in a tractable manner? [Minor] Typos: Initilization -> Initialization, Varitional -> Variational [Major] Expected an additional "baseline" in the expts -- Supervised but with the neural net policy architecture (NN approaches outperforming Supervised on LYRL dataset was baffling before realizing that Supervised is implemented using a linear CRF). [Major] Is there any guidance for picking the new regularization hyper-parameters (or at least, a sensible range for them)? [Minor] The derived bounds depend on M, an a priori upper bound on the Renyi divergence between the logging policy and any new policy. It's unclear that such a bound  can be tractably guessed (in contrast, prior work uses an upper bound on the importance weight -- which is simply 1/(Min action selection prob. by logging policy) ).
This paper provides theoretical and empirical motivations for removing the top few principle components of commonly-used word embeddings. The paper is well-written and I enjoyed reading it.  However, it does not explain how significant this result is beyond that of (Bullinaria and Levy, 2012),  who also removed the top N dimensions when benchmarking SVD-factorized word embeddings.  From what I can see, this paper provides a more detailed explanation of the phenomenon ("why" it works),  which is supported with both theoretical results and a series of empirical analyses, as well as "updating" the benchmarks and methods from the pre-neural era.  Although this contribution is relatively incremental,  I find the depth of this work very interesting,  and I think future work could perhaps rely on these insights to create better embedding algorithms that directly enforce isotropy. I have two concerns regarding the empirical section, which may be resolvable fairly quickly: 1) Are the embedding vectors L2 normalized before using them in each task?  This is known to significantly affect performance.  I am curious whether removing the top PCs is redundant or not given L2 normalization. 2) Most of the benchmarks used in this paper are "toy" tasks.  As Schnabel et al (2015) and Tsvetkov et al (2015) showed, there is often little correlation between success on these benchmarks and improvement of downstream NLP tasks.  I would like to measure the change in performance on a major NLP task that heavily relies on pre-trained word embeddings such as SQuAD. Minor Comments:* The last sentence in the first paragraph ("The success comes from the geometry of the representations...") is not true;  the success stems from the ability to capture lexical similarity.  Levy and Goldberg (2014) showed that searching for the closest word vector to (king - man + woman) is equivalent to optimizing a linear combination of 3 similarity terms [+(x,king), -(x,man), +(x, woman)].  This explanation was further demonstrated by Linzen (2016) who showed that even when removing the negative term (x, man), many analogies can still be solved, i.e. by looking for a word that is similar both to "king" and to "woman".  Add to that the fact that the analogy trick works best when the vectors are L2 normalized;  if they are all on the unit sphere, what is the geometric interpretation of (king - man + woman), which is not on the unit sphere?  I suggest removing this sentence and other references to linguistic regularities from this paper,  since they are controversial at best, and distract from the main findings. * This is also related to Bullinaria and Levy's (2012) finding that downweighting the eigenvalue matrix in SVD-based methods improves their performance.  Levy et al (2015) showed that keeping the original eigenvalues can actually degenerate SVD-based embeddings.  Perhaps there is a connection to the findings in this paper?
The paper proposes an adaptation of existing Graph ConvNets and evaluates this formulation on a several existing benchmarks of the graph neural network community. In particular, a tree structured LSTM is taken and modified. The authors describe this as adapting it to general graphs, stacking, followed by adding edge gates and residuality. My biggest concern is novelty, as the modifications are minor. In particular, the formulation can be seen in a different way. As I see it, instead of adapting Tree LSTMs to arbitary graphs, it can be seen as taking the original formulation by Scarselli and replacing the RNN by a gated version, i.e. adding the known LSTM gates (input, output, forget gate). This is a minor modification. Adding stacking and residuality are now standard operations in deep learning, and edge-gates have also already been introduced in the literature, as described in the paper. A second concern is the presentation of the paper, which can be confusing at some points. A major example is the mathematical description of the methods. When reading the description as given, one should actually infer that Graph ConvNets and Graph RNNs are the same thing, which can be seen by the fact that equations (1) and (6) are equivalent. Another example, after (2), the important point to raise is the difference to classical (sequential) RNNs, namely the fact that the dependence graph of the model is not a DAG anymore, which introduces cyclic dependencies. Generally, a clear introduction of the problem is also missing. What are the inputs, what are the outputs, what kind of problems should be solved? The update equations for the hidden states are given for all models, but how is the output calculated given the hidden states from variable numbers of nodes of an irregular graph? The model has been evaluated on standard datasets with a performance, which seems to be on par, or a slight edge, which could probably be due to the newly introduced residuality. A couple of details :- the length of a graph is not defined. The size of the set of nodes might be meant. - at the beginning of section 2.1 I do not understand the reference to word prediction and natural language processing. RNNs are not restricted to NLP and I think there is no need to introduce an application at this point. - It is unclear what does the following sentence means: "ConvNets are more pruned to deep networks than RNNs"? - What are "heterogeneous graph domains"?
The paper shows that several recently proposed interpretation techniques for neural network are performing similar processing and yield similar results. The authors show that these techniques can all be seen as a product of input activations and a modified gradient, where the local derivative of the activation function at each neuron is replaced by some fixed function. A second part of the paper looks at whether explanations are global or local. The authors propose a metric called sensitivity-n for that purpose, and make some observations about the optimality of some interpretation techniques with respect to this metric in the linear case. The behavior of each explanation w.r.t. these properties is then tested on multiple DNN models tested on real-world datasets. Results further outline the resemblance between the compared methods. In the appendix, the last step of the proof below Eq. 7 is unclear. As far as I can see, the variable g_i^LRP wasn’t defined, and the use of Eq. 5 to achieve this last could be better explained. There also seems to be some issues with the ordering i,j, where these indices alternatively describe the lower/higher layers, or the higher/lower layers.
Summary: The paper propose a method for generating adversarial examples in image recognition problems.  The Adversarial scheme is inspired in the one proposed by Goodgellow  et al 2015 (AT) that introduces small perturbations to the data in the direction that increases the error.  Such a perturbations are random (they have not structure) and lack of interpretation for a human user.  The proposal is to limit the perturbations to just three kind of global motion fields: shift, centered rotation and scale (zoom in/out).  Since the motions are small in scale,  the authors use a first-order Taylor series approximation  (as in classical optical flow).  This approximation allows to obtain close formulas for the perturbed examples; i.e. the correction factor of the Back-propagation computed derivatives w.r.t. original example.  As result, the method is computational efficient respect to the AT and the perturbations are interpretable.  Experiments demonstrate that with the MNIST database is not obtained an improvement in the error reduction but a reduction of the computational time.  However, with ta more general recognition problem conducted with the CIFAR-10 database, the use of the proposed method improves both the error and the computational time, when compared with AT and Virtual Adversarial Train.  Comments:1. The paper presents a series os typos: FILED (title), obouve, freedm, nerual,; please check carfully. 2. The Derivation of eq. (13) should be explained,  It could be said that (12) can be casted as a eigenvalue problem [for example: $ max_{\tilde v} \| \nabla_p L^T \tilde v \|^2 \;\; s.t. \| v\|=1  $] and (13) is the largest eigenvalue of $ \nabla_p L \nabla_p L^T $] 3. The improvement in the error results in the db CIFAR-10 is good enough to see merit in the proposal approach.  Maybe other perturbations with closed formula could be considered and linear combinations of them
The authors present a deep neural network that evaluates plate numbers.  The relevance of this problem is that there are auctions for plate numbers in Hong Kong, and predicting their value is a sensible activity in that context.  I find that the description of the applied problem is quite interesting;  in fact overall the paper is well written and very easy to follow.  There are some typos and grammatical problems (indicated below), but nothing really serious. So, the paper is relevant and well presented.  However, I find that the proposed solution is an application of existing techniques,  so it lacks on novelty and originality.  Even though the significance of the work is apparent given the good results of the proposed neural network,  I believe that such material is more appropriate to a focused applied meeting.  However, even for that sort of setting I think the paper requires some additional work,  as some final parts of the paper have not been tested yet  (the interesting part of explanations).  Hence I don't think the submission is ready for publication at this moment. Concerning the text, some questions/suggestions: - Abstract, line 1: I suppose "In the Chinese society..."--- are there many Chinese societies? - The references are not properly formatted;  they should appear at (XXX YYY) but appear as XXX (YYY) in many cases, mixed with the main text.  - Footnote 1, line 2: "an exchange". - Page 2, line 12: "prices. Among". - Please add commas/periods at the end of equations. - There are problems with capitalization in the references.
In this paper, the authors propose a novel tracking loss to convert the RPN to a tracker.  The internal structure of top layer features of RPN is exploited to treat feature points discriminatively.  In addition, the proposed compression network speeds up the tracking algorithm.  The experimental results on the VOT2016 dataset demonstrate its efficiency in tracking.  This work is the combination of Faster R-CNN (Ren et al. PAMI 2015) and tracking-by-detection framework.  The main contributions proposed in this paper are new tracking loss, network compression and results.  There are numerous concerns with this work: 1.	The new tracking loss shown in equation 2 is similar with the original Faster R-CNN loss shown in equation 1.  The only difference is to replace the regression loss with a predefined mask selection loss,  which is of little sense that the feature processing can be further fulfilled through one-layer CNN.  The empirical operation shown in figure 2 seems arbitrary and lack of theoretical explanation.  There is no insight of why doing so.  Simply showing the numbers in table 1 does not imply the necessity,  which ought to be put in the experiment sections.  2.	The network compression is engineering and lack insight as well.  To remove part of the CNN and retrain is a common strategy in the CNN compression methods [a] [b].  There is a lack of discussion with the relationship with prior arts. 3.	The organization is not clear.  Section 3.4 should be set in the experiments  and Section 3.5 should be set at the beginning of the algorithm.  The description of the network compression is not clear enough, especially the training details.   Meanwhile, the presentation is hard to follow.  There is no clear expression of how the tracker performs in practice. 4.	In addition, VOT 2016, the method should evaluate on the OTB dataset with the following trackers [c] [d]. 5.	The evaluation is not fair.  In Sec 6, the authors indicate that MDNet runs at 1FPS while the proposed tracker runs at 1.6FPS.  However, MDNet is based on Matlab  and the proposed tracker is based on C++ (i.e., Caffe). Reference:[a] On Compressing Deep Models by Low Rank and Sparse Decomposition. Yu et al. CVPR 2017. [b] Designing Energy-Efficient Convolutional Neural Network Using Energy-Aware Pruning. Yang et al. CVPR 2017. [c] ECO: Efficient Convolution Operators for Tracking. Danelljan et al. CVPR 2017. [d] Multi-Task Correlation Particle Filter For Robust Object Tracking. Zhang et al. CVPR 2017.
This paper proposes an evolutionary algorithm for solving the variational E step in expectation-maximization algorithm for probabilistic models with binary latent variables.  This is done by (i) considering the bit-vectors of the latent states as genomes of individuals, and by (ii) defining the fitness of the individuals as the log joint distribution of the parameters and the latent space. Pros:The paper is well written and the methodology presented is largely clear. Cons:While the reviewer is essentially fine with the idea of the method,  the reviewer is much less convinced of the empirical study.  There is no comparison with other methods such as Monte carlo sampling. It is not clear how computationally Evolutionary EM performs comparing to Variational EM algorithm  and there is neither experimental results nor analysis for the computational complexity of the proposed model. The datasets used in the experiments are quite old.  The reviewer is concerned that these datasets may not be representative of real problems. The applicability of the method is quite limited.  The proposed model is only applicable for the probabilistic models with binary latent variables,  hence it cannot be applied to more realistic complex model with real-valued latent variables.
This paper proposes several client-server neural network gradient update strategies aimed at reducing uplink usage while maintaining prediction performance. The main approaches fall into two categories: structured, where low-rank/sparse updates are learned, and sketched, where full updates are either sub-sampled or compressed before being sent to the central server. Experiments are based on the federated averaging algorithm. The work is valuable, but has room for improvement. The paper is mainly an empirical comparison of several approaches, rather than from theoretically motivated algorithms. This is not a criticism, however, it is difficult to see the reason for including the structured low-rank experiments in the paper (itAs a reader, I found it difficult to understand the actual procedures used. For example, what is the difference between the random mask update and the subsampling update (why are there no random mask experiments after figure 1, even though they performed very well)? How is the structured update "learned"? It would be very helpful to include algorithms. It seems like a good strategy is to subsample, perform Hadamard rotation, then quantise. For quantization, it appears that the HD rotation is essential for CIFAR, but less important for the reddit data. It would be interesting to understand when HD works and why, and perhaps make the paper more focused on this winning strategy, rather than including the low-rank algo. If convenient, could the authors comment on a similarly motivated paper under review at iclr 2018: VARIANCE-BASED GRADIENT COMPRESSION FOR EFFICIENT DISTRIBUTED DEEP LEARNING pros:- good use of intuition to guide algorithm choices - good compression with little loss of accuracy on best strategy - good problem for FA algorithm / well motivated cons:- some experiment choices do not appear well motivated / inclusion is not best choice - explanations of algos / lack of 'algorithms' adds to confusion a useful reference:  Strom, Nikko. "Scalable distributed dnn training using commodity gpu cloud computing." Sixteenth Annual Conference of the International Speech Communication Association. 2015.
The authors propose a generative method that can produce images along a hierarchy of specificity, i.e. both when all relevant attributes are specified, and when some are left undefined, creating a more abstract generation task. Pros:+ The results demonstrating the method's ability to generate results for (1) abstract and (2) novel/unseen attribute descriptions, are generally convincing. Both quantitative and qualitative results are provided. + The paper is fairly clear. Cons:- It is unclear how to judge diversity qualitatively, e.g. in Fig. 4(b). - Fig. 5 could be more convincing; "bushy eyebrows" is a difficult attribute to judge, and in the abstract generation when that is the only attribute specified, it is not clear how good the results are.