launch/ampere huggingface dataset:
The authors propose an extension to CNN using an autoregressive weighting for asynchronous time series applications. The method is applied to a proprietary dataset as well as a couple UCI problems and a synthetic dataset, showing improved performance over baselines in the asynchronous setting. This paper is mostly an applications paper. The method itself seems like a fairly simple extension for a particular application, although perhaps the authors have not clearly highlighted details of methodological innovation. I liked that the method was motivated to solve a real problem, and that it does seem to do so well compared to reasonable baselines. However, as an an applications paper, the bread of experiments are a little bit lacking -- with only that one potentially interesting dataset, which happens to proprietary. Given the fairly empirical nature of the paper in general, it feels like a strong argument should be made, which includes experiments, that this work will be generally significant and impactful. The writing of the paper is a bit loose with comments like: “Besides these and claims of secretive hedge funds (it can be marketing surfing on the deep learning hype), no promising results or innovative architectures were publicly published so far, to the best of our knowledge.” Parts of the also appear rush written, with some sentences half finished: “"ues of x might be heterogenous, hence On the other hand, significance network provides data-dependent weights for all regressors and sums them up in autoregressive manner.”” As a minor comment, the statement “however, due to assumed Gaussianity they are inappropriate for financial datasets, which often follow fat-tailed distributions (Cont, 2001).” Is a bit too broad. It depends where the Gaussianity appears. If the likelihood is non-Gaussian, then it often doesn’t matter if there are latent Gaussian variables.
The authors analyze show theoretical shortcomings in previous methods of explaining neural networks and propose an elegant way to remove these shortcomings in their methods PatternNet and PatternAttribution. The quest of visualizing neural network decision is now a very active field with many contributions. The contribution made by the authors stands out due to its elegant combination of theoretical insights and improved performance in application. The work is very detailed and reads very well. I am missing at least one figure with comparison with more state-of-the-art methods (e.g. I would love to see results from the method by Zintgraf et al. 2017 which unlike all included prior methods seems to produce much crisper visualizations and also is very related because it learns from the data, too). Minor questions and comments:* Fig 3: Why is the random method so good at removing correlation from fc6? And the S_w even better? Something seems special about fc6. * Fig 4: Why is the identical estimator better than the weights estimator and that one better than S_a? * It would be nice to compare the image degradation experiment with using the ranking provided by the work from Zintgraf which should by definition function as a kind of gold standard * Figure 5, 4th row (mailbox): It looks like the umbrella significantly contributes to the network decision to classify the image as "mailbox" which doesn't make too much sense. Is is a problem of the visualization  (maybe there is next to no weight on the umbrella), of PatternAttribution or a strange but interesting a artifact of the analyzed network? * page 8 "... closed form solutions (Eq (4) and Eq. (7))" The first reference seems to be wrong. I guess Eq 4. should instead reference the unnumbered equation after Eq. 3.
The paper presents a way to regularize a sequence generator by making the hidden states also predict the hidden states of an RNN working backward. Applied to sequence-to-sequence networks, the approach requires training one encoder, and two separate decoders, that generate the target sequence in forward and reversed orders.  A penalty term is added that forces an agreement between the hidden states of the two decoders.  During model evaluation only the forward decoder is used, with the backward operating decoder discarded.  The method can be interpreted to generalize other recurrent network regularizers, such as putting an L2 loss on the hidden states. Experiments indicate that the  approach is most successful when the regularized RNNs are conditional generators, which emit sequences of low entropy, such as decoders of a seq2seq speech recognition network.  Negative results were reported when the proposed regularization technique was applied to language models, whose output distribution has more entropy. The proposed regularization is evaluated with positive results on a speech recognition task and on an  image captioning task, and with negative results (no improvement, but also no deterioration) on a language modeling and sequential MNIST digit generation tasks. I have one question about baselines: is the proposed approach better than training to forward generators and force an agreement between them (in the spirit of the concurrent ICLR submission https://openreview.net/forum?id=rkr1UDeC-)?  Also, would using the backward RNN, e.g. for rescoring, bring another advantage?  In other words, what is (and is there) a gap between an ensemble of a forward and backward rnn and the forward-rnn only, but trained with the state-matching penalty? Quality:The proposed approach is well motivated  and the experiments show the limits of applicability range of the technique. Clarity:The paper is clearly written. Originality:The presented idea seems novel. Significance:The method may prove to be useful to regularize recurrent networks,  however I would like to see a comparison with ensemble methods.  Also, as the authors note the method seems to be limited to conditional sequence generators. Pros and cons:Pros: the method is simple to implement,  the paper lists for what kind of datasets it can be used. Cons: the method needs to be compared with typical ensembles of models going only forward in time,  it may turn that it using the backward RNN is not necessary
This paper proposes to automatically recognize domain names as malicious or benign by deep networks (convnets and RNNs) trained to directly classify the character sequence as such. Pros The paper addresses an important application of deep networks, comparing the performance of a variety of different types of model architectures. The tested networks seem to perform reasonably well on the task. Cons There is little novelty in the proposed method/models  -- the paper is primarily focused on comparing existing models on a new task. The descriptions of the different architectures compared are overly verbose  -- they are all simple standard convnet / RNN architectures.   The code specifying the models is also excessive for the main text  -- it should be moved to an appendix or even left for a code release. The comparisons between various architectures are not very enlightening  as they aren’t done in a controlled way  -- there are a large number of differences between any pair of models  so it’s hard to tell where the performance differences come from.  It’s also difficult to compare the learning curves among the different models (Fig 1)  as they are in separate plots with differently scaled axes. The proposed problem is an explicitly adversarial setting  and adversarial examples are a well-known issue with deep networks and other models,  but this issue is not addressed or analyzed in the paper.  (In fact, the intro claims this is an advantage of not using hand-engineered features for malicious domain detection, seemingly ignoring the literature on adversarial examples for deep nets.)  For example, in this case an attacker could start with a legitimate domain name and use black box adversarial attacks (or white box attacks, given access to the model weights) to derive a similar domain name that the models proposed here would classify as benign. While this paper addresses an important problem,  in its current form the novelty and analysis are limited  and the paper has some presentation issues.
This manuscript explores the idea of adding noise to the adversary's play in GAN dynamics over an RKHS.  This is equivalent to adding noise to the gradient update, using the duality of reproducing kernels.  Unfortunately, the evaluation here is wholly unsatisfactory to justify the manuscript's claims.  No concrete practical algorithm specification is given (only a couple of ideas to inject noise listed),  only a qualitative one on a 2-dimensional latent space in MNIST, and an inconclusive one using the much-doubted Parzen window KDE method.  The idea as stated in the abstract and introduction may well be worth pursuing,  but not on the evidence provided by the rest of the manuscript.
The collaborative block that authors propose is a generalized module that can be inserted in deep architectures for better multi-task learning. The problem is relevant as we are pushing deep networks to learn representation for multiple tasks. The proposed method while simple is novel. The few places where the paper needs improvement are: 1. The authors should test their collaborative block on multiple tasks where the tasks are less related. Ex: Scene and object classification. The current datasets where the model is evaluated is limited to Faces which is a constrained setting. It would be great if Authors provide more experiments beyond Faces to test the universality of the proposed approach. 2. The Face datasets are rather small. I wonder if the accuracy improvements hold on larger datasets and if authors can comment on any large scale experiments they have done using the proposed architecture. In it's current form I would say the experiment section and large scale experiments are two places where the paper falls short.
This paper presents a novel approach to calibrate classifiers for out of distribution samples. In additional to the original cross entropy loss, the “confidence loss”  was proposed to guarantee the out of distribution points have low confidence in the classifier. As out of distribution samples are hard to obtain, authors also propose to use GAN generating “boundary” samples as out of distribution samples. The problem setting is new and objective (1) is interesting and reasonable. However, I am not very convinced that objective (3) will generate boundary samples. Suppose that theta is set appropriately so that p_theta (y|x) gives a uniform distribution over labels for out of distribution samples. Because of the construction of U(y), which uniformly assign labels to generated out of distribution samples, the conditional probability p_g (y|x) should always be uniform so p_g (y|x) divided by p_theta (y|x) is almost always 1. The KL divergence in (a) of (3) should always be approximately 0 no matter what samples are generated. I also have a few other concerns: 1. There seems to be a related work: [1] Perello-Nieto et al., Background Check: A general technique to build more reliable and versatile classifiers, ICDM 2016, Where authors constructed a classifier, which output K+1 labels and the K+1-th label is the “background noise” label for this classification problem. Is the method in [1] applicable to this paper’s setting? Moreover, [1] did not seem to generate any out of distribution samples. 2. I am not so sure that how the actual out of distribution detection was done (did I miss something here?). Authors repeatedly mentioned “maximum prediction values”, but it was not defined throughout the paper. Algorithm 1. is called “minimization for detection and generating out of distribution (samples)”, but this is only gradient descent, right? I do not see a detection procedure. Given the title also contains “detecting”, I feel authors should write explicitly how the detection is done in the main body.
The article "Contextual Explanation Networks" introduces the class of models which learn the intermediate explanations in order to make final predictions. The contexts can be learned by, in principle, any model including neural networks, while the final predictions are supposed to be made by some simple models like linear ones. The probabilistic model allows for the simultaneous training of explanation and prediction parts as opposed to some recent post-hoc methods. The experimental part of the paper considers variety of experiments, including classification on MNIST, CIFAR-10, IMDB and also some experiments on survival analysis. I should note, that the quality of the algorithm is in general similar to other methods considered (as expected). However, while in some cases the CEN algorithm is slightly better, in other cases it appears to sufficiently loose, see for example left part of Figure 3(b) for MNIST data set. It would be interesting to know the explanation. Also, it would be interesting to have more examples of qualitative analysis to see, that the learned explanations are really useful. I am a bit worried, that while we have interpretability with respect to intermediate features, these features theirselves might be very hard to interpret. To sum up, I think that the general idea looks very natural and the results are quite supportive. However, I don't feel myself confident enough in this area of research to make strong conclusion on the quality of the paper.
In general I find this to be a good paper and vote for acceptance.  The paper is well-written and easy to follow.   The proposed approach is a useful addition to existing literature. Besides that I have not much to say except one point I would like to discuss:  In 4.2 I am not fully convinced of using an adversial model for goal generation.  RL algorithms generally suffer from poor stability   and GANs themselves can have convergence issues.  This imposes another layer of possible instability.  Besides, generating useful reward function, while not trivial, can be seen as easier than solving the full RL problem.  Can the authors argue why this model class was chosen over other, more simple, generative models?   Furthermore, did the authors do experiments with simpler models? Related: "We found that the LSGAN works better than other forms of GAN for our problem."  Was this improvement minor, or major, or didn't even work with other GAN types?  This question is important,  because for me the big question is if this model is universal and stable in a lot of applications or requires careful fine-tuning and monitoring.
Summary ======== The authors present CLEVER, an algorithm which consists in evaluating the (local) Lipschitz constant of a trained network around a data point.  This is used to compute a lower-bound on the minimal perturbation of the data point needed to fool the network. The method proposed in the paper already exists for classical function,  they only transpose it to neural networks.  Moreover, the lower bound comes from basic results in the analysis of Lipschitz continuous functions. Clarity ===== The paper is clear and well-written. Originality ========= This idea is not new:  if we search for "Lipschitz constant estimation" in google scholar, we get for example Wood, G. R., and B. P. Zhang. "Estimation of the Lipschitz constant of a function." (1996) which presents a similar algorithm (i.e., estimation of the maximum slope with reverse Weibull). Technical quality ============== The main theoretical result in the paper is the analysis of the lower-bound on \delta, the smallest perturbation to apply on a data point to fool the network.  This result is obtained almost directly by writing the bound on Lipschitz-continuous function | f(y)-f(x) | < L || y-x || where x = x_0 and y = x_0 + \delta. Comments: - Lemma 3.1: why citing Paulavicius and Zilinskas for the definition of Lipschitz continuity?  Moreover, a Lipschitz-continuous function does not need to be differentiable at all (e.g. |x| is Lipschitz with constant 1 but sharp at x=0).  Indeed, this constant can be easier obtained if the gradient exists,  but this is not a requirement. - (Flaw?) Theorem 3.2 : This theorem works for fixed target-class  since g = f_c - f_j for fixed g.  However, once g = min_j f_c - f_j, this theorem is not clear with the constant Lq.  Indeed, the function g should be  g(x) = min_{k \neq c} f_c(x) - f_k(x). Thus its Lipschitz constant is different, potentially equal to L_q = max_{k} \| L_q^k \|, where L_q^k is the Lipschitz constant of f_c-f_k.  If the theorem remains unchanged after this modification, you should clarify the proof.  Otherwise, the theorem will work with the maximum over all Lipschitz constants but the theoretical result will be weakened. - Theorem 4.1: I do not see the purpose of this result in this paper.  This should be better motivated. Numerical experiments ==================== Globally, the numerical experiments are in favor of the presented method.  The authors should also add information about the time it takes to compute the bound, the evolution of the bound in function of the number of samples and the distribution of the relative gap between the lower-bound and the best adversarial example. Moreover, the numerical experiments look to be realized in the context of targeted attack.  To show the real effectiveness of the approach, the authors should also show the effectiveness of the lower-bound in the context of non-targeted attack.
The paper presents a method to parametrize unitary matrices in an RNN as a Kronecker product of smaller matrices.  Given N inputs and output, this method allows one to specify a linear transformation with O(log(N)) parameters, and perform a forward and backward pass in O(Nlog(N)) time.   In addition a relaxation is performed allowing each constituent to deviate a bit from unitarity (“soft unitary constraint”). The paper shows nice results on a number of small tasks.  The idea is original to the best of my knowledge and is presented clearly. I especially like the idea of “soft unitary constraint” which can be applied very efficiently in this factorized setup.  I think this is the main contribution of this work. However the paper in its current form has a number of problems: - The authors state that a major shortcoming of previous (efficient) unitary RNN methods is the lack of ability to span the entire space of unitary matrices.  This method presents a family that can span the entire space, but the efficient parts of this family (which give the promised speedup) only span a tiny fraction of it,  as they require only O(log(N)) params to specify an O(N^2) unitary matrix.  Indeed in the experimental section only those members are tested. - Another claim that is made is that complex numbers are key, and again the argument is the need to span the entire space of unitary matrices,  but the same comment still hold - that is not the space this work is really dealing with,  and no experimental evidence is provided that using complex numbers was really needed. - In the experimental section an emphasis is made as to how small the number of recurrent params are,  but at the same time the input/output projections are very large, leaving the reader wondering if the workload simply shifted from the RNN to the projections.  This needs to be addressed. - Another aspect of the previous points is that it’s not clear if stacking KRU layers will work well.  This is important  as stacking LSTMs is a common practice.  Efficient KRU span a restricted subspace whose elements might not compose into structures that are expressive enough.  One way to overcome this potential problem is to add projection matrices between layers that will do some mixing,  but this will blow the number of parameters.  This needs to be explored. - The authors claim that the soft unitary constraint was key for the success of the network,  yet no details are provided as to how this constraint was applied,  and no analysis was made for its significance.
The paper solves the problem of how to do autonomous resets,  which is an important problem in real world RL.  The method is novel,  the explanation is clear,  and has good experimental results. Pros: 1. The approach is simple, solves a task of practical importance, and performs well in the experiments.  2. The experimental section performs good ablation studies wrt fewer reset thresholds, reset attempts, use of ensembles. Cons: 1. The method is evaluated only for 3 tasks, which are all in simulation, and on no real world tasks.  Additional tasks could be useful, especially for qualitative analysis of the learned reset policies. 2. It seems that while the method does reduce hard resets,  it would be more convincing if it can solve tasks which a model without a reset policy couldnt.  Right now, the methods without the reset policy perform about equally well on final reward. 3. The method wont be applicable to RL environments where we will need to take multiple non-invertible actions to achieve the goal (an analogy would be multiple levels in a game).  In such situations, one might want to use the reset policy to go back to intermediate “start” states from where we can continue again, rather than the original start state always. Conclusion/Significance: The approach is a step in the right direction,  and further refinements can make it a significant contribution to robotics work.
This paper investigates human priors for playing video games. Considering a simple video game, where an agent receives a reward when she completes a game board, this paper starts by stating that: -	Firstly, the humans perform better than an RL agent to complete the game board. -	Secondly, with a simple modification of textures the performances of human players collapse, while those of a RL agent stay the same. If I have no doubts about these results, I have a concern about the method.  In the case of human players the time needed to complete the game is plotted,  and in the case of a RL agent the number of steps needed to complete the game is plotted (fig 1).  Formally, we cannot conclude that one minute is lesser than 4 million of steps. This issue could be easily fixed.  Unfortunately, I have other concerns about the method and the conclusions. For instance, masking where objects are or suppressing visual similarity between similar objects should also deteriorate the performance of a RL agent.  So it cannot be concluded that the change of performances is due to human priors.  In these cases, I think that the change of performances is due to the increased difficulty of the game. The authors have to include RL agent in all their experiments to be able to dissociate what is due to human priors and what is due to the noise introduced in the game.
The paper suggests taking GloVe word vectors, adjust them, and then use a non-Euclidean similarity function between them. The idea is tested on very small data sets (80 and 50 examples, respectively). The proposed techniques are a combination of previously published steps, and the new algorithm fails to reach state-of-the-art on the tiny data sets. It isn't clear what the authors are trying to prove, nor whether they have successfully proven what they are trying to prove. Is the point that GloVe is a bad algorithm? That these steps are general? If the latter, then the experimental results are far weaker than what I would find convincing. Why not try on multiple different word embeddings? What happens if you start with random vectors? What happens when you try a bigger data set or a more complex problem?