launch/ampere huggingface dataset:
This paper studies how the variance of the discriminator affect the gradient signal provided to the generator and therefore how it might limit its ability to learn the true data distribution. The approach suggested in this paper models the output of the discriminator using a mixture of two Gaussians (one for “fake” and the other for “not fake”).  This seems like a rather crude approximation as the distribution of each “class” is likely to be multimodal.  Can the authors comment on this?  Could they extend their approach to use a mixture of multimodal distributions? The paper mentions that fixing the means of the distribution can be “problematic during optimization as the discriminator’s goal is to maximize the difference between these two means.“.  This relates to my previous comment where the distribution might not be unimodal.  In this case, shifting the mean doesn’t seem to be a good solution and might just yield to oscillations between different modes.  Can you please comment on this? Mode collapse: Can you comment on the behavior of your approach w.r.t. to mode collapse? Implementation details: How is the mean of the two Gaussians initialized?  Relation to instance noise and regularization techniques: Instance noise is a common trick being used to train GANs,  see e.g. http://www.inference.vc/instance-noise-a-trick-for-stabilising-gan-training/ This also relates to some regularization techniques, e.g. Roth et al., 2017 that provides a regularizer that amounts to convolving the densities with white Gaussian noise.  Can you please elaborate on the potential advantages of the proposed solution over these existing techniques? Comparison to existing baselines: Given that the paper addresses the stability problem, I would expect some empirical comparison to at least one or two of the stability methods cited in the introduction, e.g. Gulrajani et al., 2017 or Roth et al., 2017. Relation to Kernel MMD: Can the authors elaborate on how their method relates to approaches that replace the discriminator with MMD nets. e.g. - Training generative neural networks via Maximum Mean Discrepancy optimization, Dziugaite et al - Generative models and model criticism via optimized maximum mean discrepancy, Sutherland et al More explicitly, the variance in these methods can be controlled via the bandwidth of the kernel  and I therefore wonder what would one use a simple mixture of Gaussians instead?
The claimed results of  "combining transformations" in the context of RC was done in the works of Herbert Jaeger on conceptors [1],  which also should be cited here. The argument of biological plausibility is not justified.  The authors  use an echo-state neural network with standard tanh activations, which is as far away from real neuronal signal processing than  ordinary RNNs used in the field, with the difference that the recurrent weights are not trained.   If the authors want to make the case of biological plausibility, they should use spiking neural networks. The experiment on MNIST seems artificial, in particular transforming the image into a time-series and thereby imposing an artificial temporal structure.  The assumption that column_i is obtained  by information  of column_{i-k},..,column_{i-1} is not true for images.  To make a point, the authors should use a datasets with related sets of time-series data, e.g EEG or NLP data. In total this paper does not have enough novelty for acceptance  and the experiments are not well chosen for this kind of work.  Also the authors overstate the claim of biological plausibility  (just because we don't train the recurrent weights does not make a method biologically plausible). [1] H. Jaeger (2014): Controlling Recurrent Neural Networks by Conceptors. Jacobs University technical report Nr 31 (195 pages)
Let me first note that I am not very familiar with the literature on program generation, molecule design or compiler theory,  which this paper draws heavily from,  so my review is an educated guess.  This paper proposes to include additional constraints into a VAE which generates discrete sequences, namely constraints enforcing both semantic and syntactic validity.  This is an extension to the Grammar VAE of Kusner et. al, which includes syntactic constraints but not semantic ones. These semantic constraints are formalized in the form of an attribute grammar, which is provided in addition to the context-free grammar. The authors evaluate their methods on two tasks, program generation and molecule generation.  Their method makes use of additional prior knowledge of semantics,  which seems task-specific and limits the generality of their model.  They report that their method outperforms the Character VAE (CVAE) and Grammar VAE (GVAE) of Kusner et. al.   However, it isn't clear whether the comparison is appropriate:  the authors report in the appendix that they use the kekulised version of the Zinc dataset of Kusner et. al,  whereas Kusner et. al do not make any mention of this.  The baselines they compare against for CVAE and GVAE in Table 1 are taken directly from Kusner et. al though.  Can the authors clarify whether the different methods they compare in Table 1 are all run on the same dataset format? Typos: - Page 5: "while in sampling procedure" -> "while in the sampling procedure" - Page 6: "a deep convolution neural networks" -> "a deep convolutional neural network" - Page 6: "KL-divergence that proposed in" -> "KL-divergence that was proposed in"  - Page 6: "since in training time" -> "since at training time" - Page 6: "can effectively computed" -> "can effectively be computed" - Page 7: "reset for training" -> "rest for training"
This paper investigates the effect of replacing identity skip connections with trainable convolutional skip connections in ResNet.  The authors find that in their experiments, performance improves.  Therefore, the power of skip connections is due to their linearity rather than due to the fact that they represent the identity. Overall, the paper has a clear and simple message and is very readable.  The paper contains a good amount of experiments,  but in my opinion not quite enough to conclude that identity skip connections are inherently worse.  The question is then: how non-trivial is it that tandem networks work?  For someone who understands and has worked with ResNet and similar architectures, this is not a surprise.  Therefore, the paper is somewhat marginal but, I think, still worth accepting. Why did you choose a single learning rate for all architectures and datasets instead of choosing the optimal one for each archtitecture and dataset?  Was it a question of computational resources?  Using custom step sizes would strenghten your experimental results significantly.  In the absence of this, I would still ask that you create an appendix where you specify exactly how hyperparameters were chosen. Other comments:- "and that it’s easier for a layer to learn from a starting point of keeping things the same (the identity map) than from the zero map" I don't understand this comment.  Networks without skip connections are not initialized to the zero map but have nonzero, usually Gaussian, weights. - in section 2, reason (ii), you seem to imply that it is a good thing if a network behaves as an ensemble of shallower networks.  In general, this is a bad thing.  Therefore, the fact that ResNet with tandom networks is an ensemble of shallower networks is a reason for why it might perform badly, not well.  I would suggest removing reason (ii). - in section 3, reason (iii), you state that removing nonlinearities from the skip path can improve performance.  However, using tandom blocks instead of identity skip connections does not change the number of nonlinearity layers.  Therefore, I do not see how reason (iii) applies to tandem networks. - "The best blocks in each challenge were competitive with the best published results for their numbers of parameters; see Table 2 for the breakdown."  What are the best published results?  I do not see them in table 2.
This paper presents a very thorough empirical exploration of the qualities and limitations of very simple word-embedding based models.  Average and/or max pooling over word embeddings (which are initialized from pretrained embeddings) is used to obtain a fixed-length representation for natural language sequences, which is then fed through a single layer MLP classifier.  In many of the 9 evaluation tasks, this approach is found to match or outperform single-layer CNNs or RNNs. The varied findings are very clearly presented and helpfully summarized,  and for each task setting the authors perform an insightful analysis. My only criticism would be the fact that the study is limited to English,  even though the conclusions are explicitly scoped in light of this.  Moreover, I wonder how well the findings would hold in a setting with a more severe OOV problem than is perhaps present in the studied datasets. Besides concluding from the presented results that these SWEMs should be considered a strong baseline in future work,  one might also conclude that we need more challenging datasets! Minor things:- It wasn't entirely clear how the text matching tasks are encoded.  Are the two sequences combined into a single sequence before applying the model, or something else?  I might have missed this detail. - Given the two ways of using the Glove embeddings for initialization (direct update vs mapping them with an MLP into the task space), it would be helpful to know which one ended up being used (i.e. optimal) in each setting. - Something went wrong with the font size for the remainder of the text near Figure 1.
This paper proposes an optimization problem whose Lagrangian duals contain many existing objective functions for generative models.  Using this framework, the paper tries to generalize the optimization problems by defining computationally-tractable family which can be expressed in terms of existing objective functions.  The paper has interesting elements  and the results are original.  The main issue is that the significance is unclear.  The writing in Section 3 is unclear for me,  which further made it challenging to understand the consequences of the theorems presented in that section.  Here is a big-picture question that I would like to know answer for.  Do the results of sec 3 help us identify a more useful/computationally tractable model than exiting approaches?  Clarification on this will help me evaluate the significance of the paper. I have three main clarification points.  First, what is the importance of T1, T2, and T3 classes defined in Def. 7, i.e., why are these classes useful in solving some problems?  Second, is the opposite relationship in Theorem 1, 2, and 3 true as well, e.g., is every linear combination of beta-ELBO and VMI is equivalent to a likelihood-based computable-objective of KL info-encoding family?  Is the same true for other theorems? Third, the objective of section 3 is to show that "only some choices of lambda lead to a dual with a tractable equivalent form".  Could you rewrite the theorems so that they truly reflect this, rather than stating something which only indirectly imply the main claim of the paper. Some small comments:- Eq. 4. It might help to define MI to remind readers. - After Eq. 7, please add a proof (may be in the Appendix).  It is not that straightforward to see this.  Also, I suppose you are saying Eq. 3 but with f from Eq. 4. - Line after Eq. 8, D_i is "one" of the following...  Is it always the same D_i for all i or it could be different?  Make this more clear to avoid confusion. - Last line in Para after Eq. 15, "This neutrality corresponds to the observations made in.."  It might be useful to add a line explaining that particular "observation" - Def. 7, the names did not make much sense to me.  You can add a line explaining why this name is chosen. - Def. 8, the last equation is unclear.  Does the first equivalence impy the next one?  - Writing in Sec. 3.3 can be improved.  e.g., "all linear operations on log prob." is very unclear,  "stated computational constraints" which constraints?
The authors tackle the problem of estimating risk in a survival analysis setting with competing risks. They propose directly optimizing the time-dependent discrimination index using a siamese survival network. Experiments on several real-world dataset reveal modest gains in comparison with the state of the art. - The authors should clearly highlight what is their main technical contribution. For example, Eqs. 1-6 appear to be background material since the time-dependent discrimination index is taken from the literature, as the authors point out earlier. However, this is unclear from the writing. - One of the main motivations of the authors is to propose a model that is specially design to avoid the nonidentifiability issue in an scenario with competing risks. It is unclear why the authors solution is able to solve such an issue, specially given the modest reported gains in comparison with several competitive baselines. In other words, the authors oversell their own work, specially in comparison with the state of the art. - The authors use off-the-shelf siamese networks for their settting and thus it is questionable there is any novelty there. The application/setting may be novel, but not the architecture of choice. - From Eq. 4 to Eq. 5, the authors argue that the denominator does not depend on the model parameters and can be ignored. However, afterwards the objective does combine time-dependent discrimination indices of several competing risks, with different denominator values. This could be problematic if the risks are unbalanced. - The competitive gain of the authors method in comparison with other competing methods is minor. - The authors introduce F(t, D | x) as cumulative incidence function (CDF) at the beginning of section 2, however, afterwards they use R^m(t, x), which they define as risk of the subject experiencing event m before t. Is the latter a proxy for the former? How are they related?
This paper addresses multiple issues arising from the fact that commonly reported best model performance numbers are a single sample from a performance distribution.  These problems are very real,  and they deserve significant attention from the ML community.   However, I feel that the proposed solution may actually compound the issues highlighted. Firstly, the proposed metric requires calculation of multiple test set experiments for every evaluation.  In the paper up to 100 experiments were used.  This may be reasonable in scenarios where the test set is hidden, and individual test numbers are never revealed.  It also may be reasonable if we cynically assume that researchers are already running many test-set evaluations.  But I am very opposed to any suggestion that we should relax the maxim that the test set should be used only once, or as close to once as is possible.  Even the idea of researchers knowing their test set variance makes me very uneasy. Secondly, this paper tries to account for variation in results due to different degrees of hyper-parameter tuning.  This is certainly an admirable aim,  since different research groups have access to very different types of resources.  However, the suggested approach relies on randomly picking hyper-parameters from "a range that we previously found to work reasonably well".  This randomization does not account for the many experiments that were required to find this range.  And the randomization is also not extended to parameters controlling the model architecture  (I suspect that a number of experiments went into picking the 32 layers in the ResNet used by this paper).  Without a solid and consistent basis for these hyper-parameter perturbations, I worry that this approach will fail to normalize the effect of experiment numbers while also giving researchers an excuse to avoid reporting their experimental process. I think this is a nice idea  and the metric does merge the stability and low variance of mean score with the aspirations of best score.  The metric may be very useful at development time in helping researchers build a reasonable expectation of test time performance in cases where the dev and test sets are strongly correlated.  However, for the reasons outlined above, I don't think the proposed approach solves the problems that it addresses.  Ultimately, the decision about this paper is a subjective one.  Are we willing to increase the risk of inadvertent hyper-parameter tuning on the test set for the sake of a more stable metric?
This paper misses the point of what VAEs (or GANs, in general) are used for. The idea of using VAEs is not to encode and decode images (or in general any input), but to recover the generating process that created those images so we have an unlimited source of samples. The use of these techniques for compressing is still unclear and their quality today is too low. So the attack that the authors are proposing does not make sense and my take is that we should see significant changes before they can make sense. But let’s assume that at some point they can be used as the authors propose. In which one person encodes an image, send the latent variable to a friend, but a foe intercepts it on the way and tampers with it so the receiver recovers the wrong image without knowing. Now if the sender believes the sample can be tampered with, if the sender codes z with his private key would not make the attack useless? I think this will make the first attack useless. The other two attacks require that the foe is inserted in the middle of the training of the VAE. This is even less doable, because the encoder and decoder are not train remotely. They are train of the same machine or cluster in a controlled manner by the person that would use the system. Once it is train it will give away the decoder and keep the encoder for sending information.
This paper studies off-policy learning in the bandit setting.  It develops a new learning objective where the empirical risk is regularized by the squared Chi-2 divergence between the new and old policy.  This objective is motivated by a bound on the empirical risk, where this divergence appears.  The authors propose to solve this objective by using generative adversarial networks for variational divergence minimization (f-GAN).  The algorithm is then evaluated on settings derived from supervised learning tasks and compared to other algorithms. I find the paper well written and clear.  I like that the proposed method is both supported by theory and empirical results.  Minor point: I do not really agree with the discussion on the impact of the stochasticity of the logging policy in section 5.6.  Based on Figure 5 a and b, it seems that the learned policy is performing equally well no matter how stochastic the logging policy is.  So I find it a bit misleading to suggest that the learned policy are not being improved when the logging policy is more deterministic.  Rather, the gap reduces between the two policies  because the logging policy gets better.  In order to better showcase this mechanism, perhaps you could try using a logging policy that does not favor the best action. quality and clarity: ++ code made available + well written and clear - The proof of theorem 2 is not in the paper nor appendix  (the authors say it is similar to another work). originality + good extension of the work by Swaminathan & Joachims (2015a): derivation of an alternative objective and use of a deep networks.  This paper leverages a set of diverse results significance - The proposed method can only be applied if propensity scores were recorded when the data was generated. - no test on a real setting ++ The proposed method is supported both by theoretical insights and empirical experiments. + empirical improvement with respect to previous methods details/typos: 3.1, p3: R^(h) has an indexed parenthesis 5.2; and we more details 5.3: so that results more comparable
Clearly presented paper, including a number of reasonable techniques to improve LSTM-LMs. The proposed techniques are heuristic, but are reasonable and appear to yield improvements in perplexity. Some specific comments follow. re. "ASGD" for Averaged SGD: ASGD usually stands for Asynchronous SGD, have the authors considered an alternative acronym? AvSGD? re. Optimization criterion on page 2, note that SGD is usually taken to minimizing expected loss, not just empirical loss (Bottou thesis 1991). Is there any theoretical analysis of convergence for Averaged SGD? re. paragraph starting with "To prevent such inefficient data usage, we randomly select the sequence length for the forward and backward pass in two steps": the explanation is a bit unclear. What is the "base sequence length" exactly? Also, re. the motivation above this paragraph, I'm not sure what "elements" really refers to, though I can guess. What is the number of training tokens of the datasets used, PTB and WT2? Can the authors provide more explanation for what "neural cache models" are, and how they relate to "pointer models"? Why do the sections "Pointer models", "Ablation analysis", and "AWD-QRNN" come after the Experiments section?
The authors present a neural embedding technique using a hyperbolic space. The idea of embedding data into a space that is not Euclidean is not new. There have been attempts to project onto (hyper)spheres. Also, the proposal bears some resemblance with what is done in t-SNE, where an (exponential) distortion of distances is induced.  Discussing this potential similarity would certainly broaden the readership of the paper. The organisation of the paper might be improved, with a clearer red line and fewer digressions. The call to the very small appendix via eq. 17 is an example. The position of Table in the paper is odd as well. The order of examples in Fig.5 differs from the order in the list. The experiments are well illustrative but rather small sized. The qualitative assessment is always interesting  and it is completed with some label prediction task. Due the geometrical consideretations developed in the paper, other quality criteria like e.g. how well neighbourhoods are preserved in the embeddings would give some more insights. All in all the idea developed in the paper sounds interesting  but the paper organisation seems a bit loose  and additional aspects should be investigated.