launch/ampere huggingface dataset:
This paper proposes a new method to train residual networks in which one starts by training shallow ResNets, doubling the depth and warm starting from the previous smaller model in a certain way, and iterating. The authors relate this idea to a recent dynamical systems view of ResNets in which residual blocks are viewed as taking steps in an Euler discretization of a certain differential equation. This interpretation plays a role in the proposed training method by informing how the “step sizes” in the Euler discretization should change when doubling the depth of the network. The punchline of the paper is that the authors are able to achieve similar performance as “full ResNet training” but with significantly reduced training time. Overall, the proposed method is novel — even though this idea of going from shallow to deep is natural for residual networks, tying the idea to the dynamical systems perspective is elegant. Moreover the paper is clearly written. Experimental results are decent — there are clear speedups to be had based on the authors' experiments. However it is unclear if these gains in training speed are significant enough for people to flock to using this (more complicated) method of training. I only have a few small questions/comments: * A more naive way to do multi-level training would be to again iteratively double the depth, but perhaps not halve the step size. This might be a good baseline to compare against to demonstrate the value of the dynamical systems viewpoint. * One thing I’m unclear on is how convergence was assessed… my understanding is that the training proceeds for a fixed number of epochs (?) - but shouldn’t this also depend on the depth in some way? * Would the speedups be more dramatic for a larger dataset like Imagenet? * Finally, not being very familiar with multigrid methods from the numerical methods literature — I would have liked to hear about whether there are deeper connections to these methods.
This paper proposed a new parametrization scheme for weight matrices in neural network based on the Householder  reflectors to solve the gradient vanishing and exploding problems in training.  The proposed method improved two previous papers: 1) stronger expressive power than Mahammedi et al. (2017), 2) faster gradient update than Vorontsov et al. (2017). The proposed parametrization scheme is natrual from numerical linear algebra point of view and authors did a good job in Section 3 in explaining the corresponding expressive power.  The experimental results also look promising.  It would be nice if the authors can analyze the spectral properties of the saddle points in linear RNN (nonlinear is better but it's too difficult I believe).  If the authors can show the strict saddle properties then as a corollary, (stochastic) gradient descent finds a global minimum.  Overall this is a strong paper  and I recommend to accept.
The paper considers multi-task setting of machine learning. The first contribution of the paper is a novel PAC-Bayesian risk bound. This risk bound serves as an objective function for multi-task machine learning. A second contribution is an algorithm, called LAP, for minimizing a simplified version of this objective function. LAP algorithm uses several training tasks to learn a prior distribution P over hypothesis space. This prior distribution P is then used to find a posterior distribution Q that minimizes the same objective function over the test task. The third contribution is an empirical evaluation of LAP over toy dataset of two clusters and over MNIST. While the paper has the title of "life-long learning", the authors admit that all experiments are in multi-task setting, where the training is done over all tasks simultaneously. The novel risk bound and LAP algorithm can definitely be applied to life-long setting, where training tasks are available sequentially. But since there is no empirical evaluation in this setting, I suggest to adjust the title of the paper. The novel risk bound of the paper is an extension of the bound from [Pentina & Lampert, ICML 2014]. The extension seems to be quite significant. Unlike the bound of [Pentina & Lampert, ICML 2014], the new bound allows to re-use many different PAC-Bayesian complexity terms that were published previously. I liked risk bound and optimization sections of the paper. But I was less convinced by the empirical experiments. Since the paper improves the risk bound of [Pentina & Lampert, ICML 2014], I expected to see an empirical comparison of LAP and optimization  algorithm from the latter paper. To make such comparison fair, both optimization algorithms should use the same base algorithm, e.g. ridge regression, as in [Pentina & Lampert, ICML 2014]. Also I suggest to use the datasets from the latter paper. The experiment with multi-task learning over MNIST dataset looks interesting, but it is still a toy experiment. This experiment  will be more convincing with more sophisticated datasets (CIFAR-10, ImageNet) and architectures (e.g. Inception-V4, ResNet). Minor remarks: Section 6, line 4: "Combing" -> "Combining" Page 14, first equation: There should be "=" before the second expectation.
This paper addresses the question of unsupervised clustering with high classification performance. They propose a deep variational autoencoder architecture with categorical latent variables at the deepest layer and propose to train it with modifications of the standard variational approach with reparameterization gradients. The model is tested on a medical imagining dataset where the task is to distinguish healthy from pathological lymphocytes from blood samples. I am not an expert on this particular dataset, but to my eye the results look impressive. They show high sensitivity and high specificity. This paper may be an important contribution to the medical imaging community. My primary concern with the paper is the lack of novelty and relatively little in the way of contributions to the ICLR community. The proposed model is a simple variant on the standard VAE models (see for example the Ladder VAE https://arxiv.org/abs/1602.02282 for deep models with multiple stochastic layers). This would be OK if a thorough evaluation on at least two other datasets showed similar improvements as the lymphocytes dataset. As it stands, it is difficulty for me to assess the value of this model. Minor questions / concerns: - The authors claim in the first paragraph of 3.2 that deterministic mappings lack expressiveness. Would be great to see the paper take this claim seriously and investigate it. - In equation (13) it isn't clear whether you use q_phi to be the discrete mass or the concrete density. The distinction is discussed in https://arxiv.org/abs/1611.00712 - Would be nice to report the MCC in experimental results.
The authors present a variant of the adversarial feature learning (AFL) approach by Edwards & Storkey.  AFL aims to find a data representation that allows to construct a predictive model for target variable Y,  and at the same time prevents to build a predictor for sensitive variable S.  The key idea is to solve a minimax problem where the log-likelihood of a model predicting Y is maximized, and the log-likelihood of an adversarial model predicting S is minimized.  The authors suggest the use of multiple adversarial models, which can be interpreted as using an ensemble model instead of a single model. The way the log-likelihoods of the multiple adversarial models are aggregated does not yield a probability distribution as stated in Eq. 2.  While there is no requirement to have a distribution here  - a simple loss term is sufficient  - the scale of this term differs compared to calibrated log-likelihoods coming from a single adversary.  Hence, lambda in Eq. 3 may need to be chosen differently depending on the adversarial model.  Without tuning lambda for each method, the empirical experiments seem unfair.  This may also explain why, for example, the baseline method with one adversary effectively fails for Opp-L.  A better comparison would be to plot the performance of the predictor of S against the performance of Y for varying lambdas.  The area under this curve allows much better to compare the various methods. There are little theoretical contributions.  Basically, instead of a single adversarial model - e.g., a single-layer NN or a multi-layer NN - the authors propose to train multiple adversarial models on different views of the data.  An alternative interpretation is to use an ensemble learner where each learner is trained on a different (overlapping) feature set.  Though, there is no theoretical justification why ensemble learning is expected to better trade-off model capacity and robustness against an adversary.  Tuning the architecture of the single multi-layer NN adversary might be as good? In short, in the current experiments, the trade-off of the predictive performance and the effectiveness of obtaining anonymized representations effectively differs between the compared methods.  This renders the comparison unfair.  Given that there is also no theoretical argument why an ensemble approach is expected to perform better,  I recommend to reject the paper.
Summary: Based on ideas within the context of kernel theory, the authors consider post-training of NNs as an extra training step, which only optimizes the last layer of the network. This additional step makes sure that the embedding, or representation, of the data is used in the best possible way for the considered task (which is also reflected in the experiments). According to the authors, the contributions are the following: 1. Post-training step: keeping the rest of the NN frozen (after training), the method trains the last layer in order to "make sure" that the representation learned is used in the most efficient way. 2. Highlighting connections with kernel techniques and RKHS optimization (like kernel ridge regression). 3. Experimental results. Clarity: The paper is well-written, the main ideas well-clarified. Importance: While the majority of papers nowadays focuses on the representation part (i.e., how we get to \Phi_{L-1}(x)), this paper assumes this is given and proposes how to optimize the weights in the final step of the algorithm. This by itself is not enough to boost the performance universally (e.g., if \Phi_{L-1} is not well-trained, the problem is deeper than training the last layer); however, it proposes an additional step that can be used in most NN architectures. From that front (i.e., proposing to do something different than simply training a NN), I find the paper interesting, that might attract some attention at the conference. On the other hand, to my humble opinion, the experimental results do not show a significant gain in the performances of all networks (esp. Figure 3 and Table 1 are within the range of statistical error). In order to state something like this universally, either one needs to perform experiments with more than just MNIST/CIFAR datasets, or even more preferably, prove that the algorithm performs better. Originality: It would be great to have some more theory (if any) for the post-training step, or investigate more cases, rather than optimizing only the last layer. Comments: 1. I assume the authors focused in the last layer of the NN for simplicity, but is there a reason why one might want to focus only on the last layer? One reason is convexity in W of the problem (2). Any other? 2. Have the authors considered (even in practice only) to include training of the last 2 layers of the NN? The authors state this question in the future direction, but it would make the paper more complete to consider it here.
This article aims at understanding the role played by the different words in a sentence, taking into account their order in the sentence. In sentiment analysis for instance, this capacity is critical to model properly negation. As state-of-the-art approaches rely on LSTM, the authors want to understand which information comes from which gate. After a short remainder regarding LSTM, the authors propose a framework to disambiguate interactions between gates. In order to obtain an analytic formulation of the decomposition, the authors propose to linearize activation functions in the network. In the experiment section, authors compare themselves to a standard logistic regression (based on a bag of words representation). They also check the unigram sentiment scores (without context). The main issue consists in modeling the dynamics inside a sentence (when a negation or a 'used to be' reverses the sentiment). The proposed approach works fine on selected samples. The related work section is entirely focused on deep learning while the experiment section is dedicated to sentiment analysis. This section should be rebalanced. Even if the authors claim that their approach is general, they also show that it fits well the sentiment analysis task in particular. On top of that, a lot of fine-grained sentiment analysis tools has been developed outside deep-learning: the authors should refer to those works. Finally, authors should provide some quantitative analysis on sentiment classification: a lot of standard benchmarks are widely use in the literature and we need to see how the proposed method performs with respect to the state-of-the-art. Given the chosen tasks, this work should be compared to the beermind system: http://deepx.ucsd.edu/#/home/beermind and the associated publication http://arxiv.org/pdf/1511.03683.pdf
This should be the first work which introduces in the causal structure into the GAN, to solve the label dependency problem.  The idea is interesting and insightful.  The proposed method is theoretically analyzed and experimentally tested.   Two minor concerns are 1) what is the relationship between the anti-labeler and and discriminator?  2) how the tune related weight of the different objective functions.
Strengths: * Very simple approach, amounting to coupled training of "e" identical copies  of a chosen net architecture, whose predictions are fused during training.  This forces the different model instances to become more complementary. * Perhaps counterintuitively, experiments also show that coupled ensembling leads to individual nets that perform better than those produced by separate training. * The practical advantages of the proposed approach are twofold: 1. Given a fixed parameter budget, coupled ensembling leads to better accuracy than a single net or an ensemble of disjointly-trained nets. 2. For the same accuracy, coupled ensembling yields significant parameter savings. Weaknesses: * Although results are very strong,  the proposed models do not outperform the state-of-the-art, except for the models reported in Table 4,  which however were obtained by *traditional* ensembling of coupled ensembles.  * Coupled ensembling requires joint training of all nets in the ensemble  and thus is limited by the size of the model that can be fit in memory.  Conversely, traditional ensembling involves separate training of the different instances  and this enables the learning of an arbitrary number of individual nets.  * I am surprised by the results in Table 2,  which suggest that the optimal number of nets in the ensemble is remarkably low (only 3!).  It'd be valuable to understand whether this kind of result holds for other network architectures or whether it is specific to this choice of net. * Strictly speaking it is correct to refer to the individual nets in the ensembles as "branches" and "basic blocks."  Nevertheless, I find the use of these terms confusing in the context of the proposed approach,  since they are commonly used to denote concepts different from those represented here.   I would recommend refraining from using these terms here. Overall, the paper provides limited technical novelty.  Yet, it reveals some interesting empirical findings about the benefits of coordinated training of models in an ensemble.
The authors combine an ensemble of DNNs as model for the dynamics with TRPO.  The ensemble is used in two steps: First to collect imaginary roll-outs for TRPO and secondly to estimate convergence of the algorithm.  The experiments indicate superior performance over the baselines. The paper is well-written  and the experiments indicate good results.  However, idea of using ensembles in the context of (model-based) RL  is not novel,  and it comes at the cost of time complexity.   Therefore, the method should utilize the advantage an ensemble provides to its full extent.  The main strength of an ensemble is to provide lower test error, but also some from of uncertainty estimate given by the spread of the predictions.  The authors mainly utilize the first, but to a lesser extent the second advantage (the imaginary roll-outs will  utilize the spread to generate possible outcomes).  Ideally the exploration should also be guided by the uncertainty (such as VIME). Related, what where the arguments in favor of an ensemble compared to Bayesian neural networks (possibly even as simple as using MH-dropout)?  BNNs provide a stronger theoretical justification that the predictive uncertainty is meaningful. Can the authors comment on the time-complexity of the proposed methods compared to the baselines?  In Fig. 2  the x-axis is the time step of the  real data.  But I assume it took a different amount of time for each method to reach step t.  The same argument can be made for Fig. 4.  It seems here that in snake the larger ensembles reach convergence the quickest,  but I expect this effect to be reversed when considering actual training time. In total I think this paper can provide a useful addition to the literature.   However, the proposed approach does not have strong novelty  and I am not fully convinced if the additional burden on time complexity outweighs the improved performance. Minor:  In Sec. 2: "Both of these approaches assume a fixed dataset of samples which are collected before the algorithm starts operating."   This is incorrect,  while these methods consider the domain of fixed datasets,  the algorithms themselves are not limited to this context.
Summary: The paper focuses on the characterization of the landscape of deep neural networks; i.e., when and why local minima are global, what are the conditions for saddle critical points, etc.  The paper covers a somewhat wide range of deep nets (from shallow with linear activation to deeper with non-linear activation);  it focuces only on feed forward neural networks. As the authors state, this paper provides a unifying perspective to the subject  (it justifies the results of others through this unifying theory,  but also provides new results; e.g., there are results that do not depend on assumptions on the target data matrix Y). Originality: The paper provides similar results to previous work, while removing some of the assumptions made in previous work.  In that sense, the originality of the results is weak,  but definitely there is some novelty in the methodology used to get to these results.  Thus, I would say original. Importance: The paper deals with the important problem of when and why training algorithms might get to global/local/saddle critical points.  While there are no direct connections with generalization properties,  characterizing the landscape of neural networks is an important topic to make further steps into better understanding of deep learning.  It will attract some attention at the conference. Clarity: The paper is well-written  - some parts need improvement,  but overall I'm satisfied with the current version. Comments: 1. If problem (4) is not considered at all in this paper (in its full generality that considers matrix completion and matrix sensing as special cases), then the authors could just start with the model in (5). 2. Remark 1 has a nice example  - could this example be shown with Y not being the all-zeros vector? 3. In section 5, the authors make a connection with the work of Ge et al. 2016.  They state that the problems in (10)-(11) constitute generalizations of the symmetric matrix completion case, considered in Ge et al. 2016.  However, in that work, the main difficulty of proving global optimality comes from the randomness of the sampling mask operator (which introduces the notion of incoherence and requires results in expectation).  It is not clear, and maybe it is an overstatement, that the results in section 5 generalize that work.  If that is the case, could the authors describe this a bit further?
The authors propose a new regularization term modifying the VAE (Kingma et al 2013) objective to encourage learning disentangling representations. Specifically, the authors suggest to add penalization to ELBO in the form of -KL(q(z)||p(z)) , which encourages a more global criterion than the local ELBOs. In practice, the authors decide that the objective they want to optimize is unwieldy and resort to moment matching of covariances of q(z) and p(z) via gradient descent. The final objective uses a persistent estimate of the covariance matrix of q and upgrades it at each mini-batch to perform learning. The authors use this objective function to perform experiments measuring disentanglement and find minor benefits compared to other objectives in quantitative terms. Comments: 1. The originally proposed modification in Equation (4) appears to be rigorous and as far as I can tell still poses a lower bound to log(p(x)). The proof could use the result posed earlier: KL(q(z)||p(z)) is smaller than E_x KL(q(z|x)||p(z|x)). 2. The proposed moment matching scheme performing decorrelation resembles approaches for variational PCA and especially independent component analysis. The relationship to these techniques is not discussed adequately. In addition, this paper could really benefit from an empirical figure of the marginal statistics of z under the different regularizers in order to establish what type of structure is being imposed here and what it results in. 3. The resulting regularizer with the decorrelation terms could be studied as a modeling choice. In the probabilistic sense, regularizers can be seen as structural and prior assumptions on variables. As it stands, it is unnecessarily vague which assumptions this extra regularizer is making on variables. 4. Why is using the objective in Equation (4) not tried and tested and compared to? It could be thought that subsampling would be enough to evaluate this extra KL term without any need for additional variational parameters \psi. The reason for switching to the moment matching scheme seems not well motivated here without showing explicitly that Eq (4) has problems. 5. The model seems to be making on minor progress in its stated goal, disentanglement. It would be more convincing to clarify the structural properties of this regularizer in a statistical sense more clearly given that experimentally it seems to only have a minor effect. 6. Is there a relationship to NICE (Laurent Dinh et al)? 7. The infogan is also an obvious point of reference and comparison here. 8. The authors claim that there are no models which can combine GANs with inference in a satisfactory way, which is obviously not accurate nowadays given the progress on literature combining GANs and variational inference. All in all I find this paper interesting but would hope that a more careful technical justification and derivation of the model would be presented given that it seems to not be an empirically overwhelming change.
This is a well-written paper with sound experiments.  However, the research outcome is not very surprising.  - Only macro-average F-scores are reported.  Please present micro-average scores as well. - The detailed procedure of relation extraction should be described.  How do you use entity type information?  (Probably, you did not use entity types.) - Table 3: The SotA score of EU-ATR target-disease (i.e. 84.6) should be in bold face. - Section 5.3: Your system scorers in Table 3 are not consistent with Table 2 scores.  - Page 8. "Our approach outperforms ..."  The improvement is clear only for SNPPhenA and EU-ADR durg-disease. Minor comments: - TreeLSTM --> Tree-LSTM - Page 7. connexion --> connection - Page 8. four EU-ADR subtasks --> three ...  - I suggest to conduct transfer learning studies in the similar settings.