launch/ampere huggingface dataset:
This paper proposes to use Cross-Corpus training for biomedical relationship extraction from text.   - Many wording issues, like citation formats, grammar mistakes, missing words, e.g., Page 2: it as been - The description of the methods should be improved.  For instance, why the input has only two entities?  In many biomedical sentences, there are more than two entities.  How can the proposed two models handle these cases?  - The paper just presents to train on a larger labeled corpus and test on a task with a smaller labeled set.  Why is this novel?  Nothing is novel in the deep models (CNN and TreeLSTM).   - Missing refs, like: A simple neural network module for relational reasoning, Arxiv 2017
The paper addresses the problem of learning the form of the activation functions in neural networks. The authors propose to place Gaussian process (GP) priors on the functional form of each activation function (each associated with a hidden layer and unit) in the neural net. This  somehow allows to non-parametrically infer from the data the "shape" of the activation functions needed for a specific problem. The paper then proposes an inference framework (to approximately marginalize out all GP functions)  based on sparse GP methods that use inducing points and variational inference. The inducing point approximation used here is very efficient since all GP functions depend on a scalar input (as any activation function!) and therefore by just placing the inducing points in a dense grid gives a fast and accurate representation/compression of all GPs in terms of the inducing function values (denoted by U in the paper). Of course then inference involves approximating the finite posterior over inducing function values U and the paper make use of the standard Gaussian approximations. In general I like the idea and I believe that it can lead to a very useful model. However, I have found the current paper quite preliminary and incomplete. The authors need to address the following: First (very important): You need to show experimentally how your method compares against regular neural nets (with specific fixed forms for their activation functions such relus etc). At the moment in the last section you mention "We have validated networks of Gaussian Process Neurons in a set of experiments, the details of which we submit in a subsequent publication. In those experiments, our model shows to be significantly less prone to overfitting than a traditional feed-forward network of same size, despite having more parameters." ===>  Well all this needs to be included in the same paper. Secondly: Discuss the connection with Deep GPs (Damianou and Lawrence 2013). Your method seems to be connected with Deep GPs although there appear to be important differences as well. E.g. you place GPs on the scalar activation functions in an otherwise  heavily parametrized neural network (having interconnection weights between layers) while deep GPs model the full hidden layer mapping as a single GP (which does not require interconnection weights). Thirdly:  You need to better explain the propagation of uncertainly in section 3.2.2  and the central limit of distribution in section 3.2.1. This is the technical part of your paper which is a non-standard approximation. I will suggest to give a better intuition of the whole idea and move a lot of mathematical details to the appendix.
This paper proposes to adapt convnet representations to new tasks while avoiding catastrophic forgetting by learning a per-task “controller” specifying weightings of the convolution-al filters throughout the network while keeping the filters themselves fixed. Pros The proposed approach is novel and broadly applicable. By definition it maintains the exact performance on the original task, and enables the network to transfer to new tasks using a controller with a small number of parameters (asymptotically smaller than that of the base network). The method is tested on a number of datasets (each used as source and target) and shows good transfer learning performance on each one. A number of different fine-tuning regimes are explored. The paper is mostly clear and well-written (though with a few typos that should be fixed). Cons/Questions/Suggestions The distinction between the convolutional and fully-connected layers (called “classifiers”) in the approach description (sec 3) is somewhat arbitrary -- after all, convolutional layers are a generalization of fully-connected layers. (This is hinted at by the mention of fully convolutional networks.) The method could just as easily be applied to learn a task-specific rotation of the fully-connected layer weights. A more systematic set of experiments could compare learning the proposed weightings on the first K layers of the network (for K={0, 1, …, N}) and learning independent weights for the latter N-K layers, but I understand this would be a rather large experimental burden. When discussing the controller initialization (sec 4.3), it’s stated that the diagonal init works the best, and that this means one only needs to learn the diagonals to get the best results. Is this implying that the gradients wrt off-diagonal entries of the controller weight matrix are 0 under the diagonal initialization, hence the off-diagonal entries remain zero after learning? It’s not immediately clear to me whether this is the case -- it could help to clarify this in the text. If the off-diag gradients are indeed 0 under the diag init, it could also make sense to experiment with an “identity+noise” initialization of the controller matrix, which might give the best of both worlds in terms of flexibility and inductive bias to maintain the original representation. (Equivalently, one could treat the controller-weighted filters as a “residual” term on the original filters F with the controller weights W initialized to noise, with the final filters being F+(W\crossF) rather than just W\crossF.) The dataset classifier (sec 4.3.4) could be learnt end-to-end by using a softmax output of the dataset classifier as the alpha weighting. It would be interesting to see how this compares with the hard thresholding method used here. (As an intermediate step, the performance could also be measured with the dataset classifier trained in the same way but used as a soft weighting, rather than the hard version rounding alpha to 0 or 1.) Overall, the paper is clear and the proposed method is sensible, novel, and evaluated reasonably thoroughly.
This paper proposes a method of learning sparse dictionary learning by introducing new types of priors.  Specifically, they designed a novel idea of defining a metric to measure discriminative properties along with the quality of presentations. It is also presented the power of the proposed method in comparison with the existing methods in the literature. Overall, the paper deals with an important issue in dictionary learning and proposes a novel idea of utilizing a set of priors.  To this reviewer’s understanding, the thresholding parameter $\tau_{c}$ is specific for a class $c$ only,  thus different classes have different $\tau$ vectors.  If so, Eq. (6) for approximation of the measure $D(\cdot)$ is not clear how the similarity measure between ${\bf y}_{c,k}$ and ${\bf y}_{c1,k1}$, \ie, $\left\|{\bf y}_{c,k}^{+}\odot{\bf y}_{c1,k1}^{+}\right\|_{1}+\left\|{\bf y}_{c,k}^{+}\odot{\bf y}_{c1,k1}^{+}\right\|_{1}$ and $\left\|{\bf y}_{c,k}\odot{\bf y}_{c1,k1}\right\|_{2}^{2}$, works to approximate it.  It would be appreciated to give more detailed description on it and geometric illustration, if possible. There are many typos and grammatical errors,  which distract from reading and understanding the manuscript.
This paper tackles the task of learning embeddings of multi-relational graphs using a neural network. As much of previous work, the proposed architecture works on triples (h, r, t) wth h, t entities and r the relation type. Despite interesting experimental results, I find that the paper carries too many imprecisions as is. * One of the main originality of the approach is to be able for a given input triple to train by sequentially removing in turn the head h, then the tail t and finally the relation r. (called multi-shot in the paper). However, most (if not all) approaches learning embeddings of multi-relational graphs also create multiple examples given a triple. And that, at least since "Learning Structured Embeddings of Knowledge Bases" by Bordes et al. 2011 that was predicting h and t (not r). The only difference is that here it is done sequentially while most methods sample one case each time. Not really meaningful or at least not proved meaningful here. * The sequential/RNN-like structure is unclear and it is hard to see how it relates to the data. * Writing that the proposed method "unsupervised, which is distinctly different from previous works" is not true or should be rephrased. The only difference comes from that the prediction function (softmax and not ranking for instance) and the loss used. But none of the methods compared in the experiments use more information than GEN (the original graph). GEN is not the only model using a softmax by the way. * The fact of predicting indistinctly a fact or its reverse seems rather worrying to me. Predicting that "John is_father_of Paul" or that "John is_child_of Paul" is not the same..! How is assessed the fact that a prediction is conceptually correct? Using types? * The bottom part of Table 2 is surprising. How come for the task of predicting Head, the model trained only at predicting heads (GEN(t,r => h)) performs worse than the model trained only at predicting tails (GEN(h,r => t))?
This paper proposes a regularization to the softmax layer, which try to make the distribution of feature representation (inputs fed to the softmax layer) more meaningful according to the Euclidean distance. The proposed isotropic loss in equation 3 tries to equalize the squared distances from each point to the mean, so the features are encouraged to lie close to a sphere. Overall, the proposed method is a relatively simple tweak to softmax. The authors show that empirically, features learned under softmax loss + isotropic regularization outperforms other features in Euclidean metric-based tasks. My main concern with this paper is the motivation: what are the practical scenarios in which one would want to used proposed method? 1. It is true that features learned with the pure softmax loss may not presents the ideal  similarity under the  Euclidean metric (e.g. the problem depicted in Figure 1), because they are not trained to do so: their purpose is just to predict the correct label. While the proposed regularization does lead to a nicer Euclidean geometry, there is not sufficient motivation and evidence showing this regularization improves classification accuracy. 2. In table 2, the authors seem to indicate that not using the label information in the definition of Isotropic loss is an advantage. But this does not matter since you already use the labels in the softmax loss. 3. I can not easily think of scenarios in which, we would like to perform KNN in the feature space (Table 3) after training a softmax layer. In fact, Table 3 shows KNN is almost always worse than softmax in terms of classification accuracy. 4. Running kmeans or agglomerative clustering in the feature space (Table 5) *using the Euclidean metric* is again ill-posed, because the softmax layer is not trained to do this. If one really wants good clustering performance, one shall always try to learn a good metric, or , why do not you perform clustering on the softmax output (a probability vector?) 5.  The experiments on adversarial robustness and face verification seems more interesting to me, but the tasks were not carefully explained for someone not familiar with that literature. Perhaps for these tasks, multi-class classification is not the most correct objective, and maybe the proposed regularization can help, but the motivations are not given.
This paper continues a trend of incremental improvements to Wasserstein GANs (WGAN),  where the latter were proposed in order to alleviate the difficulties encountered in training GANs.  Originally, Arjovsky et al.  [1] argued that the Wasserstein distance was superior to many others typically used for GANs.  An important feature of WGANs is the requirement for the discriminator to be 1-Lipschitz,  which [1] achieved simply by clipping the network weights.  Recently, Gulrajani et al. [2] proposed a gradient penalty "encouraging" the discriminator to be 1-Lipschitz.  However, their approach estimated continuity on points between the generated and the real samples,  and thus could fail to guarantee Lipschitz-ness at the early training stages.  The paper under review overcomes this drawback by estimating the continuity on perturbations of the real samples.  Together with various technical improvements, this leads to state-of-the-art practical performance both in terms of generated images and in semi-supervised learning.   In terms of novelty, the paper provides one core conceptual idea followed by several tweaks aimed at improving the practical performance of GANs.  The key conceptual idea is to perturb each data point twice and use a Lipschitz constant to bound the difference in the discriminator’s response on the perturbed points.   The proposed method is used in eq. (6) together with the gradient penalty from [2].  The authors found that directly perturbing the data with Gaussian noise led to inferior results  and therefore propose to perturb the hidden layers using dropout.  For supervised learning they demonstrate less overfitting for both MNIST and CIFAR 10.   They also extend their framework to the semi-supervised setting of Salismans et al 2016 and report improved image generation.  The authors do an excellent comparative job in presenting their experiments.  They compare numerous techniques (e.g., Gaussian noise, dropout) and demonstrates the applicability of the approach for a wide range of tasks.  They use several criteria to evaluate their performance (images, inception score, semi-supervised learning, overfitting, weight histogram) and compare against a wide range of competing papers.  Where the paper could perhaps be slightly improved is writing clarity.  In particular, the discussion of M and M' is vital to the point of the paper,  but could be written in a more transparent manner.  The same goes for the semi-supervised experiment details and the CIFAR-10 augmentation process.  Finally, the title seems uninformative.  Almost all progress is incremental,  and the authors modestly give credit to both [1] and [2],  but the title is neither memorable nor useful in expressing the novel idea.  [1] Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein gan. [2] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans.
Summary: The paper considers second-order optimization methods for training of neural networks. In particular, the contribution of the paper is a Hessian-free method that works on blocks of parameters  (this is a user defined splitting of the parameters in blocks, e.g., parameters of each layer is one block, or parameters in several layers could constitute a block).  This results into a block-diagonal approximation to the curvature matrix, in order to improve Hessian-free convergence properties:  in the latter, a single step might require many CG steps,  so the benefit from using second-order information is not apparent. This is mainly an experimental work,  where the authors show the merits of their approach on deep autoencoders, convolutional networks and LSTMs:  results show favourable performance compared to the original Hessian-free approach and the Adam method. Originality: The paper is based on the works of Collobert (2004) and Le Roux et al. (2008), as well as the work of Martens:  the twist is that each layer of the neural network is considered a parameter block,  so that gradient interactions among weights in a single layer are more useful than those between weights in different layers.  This increases the separability of the problem and reduces the complexity.  Importance: Understanding the difference between first- and second-order methods for NN training is an important topic.  Using second-order methods could be considered at its infancy, compared to the wide variety of first-order methods.  Having new results on second-order methods with interesting results would definitely attract some attention at the conference.  Presentation/Clarity: The paper is well structured and well written.  The authors clearly place their work w.r.t. state of the art and previous works,  so that it is clear what is new and what is known. Comments: 1. It is not clear why the deficiency of first-order methods on training NNs with big batches motivates us to turn into second-order methods.  Is there a reasoning for this statement?  Or is it just because second-order methods are kind-of the only other alternative we have? 2. Assuming we can perform a second-order method, like Newton's method, on a deep NN.  Since originally Newton's method was designed to find solutions that have gradient equal to zero,  and since NNs have saddle points (probably many more than local minima),  even if we could perfectly perform second-order Newton motions, there is no guarantee whether we converge to a local minimum or a saddle point.  However, since we perform Newton's method approximately in practice,  this might help escaping saddle points.  Any comment on this aspect  (I'm not aware whether this is already commented in Schraudolph 2002, where the Gauss-Newton matrix was proposed instead of the Hessian)?
Quality This paper demonstrates that convolutional and relational neural networks fail to solve visual relation problems by training networks on artificially generated visual relation data.  This points at important limitations of current neural network architectures where architectures depend mainly on rote memorization. Clarity The rationale in the paper is straightforward.  I do think that breakdown of networks by testing on increasing image variability is expected given that there is no reason that networks should generalize well to parts of input space that were never encountered before. Originality While others have pointed out limitations before,  this paper considers relational networks for the first time. Significance  This work demonstrates failures of relational networks on relational tasks,  which is an important message.  At the same time, no new architectures are presented to address these limitations. Pros Important message about network limitations. Cons Straightforward testing of network performance on specific visual relation tasks.  No new theory development.  Conclusions drawn by testing on out of sample data may not be completely valid.
This paper introduces a new toolbox for deep neural networks learning and evaluation. The central idea is to include time in the processing of all the units in the network. For this, the authors propose a paradigm switch: form layerwise-sequential networks, where at every time frame the network is evaluated by updating each layer – from bottom to top – sequentially; to layerwise-parallel networks, where all the neurons are updated in parallel. The new paradigm implies that the layer update is achieved by using the stored previous state and the corresponding previous state of the previous layer. This has three consequences. First, every layer now use memory, a condition that already applies for RNNs in layerwise-sequential networks. Second, in order to have a consistent output, the information has to flow in the network for a number of time frames equal to the number of layers. In Neuroscience, this concept is known as reaction time. Third, since the network is not synchronized in terms of the information that is processed in a specific time frame, there are discrepancies w.r.t. the layerwise-sequential networks computation: all the techniques used to train deep NNs have to be reconsidered. Overall, the concept is interesting and timely especially for the rising field of spiking neural networks or for large and distributed architectures. The paper, however, should probably provide more examples and results in terms of architectures that can been implemented with the toolbox in comparison with other toolboxes. The paper presents a single example in which either the accuracy and the training time are not reported. While I understand that the main result of this work is the toolbox itself, more examples and results would improve the clarity and the implications for such paradigm switch. Another concern comes from the choice to use Theano as back-end, since it's known that it is going to be discontinued. Finally I suggest to improve the clarity and description of Figure 2, which is messy and confusing especially if printed in B&W.
The authors propose WAGE, which discretized weights, activations, gradients, and errors at both training and testing time.  By quantization and shifting, SGD training without momentum, and removing the softmax at output layer as well, the model managed to remove all cumbersome computations from every aspect of the model,  thus eliminating the need for a floating point unit completely.  Moreover, by keeping up to 8-bit accuracy, the model performs even better than previously proposed models.  I am eager to see a hardware realization for this method because of its promising results.  The model makes a unified discretization scheme for 4 different kinds of components,  and the accuracy for each of the kind becomes independently adjustable.  This makes the method quite flexible and has the potential to extend to more complicated networks, such as attention or memory.  One caveat is that there seem to be some conflictions in the results shown in Table 1, especially ImageNet.  Given the number of bits each of the WAGE components asked for, a 28.5% top 5 error rate seems even lower than XNOR.  I suspect it is due to the fact that gradients and errors need higher accuracy for real-valued input,  but if that is the case, accuracies on SVHN and CIFAR-10 should also reflect that.  Or, maybe it is due to hyperparameter setting or insufficient training time? Also, dropout seems not conflicting with the discretization.  If there are no other reasons, it would make sense to preserve the dropout in the network as well. In general, the paper was writ ten in good quality and in detail,  I would recommend a clear accept.