launch/ampere huggingface dataset:
This work addresses the scenario of fine-tuning a pre-trained network for new data/tasks and empirically studies various regularization techniques. Overall, the evaluation concludes with recommending that all layers of a network whose weights are directly transferred during fine-tuning should be regularized against the initial net with an L2 penalty during further training. Relationship to prior work: Regularizing a target model against a source model is not a new idea. The authors miss key connections to A-SVM [1] and PMT-SVM [2] -- two proposed transfer learning models applied to SVM weights, but otherwise very much the same as the proposed solution in this paper. Though the study here may offer new insights for deep nets, it is critical to mention prior work which also does analysis of these regularization techniques. Significance: As the majority of visual recognition problems are currently solved using variants of fine-tuning, if the findings reported in this paper generalize, then it could present a simple new regularization which improves the training of new models. The change is both conceptually simple and easy to implement so could be quickly integrated by many people. Clarity and Questions: The purpose of the paper is clear, however, some questions remain unanswered. 1) How is the regularization weight of 0.01 chosen? This is likely a critical parameter. In an experimental paper, I would expect to see a plot of performance for at least one experiment as this regularization weighting parameter is varied. 2) How does the use of L2 regularization on the last layer effect the regularization choice of other layers? What happens if you use no regularization on the last layer? L1 regularization? 3) Figure 1 is difficult to read. Please at least label the test sets on each sub-graph. 4) There seems to be some issue with the freezing experiment in Figure 2. Why does performance of L2 regularization improve as you freeze more and more layers, but is outperformed by un-freezing all. 5) Figure 3 and the discussion of linear dependence with the original model in general seems does not add much to the paper. It is clear that regularizing against the source model weights instead of 0 should result in final weights that are more similar to the initial source weights. I would rather the authors use this space to provide a deeper analysis of why this property should help performance. 6) Initializing with a source model offers a strong starting point so full from scratch learning isn’t necessary -- meaning fewer examples are needed for the continued learning (fine-tuning) phase. In a similar line of reasoning, does regularizing against the source further reduce the number of labeled points needed for fine-tuning? Can you recover L2 fine-tuning performance with fewer examples when you use L2-SP? [1] J. Yang, R. Yan, and A. Hauptmann. Adapting svm classifiers to data with shifted distributions. In ICDM Workshops, 2007. [2]  Y. Aytar and A. Zisserman. Tabula rasa: Model transfer for object category detection. In Proc. ICCV, 2011.
The authors present 3 architectures for learning representations of programs from execution traces.  In the variable trace embedding, the input to the model is given by a sequence of variable values.  The state trace embedding combines embeddings for variable traces using a second recurrent encoder.  The dependency enforcement embedding performs element-wise multiplication of embeddings for parent variables to compute the input of the GRU to compute the new hidden state of a variable.  The authors evaluate their architectures on the task of predicting error patterns for programming assignments from Microsoft DEV204.1X (an introduction to C# offered on edx) and problems on the Microsoft CodeHunt platform.  They additionally use their embeddings to decrease the search time for the Sarfgen program repair system. This is a fairly strong paper.  The proposed models make sense  and the writing is for the most part clear,  though there are a few places where ambiguity arises: - The variable "Evidence" in equation (4) is never defined.  - The authors refer to "predicting the error patterns",  but again don't define what an error pattern is.  The appendix seems to suggest that the authors are simply performing multilabel classification based on a predefined set of classes of errors,  is this correct?  - It is not immediately clear from Figures 3 and 4 that the architectures employed are in fact recurrent. - Figure 5 seems to suggest that dependencies are only enforced at points in a program where assignment is performed for a variable,  is this correct? Assuming that the authors can address these clarity issues, I would in principle be happy for the paper to appear.
This paper proposes two regularization terms to encourage learning disentangled representations.  One term is applied to weight parameters of a layer just like weight decay.  The other is applied to the activations of the target layer (e.g., the penultimate layer).  The core part of both regularization terms is a compound hinge loss of which the input is the KL divergence between two softmax-normalized input arguments.  Experiments demonstrate the proposed regularization terms are helpful in learning representations which significantly facilitate clustering performance. Pros: (1) This paper is clearly written and easy to follow. (2) Authors proposed multiple variants of the regularization term which cover both supervised and unsupervised settings. (3) Authors did a variety of classification experiments ranging from time serials, image and text data. Cons: (1) The design choice of the compound hinge loss is a bit arbitrary.  KL divergence is a natural similarity measure for probability distribution.  However, it seems that authors use softmax to force the weights or the activations of neural networks to be probability distributions just for the purpose of using KL divergence.  Have you compared with other choices of similarity measure, e.g., cosine similarity?  I think the comparison as an additional experiment would help explain the design choice of the proposed function. (2) In the binary classification experiments, it is very strange to almost randomly group several different classes of images into the same category.  I would suggest authors look into datasets where the class hierarchy is already provided, e.g., ImageNet or a combination of several fine-grained image classification datasets. Additionally, I have the following questions: (1) I am curious how the proposed method compares to other competitors in terms of the original classification setting, e.g., 10-class classification accuracy on CIFAR10.  (2) What will happen for the multi-layer loss if the network architecture is very large such that you can not use large batch size, e.g., less than 10?  (3) In drawing figure 2 and 3, if the nonlinear activation function is not ReLU, how would you exam the same behavior?  Have you tried multi-class classification for the case “without proposed loss component” and does the similar pattern still happen or not? Some typos: (1) In introduction, “when the cosine between the vectors 1” should be “when the cosine between the vectors is 1”. (2) In section 4.3, “we used the DBPedia ontology dataset dataset” should be “we used the DBPedia ontology dataset”.  I would like to hear authors’ feedback on the issues I raised.
This paper presents a new reinforcement learning architecture called Reactor by combining various improvements in deep reinforcement learning algorithms and architectures into a single model. The main contributions of the paper are to achieve a better bias-variance trade-off in policy gradient updates, multi-step off-policy updates withdistributional RL, and prioritized experience replay for transition sequences. The different modules are integrated well and the empirical results are very promising. The experiments (though limited to Atari) are well carried out and the evaluation is performed on both sample efficiency and training time. Pros: 1. Nice integration of several recent improvements in deep RL, along with a few novel tricks to improve training. 2. The empirical results on 57 Atari games are impressive, in terms of final scores as well as real-time training speed. Cons: 1. Reactor is still less sample-efficient than Rainbow, with significantly lower scores after 200M frames. While the reactor trains much faster, it does use more parallel compute, so the comparison with Rainbow on wall clock time is  not entirely fair. Would a distributed version of Rainbow perform better in this respect? 2. Empirical comparisons are restricted to the Atari domain. The conclusions of the paper will be much stronger if results are also shown on other environments like Mujoco/Vizdoom/Deepmind Lab. 3. Since the paper introduces a few new ideas like prioritized sequence replay, it would help if a more detailed analysis was performed on the impact of these individual schemes, even if in a model simpler than the Reactor. For instance, one could investigate the impact of prioritized sequence replay in models like multi-step DQN or recurrent DQN. This will help us understand  the impact of each of these ideas in a more comprehensive fashion.
Summary:This paper proposes a simple recipe to preserve proximity to zero mean for activations in deep neural networks. The proposal is to replace the non-linearity in half of the units in each layer with its "bipolar" version -- one that is obtained by flipping the function on both axes. The technique is tested on deep stacks of recurrent layers, and on convolutional networks with depth of 28, showing that improved results over the baseline networks are obtained. Clarity: The paper is easy to read. The plots in Fig. 2 and the appendix are quite helpful in improving presentation. The experimental setups are explained in detail. Quality and significance: The main idea from this paper is simple and intuitive. However, the experiments to support the idea do not seem to match the motivation of the paper. As stated in the beginning of the paper, the motivation behind having close to zero mean activations is that this is expected to speed up training using gradient descent. However, the presented results focus on the performance on held-out data instead of improvements in training speed. This is especially the case for the RNN experiments. For the CIFAR-10 experiment, the training loss curves do show faster initial progress in learning. However, it is unclear that overall training time can be reduced with the help of this technique. To evaluate this speed up effect, the dependence on the choice of learning rate and other hyperparameters should also be considered. Nevertheless, it is interesting to note the result that the proposed approach converts a deep network that does not train into one which does in many cases. The method appears to improve the training for moderately deep convolutional networks without batch normalization (although this is tested on a single dataset), but is not practically useful yet since the regularization benefits of Batch Normalization are also taken away.
Summary of paper and review: The paper presents the instability issue of training GANs for semi-supervised learning.  Then, they propose to essentially utilize a wgan for semi-supervised learning.  The novelty of the paper is minor,  since similar approaches have been done before.  The analysis is poor,  the text seems to contain mistakes,  and the results don't seem to indicate any advantage or promise of the proposed algorithm. Detailed comments: - Unless I'm grossly mistaken the loss function (2) is clearly wrong.  There is a cross-entropy term used by Salimans et al. clearly missing. - As well, if equation (4) is referring to feature matching, the expectation should be inside the norm and not outside  (this amounts to matching random specific random fake examples to specific random real examples, an imbalanced form of MMD). - Theorem 2.1 is an almost literal rewrite of Theorem 2.4 of [1], without proper attribution.  Furthermore, Theorem 2.1 is not sufficient to demonstrate existence of this issues.  This is why [1] provides an extensive batch of targeted experiments to verify this assumptions.  Analogous experiments are clearly missing.  A detailed analysis of these assumptions and its implications are missing. - In section 3, the authors propose a minor variation of the Improved GAN approach by using a wgan on the unsupervised part of the loss.  Remarkably similar algorithms (where the two discriminators are two separate heads) to this have been done before (see for example, [2], but other approaches exist after that, see for examples papers citing [2]). - Theorem 3.1 is a trivial consequence of Theorem 3 from WGAN. - The experiments leave much to be desired.  It is widely known that MNIST is a bad benchmark at this point,  and that no signal can be established from a minor success in this dataset.  Furthermore, the results in CIFAR don't seem to bring any advantage, considering the .1% difference in accuracy is 1/100 of chance in this dataset. [1]: Arjovsky & Bottou, Towards Principled Methods for Training Generative Adversarial Networks, ICLR 2017 [2]: Mroueh & Sercu, Goel, McGan: Mean and Covariance Feature Matching GAN, ICML 2017
The authors investigate different message passing schedules for GNN learning.   Their proposed approach is to partition the graph into disjoint subregions, pass many messages on the sub regions and pass fewer messages between regions (an approach that is already considered in related literature, e.g., the BP literature), with the goal of minimizing the number of messages that need to be passed to convey information between all pairs of nodes in the network.   Experimentally, the proposed approach seems to perform comparably to existing methods (or slightly worse on average in some settings).   The paper is well-written and easy to read.   My primary concern is with novelty.   Many similar ideas have been floating around in a variety of different message-passing communities.   With no theoretical reason to prefer the proposed approach, it seems like it may be of limited interest to the community if speed is its only benefit (see detailed comments below). Specific comments:1)  "When information from any one node has reached all other nodes in the graph for the first time, this problem is considered as solved." Perhaps it is my misunderstanding of the way in which GNNs work, but isn't the objective actually to reach a set of fixed point equations.   If so, then simply propagating information from one side of the graph may not be sufficient. 2)  The experimental results in Section 4.4 are almost impossible to interpret.   Perhaps it is better to plot number of edges updated versus accuracy?   This at least would put them on equal footing.    In addition, the experiments that use randomness should be repeated and plotted on average (just in case you happened to pick a bad schedule). 3)  More generally, why not consider random schedules (i.e., just pick a random edge, update, repeat) or random partitions?   I'm not certain that a fixed set will perform best independent of the types of updates being considered, and random schedules, like the fully synchronous case for an important baseline (especially if update speed is all you care about). Typos: -pg. 6, "Thm. 2" -> "Table 2"
This paper's main thesis is that automatic metrics like BLEU, ROUGE, or METEOR is suitable for task-oriented natural language generation (NLG).  In particular, the paper presents a counterargument to "How NOT To Evaluate Your Dialogue System..."  where Wei et al argue that automatic metrics are not correlated or only weakly correlated with human eval on dialogue generation.  The authors here show that the performance of various NN models as measured by automatic metrics like BLEU and METEOR is correlated with human eval. Overall, this paper presents a useful conclusion: use METEOR for evaluating task oriented NLG.  However, there isn't enough novel contribution in this paper to warrant a publication.  Many of the details unnecessary:  1) various LSTM model descriptions are unhelpful  given the base LSTM model does just as well on the presented tasks  2) Many embedding based eval methods are proposed  but no conclusions are drawn from any of these techniques.
It is clear that the problem studied in this paper is interesting.  However, after reading through the manuscript, it is not clear to me what are the real contributions made in this paper.  I also failed to find any rigorous results on generalization bounds.  In this case, I cannot recommend the acceptance of this paper.
The paper studies different methods for defining hypergraph embeddings, i.e. defining vectorial representations of the set of hyperedges of a given hypergraph.  It should be noted that the framework does not allow to compute a vectorial representation of a set of nodes not already given as an hyperedge.  A set of methods is presented : the first one is based on an auto-encoder technique ;  the second one is based on tensor decomposition ;  the third one derives from sentence embedding methods.  The fourth one extends over node embedding techniques  and the last one use spectral methods.  The two first methods use plainly the set structure of hyperedges.  Experimental results are provided on semi-supervised regression tasks.  They show very similar performance for all methods and variants.  Also run-times are compared  and the results are expected.  In conclusion, the paper gives an overview of methods for computing hypernode embeddings.  This is interesting in its own.  Nevertheless, as the target problem on hypergraphs is left unspecified,  it is difficult to infer conclusions from the study.  Therefore, I am not convinced that the paper should be published in ICLR'18. * typos * Recent surveys on graph embeddings have been published in 2017 and should be cited as "A comprehensive survey of graph embedding ..." by Cai et al * Preliminaries. The occurrence number R(g_i) are not modeled in the hypergraphs.  A graph N_a is defined but not used in the paper. * Section 3.1. the procedure for sampling hyperedges in the lattice shoud be given.  At least, you should explain how it is made efficient when the number of nodes is large. * Section 3.2. The method seems to be restricted to cases where the cardinality of hyperedges can take a small number of values.  This is discussed in Section 3.6  but the discussion is not convincing enough. * Section 3.3 The term Sen2vec is not common knowledge * Section 3.3 The length of the sentences depends on the number of permutations of $k$ elements.  How can you deal with large k ? * Section 3.4 and Section 3.5. The methods proposed in these two sections should be related with previous works on hypergraph kernels.  I.e. there should be mentions on the clique expansion and star expansion of hypergraphs.  This leads to the question why graph embeddings methods on these expansions have not be considered in the paper. * Section 4.1. Only hyperedeges of cardinality in [2,6] are considered.  This seems a rather strong limitation  and this hypothesis does not seem pertinent in many applications.  * Section 4. For online multi-player games, hypernode embeddings only allow to evaluate existing teams, i.e. already existing as hyperedges in the input hypergraph.  One of the most important problem for multi-player games is team making where team evaluation should be made for all possible teams. * Section 5. Seems redundant with the Introduction.
In this paper, the authors propose a novel method for generating adversarial examples when the model is a black-box and we only have access to its decisions (and a positive example).   It iteratively takes steps along the decision boundary while trying to minimize the distance to the original positive example. Pros:- Novel method that works under much stricter and more realistic assumptions. - Fairly thorough evaluation. - The paper is clearly written. Cons:- Need a fair number of calls to generate a small perturbation.   Would like to see more analysis of this. - Attack works for making something outside the boundary (not X),  but is less clear how to generate image to meet a specific classification (X).   3.2 attempts this slightly by using an image in the class,  but is less clear for something like FaceID. - Unclear how often the images generated look reasonable.   Do different random initializations given different quality examples?
Summary: This paper empirically studies adversarial perturbations dx and what the effects are of adversarial training (AT) with respect to shared (dx fools for many x) and singular (only for a single x) perturbations. Experiments use a (previously published) iterative fast-gradient-sign-method and use a Resnet on CIFAR. The authors conclude that in this experimental setting: - AT seems to defend models against shared dx's. - This is visible on universal perturbations, which become less effective as more AT is applied. - AT decreases the effectiveness of adversarial perturbations, e.g. AT decreases the number of adversarial perturbations that fool both an input x and x with e.g. a contrast change. - Singular perturbations are easily detected by a detector model, as such perturbations don't change much when applying AT. Pro:- Paper addresses an important problem: qualitative / quantitative understanding of the behavior of adversarial perturbations is still lacking. - The visualizations of universal perturbations as they change during AT are nice. - The basic observation wrt the behavior of AT is clearly communicated. Con:- The experiments performed are interesting directions, although unfocused and rather limited in scope. For instance, does the same phenomenon happen for different datasets? Different models? - What happens when we use adversarial attacks different from FGSM? Do we get similar results? - The papers lacks a more in-depth theoretical analysis. Is there a principled reason AT+FGSM defends against universal perturbations? Overall:- As is, it seems to me the paper lacks a significant central message (due to limited and unfocused experiments) or significant new theoretical insight into the effect of AT. A number of questions addressed are interesting starting points towards a deeper understanding of *how* the observations can be explained and more rigorous empirical investigations. Detailed: -