ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling Style and Content in Anime Illustrations

Paper content:
Existing methods for AI-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists.
We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular.
Training such model is challenging, since given a style, various content data may exist but not the other way round.
Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based on a dual-conditional generator.
We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over a thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer.
We show this unique capability as well as superior output to the current state-of-the-art.

assistant: text: An image generation method combining conditional GANs and conditional VAEs that generates high fidelity anime images with various styles from various artists.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Paper content:
Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate.
While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs.
iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD).
We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation.
We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.

assistant: text: Incremental-RNNs resolves exploding/vanishing gradient problem by updating state vectors based on difference between previous state and that predicted by an ODE.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the insufficiency of existing momentum schemes for Stochastic Optimization

Paper content:
Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD).
Rigorously speaking, fast gradient methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact.
In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain.
This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters.
These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances.
These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of minibatching.
Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance.
This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration.
Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD.
The code for implementing the ASGD Algorithm can be found at https://github.com/rahulkidambi/AccSGD.

assistant: text: Existing momentum/acceleration schemes such as heavy ball method and Nesterov's acceleration employed with stochastic gradients do not improve over vanilla stochastic gradient descent, especially when employed with small batch sizes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fast is better than free: Revisiting adversarial training

Paper content:
Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD).
In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice.
Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost.
Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy at epsilon=8/255 in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at epsilon=2/255 in 12 hours, in comparison to past work based on ``free'' adversarial training which took 10 and 50 hours to reach the same respective thresholds.

assistant: text: This paper revisits Random+FGSM method to train robust models against strong PGD evasion attacks faster than previous methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Classification of Building Noise Type/Position via Supervised Learning

Paper content:
This paper presents noise type/position classification of various impact noises generated in a building which is a serious conflict issue in apartment complexes.
For this study, a collection of floor impact noise dataset is recorded with a single microphone.
Noise types/positions are selected based on a report by the Floor Management Center under Korea Environmental Corporation.
Using a convolutional neural networks based classifier, the impact noise signals converted to log-scaled Mel-spectrograms are classified into noise types or positions.
Also, our model is evaluated on a standard environmental sound dataset ESC-50 to show extensibility on environmental sound classification.

assistant: text: This work describes the use of convolutional neural networks in a novel application area of building noise type and noise position classification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Local Stability and Performance of Simple Gradient Penalty $\mu$-Wasserstein GAN

Paper content:
Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution.
Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint.
In this study, we prove the local stability of optimizing the simple gradient penalty $\mu$-WGAN(SGP $\mu$-WGAN) under suitable assumptions regarding the equilibrium and penalty measure $\mu$.
The measure valued differentiation concept is employed to deal with the derivative of the penalty terms, which is helpful for handling abstract singular measures with lower dimensional support.
Based on this analysis, we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty.
Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results.

assistant: text: Characterizes the convergence of gradient penalized Wasserstein GAN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Leveraging Simple Model Predictions for Enhancing its Performance

Paper content:
There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations.
In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance.
Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel.
Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network, which was one of the main ideas in previous work \citep{profweight}, to other commonly used classifiers and incorporate this into our method.
The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases.
In fact, in a couple of cases, we even approach the complex model's performance.
We also conduct further experiments to validate assertions and intuitively understand why our method works.
Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.

assistant: text: The paper proposes a means of improving the predictions of a low-capacity model which shows benefits over existing approaches.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Deep Learning Models Resistant to Adversarial Attacks

Paper content:
Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.
To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.
This approach provides us with a broad and unifying view on much prior work on this topic.
Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.
In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.
These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.
They also suggest robustness against a first-order adversary as a natural security guarantee.
We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

assistant: text: We provide a principled, optimization-based re-look at the notion of adversarial examples, and develop methods that produce models that are adversarially robust against a wide range of adversaries.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Winning the Lottery with Continuous Sparsification

Paper content:
The Lottery Ticket Hypothesis from Frankle & Carbin (2019) conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts.
The proposed algorithm to search for such sub-networks (winning tickets), Iterative Magnitude Pruning (IMP), consistently finds sub-networks with 90-95% less parameters which indeed train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning.
In this paper, we propose a new algorithm to search for winning tickets, Continuous Sparsification, which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies.
We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing up to 5 times faster search, when measured in number of training epochs.

assistant: text: This work propose a new iterative pruning methods named Continuous Sparsification, which continuously prunes the current weight until it reaches the target ratio.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: NEURAL EXECUTION ENGINES

Paper content:
Turing complete computation and reasoning are often regarded as necessary pre- cursors to general intelligence.
There has been a significant body of work studying neural networks that mimic general computation, but these networks fail to generalize to data distributions that are outside of their training set.
We study this problem through the lens of fundamental computer science problems: sorting and graph processing.
We modify the masking mechanism of a transformer in order to allow them to implement rudimentary functions with strong generalization.
We call this model the Neural Execution Engine, and show that it learns, through supervision, to numerically compute the basic subroutines comprising these algorithms with near perfect accuracy.
Moreover, it retains this level of accuracy while generalizing to unseen data and long sequences outside of the training distribution.

assistant: text: This paper investigates a problem of building a program execution engine with neural networks and proposes a transformer-based model to learn basic subroutines and applies them in several standard algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Deep Latent-variable MRFs with Amortized Bethe Free Energy Minimization

Paper content:
While much recent work has targeted learning deep discrete latent variable models with variational inference, this setting remains challenging, and it is often necessary to make use of potentially high-variance gradient estimators in optimizing the ELBO.
As an alternative, we propose to optimize a non-ELBO objective derived from the Bethe free energy approximation to an MRF's partition function.
This objective gives rise to a saddle-point learning problem, which we train inference networks to approximately optimize.
The derived objective requires no sampling, and can be efficiently computed for many MRFs of interest.
We evaluate the proposed approach in learning high-order neural HMMs on text, and find that it often outperforms other approximate inference schemes in terms of true held-out log likelihood.
At the same time, we find that all the approximate inference-based approaches to learning high-order neural HMMs we consider underperform learning with exact inference by a significant margin.

assistant: text: A method for learning deep latent-variable MRF with an optimization objective that utilizes Bethe free energy, that also solves the underlying constraints of Bethe free energy optimizations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generalizing Across Domains via Cross-Gradient Training

Paper content:
We present CROSSGRAD , a method to use multi-domain training data to learn a classifier that generalizes to new domains.
CROSSGRAD does not need an adaptation phase via labeled or unlabeled data, or domain features in the new domain.
Most existing domain adaptation methods attempt to erase domain signals using techniques like domain adversarial training.
In contrast, CROSSGRAD is free to use domain signals for predicting labels, if it can prevent overfitting on training domains.
We conceptualize the task in a Bayesian setting, in which a sampling step is implemented as data augmentation, based on domain-guided perturbations of input instances.
CROSSGRAD jointly trains a label and a domain classifier on examples perturbed by loss gradients of each other’s objectives.
This enables us to directly perturb inputs, without separating and re-mixing domain signals while making various distributional assumptions.
Empirical evaluation on three different applications where this setting is natural establishes that
(1) domain-guided perturbation provides consistently better generalization to unseen domains, compared to generic instance perturbation methods, and
(2) data augmentation is a more stable and accurate method than domain adversarial training.

assistant: text: The authors introduce the CrossGrad method, which trains both a label classification task and a domain classification task.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Paper content:
We explore the concept of co-design in the context of neural network verification.
Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily.
To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task.
We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones.
Then, improving ReLU stability leads to an additional 4-13x speedup in verification times.
An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

assistant: text: We develop methods to train deep neural models that are both robust to adversarial perturbations and whose robustness is significantly easier to verify.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Model-Agnostic Feature Selection with Additional Mutual Information

Paper content:
Answering questions about data can require understanding what parts of an input X influence the response Y. Finding such an understanding can be built by testing relationships between variables through a machine learning model.
For example, conditional randomization tests help determine whether a variable relates to the response given the rest of the variables.
However, randomization tests require users to specify test statistics.
We formalize a class of proper test statistics that are guaranteed to select a feature when it provides information about the response even when the rest of the features are known.
We show that f-divergences provide a broad class of proper test statistics.
In the class of f-divergences, the KL-divergence yields an easy-to-compute proper test statistic that relates to the AMI.
Questions of feature importance can be asked at the level of an individual sample.
We show that estimators from the same AMI test can also be used to find important features in a particular instance.
We provide an example to show that perfect predictive models are insufficient for instance-wise feature selection.
We evaluate our method on several simulation experiments, on a genomic dataset, a clinical dataset for hospital readmission, and on a subset of classes in ImageNet.
Our method outperforms several baselines in various simulated datasets, is able to identify biologically significant genes, can select the most important predictors of a hospital readmission event, and is able to identify distinguishing features in an image-classification task.

assistant: text: This paper proposes a practical improvement of the conditional randomization test and a new test statistic, proves f-divergence is one possible choice, and shows that KL-divergence cancels out some conditional distributions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Gradient Push for Distributed Deep Learning

Paper content:
Large mini-batch parallel SGD is commonly used for distributed training of deep networks.
Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.
In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.
SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios.
The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.
We prove that SGP converges to a stationary point of smooth, non-convex objective functions.
Furthermore, we validate empirically the potential of SGP.
For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.5 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

assistant: text: The paper proves the convergence of SGP for nonconvex smooth functions and shows the SGP can achieve a significant speed-up in the low-latency environment without sacrificing too much predictive performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TOWARDS ROBOT VISION MODULE DEVELOPMENT WITH EXPERIENTIAL ROBOT LEARNING

Paper content:
n this paper we present a thrust in three directions of visual development us- ing supervised and semi-supervised techniques.
The first is an implementation of semi-supervised object detection and recognition using the principles of Soft At- tention and Generative Adversarial Networks (GANs).
The second and the third are supervised networks that learn basic concepts of spatial locality and quantity respectively using Convolutional Neural Networks (CNNs).
The three thrusts to- gether are based on the approach of Experiential Robot Learning, introduced in previous publication.
While the results are unripe for implementation, we believe they constitute a stepping stone towards autonomous development of robotic vi- sual modules.

assistant: text: 3 thrusts serving as stepping stones for robot experiential learning of vision module
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generative Question Answering: Learning to Answer the Whole Question

Paper content:
Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.
We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.
Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answer—allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.
Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.
The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.
We will release our code.

assistant: text: This paper proposes a generative approach to textual and visual QA, where a joint distribution over the question and answer space given the context is learned, which captures more complex relationships.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators

Paper content:
Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain.
We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.
The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning.
This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time.
Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way.
This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well.
Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains.
They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

assistant: text: This paper tackles few-shot classification with many different domains by building a pool of embedding models to capture domain-invariant and domain-specific features without a significant increase in the number of parameters.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training individually fair ML models with sensitive subspace robustness

Paper content:
We propose an approach to training machine learning models that are fair in the sense that their performance is invariant under certain perturbations to the features.
For example, the performance of a resume screening system should be invariant under changes to the name of the applicant.
We formalize this intuitive notion of fairness by connecting it to the original  notion of individual fairness put forth by Dwork et al and show that the proposed approach achieves this notion of fairness.
We also demonstrate the effectiveness of the approach on two machine learning tasks that are susceptible to gender and racial biases.

assistant: text: Algorithm for training individually fair classifier using adversarial robustness
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Benchmarking Adversarial Robustness

Paper content:
Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.
While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.
In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.
After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.
Based on the evaluation results, we draw several important findings and provide insights for future research.

assistant: text: A large-scale empirical study comparing different adversarial attack and defense techniques, and use of accuracy vs. perturbation budget and accuracy vs. attack strength curves to evaluate attacks and defenses.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Benchmarking Adversarial Robustness

Paper content:
Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important problems in the development of deep learning.
While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms.
In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks.
After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods.
Based on the evaluation results, we draw several important findings and provide insights for future research.

assistant: text: We provide a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness of deep learning models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Few-Shot Learning with Adversarially Queried Meta-Learners

Paper content:
Previous work on adversarially robust neural networks requires large training sets and computationally expensive training procedures.
On the other hand, few-shot learning methods are highly vulnerable to adversarial examples.
The goal of our work is to produce networks which both perform well at few-shot tasks and are simultaneously robust to adversarial examples.
We adapt adversarial training for meta-learning, we adapt robust architectural features to small networks for meta-learning, we test pre-processing defenses as an alternative to adversarial training for meta-learning, and we investigate the advantages of robust meta-learning over robust transfer-learning for few-shot tasks.
This work provides a thorough analysis of adversarially robust methods in the context of meta-learning, and we lay the foundation for future work on defenses for few-shot tasks.

assistant: text: The authors of this paper propose a novel approach for training a robust few-shot model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Loss Scaling for Mixed Precision Training

Paper content:
Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs.
MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training.
Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages.
We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter.
We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods.
We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy, compared with using the existing state-of-the-art MPT and single-precision floating point.

assistant: text: The authors propose a method to train models in FP16 precision that adopts a more elaborate way to minimize underflow in every layer simultaneously and automatically.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Three Mechanisms of Weight Decay Regularization

Paper content:
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_2$ regularization.
Literal weight decay has been shown to outperform $L_2$ regularization for optimizers for which they differ.
We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures.
We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization.
Our results provide insight into how to improve the regularization of neural networks.

assistant: text: Discusses the effect of weight decay on the training of deep network models with and without batch normalization and when using first/second order optimization methods and hypothesizes that a larger learning rate has a regularization effect.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION

Paper content:
Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.
SGD estimates the gradient by uniform sampling with sample size one.
There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.
Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.
As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.
In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.
Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.
As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.
We demonstrate the benefits of our proposal on both SGD and AdaGrad.

assistant: text: Authors propose sampling stochastic gradients from a monotonic function proportional to gradient magnitudes by using LSH.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

Paper content:
Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts.
We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth.
We propose robust growing and stopping policies to generalize to different network architectures and datasets.
Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet.
For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts.
Our AutoGrow is efficient.
It discovers depth within similar time of training a single DNN.

assistant: text: A method that automatically grows layers in neural networks to discover optimal depth.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization

Paper content:
Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field.
But its performance can awfully degrade with insufficient batch size.
This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption.
Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption.
In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before.
The extra batch statistics associated with gradients also can severely affect the training of deep neural network.
Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN).
MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure.
We prove the benefits of MABN by both theoretical analysis and experiments.
Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO.
The code has been released in https://github.com/megvii-model/MABN.

assistant: text: We propose a novel normalization method to handle small batch size cases.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bayesian Sparsification of Gated Recurrent Neural Networks

Paper content:
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e.
g.
neurons.
We apply and further develop this approach for gated recurrent architectures.
Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM.
It makes some gates and information flow components constant, speeds up forward pass and improves compression.
Moreover, the resulting structure of gate sparsity is interpretable and depends on the task.

assistant: text: We propose to sparsify preactivations of gates and information flow in LSTM to make them constant and boost the neuron sparsity level
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Imbalanced Classification via Adversarial Minority Over-sampling

Paper content:
In most real-world scenarios, training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion.
In this paper, we explore a novel yet simple way to alleviate this issue via synthesizing less-frequent classes with adversarial examples of other classes.
Surprisingly, we found this counter-intuitive method can effectively learn generalizable features of minority classes by transferring and leveraging the diversity of the majority information.
Our experimental results on various types of class-imbalanced datasets in image classification and natural language processing show that the proposed method not only improves the generalization of minority classes significantly compared to other re-sampling or re-weighting methods, but also surpasses other methods of state-of-art level for the class-imbalanced classification.

assistant: text: Proposes a new optimization objective that generates synthetic samples by over-sampling the majority classes instead of minority classes, solving the problem of overfitting minority classes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-agent Reinforcement Learning for Networked System Control

Paper content:
This paper considers multi-agent reinforcement learning (MARL) in networked system control.
Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors.
We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent.
Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL.
Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

assistant: text: This paper proposes a new formulation and a new communication protocol for networked multi-agent control problems
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unifying semi-supervised and robust learning by mixup

Paper content:
Supervised deep learning methods require cleanly labeled large-scale datasets, but collecting such data is difficult and sometimes impossible.
There exist two popular frameworks to alleviate this problem: semi-supervised learning and robust learning to label noise.
Although these frameworks relax the restriction of supervised learning, they are studied independently.
Hence, the training scheme that is suitable when only small cleanly-labeled data are available remains unknown.
In this study, we consider learning from bi-quality data as a generalization of these studies, in which a small portion of data is cleanly labeled, and the rest is corrupt.
Under this framework, we compare recent algorithms for semi-supervised and robust learning.
The results suggest that semi-supervised learning outperforms robust learning with noisy labels.
We also propose a training strategy for mixing mixup techniques to learn from such bi-quality data effectively.

assistant: text: We propose to compare semi-supervised and robust learning to noisy label under a shared setting
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Random Partition Relaxation for Training Binary and Ternary Weight Neural Network

Paper content:
We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values.
Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation.
We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks.
We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.

assistant: text: The paper proposes a new training scheme of optimizing a ternary neural network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Boosting Generative Models by Leveraging Cascaded Meta-Models

Paper content:
A deep generative model is a powerful method of learning a data distribution, which has achieved tremendous success in numerous scenarios.
However, it is nontrivial for a single generative model to faithfully capture the distributions of the complex data such as images with complicate structures.
In this paper, we propose a novel approach of cascaded boosting for boosting generative models, where meta-models (i.e., weak learners) are cascaded together to produce a stronger model.
Any hidden variable meta-model can be leveraged as long as it can support the likelihood evaluation.
We derive a decomposable variational lower bound of the boosted model, which allows each meta-model to be trained separately and greedily.
We can further improve the learning power of the generative models by combing our cascaded boosting framework with the multiplicative boosting framework.

assistant: text: This paper proposed a novel approach of cascaded boosting for boosting generative models which allows each each meta-model to be trained separately and greedily.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An Adversarial Learning Framework for a Persona-based Multi-turn Dialogue Model

Paper content:
In this paper, we extend the persona-based sequence-to-sequence (Seq2Seq) neural network conversation model to a multi-turn dialogue scenario by modifying the state-of-the-art hredGAN architecture to simultaneously capture utterance attributes such as speaker identity, dialogue topic, speaker sentiments and so on.
The proposed system, phredGAN has a persona-based HRED generator (PHRED) and a conditional discriminator.
We also explore two approaches to accomplish the conditional discriminator: (1) $phredGAN_a$, a system that passes the attribute representation as an additional input into a traditional adversarial discriminator, and (2) $phredGAN_d$, a dual discriminator system which in addition to the adversarial discriminator, collaboratively predicts the attribute(s) that generated the input utterance.
To demonstrate the superior performance of phredGAN over the persona SeqSeq model, we experiment with two conversational datasets, the Ubuntu Dialogue Corpus (UDC) and TV series transcripts from the Big Bang Theory and Friends.
Performance comparison is made with respect to a variety of quantitative measures as well as crowd-sourced human evaluation.
We also explore the trade-offs from using either variant of $phredGAN$ on datasets with many but weak attribute modalities (such as with Big Bang Theory and Friends) and ones with few but strong attribute modalities (customer-agent interactions in Ubuntu dataset).

assistant: text: This paper proposes an extension to hredGAN to simultaneously learn a set of attribute embeddings that represent the persona of each speaker and generate persona-based responses
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Neural Forests: An Architecture for Tabular Data

Paper content:
Deep neural models, such as convolutional and recurrent networks, achieve phenomenal results over spatial data such as images and text.
However, when considering tabular data, gradient boosting of decision trees (GBDT) remains the method of choice.
Aiming to bridge this gap, we propose \emph{deep neural forests} (DNF)
--  a novel architecture that combines elements from decision trees as well as dense residual connections.
We present the results of extensive empirical study in which we examine the performance of GBDTs, DNFs and (deep) fully-connected networks.
These results indicate that DNFs achieve comparable results to GBDTs on tabular data, and open the door to end-to-end neural modeling of multi-modal data.
To this end, we present a successful application of DNFs as part of a hybrid architecture for a multi-modal driving scene understanding classification task.

assistant: text: This paper proposes deep neural forest, an algorithm which targets tabular data and integrates strong points of gradient boosting of decision trees.