ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learnable Higher-order Representation for Action Recognition

Paper content:
Capturing spatiotemporal dynamics is an essential topic in video recognition.
In this paper, we present learnable higher-order operation as a generic family of building blocks for capturing higher-order correlations from high dimensional input video space.
We prove that several successful architectures for visual classification tasks are in the family of higher-order neural networks, theoretical and experimental analysis demonstrates their underlying mechanism is higher-order.
On the task of video recognition, even using RGB only without fine-tuning with other video datasets, our higher-order models can achieve results on par with or better than the existing state-of-the-art methods on both Something-Something (V1 and V2) and Charades datasets.

assistant: text: Proposes a new 3D convolutional block which convolves video input with its context, based on the assumpton that relevant context is present around the image's object.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Efficient Receptive Field Learning by Dynamic Gaussian Structure

Paper content:
The visual world is vast and varied, but its variations divide into structured and unstructured factors.
Structured factors, such as scale and orientation, admit clear theories and efficient representation design.
Unstructured factors, such as what it is that makes a cat look like a cat, are too complicated to model analytically, and so require free-form representation learning.
We compose structured Gaussian filters and free-form filters, optimized end-to-end, to factorize the representation for efficient yet general learning.
Our experiments on dynamic structure, in which the structured filters vary with the input, equal the accuracy of dynamic inference with more degrees of freedom while improving efficiency.
(Please see https://arxiv.org/abs/1904.11487 for the full edition.)
Although the visual world is varied, there is nevertheless ubiquitous structure.
Free-form learned representations are structure-agnostic, making them general, but their not harnessing structure is computationally and statistically inefficient.
Structured representations like steerable filtering BID5 BID6 , scattering BID0 , and steerable networks BID1 efficiently express certain structures, but are constrained.
We propose the semi-structured composition of Gaussian and free-form filters to blur the line between free-form and structured representations.The effectiveness of strongly structured representations hinges on whether they encompass the true structure of the data.
If not, the representation is limiting, and subject to error.
At least, such is the case when structure substitutes for learning.
In this work we compose structured and free-form filters and learn both end-to-end ( FIG1 ).
The free-form parameters are not constrained by our composition for generality.
The structured parameters are low-dimensional for efficiency.We choose Gaussian structure to represent the spatial structures of scale, aspect, and orientation through covariance BID7 .
Optimizing these structured covariance parameters carries out a form of differentiable architecture search over receptive fields.
Since this structure is lowdimensional, it is computationally efficient and could be learned from limited data.

assistant: text: Dynamic receptive fields with spatial Gaussian structure are accurate and efficient.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking learning rate schedules for stochastic optimization

Paper content:
There is a stark disparity between the learning rate schedules used in the practice of large scale machine learning and what are considered admissible learning rate schedules prescribed in the theory of stochastic approximation.
Recent results, such as in the 'super-convergence' methods which use oscillating learning rates, serve to emphasize this point even more.
One plausible explanation is that non-convex neural network training procedures are better suited to the use of fundamentally different learning rate  schedules, such as the ``cut the learning rate every constant number of epochs'' method (which more closely resembles an exponentially decaying learning rate schedule); note that this widely used schedule is in stark contrast to the polynomial decay schemes prescribed in the stochastic approximation literature, which are indeed shown to be (worst case) optimal for classes of convex optimization problems.
The main contribution of this work shows that the picture is far more nuanced, where we do not even need to move to non-convex optimization to show other learning rate schemes can be far more effective.
In fact, even for the simple case of stochastic linear regression with a fixed time horizon, the rate achieved by any polynomial decay scheme is sub-optimal compared to the statistical minimax rate (by a factor of condition number); in contrast the ```''cut the learning rate every constant number of epochs'' provides an exponential improvement (depending only logarithmically on the condition number) compared to any polynomial decay scheme.
Finally, it is important to ask if our theoretical insights are somehow fundamentally tied to quadratic loss minimization (where we have circumvented minimax lower bounds for more general convex optimization problems)?
Here, we conjecture that recent results which make the gradient norm small at a near optimal rate, for both convex and non-convex optimization, may also provide more insights into learning rate schedules used in practice.

assistant: text: This paper presents a theoretical study of different learning rate schedules that resulted in statistical minimax lower bounds for both polynomial and constant-and-cut schemes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Train Neural Network by Embedding Space Probabilistic Constraint 

Paper content:
Using higher order knowledge to reduce training data has become a popular research topic.
However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.
Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.
We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.
While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.
We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

assistant: text: This paper introduces a method to perform semi-supervised learning with deep neural networks, and the model achieves relatively high accuracy, given a small training size.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Defense by Learning to Attack

Paper content:
Adversarial training provides a principled approach for training robust neural networks.
From an optimization perspective, the adversarial training is essentially solving a minmax robust optimization problem.
The outer minimization is trying to learn a robust classifier, while the inner maximization is trying to generate adversarial samples.
Unfortunately, such a minmax problem is very difficult to solve due to the lack of convex-concave structure.
This work proposes a new adversarial training method based on a general learning-to-learn framework.
Specifically, instead of applying the existing hand-design algorithms for the inner problem, we learn an optimizer, which is parametrized as a convolutional neural network.
At the same time, a robust classifier is learned to defense the adversarial attack generated by the learned optimizer.
From the perspective of generative learning, our proposed method can be viewed as learning a deep generative model for generating adversarial samples, which is adaptive to the robust classification.
Our experiments demonstrate that our proposed method significantly outperforms existing adversarial training methods on CIFAR-10 and CIFAR-100 datasets.

assistant: text: Don't know how to optimize? Then just learn to optimize!
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Swoosh! Rattle! Thump! - Actions that Sound

Paper content:
Truly intelligent agents need to capture the interplay of all their senses to build a rich physical understanding of their world.
In robotics, we have seen tremendous progress in using visual and tactile perception; however we have often ignored a key sense: sound.
This is primarily due to lack of data that captures the interplay of action and sound.
In this work, we perform the first large-scale study of the interactions between sound and robotic action.
To do this, we create the largest available sound-action-vision dataset with 15,000 interactions on 60 objects using our robotic platform Tilt-Bot.
By tilting objects and allowing them to crash into the walls of a robotic tray, we collect rich four-channel audio information.
Using this data, we explore the synergies between sound and action, and present three key insights.
First, sound is indicative of fine-grained object class information, e.g., sound can differentiate a metal screwdriver from a metal wrench.
Second, sound also contains information about the causal effects of an action, i.e. given the sound produced, we can predict what action was applied on the object.
Finally, object representations derived from audio embeddings are indicative of implicit physical properties.
We demonstrate that on previously unseen objects, audio embeddings generated through interactions can predict forward models 24% better than passive visual embeddings.

assistant: text: This paper studies the role of audio in object and action perception, as well as how auditory information can help learning forward and inverse dynamics models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SVDocNet: Spatially Variant U-Net for Blind Document Deblurring

Paper content:
Blind document deblurring is a fundamental task in the field of document processing and restoration, having wide enhancement applications in optical character recognition systems, forensics, etc.
Since this problem is highly ill-posed, supervised and unsupervised learning methods are well suited for this application.
Using various techniques, extensive work has been done on natural-scene deblurring.
However, these extracted features are not suitable for document images.
We present SVDocNet, an end-to-end trainable U-Net based spatial recurrent neural network (RNN) for blind document deblurring where the weights of the RNNs are determined by different convolutional neural networks (CNNs).
This network achieves state of the art performance in terms of both quantitative measures and qualitative results.

assistant: text: We present SVDocNet, an end-to-end trainable U-Net based spatial recurrent neural network (RNN) for blind document deblurring.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Quantized Activation: To prevent Overfitting in Fast Adversarial Training

Paper content:
Existing neural networks are vulnerable to "adversarial examples"---created by adding maliciously designed small perturbations in inputs to induce a misclassification by the networks.
The most investigated defense strategy is adversarial training which augments training data with adversarial examples.
However, applying single-step adversaries in adversarial training does not support the robustness of the networks, instead, they will even make the networks to be overfitted.
In contrast to the single-step, multi-step training results in the state-of-the-art performance on MNIST and CIFAR10, yet it needs a massive amount of time.
Therefore, we propose a method, Stochastic Quantized Activation (SQA) that solves overfitting problems in single-step adversarial training and fastly achieves the robustness comparable to the multi-step.
SQA attenuates the adversarial effects by providing random selectivity to activation functions and allows the network to learn robustness with only single-step training.
Throughout the experiment, our method demonstrates the state-of-the-art robustness against one of the strongest white-box attacks as PGD training, but with much less computational cost.
Finally, we visualize the learning process of the network with SQA to handle strong adversaries, which is different from existing methods.

assistant: text: This paper proposes Stochastic Quantized Activation that solves overfitting problems in FGSM adversarial training and fastly achieves the robustness comparable to multi-step training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories

Paper content:
We address the problem of learning to discover 3D parts for objects in unseen categories.
Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches.
Formulated as a contextual bandit problem, we propose a learning-based iterative grouping framework which learns a grouping policy to progressively merge small part proposals into bigger ones in a bottom-up fashion.
At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to novel categories.
On a recently proposed large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples.
Quantitative comparisons against four strong shape segmentation baselines show that we achieve the state-of-the-art performance.

assistant: text: A zero-shot segmentation framework for 3D object part segmentation. Model the segmentation as a decision-making process and solve as a contextual bandit problem.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Graph Matching Consensus

Paper content:
This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs.
First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes.
Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs.
We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process.
Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently.
We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

assistant: text: A framework for answering graph matching questions consisting of local node embeddings with a message passing refinement step.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Difficulty of Training Sparse Neural Networks

Paper content:
We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime.
Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning.
We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the ``good'' solution.
Additionally, our attempts to find a decreasing objective path from ``bad'' solutions to the ``good'' ones in the sparse subspace fail.
However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions.
These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

assistant: text: In this paper we highlight  the difficulty of training sparse neural networks by doing interpolation experiments in the energy landscape
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Structured Semi-Implicit Variational Inference

Paper content:
In this work we construct flexible joint distributions from low-dimensional conditional semi-implicit distributions.
Explicitly defining the structure of the approximation allows to make the variational lower bound tighter, resulting in more accurate inference.

assistant: text: Utilizing the structure of distributions improves semi-implicit variational inference
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Difference-Seeking Generative Adversarial Network

Paper content:
We propose a novel algorithm, Difference-Seeking Generative Adversarial Network (DSGAN), developed from traditional GAN.
DSGAN considers the scenario that the training samples of target distribution, $p_{t}$, are difficult to collect.
Suppose there are two distributions  $p_{\bar{d}}$ and $p_{d}$ such that the density of the target distribution can be the differences between the densities of $p_{\bar{d}}$ and $p_{d}$.
We show how to learn the target distribution $p_{t}$ only via samples from $p_{d}$ and $p_{\bar{d}}$ (relatively easy to obtain).
DSGAN has the flexibility to produce samples from various target distributions (e.g. the out-of-distribution).
Two key applications, semi-supervised learning and adversarial training, are taken as examples to validate the effectiveness of DSGAN.
We also provide theoretical analyses about the convergence of DSGAN.

assistant: text: We proposed "Difference-Seeking Generative Adversarial Network" (DSGAN) model to learn the target distribution which is hard to collect training data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards an Adversarially Robust Normalization Approach

Paper content:
Batch Normalization (BatchNorm) has shown to be effective for improving and accelerating the training of deep neural networks.
However, recently it has been shown that it is also vulnerable to adversarial perturbations.
In this work, we aim to investigate the cause of adversarial vulnerability of the BatchNorm.
We hypothesize that the use of different normalization statistics during training and inference (mini-batch statistics for training and moving average of these values at inference) is the main cause of this adversarial vulnerability in the BatchNorm layer.
We empirically proved this by experiments on various neural network architectures and datasets.
Furthermore, we introduce Robust Normalization (RobustNorm) and experimentally show that it is not only resilient to adversarial perturbation but also inherit the benefits of BatchNorm.

assistant: text: This paper investigates the reason behind the vulnerability of BatchNorm and proposes Robust Normalization, a normalization method that achieves significantly better results under a variety of attack methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: FigureQA: An Annotated Figure Dataset for Visual Reasoning

Paper content:
We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images.
The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts.
We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection.
To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure.
To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives.
In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements.
We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as strong baseline.
Preliminary results indicate that the task poses a significant machine learning challenge.
We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

assistant: text: We present a question-answering dataset, FigureQA, as a first step towards developing models that can intuitively recognize patterns from visual representations of data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit

Paper content:
The loss of a few neurons in a brain rarely results in any visible loss of function.
However, the insight into what “few” means in this context is unclear.
How many random neuron failures will it take to lead to a visible loss of function?
In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces.
We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting.
We give provable guarantees on the robustness of the network to these crashes.
Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar.
The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks.
We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture.
We design an algorithm achieving fault tolerance using a reasonable number of neurons.
In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.

assistant: text: We give a bound for NNs on the output error in case of random weight failures using a Taylor expansion in the continuous limit where nearby neurons are similar
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum

Paper content:
Long short-term memory networks (LSTMs) were introduced to combat vanishing gradients in simple recurrent neural networks (S-RNNs) by augmenting them with additive recurrent connections controlled by gates.
We present an alternate view to explain the success of LSTMs: the gates themselves are powerful recurrent models that provide more representational power than previously appreciated.
We do this by showing that the LSTM's gates can be decoupled from the embedded S-RNN, producing a restricted class of RNNs where the main recurrence computes an element-wise weighted sum of context-independent functions of the inputs.
Experiments on a range of challenging NLP problems demonstrate that the simplified gate-based models work substantially better than S-RNNs, and often just as well as the original LSTMs, strongly suggesting that the gates are doing much more in practice than just alleviating vanishing gradients.

assistant: text: Gates do all the heavy lifting in LSTMs by computing element-wise weighted sums, and removing the internal simple RNN does not degrade model performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Retrieving Signals with Deep Complex Extractors

Paper content:
Recent advances have made it possible to create deep complex-valued neural networks.
Despite this progress, many challenging learning tasks have yet to leverage the power of complex representations.
Building on recent advances, we propose a new deep complex-valued method for signal retrieval and extraction in the frequency domain.
As a case study, we perform audio source separation in the Fourier domain.
Our new method takes advantage of the convolution theorem which states that the Fourier transform of two convolved signals is the elementwise product of their Fourier transforms.
Our novel method is based on a complex-valued version of Feature-Wise Linear Modulation (FiLM) and serves as the keystone of our proposed signal extraction method.
We also introduce a new and explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram.
Using the Wall Street Journal Dataset, we compared our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss.

assistant: text: New Signal Extraction Method in the Fourier Domain
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: WEAKLY SEMI-SUPERVISED NEURAL TOPIC MODELS

Paper content:
We consider the problem of topic modeling in a weakly semi-supervised setting.
In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics.
In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.
Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework.
We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective.
After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models.

assistant: text: A flexible method of weakly supervising a topic model to achieve better alignment with user intuition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Does the neuronal noise in cortex help generalization?

Paper content:
Neural activity is highly variable in response to repeated stimuli.
We used an open dataset, the Allen Brain Observatory, to quantify the distribution of responses to repeated natural movie presentations.
A large fraction of responses are best fit by log-normal distributions or Gaussian mixtures with two components.
These distributions are similar to those from units in deep neural networks with dropout.
Using a separate set of electrophysiological recordings, we constructed a population coupling model as a control for state-dependent activity fluctuations and found that the model residuals also show non-Gaussian distributions.
We then analyzed responses across trials from multiple sections of different movie clips and observed that the noise in cortex aligns better with in-clip versus out-of-clip stimulus variations.
We argue that noise is useful for generalization when it moves along representations of different exemplars in-class, similar to the structure of cortical noise.

assistant: text: We study the structure of noise in the brain and find it may help generalization by moving representations along in-class stimulus variations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Gradients as Features for Deep Representation Learning

Paper content:
We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks.
Specifically, we propose to explore gradient-based features.
These features are gradients of the model parameters with respect to a task-specific loss given an input sample.
Our key innovation is the design of a linear model that incorporates both gradient features and the activation of the network.
We show that our model provides a local linear approximation to a underlying deep model, and discuss important theoretical insight.
Moreover, we present an efficient algorithm for the training and inference of our model without computing the actual gradients.
Our method is evaluated across a number of representation learning tasks on several datasets and using different network architectures.
We demonstrate strong results in all settings.
And our results are well-aligned with our theoretical insight.

assistant: text: This paper proposes to use the gradients of specific layers of convolutional networks as features in a linearized model for transfer learning and fast adaptation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling neural mechanisms for perceptual grouping

Paper content:
Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence.
This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons.
However, the relative contributions of these connections to perceptual grouping are poorly understood.
We address this question by systematically evaluating neural network architectures featuring combinations of these connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping.
We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up processing.
Horizontal connections resolve this limitation on tasks with Gestalt cues by supporting incremental spatial propagation of activities, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object.
Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.

assistant: text: Using neural networks as a computational model of the brain, examines the efficiency of different strategies for solving two visual challenges.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fixing Weight Decay Regularization in Adam

Paper content:
We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential benefit of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t. the loss function.
We provide empirical evidence that our proposed modification
(i) 
decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and
(ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter).
We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.
Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32.
Our source code will become available after the review process.

assistant: text: The paper presents an alternative way to implement weight decay in Adam with empirical results shown
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: {COMPANYNAME}11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery

Paper content:
We release the largest public ECG dataset of continuous raw signals for representation learning containing over 11k patients and 2 billion labelled beats.
Our goal is to enable semi-supervised ECG models to be made as well as to discover unknown subtypes of arrhythmia and anomalous ECG signal events.
To this end, we propose an unsupervised representation learning task, evaluated in a semi-supervised fashion.
We provide a set of baselines for different feature extractors that can be built upon.
Additionally, we perform qualitative evaluations on results from PCA embeddings, where we identify some clustering of known subtypes indicating the potential for representation learning in arrhythmia sub-type discovery.

assistant: text: We release a dataset constructed from single-lead ECG data from 11,000 patients who were prescribed to use the {DEVICENAME}(TM) device.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: RotationOut as a Regularization Method for Neural Network

Paper content:
In this paper, we propose a novel regularization method, RotationOut, for neural networks.
Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector.
RotationOut can also be used in convolutional layers and recurrent layers with a small modification.
We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction.
Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization.
Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method.
Codes will be available.

assistant: text: This paper proposes a novel regularization method for training neural networks, which adds noise neurons in an inter-dependent fashion.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Synthetic vs Real: Deep Learning on Controlled Noise

Paper content:
Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels.
Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting.
To this end, this paper establishes a benchmark of real-world noisy labels at 10 controlled noise levels.
As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods, and training settings.
Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise.
(2) DNNs may not learn patterns first on real-world noisy data.
(3) When networks are fine-tuned, ImageNet architectures generalize well on noisy data.
(4) Real-world noise appears to be less harmful, yet it is more difficult for robust DNN methods to improve.
(5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa.
We hope our benchmark, as well as our findings, will facilitate deep learning research on noisy data.

assistant: text: This paper compares 6 existing noisy label learning methods in two training settings: from scratch, and finetuning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Effect of top-down connections in Hierarchical Sparse Coding

Paper content:
Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images.
The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems.
However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers.
In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection.
In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers.
A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer.
First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers.
Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model.
Third, we show that the SDPC also accelerates the learning process.
Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

assistant: text: This paper experimentally demonstrates the beneficial effect of top-down connections in Hierarchical Sparse Coding algorithm.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ACIQ: Analytical Clipping for Integer Quantization of neural networks

Paper content:
We analyze the trade-off between quantization noise and clipping distortion in low precision networks.
We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping.
By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping.
For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision.
Our results have many applications for the quantization of neural networks at both training and inference time.

assistant: text: Derives a formula for finding the minimum and maximum clipping values for uniform quantization which minimize the square error resulting from quantization, for either a Laplace or Gaussian distribution over pre-quantized value.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AdamT: A Stochastic Optimization with Trend Correction Scheme

Paper content:
Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning.
Such methods are appealing for capability on large-scale sparse datasets.
On top of that, they are computationally efficient and insensitive to the hyper-parameter settings.
In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT.
Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients.
The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly.
We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.

assistant: text: We present a new framework for adapting Adam-typed methods, namely AdamT, to include the trend information when updating the parameters with the adaptive step size and gradients.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A closer look at the approximation capabilities of neural networks

Paper content:
The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions σ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold ε, if and only if σ is non-polynomial.
In this paper, we give a direct algebraic proof of the theorem.
Furthermore we shall explicitly quantify the number of hidden units required for approximation.
Specifically, if X in R^n is compact, then a neural network with n input units, m output units, and a single hidden layer with {n+d choose d} hidden units (independent of m and ε), can uniformly approximate any polynomial function f:X -> R^m whose total degree is at most d for each of its m coordinate functions.
In the general case that f is any continuous function, we show there exists some N in O(ε^{-n}) (independent of m), such that N hidden units would suffice to approximate f.
We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights.
We highlight several consequences:
(i) For any δ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < δ.
(ii) There exists some λ>0 (depending only on f and σ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w|>λ.
(iii) If the non-bias weights in the first layer are *fixed* and randomly chosen from a suitable range, then the UAP holds with probability 1.

assistant: text: A quantitative refinement of the universal approximation theorem via an algebraic approach.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Neural Pruning

Paper content:
Despite the remarkable performance of deep neural networks (DNNs) on various tasks, they are susceptible to adversarial perturbations which makes it difficult to deploy them in real-world safety-critical applications.
In this paper, we aim to obtain robust networks by sparsifying DNN's latent features sensitive to adversarial perturbation.
Specifically, we define vulnerability at the latent feature space and then propose a Bayesian framework to prioritize/prune features based on their contribution to both the original and adversarial loss.
We also suggest regularizing the features' vulnerability during training to improve robustness further.
While such network sparsification has been primarily studied in the literature for computational efficiency and regularization effect of DNNs, we confirm that it is also useful to design a defense mechanism through quantitative evaluation and qualitative analysis.
We validate our method, \emph{Adversarial Neural Pruning (ANP)} on multiple benchmark datasets, which results in an improvement in test accuracy and leads to state-of-the-art robustness.
ANP also tackles the practical problem of obtaining sparse and robust networks at the same time, which could be crucial to ensure adversarial robustness on lightweight networks deployed to computation and memory-limited devices.

assistant: text: This paper proposes "adversarial neural pruning" method of training a pruning mask and a new vulnerability suppression loss to improve accuracy and adversarial robustness.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Deep Learning Models Resistant to Adversarial Attacks

Paper content:
Recent work has demonstrated that neural networks are vulnerable to adversarial examples, i.e., inputs that are almost indistinguishable from natural data and yet classified incorrectly by the network.
To address this problem, we study the adversarial robustness of neural networks through the lens of robust optimization.
This approach provides us with a broad and unifying view on much prior work on this topic.
Its principled nature also enables us to identify methods for both training and attacking neural networks that are reliable and, in a certain sense, universal.
In particular, they specify a concrete security guarantee that would protect against a well-defined class of adversaries.
These methods let us train networks with significantly improved resistance to a wide range of adversarial attacks.
They also suggest robustness against a first-order adversary as a natural security guarantee.
We believe that robustness against such well-defined classes of adversaries is an important stepping stone towards fully resistant deep learning models.

assistant: text: Investigates a minimax formulation of deep network learning to increase their robustness, using projected gradient descent as the main adversary.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Stochastic Gradient Push for Distributed Deep Learning

Paper content:
Large mini-batch parallel SGD is commonly used for distributed training of deep networks.
Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication.
In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training.
SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios.
The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies.
We prove that SGP converges to a stationary point of smooth, non-convex objective functions.
Furthermore, we validate empirically the potential of SGP.
For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.5 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

assistant: text: The authors propose using gossip algorithms as a general method of computing approximate average over a set of workers approximately
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Self-Supervised GAN Compression

Paper content:
Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters.
These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements.
Some model compression methods have been successfully applied on image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks.
In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods.
We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator.
We show that this framework has a compelling performance to high degrees of sparsity, generalizes well to new tasks and models, and enables meaningful comparisons between different pruning granularities.

assistant: text: Existing pruning methods fail when applied to GANs tackling complex tasks, so we present a simple and robust method to prune generators that works well for a wide variety of networks and tasks.