ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification

Paper content:
Modern deep neural networks (DNNs) require high memory consumption and large computational loads.
In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods.
Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices.
However, it is hard to measure the ranks of DNN layers during the training process.
Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step.
The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate.
In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights.
To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding.
Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer.
Singular value pruning is applied at the end to reach a low-rank model.
We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.

assistant: text: Proposal to parametrize each layer of a deep neural network, before training, with a low-rank matrix decomposition, accordingly replace convolutions with two consecutive convolutions, and then train the decomposed method.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Toward predictive machine learning for active vision

Paper content:
We develop a comprehensive description of the active inference framework, as proposed by Friston (2010), under a machine-learning compliant perspective.
Stemming from a biological inspiration and the auto-encoding principles, a sketch of a cognitive architecture is proposed that should provide ways to implement estimation-oriented control policies.
Computer simulations illustrate the effectiveness of the approach through a foveated inspection of the input data.
The pros and cons of the control policy are analyzed in detail, showing interesting promises in terms of processing compression.
Though optimizing future posterior entropy over the actions set is shown enough to attain locally optimal action selection, offline calculation using class-specific saliency maps is shown better for it saves processing costs through saccades pathways pre-processing, with a negligible effect on the recognition/compression rates.

assistant: text: Presents a computational framework for the active vision problem and explains how the control policy can be learned to reduce the entropy of the posterior belief.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An Empirical Study of Encoders and Decoders in Graph-Based Dependency Parsing

Paper content:
Graph-based dependency parsing consists of two steps: first, an encoder produces a feature representation for each parsing substructure of the input sentence, which is then used to compute a score for the substructure; and second, a decoder} finds the parse tree whose substructures have the largest total score.
Over the past few years, powerful neural techniques have been introduced into the encoding step which substantially increases parsing accuracies.
However, advanced decoding techniques, in particular high-order decoding, have seen a decline in usage.
It is widely believed that contextualized features produced by neural encoders can help capture high-order decoding information and hence diminish the need for a high-order decoder.
In this paper, we empirically evaluate the combinations of different neural and non-neural encoders with first- and second-order decoders and provide a comprehensive analysis about the effectiveness of these combinations with varied training data sizes.
We find that: first, when there is large training data, a strong neural encoder with first-order decoding is sufficient to achieve high parsing accuracy and only slightly lags behind the combination of neural encoding and second-order decoding; second, with small training data, a non-neural encoder with a second-order decoder outperforms the other combinations in most cases.

assistant: text: An empirical study that examines the effectiveness of different encoder-decoder combinations for the task of dependency parsing
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AutoQ: Automated Kernel-Wise Neural Network Quantization 

Paper content:
Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices.
Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy.
The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead.
To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs.
However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large.
The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results.
It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy.
In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer.
Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

assistant: text: Accurate, Fast and Automated Kernel-Wise Neural Network Quantization with Mixed Precision using Hierarchical Deep Reinforcement Learning
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Analyzing Inverse Problems with Invertible Neural Networks

Paper content:
For many applications, in particular in natural science, the task is to
determine hidden system parameters from a set of measurements.
Often,
the forward process from parameter- to measurement-space is well-defined,
whereas the inverse problem is ambiguous: multiple parameter sets can
result in the same measurement.
To fully characterize this ambiguity, the full
posterior parameter distribution, conditioned on an observed measurement,
has to be determined.
We argue that a particular class of neural networks
is well suited for this task – so-called Invertible Neural Networks (INNs).
Unlike classical neural networks, which attempt to solve the ambiguous
inverse problem directly, INNs focus on learning the forward process, using
additional latent output variables to capture the information otherwise
lost.
Due to invertibility, a model of the corresponding inverse process is
learned implicitly.
Given a specific measurement and the distribution of
the latent variables, the inverse pass of the INN provides the full posterior
over parameter space.
We prove theoretically and verify experimentally, on
artificial data and real-world problems from medicine and astrophysics, that
INNs are a powerful analysis tool to find multi-modalities in parameter space,
uncover parameter correlations, and identify unrecoverable parameters.

assistant: text: The author proposes to use invertible networks to solve ambiguous inverse problems and suggest to not only train the forward model, but also the inverse model with an MMD critic.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Approximation capability of neural networks on sets of probability measures and tree-structured data

Paper content:
This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.
By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces.
The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions.
The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer.
This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.
This work has been motivated by recently proposed solutions to multi-instance learning BID28 ; BID19 ; BID5 and by mean-map embedding of probability measures BID23 .
It generalizes the universal approximation theorem of neural networks to compact sets of probability measures over compact subsets of Euclidean spaces.
Therefore, it can be seen as an adaptation of the mean-map framework to the world of neural networks, which is important for comparing probability measures and for multi-instance learning, and it proves the soundness of the constructions of BID19 ; BID5 .The
universal approximation theorem is extended to inputs with a tree schema (structure) which, being the basis of many data exchange formats like JSON, XML, ProtoBuffer, Avro, etc., are nowadays ubiquitous. This
theoretically justifies applications of (MIL) neural networks in this setting.As the presented proof relies on the Stone-Weierstrass theorem, it restricts non-linear functions in neural networks to be continuous in all but the last non-linear layer. Although
this does not have an impact on practical applications (all commonly use nonlinear functions within neural networks are continuous) it would be interesting to generalize the result to non-continuous non-linearities, as has been done for feed-forward neural networks in BID14 .

assistant: text: This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings

Paper content:
Style transfer usually refers to the task of applying color and texture information from a specific style image to a given content image while preserving the structure of the latter.
Here we tackle the more generic problem of semantic style transfer: given two unpaired collections of images, we aim to learn a mapping between the corpus-level style of each collection, while preserving semantic content shared across the two domains.
We introduce XGAN ("Cross-GAN"), a dual adversarial autoencoder, which captures a shared representation of the common domain semantic content in an unsupervised way, while jointly learning the domain-to-domain image translations in both directions.
We exploit ideas from the domain adaptation literature and define a semantic consistency loss which encourages the model to preserve semantics in the learned embedding space.
We report promising qualitative results for the task of face-to-cartoon translation.
The cartoon dataset we collected for this purpose will also be released as a new benchmark for semantic style transfer.

assistant: text: XGAN is an unsupervised model for feature-level image-to-image translation applied to semantic style transfer problems such as the face-to-cartoon task, for which we introduce a new dataset.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compositional Transfer in Hierarchical Reinforcement Learning

Paper content:
The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency.
To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time.
To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL).
We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference.
Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions.
Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives.
We design an RL algorithm that enables stable and fast learning of structured policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting.
Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and  demonstrate substantial improvements in data data-efficiency over competitive baselines.

assistant: text: The authors introduce a hierarchical policy structure for use in both single task and multitask reinforcement learning, and assess the structure's usefulness on complex robotic tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Intrinsic Motivation for Encouraging Synergistic Behavior

Paper content:
We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually.
Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent.
We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy.
While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken.
We validate our approach in robotic bimanual manipulation tasks with sparse rewards; we find that our approach yields more efficient learning than both
1) training with only the sparse reward and
2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior.
Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.

assistant: text: We propose a formulation of intrinsic motivation that is suitable as an exploration bias in multi-agent sparse-reward synergistic tasks, by encouraging agents to affect the world in ways that would not be achieved if they were acting individually.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dimensional Reweighting Graph Convolution Networks

Paper content:
In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs.
We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.
However, practically, we find that the degrees DrGCNs help vary severely on different datasets.
We revisit the problem and develop a new measure K to quantify the effect.
This measure guides when we should use dimensional reweighting in GCNs and how much it can help.
Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs.
The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants.
Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches.
Significant improvements can also be observed on a large scale industrial dataset.

assistant: text: We propose a simple yet effective reweighting scheme for GCNs, theoretically supported by the mean field theory.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Frequency-based Search-control in Dyna

Paper content:
Model-based reinforcement learning has been empirically demonstrated as a successful strategy to improve sample efficiency.
Particularly, Dyna architecture, as an elegant model-based architecture integrating learning and planning, provides huge flexibility of using a model.
One of the most important components in Dyna is called search-control, which refers to the process of generating state or state-action pairs from which we query the model to acquire simulated experiences.
Search-control is critical to improve learning efficiency.
In this work, we propose a simple and novel search-control strategy by searching high frequency region on value function.
Our main intuition is built on Shannon sampling theorem from signal processing, which indicates that a high frequency signal requires more samples to reconstruct.
We empirically show that a high frequency function is more difficult to approximate.
This suggests a search-control strategy: we should use states in high frequency region of the value function to query the model to acquire more samples.
We develop a simple strategy to locally measure the frequency of a function by gradient norm, and provide theoretical justification for this approach.
We then apply our strategy to search-control in Dyna, and conduct experiments to show its property and effectiveness on benchmark domains.

assistant: text: Acquire states from high frequency region for search-control in Dyna.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Context Dependent Modulation of Activation Function

Paper content:
We propose a modification to traditional Artificial Neural Networks (ANNs), which provides the ANNs with new aptitudes motivated by biological neurons.
Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information.
A biological neuron change firing modes accordingly to peripheral factors (e.g., neuromodulators) as well as intrinsic ones.
Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns.
In this manner, we enable the slope of the activation function to be context dependent.
This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.

assistant: text: A method to scale the activations of a layer of neurons in an ANN depending on inputs to that layer that reports improvements above the baselines.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ridge Regression: Structure, Cross-Validation, and Sketching

Paper content:
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator?
(2) how to correctly use cross-validation to choose the regularization parameter?
and (3) how to accelerate computation without losing too much accuracy?
We consider the three problems in a unified large-data linear model.
We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise.
We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.
We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.
Our results are illustrated by simulations and by analyzing empirical data.
Ridge or 2 -regularized regression is a widely used method for prediction and estimation when the data dimension p is large compared to the number of datapoints n.
This is especially so in problems with many good features, where sparsity assumptions may not be justified.
A great deal is known about ridge regression.
It is Bayes optimal for any quadratic loss in a Bayesian linear model where the parameters and noise are Gaussian.
The asymptotic properties of ridge have been widely studied (e.g., Tulino & Verdú, 2004; Serdobolskii, 2007; Couillet & Debbah, 2011; Dicker, 2016; Dobriban & Wager, 2018, etc) .
For choosing the regularization parameter in practice, cross-validation (CV) is widely used.
In addition, there is an exact shortcut (e.g., Hastie et al., 2009, p. 243) , which has good consistency properties (Hastie et al., 2019) .
There is also a lot of work on fast approximate algorithms for ridge, e.g., using sketching methods (e.g., el Alaoui & Mahoney, 2015; Chen et al., 2015; Wang et al., 2018; Chowdhury et al., 2018, among

assistant: text: A theoretical study of ridge regression by exploiting a new asymptotic characterisation of the ridge regression estimator.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AutoQ: Automated Kernel-Wise Neural Network Quantization 

Paper content:
Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices.
Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy.
The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead.
To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs.
However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large.
The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results.
It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy.
In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer.
Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

assistant: text: This paper proposes to automatically search quantization schemes for each kernel in the neural network, using hierarchial RL to guide the search.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Multi-agent Reinforcement Learning for Networked System Control

Paper content:
This paper considers multi-agent reinforcement learning (MARL) in networked system control.
Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors.
We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent.
Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL.
Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

assistant: text: Concerned with N-MARL's where agents update their policy based only on messages from neighboring nodes, showing that introducing a spatial discount factor stabilizes learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Amharic Light Stemmer

Paper content:
Stemming is the process of removing affixes( i.e. prefixes, infixes and suffixes) that improve the accuracy and performance of information retrieval systems.This paper presents the reduction of Amharic words to corresponding stem where with the intention that it preserves semantic information.
The proposed approach efficiently removes affixes from an Amharic word.
The process of removing such affixes (prefixes, infixes and suffixes) from a word to its base form is called stemming.
While many stemmers exist for dominant languages such as English, under resourced languages such as Amharic which lacks such powerful tool support.
In this paper, we design a light Amharic stemmer relying on the rules that receives an Amharic word and then it finds a match to the beginning of a word to the possible prefixes and to its ending with the possible suffixes and finally it checks whether it has infix.
The final result is the stem if there is any prefix, infix or/and suffix, otherwise it remains in one of the earlier states.
The technique does not rely on any additional resource (e.g. dictionary) to verify the generated stem.
The performance of the generated stemmer is evaluated using manually annotated Amharic words.
The result is compared with current state-of-the-art stemmer for Amharic showing an increase of 7% in stemmer correctness.

assistant: text: This paper proposes a technique for Amharic light stemming using a cascade of transformations that standardize the form, remove suffixes, prefixes, and infixes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Certified Robustness for Top-k Predictions against Adversarial Perturbations via Randomized Smoothing

Paper content:
It is well-known that  classifiers are vulnerable to adversarial perturbations.
To defend against adversarial perturbations, various certified robustness results have been derived.
However, existing certified robustnesses are limited to top-1 predictions.
In many real-world applications, top-$k$ predictions are more relevant.
In this work, we aim to derive certified robustness for top-$k$ predictions.
In particular, our certified robustness is based on randomized smoothing, which turns any classifier to a new classifier via adding noise to an input example.
We adopt randomized smoothing because it is scalable to large-scale neural networks and applicable to any classifier.
We derive a tight robustness in $\ell_2$ norm for top-$k$ predictions  when using randomized smoothing with Gaussian noise.
We find that generalizing the certified robustness  from top-1 to top-$k$ predictions faces significant technical challenges.
We also empirically evaluate our method on CIFAR10 and ImageNet.
For example, our method can obtain an ImageNet classifier with a certified top-5 accuracy of 62.8\% when the $\ell_2$-norms of the adversarial perturbations are less than 0.5 (=127/255).
Our code is publicly available at: \url{https://github.com/jjy1994/Certify_Topk}.

assistant: text: This paper builds upon the random smoothing technique for top-1 prediction, and aims to provide certification on top-k predictions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Characterizing Sparse Connectivity Patterns in Neural Networks

Paper content:
We propose a novel way of reducing the number of parameters in the storage-hungry fully connected layers of a neural network by using pre-defined sparsity, where the majority of connections are absent prior to starting training.
Our results indicate that convolutional neural networks can operate without any loss of accuracy at less than 0.5% classification layer connection density, or less than 5% overall network connection density.
We also investigate the effects of pre-defining the sparsity of networks with only fully connected layers.
Based on our sparsifying technique, we introduce the `scatter' metric to characterize the quality of a particular connection pattern.
As proof of concept, we show results on CIFAR, MNIST and a new dataset on classifying Morse code symbols, which highlights some interesting trends and limits of sparse connection patterns.

assistant: text: This paper examines sparse connection patterns in upper layers of convolutional image classification networks, and introduces heuristics for distributing connections among windows/groups and a measure called scatter to construct connectivity masks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Accelerating Convolutional Neural Networks using Iterative Two-Pass Decomposition

Paper content:
We present the iterative two-pass decomposition flow to accelerate existing convolutional neural networks (CNNs).
The proposed rank selection algorithm can effectively determine the proper ranks of the target convolutional layers for the low rank approximation.
Our two-pass CP-decomposition helps prevent from the instability problem.
The iterative flow makes the decomposition of the deeper networks systematic.
The experiment results shows that VGG16 can be accelerated with a 6.2x measured speedup while the accuracy drop remains only 1.2%.

assistant: text: This paper addresses the problem of learning a low rank tensor filter operation for filtering layers in deep neural networks (DNNs).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: PRUNING IN TRAINING: LEARNING AND RANKING SPARSE CONNECTIONS IN DEEP CONVOLUTIONAL NETWORKS

Paper content:
This paper proposes a Pruning in Training (PiT) framework of learning to reduce the parameter size of networks.
Different from existing works, our PiT framework employs the sparse penalties to train networks and thus help rank the importance of weights and filters.
Our PiT algorithms can directly prune the network without any fine-tuning.
The pruned networks can still achieve comparable performance to the original networks.
In particular, we introduce the (Group) Lasso-type Penalty (L-P /GL-P), and (Group) Split LBI Penalty (S-P / GS-P) to regularize the networks, and a pruning strategy proposed  is used in help prune the network.
We conduct the extensive experiments on MNIST, Cifar-10, and miniImageNet.
The results validate the efficacy of our proposed methods.
Remarkably, on MNIST dataset, our PiT framework can save 17.5% parameter size of LeNet-5, which achieves the 98.47% recognition accuracy.

assistant: text: This paper introduces an approach to pruning while training a network using lasso and split LBI penalties
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: signSGD with Majority Vote is Communication Efficient and Fault Tolerant

Paper content:
Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines.
As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable.
The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults.
We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD.
Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote.
This algorithm uses 32x less communication per iteration than full-precision, distributed SGD.
Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct.
Aggregating sign gradients by majority vote means that no individual worker has too much power.
We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially.
The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate.
On the practical side, we built our distributed training system in Pytorch.
Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

assistant: text: Presents a distributed implementation of signSGD with majority vote as aggregation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: An implicit function learning approach for parametric modal regression

Paper content:
For multi-valued functions---such as when the conditional distribution on targets given the inputs is multi-modal---standard regression approaches are not always desirable because they provide the conditional mean.
Modal regression approaches aim to instead find the conditional mode, but are restricted to nonparametric approaches.
Such approaches can be difficult to scale, and make it difficult to benefit from parametric function approximation, like neural networks, which can learn complex relationships between inputs and targets.
In this work, we propose a parametric modal regression algorithm, by using the implicit function theorem to develop an objective for learning a joint parameterized function over inputs and targets.
We empirically demonstrate on several synthetic problems that our method
(i) can learn multi-valued functions and produce the conditional modes,
(ii) scales well to high-dimensional inputs and
(iii) is even more effective for certain unimodal problems, particularly for high frequency data where the joint function over inputs and targets can better capture the complex relationship between them.
We conclude by showing that our method provides small improvements on two regression datasets that have asymmetric distributions over the targets.

assistant: text: We introduce a simple and novel modal regression algorithm which is easy to scale to large problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Paper content:
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses).
When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed.
While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents.
This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated.
Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

assistant: text: This paper proposes ON-LSTM, a new RNN unit that integrates the latent tree structure into recurrent models and that has good results on language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Autonomous Scheduling of Agile Spacecraft Constellations with Delay Tolerant Networking for Reactive Imaging

Paper content:
Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.
When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.
In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.
The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.
The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.
In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.
We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.
We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.
The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.
Response and revisit requirements for Earth Observation (EO) vary significantly by application, ranging from less than an hour to monitor disasters, to daily for meteorology, to weekly for land cover monitoring BID33 .
Geostationary satellites provide frequent revisits, but at the cost of coarse spatial resolution, extra launch costs and no polar access.
Lower Earth Orbit satellites overcome these shortcomings, but need numbers and coordination to make up for response characteristics.
Adding agility to satellites and autonomy to the constellation improves the revisit/response for the same number of satellites Copyright © 2019.
All rights reserved.
in given orbits.
However, human operators are expected to scale linearly with constellation nodes BID9 ) and operations staffing may be very costly.

assistant: text: This paper proposes a communication module to optimize the schedule of communication for the problem of spacecraft constellations, and compares the algorithm in distributed and centralized settings.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing the Transferability of Adversarial Examples with Noise Reduced Gradient

Paper content:
Deep neural networks provide state-of-the-art performance for many applications of interest.
Unfortunately they are known to be vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs.
Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models.
Consequently  the adversary can leverage it to attack against the deployed black-box systems.
In this work, we demonstrate that the adversarial perturbation can be decomposed into two components: model-specific and data-dependent one, and it is the latter that mainly contributes to the transferability.
Motivated by this understanding, we propose to craft adversarial examples by utilizing the noise reduced gradient (NRG) which approximates the data-dependent component.
Experiments on various classification models trained on ImageNet demonstrates that the new approach enhances the transferability dramatically.
We also find that low-capacity models have more powerful attack capability than high-capacity counterparts, under the condition that they have comparable test performance.
These insights give rise to a principled manner to construct adversarial examples with high success rates and could potentially provide us guidance for designing effective defense approaches against black-box attacks.

assistant: text: This paper postulates that an adversarial perturbation consists of a model-specific and data-specific component, and that amplification of the latter is best suited for adversarial attacks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reinforcement Learning with Chromatic Networks

Paper content:
We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS and ES in a highly scalable and intuitive way.
By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings.
For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing >90 % compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices, while still maintaining good reward.
We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics with limited storage and computational resources.

assistant: text: We show that ENAS with ES-optimization in RL is highly scalable, and use it to compactify neural network policies by weight sharing.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Paper content:
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure.
The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections.
In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth.
To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training.
We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus.
On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB.
Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

assistant: text: we find 99.9% of the gradient exchange in distributed SGD is redundant; we reduce the communication bandwidth by two orders of magnitude without losing accuracy.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Random Partition Relaxation for Training Binary and Ternary Weight Neural Network

Paper content:
We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values.
Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation.
We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks.
We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.

assistant: text: State-of-the-art training method for binary and ternary weight networks based on alternating optimization of randomly relaxed weight partitions
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Representing dynamically: An active process for describing sequential data

Paper content:
We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions.
The method simultaneously acquires representations of input data and its dynamics.
It is based on a hierarchical generative model composed of two levels.
In the first level, a model learns representations to generate observed data.
In the second level, representational states encode the dynamics of the lower one.
The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models.
The method actively explores the latent space guided by its knowledge and the uncertainty about it.
That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space.
So, no encoder or inference models are used since the generators also serve as their inverse transformations.
The method is evaluated in two scenarios, with static images and with videos.
The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders.
With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

assistant: text: A method that build representations of sequential data and its dynamics through generative models with an active process