ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Examining Interpretable Feature Relationships in Deep Networks for Action recognition

Paper content:
A number of recent methods to understand neural networks have focused on quantifying the role of individual features.
One such method, NetDissect identifies interpretable features of a model using the Broden dataset of visual semantic labels (colors, materials, textures, objects and scenes).
Given the recent rise of a number of action recognition datasets, we propose extending the Broden dataset to include actions to better analyze learned action models.
We describe the annotation process, results from interpreting action recognition models on the extended Broden dataset and examine interpretable feature paths to help us understand the conceptual hierarchy used to classify an action.
The success of Deep convolutional neural networks (DNNs) is partly due to their ability to learn hidden representations that capture the important factors of variation in the data.
Previous works have visualized the units of deep convolutional networks by sampling image patches that maximize the activation of each feature BID6 or by generating images that maximize each feature activation.
Such visualizations show that individual features act as visual concept detectors.
Features at lower layers detect concrete patterns such as textures or shapes while features at higher layers detect more semantically meaningful concepts such as dog heads or bicycle wheels.
One tool for network interpretability (NetDissect) BID0 BID8 uses the Broden dataset (consists of objects, scenes, object parts, textures and materials) to evaluate individual units.Recently, DNNs have shown significant progress in action recognition with the introduction of large-scale video datasets.
However, while NetDissect with the Broden dataset is appropriate for networks trained on object or scene recognition, it does not include the ability to detect learned action concepts.In this paper, we propose extending the Broden dataset to include actions so that we can more appropriately interpret action recognition networks.
We describe our annotation process to collect images across action classes and select Sample videos Example frames from a few videos to show intraclass action variation Action Regions Spatial localization of actions in single frames for network interpretation Action Interpretation Identifying interpretable action features regions of importance for identifying each action.
We then show results using our Action Region dataset together with the existing Broden set to identify interpretable action features in deep networks trained for action recognition.
The Action Region dataset presented, and the code for integrating with NetDissect, will be made available online.

assistant: text: We expand Network Dissection to include action interpretation and examine interpretable feature paths to understand the conceptual hierarchy used to classify an action.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS

Paper content:
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.
The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium.
We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
The introduction of normalizing layers to neural networks has in no small part contributed to the deep learning revolution in machine learning.
The most successful of these techniques in the image classification domain is the batch normalization (BatchNorm) layer BID4 , which works by normalizing the univariate first and second order statistics between layers.Batchnorm has seen near universal adoption in image classification tasks due to its surprisingly multifaceted benefits.
Compared to an unnormalized network, its has been widely observed that using batch norm empirically results in:• Stability over a wide range of step sizes • Faster convergence (particularly with larger step sizes)• Improved generalizationThe multiple effects of BatchNorm make it both hard to replace and hard to analyze.
In this paper we introduce Equilibrium Normalization (EquiNorm), a normalization that works in weight space and still uses a form of batch statistics unlike previous weight space approaches.
EquiNorm results in very rapid convergence, even more so than BatchNorm, however as we will show in our experiments, this also results in a tendency to overfit.
When combined with additional regularisation, EquiNorm can significantly outperform BatchNorm, which benefits less from this additional regularisation.

assistant: text: An alternative normalization technique to batch normalization
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Deep Mean Field Games for Modeling Large Population Behavior

Paper content:
We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space.
A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions.
We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP.
This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning.
Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.

assistant: text: The authors deal with inference in models of collective behavior by using inverse reinforcement learning to learn the reward functions of agents in the model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Random Partition Relaxation for Training Binary and Ternary Weight Neural Network

Paper content:
We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values.
Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation.
We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks.
We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.

assistant: text: Authors propose RPR, a way to randomly partition and quantize weights and train the remaining parameters followed by relaxation in alternate cycles to train quantized models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Role of two learning rates in convergence of model-agnostic meta-learning

Paper content:
Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method.
However, MAML is notorious for being hard to train because of the existence of two learning rates.
Therefore, in this paper, we derive the conditions that inner learning rate $\alpha$ and meta-learning rate $\beta$ must satisfy for MAML to converge to minima with some simplifications.
We find that the upper bound of $\beta$ depends on $ \alpha$, in contrast to the case of using the normal gradient descent method.
Moreover, we show that the threshold of $\beta$ increases as $\alpha$ approaches its own upper bound.
This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network.
Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible $\alpha$; next, tune $\beta$ based on the chosen value of $\alpha$.

assistant: text: We analyzed the role of two learning rates in model-agnostic meta-learning in convergence.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Fast is better than free: Revisiting adversarial training

Paper content:
Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD).
In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice.
Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost.
Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy at epsilon=8/255 in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at epsilon=2/255 in 12 hours, in comparison to past work based on ``free'' adversarial training which took 10 and 50 hours to reach the same respective thresholds.

assistant: text: FGSM-based adversarial training, with randomization, works just as well as PGD-based adversarial training: we can use this to train a robust classifier in 6 minutes on CIFAR10, and 12 hours on ImageNet, on a single machine.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Flexible, Extensible Software Framework for Neural Net Compression

Paper content:
We propose a software framework based on ideas of the Learning-Compression algorithm , that allows one to compress any neural network by different compression mechanisms (pruning, quantization, low-rank, etc.).
By design, the learning of the neural net (handled by SGD) is decoupled from the compression of its parameters (handled by a signal compression function), so that the framework can be easily extended to handle different combinations of neural net and compression type.
In addition, it has other advantages, such as easy integration with deep learning frameworks, efficient training time, competitive practical performance in the loss-compression tradeoff, and reasonable convergence guarantees.
Our toolkit is written in Python and Pytorch and we plan to make it available by the workshop time, and eventually open it for contributions from the community.

assistant: text: This paper presents the design of a software library that makes it easier for the user to compress their networks by hiding away the details of the compression methods.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Combination of Supervised and Reinforcement Learning For Vision-Based Autonomous Control

Paper content:
Reinforcement learning methods have recently achieved impressive results on a wide range of control problems.
However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution.
This limitation largely prohibits  their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control.
Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account reward-based control policies and is not capable to provide independent control policies.
In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments.
We use SpeedDreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality.
Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

assistant: text: The new combination of reinforcement and supervised learning, dramatically decreasing the number of required samples for training on video
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dynamic Channel Pruning: Feature Boosting and Suppression

Paper content:
Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.
In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time.
FBS introduces small auxiliary connections to existing convolutional layers.
In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels.
FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs.
We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification.
Experiments show that FBS can respectively provide 5× and 2× savings in compute on VGG-16 and ResNet-18, both with less than 0.6% top-5 accuracy loss.

assistant: text: A feature boosting and suppression method for dynamic channel pruning that predicts the importance of each channel and then uses an affine function to amplify/suppress channel importance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Train Neural Network by Embedding Space Probabilistic Constraint 

Paper content:
Using higher order knowledge to reduce training data has become a popular research topic.
However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.
Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.
We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.
While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.
We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

assistant: text: We introduce an embedding space approach to constrain neural network output probability distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Neural Trees

Paper content:
Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures.
We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers).
ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects.
We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset.
Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.

assistant: text: The authors proposed a new model, Adaptive Neural Trees, by combining the representation learning and gradient optimization of neural networks with architecture learning of decision trees
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Probing Emergent Semantics in Predictive Agents via Question Answering

Paper content:
Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments.
We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling – action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019).
After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent.
The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment.
Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach.
By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

assistant: text: We use question-answering to evaluate how much knowledge about the environment can agents learn by self-supervised prediction.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Robust Few-Shot Learning with Adversarially Queried Meta-Learners

Paper content:
Previous work on adversarially robust neural networks requires large training sets and computationally expensive training procedures.
On the other hand, few-shot learning methods are highly vulnerable to adversarial examples.
The goal of our work is to produce networks which both perform well at few-shot tasks and are simultaneously robust to adversarial examples.
We adapt adversarial training for meta-learning, we adapt robust architectural features to small networks for meta-learning, we test pre-processing defenses as an alternative to adversarial training for meta-learning, and we investigate the advantages of robust meta-learning over robust transfer-learning for few-shot tasks.
This work provides a thorough analysis of adversarially robust methods in the context of meta-learning, and we lay the foundation for future work on defenses for few-shot tasks.

assistant: text: We develop meta-learning methods for adversarially robust few-shot learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adversarial Examples Are a Natural Consequence of Test Error in Noise

Paper content:
Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to making small modifications of a correctly handled input.
At the same time, less surprisingly, image classifiers lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise.
In this work, we show that these are two manifestations of the same underlying phenomenon.
We establish this connection in several ways.
First, we find that adversarial examples exist at the same distance scales we would expect from a linear model with the same performance on corrupted images.
Next, we show that Gaussian data augmentation during training improves robustness to small adversarial perturbations and that adversarial training improves robustness to several types of image corruptions.
Finally, we present a model-independent upper bound on the distance from a corrupted image to its nearest error given test performance and show that in practice we already come close to achieving the bound, so that improving robustness further for the corrupted image distribution requires significantly reducing test error.
All of this suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions.
This yields a computationally tractable evaluation metric for defenses to consider: test error in noisy image distributions.

assistant: text: This paper proposes an alternative view for adversarial examples in high dimension spaces by considering the "error rate" in a Gaussian distribution centered at each test point.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Study of a Simple, Expressive and Consistent Graph Feature Representation

Paper content:
Graphs possess exotic features like variable size and absence of natural ordering of the nodes that make them difficult to analyze and compare.
To circumvent this problem and learn on graphs, graph feature representation is required.
Main difficulties with feature extraction lie in the trade-off between expressiveness, consistency and efficiency, i.e. the capacity to extract features that represent the structural information of the graph while being deformation-consistent and isomorphism-invariant.
While state-of-the-art methods enhance expressiveness with powerful graph neural-networks, we propose to leverage natural spectral properties of graphs to study a simple graph feature: the graph Laplacian spectrum (GLS).
We analyze the representational power of this object that satisfies both isomorphism-invariance, expressiveness and deformation-consistency.
In particular, we propose a theoretical analysis based on graph perturbation to understand what kind of comparison between graphs we do when comparing GLS.
To do so, we derive bounds for the distance between GLS that are related to the divergence to isomorphism, a standard computationally expensive graph divergence.
Finally, we experiment GLS as graph representation through consistency tests and classification tasks, and show that it is a strong graph feature representation baseline.

assistant: text: We study theoretically the consistency the Laplacian spectrum and use it as whole-graph embeddding
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Context-Gated Convolution

Paper content:
As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.
Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.
In these works, the global context information is incorporated into local features before they are fed into convolutional layers.
However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.
Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.
As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.
Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

assistant: text: This paper uses global context to modulate the weights of convolutional layers and help CNNs capture more discriminative features with high performance and fewer parameters than feature map modulating.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training-Free Uncertainty Estimation for Neural Networks

Paper content:
Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas.
However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it.
None of the previous methods are able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it.
To address this gap, we perform the first systematic exploration into training-free uncertainty estimation.
We propose three simple and scalable methods to analyze the variance of output from a trained network under tolerable perturbations: infer-transformation, infer-noise, and infer-dropout.
They operate solely during inference, without the need to re-train, re-design, or fine-tune the model, as typically required by other state-of-the-art uncertainty estimation methods.
Surprisingly, even without involving such perturbations in training, our methods produce comparable or even better uncertainty estimation when compared to other training-required state-of-the-art methods.
Last but not least, we demonstrate that the uncertainty from our proposed methods can be used to improve the neural network training.

assistant: text: A set of methods to obtain uncertainty estimation of any given model without re-designing, re-training, or to fine-tuning it.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scalable Rule Learning in Probabilistic Knowledge Bases

Paper content:
Knowledge Bases (KBs) are becoming increasingly large, sparse and probabilistic.
These KBs are typically used to perform query inferences and rule mining.
But their efficacy is only as high as their completeness.
Efficiently utilizing incomplete KBs remains a major challenge as the current KB completion techniques either do not take into account the inherent uncertainty associated with each KB tuple or do not scale to large KBs.
Probabilistic rule learning not only considers the probability of every KB tuple but also tackles the problem of KB completion in an explainable way.
For any given probabilistic KB, it learns probabilistic first-order rules from its relations to identify interesting patterns.
But, the current probabilistic rule learning techniques perform grounding to do probabilistic inference for evaluation of candidate rules.
It does not scale well to large KBs as the time complexity of inference using grounding is exponential over the size of the KB.
In this paper, we present SafeLearner -- a scalable solution to probabilistic KB completion that performs probabilistic rule learning using lifted probabilistic inference -- as faster approach instead of grounding.
We compared SafeLearner to the state-of-the-art probabilistic rule learner ProbFOIL+ and to its deterministic contemporary AMIE+ on standard probabilistic KBs of NELL (Never-Ending Language Learner) and Yago.
Our results demonstrate that SafeLearner scales as good as AMIE+ when learning simple rules and is also significantly faster than ProbFOIL+.

assistant: text: A model for probabilistic rule learning to automate the completion of probabilistic databases that uses AMIE+ and lifted inference to help computational efficiency.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Real or Fake: An Empirical Study and Improved Model for Fake Face Detection

Paper content:
Now GANs can generate more and more realistic face images that can easily fool human beings.
In contrast, a common convolutional neural network(CNN), e.g. ResNet-18, can achieve more than 99.9% accuracy in discerning fake/real faces if training and testing faces are from the same source.
In this paper, we performed both human studies and CNN experiments, which led us to two important findings.
One finding is that the textures of fake faces are substantially different from real ones.
CNNs can capture local image texture information for recognizing fake/real face, while such cues are easily overlooked by humans.
The other finding is that global image texture information is more robust to image editing and generalizable to fake faces from different GANs and datasets.
Based on the above findings, we propose  a  novel  architecture  coined  as  Gram-Net,  which  incorporates  “Gram Block” in multiple semantic levels to extract global image texture representations.
Experimental results demonstrate that our Gram-Net performs better than existing approaches for fake face detection.
Especially, our Gram-Net is more robust to image editing, e.g.  downsampling, JPEG compression, blur, and noise.
More importantly, our Gram-Net generalizes significantly better in detecting fake faces from GAN models not seen in the training phase.

assistant: text: The paper proposes a way to improve model performance for fake face detection in images generated by a GAN to be more generalizable based on texture information.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Meta-Teaching: Curriculum Generation for Lifelong Learning

Paper content:
Meta-learning will be crucial to creating lifelong, generalizable AI.
In practice, however, it is hard to define the meta-training task distribution that is used to train meta-learners.
If made too small, tasks are too similar for a model to meaningfully generalize.
If made too large, generalization becomes incredibly difficult.
We argue that both problems can be alleviated by introducing a teacher model that controls the sequence of tasks that a meta-learner is trained on.
This teacher model is incentivized to start the student meta-learner on simple tasks then adaptively increase task difficulty in response to student progress.
While this approach has been previously studied in curriculum generation, our main contribution is in extending it to meta-learning.

assistant: text: Teacher that trains meta-learners like humans
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Do deep neural networks possess concept space grid cells?

Paper content:
Place and grid-cells are known to aid navigation in animals and humans.
Together with concept cells, they allow humans to form an internal representation of the external world, namely the concept space.
We investigate the presence of such a space in deep neural networks by plotting the activation profile of its hidden layer neurons.
Although place cell and concept-cell like properties are found, grid-cell like firing patterns are absent thereby indicating a lack of path integration or feature transformation functionality in trained networks.
Overall, we present a plausible inadequacy in current deep learning practices that restrict deep networks from performing analogical reasoning and memory retrieval tasks.

assistant: text: We investigated if simple deep networks possess grid cell-like artificial neurons while memory retrieval in the learned concept space.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

Paper content:
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients.
Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs).
We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices.
This is done by parametrizing with a skew-symmetric matrix using the Cayley transform.
Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones.
The proposed training scheme involves a straightforward gradient calculation and update step.
In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

assistant: text: A novel approach to maintain orthogonal recurrent weight matrices in a RNN.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Explanation  by Progressive  Exaggeration

Paper content:
As machine learning methods see greater adoption and implementation in high stakes applications such
as medical image diagnosis, the need for model interpretability and explanation has become more
critical.
Classical approaches that assess feature importance (eg saliency maps) do not explain how and why a particular region of an image is relevant to the prediction.
We propose
a method that explains the outcome of a classification black-box by gradually exaggerating
the semantic effect of a given class.
Given a query input to a classifier, our method produces a
progressive set of plausible variations of that query, which gradually change the posterior probability
from its original class to its negation.
These counter-factually generated samples preserve features
unrelated to the classification decision, such that a user can employ our method as a ``tuning knob'' to traverse a data manifold while crossing the decision boundary.
Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.

assistant: text: A method to explain a classifier, by generating visual perturbation of an image by exaggerating  or diminishing the semantic features that the classifier associates with a target label.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improved Modeling of Complex Systems Using Hybrid Physics/Machine Learning/Stochastic Models

Paper content:
Combining domain knowledge models with neural models has been challenging.
End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.
In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems.
The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.
We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.
Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.
Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

assistant: text: Improved modeling of complex systems uses hybrid neural/domain model composition, new decorrelation loss functions and extrapolative test sets
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: STRUCTURED ALIGNMENT NETWORKS

Paper content:
Many tasks in natural language processing involve comparing two sentences to compute some notion of relevance, entailment, or similarity.
Typically this comparison is done either at the word level or at the sentence level, with no attempt to leverage the inherent structure of the sentence.
When sentence structure is used for comparison, it is obtained during a non-differentiable pre-processing step, leading to propagation of errors.
We introduce a model of structured alignments between sentences, showing how to compare two sentences by matching their latent structures.
Using a structured attention mechanism, our model matches possible spans in the first sentence to possible spans in the second sentence, simultaneously discovering the tree structure of each sentence and performing a comparison, in a model that is fully differentiable and is trained only on the comparison objective.
We evaluate this model on two sentence comparison tasks: the Stanford natural language inference dataset and the TREC-QA dataset.
We find that comparing spans results in superior performance to comparing words individually, and that the learned trees are consistent with actual linguistic structures.

assistant: text: This paper introduces a structured attention mechanisms to compute alignment scores among all possible spans in two given sentences
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Paper content:
A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner.
These representations are typically used as general purpose features for words across a range of NLP problems.
However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem.
Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations.
In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model.
We train this model on several data sources with multiple training objectives on over 100 million sentences.
Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods.
We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

assistant: text: A large-scale multi-task learning framework with diverse training objectives to learn fixed-length sentence representations
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On the Sensitivity of Adversarial Robustness to Input Data Distributions

Paper content:
Neural networks are vulnerable to small adversarial perturbations.
Existing literature largely focused on understanding and mitigating the vulnerability of learned models.
In this paper, we demonstrate an intriguing phenomenon about the most popular robust training method in the literature, adversarial training: Adversarial robustness, unlike clean accuracy, is sensitive to the input data distribution.
Even a semantics-preserving transformations on the input data distribution can cause a significantly different robustness for the adversarial trained model that is both trained and evaluated on the new distribution.
Our discovery of such sensitivity on data distribution is based on a study which disentangles the behaviors of clean accuracy and robust accuracy of the Bayes classifier.
Empirical investigations further confirm our finding.
We construct semantically-identical variants for MNIST and CIFAR10 respectively, and show that standardly trained models achieve comparable clean accuracies on them, but adversarially trained models achieve significantly different robustness accuracies.
This counter-intuitive phenomenon indicates that input data distribution alone can affect the adversarial robustness of trained neural networks, not necessarily the tasks themselves.
Lastly, we discuss the practical implications on evaluating adversarial robustness, and make initial attempts to understand this complex phenomenon.

assistant: text: Paper clarifies the difference between clean and robust accuracy and shows that changing the marginal distribution of the input data P(x) while preserving its semantic P(y|x) affects the robustness of the model.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Compression of Deep Neural Networks by combining pruning and low rank decomposition

Paper content:
Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments such as, mobile phones, IOT edge devices as well as "inferencing as a service" environments on the cloud.
Prior work has considered reduction in the size of the models, through compression techniques like weight pruning, filter pruning, etc. or through low-rank decomposition of the convolution layers.
In this paper, we demonstrate the use of multiple techniques to achieve not only higher model compression but also reduce the compute resources required during inferencing.
We do filter pruning followed by low-rank decomposition using Tucker decomposition for model compression.
We show that our approach achieves upto 57\% higher model compression when compared to either Tucker Decomposition or Filter pruning alone  at similar accuracy for GoogleNet.
Also, it reduces the Flops by upto 48\% thereby making the inferencing faster.

assistant: text: Combining orthogonal model compression techniques to get significant reduction in model size and number of flops required during inferencing.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discrete Transformer

Paper content:
The transformer has become a central model for many NLP tasks from translation to language modeling to representation learning.
Its success demonstrates the effectiveness of stacked attention as a replacement for recurrence for many tasks.
In theory attention also offers more insights into the model’s internal decisions; however, in practice when stacked it quickly becomes nearly as fully-connected as recurrent models.
In this work, we propose an alternative transformer architecture, discrete transformer, with the goal of better separating out internal model decisions.
The model uses hard attention to ensure that each step only depends on a fixed context.
Additionally, the model uses a separate “syntactic” controller to separate out network structure from decision making.
Finally we show that this approach can be further sparsified with direct regularization.
Empirically, this approach is able to maintain the same level of performance on several datasets, while discretizing reasoning decisions over the data.

assistant: text: Discrete transformer which uses hard attention to ensure that each step only depends on a fixed context.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improved Modeling of Complex Systems Using Hybrid Physics/Machine Learning/Stochastic Models

Paper content:
Combining domain knowledge models with neural models has been challenging.
End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.
In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems.
The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.
We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.
Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.
Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling.

assistant: text: This paper presents approaches for combining neural network with non-NN models to predict behavior of complex physical systems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generative Question Answering: Learning to Answer the Whole Question

Paper content:
Discriminative  question  answering  models  can  overfit  to  superficial  biases  in datasets,  because their loss function saturates when any clue makes the answer likely.
We introduce generative models of the joint distribution of questions and answers, which are trained to explain the whole question, not just to answer it.
Our  question  answering  (QA)  model  is  implemented  by  learning  a  prior  over answers,  and  a  conditional  language  model  to  generate  the  question  given  the answer—allowing scalable and interpretable many-hop reasoning as the question is generated word-by-word.
Our model achieves competitive performance with specialised discriminative models on the SQUAD and CLEVR benchmarks, indicating that it is a more general architecture for language understanding and reasoning than previous work.
The model greatly improves generalisation both from biased training data and to adversarial testing data, achieving a new state-of-the-art on ADVERSARIAL SQUAD.
We will release our code.

assistant: text: This paper introduces a generative model for question answering and proposes to model p(q,a|c), factorized as p(a|c) * p(q|a,c).