ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Context-Gated Convolution

Paper content:
As the basic building block of Convolutional Neural Networks (CNNs), the convolutional layer is designed to extract local patterns and lacks the ability to model global context in its nature.
Many efforts have been recently made to complement CNNs with the global modeling ability, especially by a family of works on global feature interaction.
In these works, the global context information is incorporated into local features before they are fed into convolutional layers.
However, research on neuroscience reveals that, besides influences changing the inputs to our neurons, the neurons' ability of modifying their functions dynamically according to context is essential for perceptual tasks, which has been overlooked in most of CNNs.
Motivated by this, we propose one novel Context-Gated Convolution (CGC) to explicitly modify the weights of convolutional layers adaptively under the guidance of global context.
As such, being aware of the global context, the modulated convolution kernel of our proposed CGC can better extract representative local patterns and compose discriminative features.
Moreover, our proposed CGC is lightweight, amenable to modern CNN architectures, and consistently improves the performance of CNNs according to extensive experiments on image classification, action recognition, and machine translation.

assistant: text: A novel Context-Gated Convolution which incorporates global context information into CNNs by explicitly modulating convolution kernels, and thus captures more representative local patterns and extract discriminative features.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Feature-Robustness, Flatness and Generalization Error for Deep Neural Networks

Paper content:
The performance of deep neural networks is often attributed to their automated, task-related feature construction.
It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.
Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.
For several flatness measures, this correlation has been empirically validated.
However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily.
This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization.
The proposed measures imply a robustness of the network to changes in the input and the hidden layers.
Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data.
With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

assistant: text: We introduce a novel measure of flatness at local minima of the loss surface of deep neural networks which is invariant with respect to layer-wise reparameterizations and we connect flatness to feature robustness and generalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Classification-Based Anomaly Detection for General Data

Paper content:
Anomaly detection, finding patterns that substantially deviate from those seen previously, is one of the fundamental problems of artificial intelligence.
Recently, classification-based methods were shown to achieve superior results on this task.
In this work, we present a unifying view and propose an open-set method to relax current generalization assumptions.
Furthermore, we extend the applicability of transformation-based methods to non-image data using random affine transformations.
Our method is shown to obtain state-of-the-art accuracy and is applicable to broad data types.
The strong performance of our method is extensively validated on multiple datasets from different domains.

assistant: text: This paper proposes an approach to classification-based anomaly detection for general data by using the affine transformation y = Wx+b.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Execution of Graph Algorithms

Paper content:
Graph Neural Networks (GNNs) are a powerful representational tool for solving problems on graph-structured inputs.
In almost all cases so far, however, they have been applied to directly recovering a final solution from raw inputs, without explicit guidance on how to structure their problem-solving.
Here, instead, we focus on learning in the space of algorithms: we train several state-of-the-art GNN architectures to imitate individual steps of classical graph algorithms, parallel (breadth-first search, Bellman-Ford) as well as sequential (Prim's algorithm).
As graph algorithms usually rely on making discrete decisions within neighbourhoods, we hypothesise that maximisation-based message passing neural networks are best-suited for such objectives, and validate this claim empirically.
We also demonstrate how learning in the space of algorithms can yield new opportunities for positive transfer between tasks---showing how learning a shortest-path algorithm can be substantially improved when simultaneously learning a reachability algorithm.

assistant: text: Suggests training neural networks to imitate graph algorithms by learning primitives and subroutines rather than the final output.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Word2net: Deep Representations of Language

Paper content:
Word embeddings extract semantic features of words from large datasets of text.
Most embedding methods rely on a log-bilinear model to predict the occurrence
of a word in a context of other words.
Here we propose word2net, a method that
replaces their linear parametrization with neural networks.
For each term in the
vocabulary, word2net posits a neural network that takes the context as input and
outputs a probability of occurrence.
Further, word2net can use the hierarchical
organization of its word networks to incorporate additional meta-data, such as
syntactic features, into the embedding model.
For example, we show how to share
parameters across word networks to develop an embedding model that includes
part-of-speech information.
We study word2net with two datasets, a collection
of Wikipedia articles and a corpus of U.S. Senate speeches.
Quantitatively, we
found that word2net outperforms popular embedding methods on predicting held-
out words and that sharing parameters based on part of speech further boosts
performance.
Qualitatively, word2net learns interpretable semantic representations
and, compared to vector-based methods, better incorporates syntactic information.

assistant: text: A method to use non-linear combination of context vectors for learning vector representation of words, where the main idea is to replace each word embedding by a neural network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Combining adaptive algorithms and hypergradient method: a performance and robustness study

Paper content:
Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014).
In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.
As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017).
In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters.
Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.

assistant: text: Reports the results of testing several stepsize adjustment related methods including vanilla SGD, SGD with Neserov momentum, and ADAM and compares those methods with hypergradient and without.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

Paper content:
Training large deep neural networks on massive datasets is  computationally very challenging.
There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue.
The most prominent algorithm in this line of research is LARS, which by  employing layerwise adaptive learning rates trains ResNet on ImageNet in a few minutes.
However, LARS performs poorly for attention models like BERT, indicating that its performance gains are not consistent across tasks.
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Using this strategy, we develop a new layerwise adaptive large batch optimization technique called LAMB; we then provide convergence analysis of LAMB as well as LARS, showing convergence to a stationary point in general nonconvex settings.
Our empirical results demonstrate the superior performance of LAMB across various tasks such as BERT and ResNet-50 training with very little hyperparameter tuning.
In particular, for BERT training, our optimizer enables use of very large batch sizes of 32868 without any degradation of performance.
By increasing the batch size to the memory limit of a TPUv3 Pod, BERT training time can be reduced from 3 days to just 76 minutes (Table 1).

assistant: text: In this paper, the authors made a study on large-batch training for the BERT, and successfully trained a BERT model in 76 minutes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bayesian Inference for Large Scale Image Classification

Paper content:
Bayesian inference promises to ground and improve the performance of deep neural networks.
It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness.
Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters.
Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind  optimization methods for large scale deep learning tasks.
We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network.
ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients.
We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the  absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood.
We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

assistant: text: We scale Bayesian Inference to ImageNet classification and achieve competitive results accuracy and uncertainty calibration.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition

Paper content:
In this paper we study the problem of learning the weights of a deep convolutional neural network.
We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer.
We develop an algorithm for simultaneously learning all the kernels from the training data.
Our approach dubbed Deep Tensor Decomposition (DeepTD) is based on a rank-1 tensor decomposition.
We theoretically investigate DeepTD under a realizable model for the training data where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted convolutional kernels.
We show that DeepTD is data-efficient and provably works as soon as the sample size exceeds the total number of convolutional weights in the network.
Our numerical experiments demonstrate the effectiveness of DeepTD and verify our theoretical findings.

assistant: text: Provides theoretical guarantees for learning deep convolutional neural networks using rank-one tensor decomposition.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Collapse of deep and narrow neural nets

Paper content:
Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem.
This problem can be typically resolved by the rectified linear unit (ReLU) activation.
However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability.
Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives.
We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse.
We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states.
Finally, we examine different ways of initialization and normalization that may avoid the collapse problem.
Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.

assistant: text: This paper shows that the training of deep ReLU neural networks will converge to a constant classifier with high probability over random initialization if hidden layer widths are too small.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Soft Value Iteration Networks for Planetary Rover Path Planning

Paper content:
Value iteration networks are an approximation of the value iteration (VI) algorithm implemented with convolutional neural networks to make VI fully differentiable.
In this work, we study these networks in the context of robot motion planning, with a focus on applications to planetary rovers.
The key challenging task in learning-based motion planning is to learn a transformation from terrain observations to a suitable navigation reward function.
In order to deal with complex terrain observations and policy learning, we propose a value iteration recurrence, referred to as the soft value iteration network (SVIN).
SVIN is designed to produce more effective training gradients through the value iteration network.
It relies on a soft policy model, where the policy is represented with a probability distribution over all possible actions, rather than a deterministic policy that returns only the best action.
We demonstrate the effectiveness of the proposed method in robot motion planning scenarios.
In particular, we study the application of SVIN to very challenging problems in planetary rover navigation and present early training results on data gathered by the  Curiosity rover that is currently operating on Mars.

assistant: text: We propose an improvement to value iteration networks, with applications to planetary rover path planning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Disentangling Content and Style via Unsupervised Geometry Distillation

Paper content:
It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation in a different and unpredictable way.
It is rare for one to have access to a large number of data to help separate the influences.
In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner.
We address this problem in a two-branch Autoencoder framework.
For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge.
This encourages the branch to distill geometry information.
Another branch learns the complementary style information.
The two branches form an effective framework that can disentangle object's content-style representation without any human annotation.
We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data.
We are able to generate photo-realistic images with 256x256 resolution that are clearly disentangled in content and style.

assistant: text: Propose model based on autoencoder framework to disentangle an object's representation, results show that model can produce representations capturing content and style.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Scheduling with Complex Consumptive Resources for a Planetary Rover

Paper content:
Generating and scheduling activities is particularly challenging
when considering both consumptive resources and
complex resource interactions such as time-dependent resource
usage.We present three methods of determining valid
temporal placement intervals for an activity in a temporally
grounded plan in the presence of such constraints.
We introduce
the Max Duration and Probe algorithms which are
sound, but incomplete, and the Linear algorithm which is
sound and complete for linear rate resource consumption.
We apply these techniques to the problem of scheduling
awakes for a planetary rover where the awake durations
are affected by existing activities.
We demonstrate how the
Probe algorithm performs competitively with the Linear algorithm
given an advantageous problem space and well-defined
heuristics.
We show that the Probe and Linear algorithms
outperform the Max Duration algorithm empirically.
We then empirically present the runtime differences between
the three algorithms.
The Probe algorithm is currently base-lined
for use in the onboard scheduler for NASA’s next planetary
rover, the Mars 2020 rover.

assistant: text: The paper presents three approaches for on-board scheduling of activities in a planetary rover under reservoir resource constraints.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Adaptive Estimators Show Information Compression in Deep Neural Networks

Paper content:
To improve how neural networks function it is crucial to understand their learning process.
The information bottleneck theory of deep learning proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task.
However, empirical evidence for this theory is conflicting, as compression was only observed when networks used saturating activation functions.
In contrast, networks with non-saturating activation functions achieved comparable levels of task performance but did not show compression.
In this paper we developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions.
Using these adaptive estimation techniques, we explored compression in networks with a range of different activation functions.
With two improved methods of estimation, firstly, we show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.
We also find that there is a large amount of variation in compression between different network initializations.
Secondary, we see that L2 regularization leads to significantly increased compression, while preventing overfitting.
Finally, we show that only compression of the last layer is positively correlated with generalization.

assistant: text: This paper studied the popular belief that deep neural networks do information compression for supervised tasks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A novel text representation which enables image classifiers to perform text classification

Paper content:
We introduce a novel method for converting text data into abstract image representations, which allows image-based processing techniques (e.g. image classification networks) to be applied to text-based comparison problems.
We apply the technique to entity disambiguation of inventor names in US patents.
The method involves converting text from each pairwise comparison between two inventor name records into a 2D RGB (stacked) image representation.
We then train an image classification neural network to discriminate between such pairwise comparison images, and use the trained network to label each pair of records as either matched (same inventor) or non-matched (different inventors), obtaining highly accurate results (F1: 99.09%, precision: 99.41%, recall: 98.76%).
Our new text-to-image representation method could potentially be used more broadly for other NLP comparison problems, such as disambiguation of academic publications, or for problems that require simultaneous classification of both text and images.

assistant: text: A method to map a pair of textual information into a 2D RGB image that can be fed to 2D convoutional neural networks (image classifiers).
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ISA-VAE: Independent Subspace Analysis with Variational Autoencoders

Paper content:
Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way.
These methods have focussed largely on modifying the variational cost function to achieve this goal.
However, we show that methods like beta-VAE simplify the tendency of variational inference to underfit causing pathological over-pruning and over-orthogonalization of learned components.
In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered.
Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space.
This means there is no pressure to identify each true factor of variation with a latent variable.
We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry.
Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like beta-VAE and TCVAE between reconstruction loss and disentanglement.
The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

assistant: text: The authors point out issues in current VAE approaches and provide a new perspective on the tradeoff between reconstruction and orthogonalization for VAE, beta-VAE, and beta-TCVAE.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: DRASIC: Distributed Recurrent Autoencoder for Scalable Image Compression

Paper content:
We propose a new architecture for distributed image compression from a group of distributed data sources.
The work is motivated by practical needs of data-driven codec design, low power consumption, robustness, and data privacy.
The proposed architecture, which we refer to as Distributed Recurrent Autoencoder for Scalable Image Compression (DRASIC), is able to train distributed encoders and one joint decoder on correlated data sources.
Its compression capability is much better than the method of training codecs separately.
Meanwhile, for 10 distributed sources, our distributed system remarkably performs within 2 dB peak signal-to-noise ratio (PSNR) of that of a single codec trained with all data sources.
We experiment distributed sources with different correlations and show how our methodology well matches the Slepian-Wolf Theorem in Distributed Source Coding (DSC).
Our method is also shown to be robust to the lack of presence of encoded data from a number of distributed sources.
Moreover, it is scalable in the sense that codes can be decoded simultaneously at more than one compression quality level.
To the best of our knowledge, this is the first data-driven DSC framework for general distributed code design with deep learning.

assistant: text: The paper proposed a distributed recurrent auto-encoder for image compression that uses a ConvLSTM to learn binary codes that are constructed progressively from residuals of previously encoded information
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: What do you learn from context? Probing for sentence structure in contextualized word representations

Paper content:
Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks.
Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline.
We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena.
We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

assistant: text: Proposes the "edge probing" method and focuses on the relationship between spans rather than individual words, enabling the authors to look at syntactic constituency, dependencies, entity labels, and semantic role labeling.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Scalable Imitation Learning for Multi-Agent Systems with Graph Neural Networks

Paper content:
We propose an implementation of GNN that predicts and imitates the motion be- haviors from observed swarm trajectory data.
The network’s ability to capture interaction dynamics in swarms is demonstrated through transfer learning.
We finally discuss the inherent availability and challenges in the scalability of GNN, and proposed a method to improve it with layer-wise tuning and mixing of data enabled by padding.

assistant: text: The paper proposes a new time series model for learning a sequence of graphs.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep reinforcement learning with relational inductive biases

Paper content:
We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.
Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene.
In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmaster-level on four.
In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training.
Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions.
The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases.
Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

assistant: text: A shared relational network architecture for parameterizing the actor and critic network, focused on distributed advantage actor-critic algorithms, that enhances model-free deep reinforcement techniques with relational knowledge about the environment so agents can learn interpretable state representations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Improving Sample Complexity with Observational Supervision

Paper content:
Supervised machine learning models for high-value computer vision applications such as medical image classification often require large datasets labeled by domain experts, which are slow to collect, expensive to maintain, and static with respect to changes in the data distribution.
In this context, we assess the utility of observational supervision, where we take advantage of passively-collected signals such as eye tracking or “gaze” data, to reduce the amount of hand-labeled data needed for model training.
Specifically, we leverage gaze information to directly supervise a visual attention layer by penalizing disagreement between the spatial regions the human labeler looked at the longest and those that most heavily influence model output.
We present evidence that constraining the model in this way can reduce the number of labeled examples required to achieve a given performance level by as much as 50%, and that gaze information is most helpful on more difficult tasks.

assistant: text: A method to use gaze information to reduce the sample complexity of a model and the needed labeling effort to get a target performance, with improved results in middle-sized samples and harder tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Depthwise Separable Convolutions for Neural Machine Translation

Paper content:
Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.
They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures).
Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results.
In this work, we study how depthwise separable convolutions can be applied to neural machine translation.
We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.
In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation.
We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models.

assistant: text: Depthwise separable convolutions improve neural machine translation: the more separable the better.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Detecting Topological Defects in 2D Active Nematics Using Convolutional Neural Networks

Paper content:
Active matter consists of active agents which transform energy extracted from surroundings into momentum, producing a variety of collective phenomena.
A model, synthetic active system composed of microtubule polymers driven by protein motors spontaneously forms a liquid-crystalline nematic phase.
Extensile stress created by the protein motors precipitates continuous buckling and folding of the microtubules creating motile topological defects and turbulent fluid flows.
Defect motion is determined by the rheological properties of the material; however, these remain largely unquantified.
Measuring defects dynamics can yield fundamental insights into active nematics, a class of materials that include bacterial films and animal cells.
Current methods for defect detection lack robustness and precision, and require fine-tuning for datasets with different visual quality.
In this study, we applied Deep Learning to train a defect detector to automatically analyze microscopy videos of the microtubule active nematic.
Experimental results indicate that our method is robust and accurate.
It is expected to significantly increase the amount of video data that can be processed.

assistant: text: An interesting application of CNN in soft condensed matter physics experiments.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Contrastive Representation Distillation

Paper content:
Often we wish to transfer representational knowledge from one neural network to another.
Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator.
Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network.
We demonstrate that this objective ignores important structural knowledge of the teacher network.
This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data.
We formulate this objective as contrastive learning.
Experiments demonstrate that our resulting new objective outperforms knowledge distillation on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer.
When combined with knowledge distillation, our method sets a state of the art in many transfer tasks, sometimes even outperforming the teacher network.

assistant: text: This paper combines a contrastive objective measuring the mutual information between the representations learned by teacher and student networks for model distillation, and proposes a model with improvement over existing alternatives on distillation tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Black Box Recursive Translations for Molecular Optimization

Paper content:
Machine learning algorithms for generating molecular structures offer a promising new approach to drug discovery.
We cast molecular optimization as a translation problem, where the goal is to map an input compound to a target compound with improved biochemical properties.
Remarkably, we observe that when generated molecules are iteratively fed back into the translator, molecular compound attributes improve with each step.
We show that this finding is invariant to the choice of translation model, making this a "black box" algorithm.
We call this method Black Box Recursive Translation (BBRT), a new inference method for molecular property optimization.
This simple, powerful technique operates strictly on the inputs and outputs of any translation model.
We obtain new state-of-the-art results for molecular property optimization tasks using our simple drop-in replacement with well-known sequence and graph-based models.
Our method provides a significant boost in performance relative to its non-recursive peers with just a simple "``for" loop.
Further, BBRT is highly interpretable, allowing users to map the evolution of newly discovered compounds from known starting points.

assistant: text: The authors frame molecule optimization as a sequence-to-sequence problem, and extend existing methods for improving molecules, showing that it is beneficial for optimizing logP but not QED.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dynamic Channel Pruning: Feature Boosting and Suppression

Paper content:
Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.
In this paper, we reduce this cost by exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent, and propose feature boosting and suppression (FBS), a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time.
FBS introduces small auxiliary connections to existing convolutional layers.
In contrast to channel pruning methods which permanently remove channels, it preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels.
FBS-augmented networks are trained with conventional stochastic gradient descent, making it readily available for many state-of-the-art CNNs.
We compare FBS to a range of existing channel pruning and dynamic execution schemes and demonstrate large improvements on ImageNet classification.
Experiments show that FBS can respectively provide 5× and 2× savings in compute on VGG-16 and ResNet-18, both with less than 0.6% top-5 accuracy loss.

assistant: text: We make convolutional layers run faster by dynamically boosting and suppressing channels in feature computation.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Variational lower bounds on mutual information based on nonextensive statistical mechanics

Paper content:
This paper aims to address the limitations of mutual information estimators based on variational optimization.
By redefining the cost using generalized functions from nonextensive statistical mechanics we raise the upper bound of previous estimators and enable the control of the bias variance trade off.
Variational based estimators outperform previous methods especially in high dependence high dimensional scenarios found in machine learning setups.
Despite their performance, these estimators either exhibit a high variance or are upper bounded by log(batch size).
Our approach inspired by nonextensive statistical mechanics uses different generalizations for the logarithm and the exponential in the partition function.
This enables the estimator to capture changes in mutual information over a wider range of dimensions and correlations of the input variables whereas previous estimators saturate them.

assistant: text: Mutual information estimator based nonextensive statistical mechanics
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Neural Arithmetic Unit by reusing many small pre-trained networks

Paper content:
We propose a solution for evaluation of mathematical expression.
However, instead of designing a single end-to-end model we propose a Lego bricks style architecture.
In this architecture instead of training a complex end-to-end neural network, many small networks can be trained independently each accomplishing one specific operation and acting a single lego brick.
More difficult or complex task can then be solved using a combination of these smaller network.
In this work we first identify 8 fundamental operations that are commonly used to solve arithmetic operations (such as 1 digit multiplication, addition, subtraction, sign calculator etc).
These fundamental operations are then learned using simple feed forward neural networks.
We then shows that different operations can be designed simply by reusing these smaller networks.
As an example we reuse these smaller networks to develop larger and a more complex network to solve n-digit multiplication, n-digit division, and cross product.
This bottom-up strategy not only introduces reusability, we also show that it allows to generalize for computations involving n-digits and we show results for up to 7 digit numbers.
Unlike existing methods, our solution also generalizes for both positive as well as negative numbers.

assistant: text: The paper proposes a method to design a NN based mathematical expression evaluation engine.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A∗ Search and Bound-Sensitive Heuristics for Oversubscription Planning

Paper content:
Oversubscription planning (OSP) is the problem of finding plans that maximize the utility value of their end state while staying within a specified cost bound.
Recently, it has been shown that OSP problems can be reformulated as classical planning problems with multiple cost functions but no utilities.
Here we take advantage of this reformulation to show that OSP problems can be solved optimally using the A* search algorithm, in contrast to previous approaches that have used variations on branch-and-bound search.
This allows many powerful techniques developed for classical planning to be applied to OSP problems.
We also introduce novel bound-sensitive heuristics, which are able to reason about the primary cost of a solution while taking into account secondary cost functions and bounds, to provide superior guidance compared to heuristics that do not take these bounds into account.
We implement two such bound-sensitive variants of existing classical planning heuristics, and show experimentally that the resulting search is significantly more informed than comparable heuristics that do not consider bounds.

assistant: text: Presents an approach to solve oversubscription planning (OSP) tasks optimally by using a translation to classical planning with multiple cost functions.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: The Cakewalk Method

Paper content:
Combinatorial optimization is a common theme in computer science.
While in general such problems are NP-Hard, from a practical point of view, locally optimal solutions can be useful.
In some combinatorial problems however, it can be hard to define meaningful solution neighborhoods that connect large portions of the search space, thus hindering methods that search this space directly.
We suggest to circumvent such cases by utilizing a policy gradient algorithm that transforms the problem to the continuous domain, and to optimize a new surrogate objective that renders the former as generic stochastic optimizer.
This is achieved by producing a surrogate objective whose distribution is fixed and predetermined, thus removing the need to fine-tune various hyper-parameters in a case by case manner.
Since we are interested in methods which can successfully recover locally optimal solutions, we use the problem of finding locally maximal cliques as a challenging experimental benchmark, and we report results on a large dataset of graphs that is designed to test clique finding algorithms.
Notably, we show in this benchmark that fixing the distribution of the surrogate is key to consistently recovering locally optimal solutions, and that our surrogate objective leads to an algorithm that outperforms other methods we have tested in a number of measures.

assistant: text: The paper propose to replace the reward term in the policy gradient algorithm with its centered empirical cumulative distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Implementing Inductive bias for different navigation tasks through diverse RNN attrractors

Paper content:
Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map.
The precise form of this representation is often considered to be a metric representation of space.
An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks.
Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment.
To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output.
We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state.
These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities.
Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods.
Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.

assistant: text: Task agnostic pre-training can shape RNN's attractor landscape, and form diverse inductive bias for different navigation tasks
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Enhancing Batch Normalized Convolutional Networks using Displaced Rectifier Linear Units: A Systematic Comparative Study

Paper content:
In this paper, we turn our attention to the interworking between the activation functions and the batch normalization, which is a virtually mandatory technique to train deep networks currently.
We propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization.
Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of standardized VGG and Residual Networks state-of-the-art models.
These convolutional neural networks were trained on CIFAR-100 and CIFAR-10, the most commonly used deep learning computer vision datasets.
The results showed DReLU speeded up learning in all models and datasets.
Besides, statistical significant performance assessments (p<0.05) showed DReLU enhanced the test accuracy presented by ReLU in all scenarios.
Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance.
Therefore, this work demonstrates that it is possible to increase performance replacing ReLU by an enhanced activation function.

assistant: text: A new activation function called Displaced Rectifier Linear Unit is proposed. It is showed to enhance the training and inference performance of batch normalized convolutional neural networks.