ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Layer rotation: a surprisingly simple indicator of generalization in deep networks?

Paper content:
Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.
Compared to previously studied indicators of generalization, we show that layer rotation has the additional benefit of being easily monitored and controlled, as well as having a network-independent optimum: the training procedures during which all layers' weights reach a cosine distance of 1 from their initialization consistently outperform other configurations -by up to 20% test accuracy.
Finally, our results also suggest that the study of layer rotation can provide a unified framework to explain the impact of weight decay and adaptive gradient methods on generalization.

assistant: text: This paper presents empirical evidence supporting the discovery of an indicator of generalization: the evolution across training of the cosine distance between each layer's weight vector and its initialization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses

Paper content:
Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs).
Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy.
Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences.
In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks.
This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.

assistant: text: We replace some gradients paths in hierarchical RNN's by an auxiliary loss. We show that this can reduce the memory cost while preserving performance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Depthwise Separable Convolutions for Neural Machine Translation

Paper content:
Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.
They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures).
Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results.
In this work, we study how depthwise separable convolutions can be applied to neural machine translation.
We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.
In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation.
We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models.

assistant: text: This paper proposes to use depthwise separable convolution layers in a fully convolutional neural machine translation model, and introduces a new super-separable convolution layer which further reduces computational cost.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Construction of Macro Actions for Deep Reinforcement Learning

Paper content:
Conventional deep reinforcement learning typically determines an appropriate primitive action at each timestep, which requires enormous amount of time and effort for learning an effective policy, especially in large and complex environments.
To deal with the issue fundamentally, we incorporate macro actions, defined as sequences of primitive actions, into the primitive action space to form an augmented action space.
The problem lies in how to find an appropriate macro action to augment the primitive action space.
The agent using a proper augmented action space is able to jump to a farther state and thus speed up the exploration process as well as facilitate the learning procedure.
In previous researches, macro actions are developed by mining the most frequently used action sequences or repeating previous actions.
However, the most frequently used action sequences are extracted from a past policy, which may only reinforce the original behavior of that policy.
On the other hand, repeating actions may limit the diversity of behaviors of the agent.
Instead, we propose to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent.
Our approach appends a macro action to the primitive action space once at a time and evaluates whether the augmented action space leads to promising performance or not.
We perform extensive experiments and show that the constructed macro actions are able to speed up the learning process for a variety of deep reinforcement learning methods.
Our experimental results also demonstrate that the macro actions suggested by our approach are transferable among deep reinforcement learning methods and similar environments.
We further provide a comprehensive set of ablation analysis to validate our methodology.

assistant: text: We propose to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training Neural Networks for Aspect Extraction Using Descriptive Keywords Only

Paper content:
Aspect extraction in online product reviews is a key task in sentiment analysis and opinion mining.
Training supervised neural networks for aspect extraction is not possible when ground truth aspect labels are not available, while the unsupervised neural topic models fail to capture the particular aspects of interest.
In this work, we propose a weakly supervised approach for training neural networks for aspect extraction in cases where only a small set of seed words, i.e., keywords that describe an aspect, are available.
Our main contributions are as follows.
First, we show that current weakly supervised networks fail to leverage the predictive power of the available seed words by comparing them to a simple bag-of-words classifier.
Second, we propose a distillation approach for aspect extraction where the seed words are considered by the bag-of-words classifier (teacher) and distilled to the parameters of a neural network (student).
Third, we show that regularization encourages the student to consider non-seed words for classification and, as a result, the student outperforms the teacher, which only considers the seed words.
Finally, we empirically show that our proposed distillation approach outperforms (by up to 34.4% in F1 score) previous weakly supervised approaches for aspect extraction in six domains of Amazon product reviews.

assistant: text: We effectively leverage a few keywords as weak supervision for training neural networks for aspect extraction.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Influence-Directed Explanations for Deep Convolutional Networks

Paper content:
We study the problem of explaining a rich class of behavioral properties of deep neural networks.
Our influence-directed explanations approach this problem by peering inside the network to identify neurons with high influence on the property of interest using an axiomatically justified influence measure, and then providing an interpretation for the concepts these neurons represent.
We evaluate our approach by training convolutional neural networks on Pubfig, ImageNet, and Diabetic Retinopathy datasets.
Our evaluation demonstrates that influence-directed explanations (1) localize features used by the network, (2) isolate features distinguishing related instances, (3) help extract the essence of what the network learned about the class, and (4) assist in debugging misclassifications.

assistant: text: We present an influence-directed approach to constructing explanations for the behavior of deep convolutional networks, and show how it can be used to answer a broad set of questions that could not be addressed by prior work.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Part-Based 3D Face Morphable Model with Anthropometric Local Control

Paper content:
We propose an approach to construct realistic 3D facial morphable models (3DMM) that allows an intuitive facial attribute
editing workflow.
Current face modeling methods using 3DMM suffer from the lack of local control.
We thus create a 3DMM by
combining local part-based 3DMM for the eyes, nose, mouth, ears, and facial mask regions.
Our local PCA-based approach
uses a novel method to select the best eigenvectors from the local 3DMM to ensure that the combined 3DMM is expressive
while allowing accurate reconstruction.
The editing controls we provide to the user are intuitive as they are extracted from
anthropometric measurements found in the literature.
Out of a large set of possible anthropometric measurements, we filter the
ones that have meaningful generative power given the face data set.
We bind the measurements to the part-based 3DMM through
mapping matrices derived from our data set of facial scans.
Our part-based 3DMM is compact yet accurate, and compared to
other 3DMM methods, it provides a new trade-off between local and global control.
We tested our approach on a data set of 135
scans used to derive the 3DMM, plus 19 scans that served for validation.
The results show that our part-based 3DMM approach
has excellent generative properties and allows intuitive local control to the user.

assistant: text: Proposes a piecewise morphable model for human face meshes and also proposes a mapping between anthropometric measurements of the face and the parameters of the model in order to synthesize and edit faces with desired attributes.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Autonomous Scheduling of Agile Spacecraft Constellations with Delay Tolerant Networking for Reactive Imaging

Paper content:
Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.
When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.
In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.
The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.
The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.
In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.
We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.
We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.
The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.

assistant: text: This paper proposes a communication module to optimize the schedule of communication for the problem of spacecraft constellations, and compares the algorithm in distributed and centralized settings.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Autonomous Scheduling of Agile Spacecraft Constellations with Delay Tolerant Networking for Reactive Imaging

Paper content:
Small spacecraft now have precise attitude control systems available commercially, allowing them to slew in 3 degrees of freedom, and capture images within short notice.
When combined with appropriate software, this agility can significantly increase response rate, revisit time and coverage.
In prior work, we have demonstrated an algorithmic framework that combines orbital mechanics, attitude control and scheduling optimization to plan the time-varying, full-body orientation of agile, small spacecraft in a constellation.
The proposed schedule optimization would run at the ground station autonomously, and the resultant schedules uplinked to the spacecraft for execution.
The algorithm is generalizable over small steerable spacecraft, control capability, sensor specs, imaging requirements, and regions of interest.
In this article, we modify the algorithm to run onboard small spacecraft, such that the constellation can make time-sensitive decisions to slew and capture images autonomously, without ground control.
We have developed a communication module based on Delay/Disruption Tolerant Networking (DTN) for onboard data management and routing among the satellites, which will work in conjunction with the other modules to optimize the schedule of agile communication and steering.
We then apply this preliminary framework on representative constellations to simulate targeted measurements of episodic precipitation events and subsequent urban floods.
The command and control efficiency of our agile algorithm is compared to non-agile (11.3x improvement) and non-DTN (21% improvement) constellations.

assistant: text: We propose an algorithmic framework to schedule constellations of small spacecraft with 3-DOF re-orientation capabilities, networked with inter-sat links.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Generating Multi-Agent Trajectories using Programmatic Weak Supervision

Paper content:
We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as  offensive basketball gameplay.
When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables.
Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way.
We present a hierarchical framework that can effectively learn such sequential generative models.
Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.
In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.
We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

assistant: text: Proposes multi-agent sequential generative models.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep reinforcement learning with relational inductive biases

Paper content:
We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability.
Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene.
In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmaster-level on four.
In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training.
Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions.
The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases.
Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

assistant: text: Relational inductive biases improve out-of-distribution generalization capacities in model-free reinforcement learning agents
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION

Paper content:
Stochastic Gradient Descent or SGD is the most popular optimization algorithm for large-scale problems.
SGD estimates the gradient by uniform sampling with sample size one.
There have been several other works that suggest faster epoch wise convergence by using weighted non-uniform sampling for better gradient estimates.
Unfortunately, the per-iteration cost of maintaining this adaptive distribution for gradient estimation is more than calculating the full gradient.
As a result, the false impression of faster convergence in iterations leads to slower convergence in time, which we call a chicken-and-egg loop.
In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.
Such an algorithm is possible due to the sampling view of Locality Sensitive Hashing (LSH), which came to light recently.
As a consequence of superior and fast estimation, we reduce the running time of all existing gradient descent algorithms.
We demonstrate the benefits of our proposal on both SGD and AdaGrad.

assistant: text: We improve the running of all existing gradient descent algorithms.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: When Does Self-supervision Improve Few-shot Learning?

Paper content:
We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions.
Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown.
We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets.
The improvements are greater when the training set is smaller or the task is more challenging.
Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL.
Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements.
We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

assistant: text: Self-supervision improves few-shot recognition on small and challenging datasets without relying on extra data; Extra data helps only when it is from the same or similar domain.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: On Understanding Knowledge Graph Representation

Paper content:
Many methods have been developed to represent knowledge graph data, which implicitly exploit low-rank latent structure in the data to encode known information and enable unknown facts to be inferred.
To predict whether a relationship holds between entities, their embeddings are typically compared in the latent space following a relation-specific mapping.
Whilst link prediction has steadily improved, the latent structure, and hence why such models capture semantic information, remains unexplained.
We build on recent theoretical interpretation of word embeddings as a basis to consider an explicit structure for representations of relations between entities.
For identifiable relation types, we are able to predict properties and justify the relative performance of leading knowledge graph representation methods, including their often overlooked ability to make independent predictions.

assistant: text: Understanding the structure of knowledge graph representation using insight from word embeddings.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Information Plane Analysis of Deep Neural Networks via Matrix--Based Renyi's Entropy and Tensor Kernels

Paper content:
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability.
However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP.
For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers.
MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks.
None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16.
In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data.
The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach.
Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

assistant: text: The authors propose a tensor-kernel based estimator for mutual information estimation between high-dimensional layers in a neural network.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Neural Machine Translation

Paper content:
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.
There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal.
In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora.
Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation.
Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French-to-English and German-to-English translation.
The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively.
Our implementation is released as an open source project.

assistant: text: This is a paper on unsupervised MT which trains a standard architecture using word embeddings in a shared embedding space only with bilingual word papers and an encoder-decoder trained using monolingual data.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

Paper content:
Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements.
For example, air quality monitoring system records PM2.5, CO, etc.
The resulting time-series data often has missing values due to device outages or communication errors.
In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps.
Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly.
In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data.
In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner.
On three real-world datasets, including one our newly collected NYC-traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.

assistant: text: This paper proposes the problem of applying the transformer network to spatiotemporal data in a compuationally efficient way, and investigates ways of implementing 3D attention.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TabNN: A Universal Neural Network Solution for Tabular Data

Paper content:
Neural Network (NN) has achieved state-of-the-art performances in many tasks within image, speech, and text domains.
Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency.
Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data.
Nevertheless, there are many applications with all kinds of tabular data in other domains.
Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all.
Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains.
To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically.
Specifically, the design of TabNN follows two principles: \emph{to explicitly leverages expressive feature combinations} and \emph{to reduce model complexity}.
Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN.
Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.

assistant: text: We propose a universal neural network solution to derive effective NN architectures for tabular data automatically.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Spectral Inference Networks: Unifying Deep and Spectral Learning

Paper content:
We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization.
Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics.
As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data.
We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions.
We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets.
Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.

assistant: text: This paper presents a framework to learn eigenfunctions via a stochastic process and proposes to tackle the challenge of computing eigenfunctions in a large-scale context by approximating then using a two-phase stochastic optimization process.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Policy Message Passing: A New Algorithm for Probabilistic Graph Inference

Paper content:
A general graph-structured neural network architecture operates on graphs through two core components: (1) complex enough message functions; (2) a fixed information aggregation process.
In this paper, we present the Policy Message Passing algorithm, which takes a probabilistic perspective and reformulates the whole information aggregation as stochastic sequential processes.
The algorithm works on a much larger search space, utilizes reasoning history to perform inference, and is robust to noisy edges.
We apply our algorithm to multiple complex graph reasoning and prediction tasks and show that our algorithm consistently outperforms state-of-the-art graph-structured models by a significant margin.

assistant: text: An probabilistic inference algorithm driven by neural network for graph-structured models
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Event Discovery for History Representation in Reinforcement Learning

Paper content:
Environments in Reinforcement Learning (RL) are usually only partially observable.
To address this problem, a possible solution is to provide the agent with information about past  observations.
While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode.
Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision.
We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower.
Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.

assistant: text: event discovery to represent the history for the agent in RL
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Deep Generative Inpainting with Comparative Sample Augmentation

Paper content:
Recent advancements in deep learning techniques such as Convolutional Neural Networks(CNN) and Generative Adversarial Networks(GAN) have achieved breakthroughs in the problem of semantic image inpainting, the task of reconstructing missing pixels in given images.
While much more effective than conventional approaches, deep learning models require large datasets and great computational resources for training, and inpainting quality varies considerably when training data vary in size and diversity.
To address these problems, we present in this paper a inpainting strategy of \textit{Comparative Sample Augmentation}, which enhances the quality of training set by filtering out irrelevant images and constructing additional images using information about the surrounding regions of the images to be inpainted.
Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying sizes, while maintaining inpainting quality as measured by qualitative and quantitative metrics for a large class of deep models, with little need for model-specific consideration.

assistant: text: We introduced a strategy which enables inpainting models on datasets of various sizes
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Y-net: A Physics-constrained and Semi-supervised Learning Approach to the Phase Problem in Computational Electron Imaging

Paper content:
The phase problem in diffraction physics is one of the oldest inverse problems in all of science.
The central difficulty that any approach to solving this inverse problem must overcome is that half of the information, namely the phase of the diffracted beam, is always missing.
In the context of electron microscopy, the phase problem is generally non-linear and solutions provided by phase-retrieval techniques are known to be poor approximations to the physics of electrons interacting with matter.
Here, we show that a semi-supervised learning approach can effectively solve the phase problem in electron microscopy/scattering.
In particular, we introduce a new Deep Neural Network (DNN), Y-net, which simultaneously learns a reconstruction algorithm via supervised training in addition to learning a physics-based regularization via unsupervised training.
We demonstrate that this constrained, semi-supervised approach is an order of magnitude more data-efficient and accurate than the same model trained in a purely supervised fashion.
In addition, the architecture of the Y-net model provides for a straightforward evaluation of the consistency of the model's prediction during inference and is generally applicable to the phase problem in other settings.

assistant: text: We introduce a semi-supervised deep neural network to approximate the solution of the phase problem in electron microscopy
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Load Balancing in Large-Scale Heterogeneous Systems with Multiple Dispatchers

Paper content:
In recent years, the efficiency and even the feasibility of traditional load-balancing policies are challenged by the rapid growth of cloud infrastructure with increasing levels of server heterogeneity and increasing size of cloud services and applications.
In such many software-load-balancers heterogeneous systems, traditional solutions, such as JSQ, incur an increasing communication overhead, whereas low-communication alternatives, such as JSQ(d) and the recently proposed JIQ scheme are either unstable or provide poor performance.
We argue that a better low-communication load balancing scheme can be established by allowing each dispatcher to have a different view of the system and keep using JSQ, rather than greedily trying to avoid starvation on a per-decision basis.
accordingly, we introduce the Loosely-Shortest -Queue family of load balancing algorithms.
Roughly speaking, in Loosely-shortest -Queue, each dispatcher keeps a different approximation of the server queue lengths and routes jobs to the shortest among them.
Communication is used only to update the approximations and make sure that they are not too far from the real queue lengths in expectation.
We formally establish the strong stability of any Loosely-Shortest -Queue policy and provide an easy-to-verify sufficient condition for verifying that a policy is Loosely-Shortest -Queue.
We further demonstrate that the Loosely-Shortest -Queue approach allows constructing throughput optimal policies with an arbitrarily low communication budget.
Finally, using extensive simulations that consider homogeneous, heterogeneous and highly skewed heterogeneous systems in scenarios with a single dispatcher as well as with multiple dispatchers, we show that the examined Loosely-Shortest -Queue example policies are always stable as dictated by theory.
Moreover, it exhibits an appealing performance and significantly outperforms well-known low-communication policies, such as JSQ(d) and JIQ, while using a similar communication budget.

assistant: text: Scalable and low communication load balancing solution for heterogeneous-server multi-dispatcher systems with strong theoretical guarantees and promising empirical results.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Uncertainty-aware Variational-Recurrent Imputation Network for Clinical Time Series

Paper content:
Electronic Health Records (EHR) comprise of longitudinal clinical observations portrayed with sparsity, irregularity, and high-dimensionality which become the major obstacles in drawing reliable downstream outcome.
Despite greatly numbers of imputation methods are being proposed to tackle these issues, most of the existing methods ignore correlated features or temporal dynamics and entirely put aside the uncertainty.
In particular, since the missing values estimates have the risk of being imprecise, it motivates us to pay attention to reliable and less certain information differently.
In this work, we propose a novel variational-recurrent imputation network (V-RIN), which unified imputation and prediction network, by taking into account the correlated features, temporal dynamics, and further utilizing the uncertainty to alleviate the risk of biased missing values estimates.
Specifically, we leverage the deep generative model to estimate the missing values based on the distribution among variables and a recurrent imputation network to exploit the temporal relations in conjunction with utilization of the uncertainty.
We validated the effectiveness of our proposed model with publicly available real-world EHR dataset, PhysioNet Challenge 2012, and compared the results with other state-of-the-art competing methods in the literature.

assistant: text: Our variational-recurrent imputation network (V-RIN) takes into account the correlated features, temporal dynamics, and further utilizes the uncertainty to alleviate the risk of biased missing values estimates.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Bayesian Sparsification of Gated Recurrent Neural Networks

Paper content:
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e.
g.
neurons.
We apply and further develop this approach for gated recurrent architectures.
Specifically, in addition to sparsification of individual weights and neurons, we propose to sparsify preactivations of gates and information flow in LSTM.
It makes some gates and information flow components constant, speeds up forward pass and improves compression.
Moreover, the resulting structure of gate sparsity is interpretable and depends on the task.

assistant: text: This paper proposed a sparsification method for recurrent neural networks by eliminating neurons with zero preactivations to obtain compact networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Unsupervised Domain Adaptation for Distance Metric Learning

Paper content:
Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain.
However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space.
This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated.
Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one.
To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space.
Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain.
In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.

assistant: text: A new theory of unsupervised domain adaptation for distance metric learning and its application to face recognition across diverse ethnicity variations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Bayesian Nonparametric Topic Model with Variational Auto-Encoders

Paper content:
Topic modeling of text documents is one of the most important tasks in representation learning.
In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders.
On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically.
On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner.
Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically.
Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks.
Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.

assistant: text: A Bayesian Nonparametric Topic Model with Variational Auto-Encoders which achieves the state-of-the-arts on public benchmarks in terms of perplexity, topic coherence and retrieval tasks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning to Learn via Gradient Component Corrections

Paper content:
Gradient-based meta-learning algorithms require several steps of gradient descent to adapt to newly incoming tasks.
This process becomes more costly as the number  of  samples  increases.
Moreover,  the  gradient  updates  suffer  from  several sources of noise leading to a degraded performance
.   In this work,  we propose a meta-learning algorithm equipped with the GradiEnt Component COrrections, aGECCO cell for short, which generates a multiplicative corrective low-rank matrix which (after vectorization) corrects the estimated gradients
. GECCO contains a simple decoder-like network with learnable parameters, an attention module and a so-called context input parameter
. The context parameter of GECCO is updated to  generate  a  low-rank  corrective  term  for  the  network  gradients
.   As  a  result, meta-learning requires only a few of gradient updates to absorb new task (often, a single update is sufficient in the few-shot scenario
). While previous approaches address this problem by altering the learning rates, factorising network parameters or directly learning feature corrections from features and/or gradients, GECCO is an off-the-shelf generator-like unit that performs element-wise gradient corrections without the need to ‘observe’ the features and/or the gradients directly
.  We show that our GECCO
(i) accelerates learning,
(ii) performs robust corrections of the gradients corrupted by a noise, and
(iii) leads to notable improvements over existing gradient-based meta-learning algorithms.

assistant: text: We propose a meta-learner to adapt quickly on multiple tasks even one step in a few-shot setting.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Role of two learning rates in convergence of model-agnostic meta-learning

Paper content:
Model-agnostic meta-learning (MAML) is known as a powerful meta-learning method.
However, MAML is notorious for being hard to train because of the existence of two learning rates.
Therefore, in this paper, we derive the conditions that inner learning rate $\alpha$ and meta-learning rate $\beta$ must satisfy for MAML to converge to minima with some simplifications.
We find that the upper bound of $\beta$ depends on $ \alpha$, in contrast to the case of using the normal gradient descent method.
Moreover, we show that the threshold of $\beta$ increases as $\alpha$ approaches its own upper bound.
This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network.
Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible $\alpha$; next, tune $\beta$ based on the chosen value of $\alpha$.

assistant: text: This paper studies a method to help tune the two learning rates used in the MAML training algorithm.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Prototypical Bregman Networks

Paper content:
In this work, we approach one-shot and few-shot learning problems as methods for finding good prototypes for each class, where these prototypes are generalizable to new data samples and classes.
We propose a metric learner that learns a Bregman divergence by learning its underlying convex function.
Bregman divergences are a good candidate for this framework given they are the only class of divergences with the property that the best representative of a set of points is given by its mean.
We propose a flexible extension to prototypical networks to enable joint learning of the embedding and the divergence, while preserving computational efficiency.
Our preliminary results are comparable with the prior work on the Omniglot and Mini-ImageNet datasets, two standard benchmarks for one-shot and few-shot learning.
We argue that our model can be used for other tasks that involve metric learning or tasks that require approximate convexity such as structured prediction and data completion.

assistant: text: Bregman divergence learning for few-shot learning.