ali5341/scitldr-chat-format huggingface dataset:
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Feature-Robustness, Flatness and Generalization Error for Deep Neural Networks

Paper content:
The performance of deep neural networks is often attributed to their automated, task-related feature construction.
It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples.
Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error.
For several flatness measures, this correlation has been empirically validated.
However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization: if a network uses ReLU activations, the network function can be reparameterized without changing its output in such a way that flatness is changed almost arbitrarily.
This paper proposes a natural modification of existing flatness measures that results in invariance to reparameterization.
The proposed measures imply a robustness of the network to changes in the input and the hidden layers.
Connecting this feature robustness to generalization leads to a generalized definition of the representativeness of data.
With this, the generalization error of a model trained on representative data can be bounded by its feature robustness which depends on our novel flatness measure.

assistant: text: The authors propose a notion of feature robustness which is invariant with respect to rescaling the weight and discuss the notion's relationship to generalization.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dynamic parameter reallocation improves trainability of deep convolutional networks

Paper content:
Network pruning has emerged as a powerful technique for reducing the size of deep neural networks.
Pruning uncovers high-performance subnetworks by taking a trained dense network and gradually removing unimportant connections.
Recently, alternative techniques have emerged for training sparse networks directly without having to train a large dense model beforehand, thereby achieving small memory footprints during both training and inference.These techniques are based on dynamic reallocation of non-zero parameters during training.
Thus, they are in effect executing a training-time search for the optimal subnetwork.
We investigate a most recent one of these techniques and conduct additional experiments to elucidate its behavior in training sparse deep convolutional networks.
Dynamic parameter reallocation converges early during training to a highly trainable subnetwork.
We show that neither the structure, nor the initialization of the discovered high-performance subnetwork is sufficient to explain its good performance.
Rather, it is the dynamics of parameter reallocation that are responsible for successful learning.
Dynamic parameter reallocation thus improves the trainability of deep convolutional networks, playing a similar role as overparameterization, without incurring the memory and computational cost of the latter.

assistant: text: Dynamic parameter-reallocation enables the successful direct training of compact sparse networks, and it plays an indispensable role even when we know the optimal sparse network a-priori
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Ranking Policy Gradient

Paper content:
Sample inefficiency is a long-lasting problem in reinforcement learning (RL).
The state-of-the-art uses action value function to derive policy while it usually involves an extensive search over the state-action space and unstable optimization.
Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions.
To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles.
These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency.
We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.

assistant: text: This paper presents a new view on policy gradient methods from the perspective of ranking.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: $\textrm{D}^2$GAN: A Few-Shot Learning Approach with Diverse and Discriminative Feature Synthesis

Paper content:
The rich and accessible labeled data fuel the revolutionary success of deep learning.
Nonetheless, massive supervision remains a luxury for many real applications, boosting great interest in label-scarce techniques such as few-shot learning (FSL).
An intuitively feasible approach to FSL is to conduct data augmentation via synthesizing additional training samples.
The key to this approach is how to guarantee both discriminability and diversity of the synthesized samples.
In this paper, we propose a novel FSL model, called $\textrm{D}^2$GAN, which synthesizes Diverse and Discriminative features based on Generative Adversarial Networks (GAN).
$\textrm{D}^2$GAN secures discriminability of the synthesized features by constraining them to have high correlation with real features of the same classes while low correlation with those of different classes.
Based on the observation that noise vectors that are closer in the latent code space are more likely to be collapsed into the same mode when mapped to feature space, $\textrm{D}^2$GAN incorporates a novel anti-collapse regularization term, which encourages feature diversity by penalizing the ratio of the logarithmic similarity of two synthesized features and the logarithmic similarity of the latent codes generating them.
Experiments on three common benchmark datasets verify the effectiveness of $\textrm{D}^2$GAN by comparing with the state-of-the-art.

assistant: text: A new GAN based few-shot learning algorithm by synthesizing  diverse and discriminative Features
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

Paper content:
We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks.
The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks.
Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped).
Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.

assistant: text: A set of ideas related to theoretical understanding generalization properties of multilayer neural networks, and a qualitative analogy between behaviours in deep learning and results from quantitative statistical physics analysis of single and two-layer neural networks.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Discriminative out-of-distribution detection for semantic segmentation

Paper content:
Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes.
However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input.
These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive.
We propose to address this issue by discriminative detection
of OOD pixels in input data.
Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task.
Instead, we train a dedicated OOD model
which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world.
We perform our experiments on high resolution natural images in a dense prediction setup.
We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset.
We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.
The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

assistant: text: This paper addresses out-of-distribution detection for helping the segmentation process, and proposes an approach of training a binary classifier that distinguishes image patches from a known set of classes from those of an unknown.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Training-Free Uncertainty Estimation for Neural Networks

Paper content:
Uncertainty estimation is an essential step in the evaluation of the robustness for deep learning models in computer vision, especially when applied in risk-sensitive areas.
However, most state-of-the-art deep learning models either fail to obtain uncertainty estimation or need significant modification (e.g., formulating a proper Bayesian treatment) to obtain it.
None of the previous methods are able to take an arbitrary model off the shelf and generate uncertainty estimation without retraining or redesigning it.
To address this gap, we perform the first systematic exploration into training-free uncertainty estimation.
We propose three simple and scalable methods to analyze the variance of output from a trained network under tolerable perturbations: infer-transformation, infer-noise, and infer-dropout.
They operate solely during inference, without the need to re-train, re-design, or fine-tune the model, as typically required by other state-of-the-art uncertainty estimation methods.
Surprisingly, even without involving such perturbations in training, our methods produce comparable or even better uncertainty estimation when compared to other training-required state-of-the-art methods.
Last but not least, we demonstrate that the uncertainty from our proposed methods can be used to improve the neural network training.

assistant: text: Describes several approaches for measuring uncertainty in arbitrary neural networks when there is an absence of distortion during training.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Minimizing Change in Classifier Likelihood to Mitigate Catastrophic Forgetting

Paper content:
Continual learning is a longstanding goal of artificial intelligence, but is often counfounded by catastrophic forgetting that prevents neural networks from learning tasks sequentially.
Previous methods in continual learning have demonstrated how to mitigate catastrophic forgetting, and learn new tasks while retaining performance on the previous tasks.
We analyze catastrophic forgetting from the perspective of change in classifier likelihood and propose a simple L1 minimization criterion which can be adapted to different use cases.
We further investigate two ways to minimize forgetting as quantified by this criterion and propose strategies to achieve finer control over forgetting.
Finally, we evaluate our strategies on 3 datasets of varying difficulty and demonstrate improvements over previously known L2 strategies for mitigating catastrophic forgetting.

assistant: text: Another perspective on catastrophic forgetting
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets

Paper content:
Adaptive gradient algorithms perform gradient-based updates using the history of gradients and are ubiquitous in training deep neural networks.
While adaptive gradient methods theory is well understood for minimization problems, the underlying factors driving their empirical success in min-max problems such as GANs remain unclear.
In this paper, we aim at bridging  this gap from both theoretical and empirical perspectives.
First, we analyze a variant of Optimistic Stochastic Gradient (OSG) proposed in~\citep{daskalakis2017training} for solving a class of non-convex non-concave min-max problem and establish $O(\epsilon^{-4})$ complexity for finding $\epsilon$-first-order stationary point, in which the algorithm only requires invoking one stochastic first-order oracle while enjoying state-of-the-art iteration complexity achieved by stochastic extragradient method by~\citep{iusem2017extragradient}.
Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an \emph{improved} adaptive complexity $\widetilde{O}\left(\epsilon^{-\frac{2}{1-\alpha}}\right)$~\footnote{Here $\widetilde{O}(\cdot)$ compresses a logarithmic factor of $\epsilon$.
}, where $\alpha$ characterizes the growth rate of the cumulative stochastic gradient and $0\leq \alpha\leq 1/2$.
To the best of our knowledge, this is the first work for establishing adaptive complexity in non-convex non-concave min-max optimization.
Empirically, our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training.
Moreover, this observation can be explained by the slow growth rate of the cumulative stochastic gradient, as observed empirically.

assistant: text: Develops algorithms for the solution of variational inequalities in the stochastic setting, proposing a variation of the extragradient method.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Infinite-horizon Off-Policy Policy Evaluation with Multiple Behavior Policies

Paper content:
We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies.
Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation.
We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities.
With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction.
In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.

assistant: text: An algorithm to solve infinite horizon off policy evaluation with multiple behavior policies by estimating a mixed policy under regression, and theoretical proof that an estimated policy ratio can reduce variance.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Min-max Entropy for Weakly Supervised Pointwise Localization

Paper content:
Pointwise localization allows more precise localization and accurate interpretability, compared to bounding box, in applications where objects are highly unstructured such as in medical domain.
In this work, we focus on  weakly supervised localization (WSL) where a model is trained to classify an image and localize regions of interest at pixel-level using only global image annotation.
Typical convolutional attentions maps are prune to high false positive regions.
To alleviate this issue, we propose a new deep learning method for WSL, composed of a localizer and a classifier, where the localizer is constrained to determine relevant and irrelevant regions using conditional entropy (CE) with the aim to reduce false positive regions.
Experimental results on a public medical dataset and two natural datasets, using Dice index, show that, compared to state of the art WSL methods, our proposal can provide significant improvements in terms of image-level classification and pixel-level localization (low false positive) with robustness to overfitting.
A public reproducible PyTorch implementation is provided.

assistant: text: This work explores the problem of WSL using a novel design of regularization terms and a recursive erasing algorithm.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning From Noisy Singly-labeled Data

Paper content:
Supervised learning depends on annotated examples, which are taken to be the ground truth.
But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk.
Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem).
Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples.
This raises two fundamental questions: (1) How can we best learn from noisy workers?
(2) How should we allocate our labeling budget to maximize the performance of a classifier?
We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data.
The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality.
Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality.
We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold.
Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

assistant: text: A new approach for learning a model from noisy crowdsourced annotations.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: MEMO: A Deep Network for Flexible Combination of Episodic Memories

Paper content:
Recent research developing neural network architectures with external memory have often used the benchmark bAbI question and answering dataset which provides a challenging number of tasks requiring reasoning.
Here we employed a classic associative inference task from the human neuroscience literature in order to more carefully probe the reasoning capacity of existing memory-augmented architectures.
This task is thought to capture the essence of reasoning -- the appreciation of distant relationships among elements distributed across multiple facts or memories.
Surprisingly, we found that current architectures struggle to reason over long distance associations.
Similar results were obtained on a more complex task involving finding the shortest path between nodes in a path.
We therefore developed a novel architecture, MEMO, endowed with the capacity to reason over longer distances.
This was accomplished with the addition of two novel components.
First, it introduces a separation between memories/facts stored in external memory and the items that comprise these facts in external memory.
Second, it makes use of an adaptive retrieval mechanism, allowing a variable number of ‘memory hops’ before the answer is produced.
MEMO is capable of solving our novel reasoning tasks, as well as all 20 tasks in bAbI.

assistant: text: This paper proposes changes to the End2End Memory Network architecture, introduces a new Paired Associative Inference task that most existing models struggle to solve, and shows that their proposed architecture solves the task better.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Paper content:
Domain adaptation is critical for success in new, unseen environments.
Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difficult to visualize and sometimes fail to capture pixel-level and low-level domain shifts.
Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at  mapping images between domains, even without the use of aligned image pairs.
We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain Adaptation model.
CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs.
Our model can be applied in a variety of visual recognition and prediction settings.
We show new state-of-the-art results across multiple adaptation tasks, including digit classification and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains.

assistant: text: This paper makes a novel extension to the previous work on CycleGAN by coupling it with adversarial adaptation approaches, including a new feature and semantic loss in the overall objective of the CycleGAN, with clear benefits.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Universal Deep Neural Network Compression

Paper content:
In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment.
Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding.
In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.
Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.
Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively.

assistant: text: We introduce the universal deep neural network compression scheme, which is applicable universally for compression of any models and can perform near-optimally regardless of their weight distribution.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring  for Minimax Problems and GANs

Paper content:
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood.
In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations.
First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable.
Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients.
Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.

assistant: text: This paper analyzes the dynamics of stochastic gradient descent when applied to convex-concave games, as well as GD with optimism and a new anchored GD algorithm that converges under weaker assumptions than SGD or SGD with optimism.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Train Neural Network by Embedding Space Probabilistic Constraint 

Paper content:
Using higher order knowledge to reduce training data has become a popular research topic.
However, the ability for available methods to draw effective decision boundaries is still limited: when training set is small, neural networks will be biased to certain labels.
Based on this observation, we consider constraining output probability distribution as higher order domain knowledge.
We design a novel algorithm that jointly optimizes output probability distribution on a clustered embedding space to make neural networks draw effective decision boundaries.
While directly applying probability constraint is not effective, users need to provide additional very weak supervisions: mark some batches that have output distribution greatly differ from target probability distribution.
We use experiments to empirically prove that our model can converge to an accuracy higher than other state-of-art semi-supervised learning models with less high quality labeled training examples.

assistant: text: This paper incorporates label distribution into model learning when a limited number of training instances is available, and proposes two techniques for handling the problem of output label distribution being wrongly biased.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Extractor-Attention Network: A New Attention Network with Hybrid Encoders for Chinese Text Classification

Paper content:
Chinese text classification has received more and more attention today.
However, the problem of Chinese text representation still hinders the improvement of Chinese text classification, especially the polyphone and the homophone in social media.
To cope with it effectively, we propose a new structure, the Extractor, based on attention mechanisms and design novel attention networks named Extractor-attention network (EAN).
Unlike most of previous works, EAN uses a combination of a word encoder and a Pinyin character encoder instead of a single encoder.
It improves the capability of Chinese text representation.
Moreover, compared with the hybrid encoder methods, EAN has more complex combination architecture and more reducing parameters structures.
Thus, EAN can take advantage of a large amount of information that comes from multi-inputs and alleviates efficiency issues.
The proposed model achieves the state of the art results on 5 large datasets for Chinese text classification.

assistant: text: Proposal for an attention network where both word and pinyin are considered for Chinese representation, with improved results shown in several datasets for text classification.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Permutation Equivariant Models for Compositional Generalization in Language

Paper content:
Humans understand novel sentences by composing meanings and roles of core language components.
In contrast, neural network models for natural language modeling fail when such compositional generalization is required.
The main contribution of this paper is to hypothesize that language compositionality is a form of group-equivariance.
Based on this hypothesis, we propose a set of tools for constructing equivariant sequence-to-sequence models.
Throughout a variety of experiments on the SCAN tasks, we analyze the behavior of existing models under the lens of equivariance, and demonstrate that our equivariant architecture is able to achieve the type compositional generalization required in human language understanding.

assistant: text: This work focuses on learning locally equivariant representations and functions over input/output words for the purposes of SCAN task.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Split LBI for Deep Learning: Structural Sparsity via Differential Inclusion Paths

Paper content:
Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error.
However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima.
In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on \emph{differential inclusions of inverse scale spaces}, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity.
It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks.
Experimental evidence shows that\ SplitLBI may achieve state-of-the-art performance in large scale training on ImageNet-2012 dataset etc., while with \emph{early stopping} it unveils effective subnet architecture with comparable test accuracies to dense models after retraining instead of pruning well-trained ones.

assistant: text: SplitLBI is applied to deep learning to explore model structural sparsity, achieving state-of-the-art performance in ImageNet-2012 and unveiling effective subnet architecture.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: A Simple Geometric Proof for the Benefit of Depth in ReLU Networks

Paper content:
We present a simple proof for the benefit of depth in multi-layer feedforward network with rectifed activation (``"depth separation").
Specifically we present a sequence of classification problems f_i such that
(a) for any fixed depth rectified network we can find an index m such that problems with index > m require exponential network width to fully represent the function f_m; and
(b) for any problem f_m in the family, we present a concrete neural network with linear depth and bounded width that fully represents it.
While there are several previous work showing similar results, our proof uses substantially simpler tools and techniques, and should be accessible to undergraduate students in computer science and people with similar backgrounds.
We present a simple, geometric proof of the benefit of depth in deep neural networks.
We prove that there exist a set of functions indexed by m, each of which can be efficiently represented by a depth m rectified MLP network requiring O(m) parameters.
However, for any bounded depth rectified MLP network, there is a function f m in this set that representing it will require an exponential number of parameters in m.
More formally, let G d be the set of multi-layer perceptron (MLP) networks with rectified activation and d hidden layers, and let g Θ be such an MLP with parameters Θ.
We will prove the following theorem: Theorem 1 (Depth Separation).
There exists a set of functions f 1 , f 2 , ..., f i : R 2 → {−1, 1} such that: While this is not a novel result, a main characteristic of our proof is its simplicity.
In contrast to previous work, our proof uses only basic algebra, geometry and simple combinatorial arguments.
As such, it can be easily read and understood by newcomers and practitioners, or taught in an undergraduate class, without requiring extensive background.
Tailoring to these crowds, our presentation style is more verbose then is usual in papers of this kind, attempting to spell out all steps explicitly.
We also opted to trade generality for proof simplicity, remaining in input space R 2 rather than the more general R n , thus allowing us to work with lines rather than hyperplanes.
Beyond being easy to visualize, it also results in simple proofs of the different lemmas.

assistant: text: A proof that deeper networks need less units than shallower ones for a family of problems.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: V1Net: A computational model of cortical horizontal connections

Paper content:
The primate visual system builds robust, multi-purpose representations of the external world in order to support several diverse downstream cortical processes.
Such representations are required to be invariant to the sensory inconsistencies caused by dynamically varying lighting, local texture distortion, etc.
A key architectural feature combating such environmental irregularities is ‘long-range horizontal connections’ that aid the perception of the global form of objects.
In this work, we explore the introduction of such horizontal connections into standard deep convolutional networks; we present V1Net -- a novel convolutional-recurrent unit that models linear and nonlinear horizontal inhibitory and excitatory connections inspired by primate visual cortical connectivity.
We introduce the Texturized Challenge -- a new benchmark to evaluate object recognition performance under perceptual noise -- which we use to evaluate V1Net against an array of carefully selected control models with/without recurrent processing.
Additionally, we present results from an ablation study of V1Net demonstrating the utility of diverse neurally inspired horizontal connections for state-of-the-art AI systems on the task of object boundary detection from natural images.
We also present the emergence of several biologically plausible horizontal connectivity patterns, namely center-on surround-off, association fields and border-ownership connectivity patterns in a V1Net model trained to perform boundary detection on natural images from the Berkeley Segmentation Dataset 500 (BSDS500).
Our findings suggest an increased representational similarity between V1Net and biological visual systems, and highlight the importance of neurally inspired recurrent contextual processing principles for learning visual representations that are robust to perceptual noise and furthering the state-of-the-art in computer vision.

assistant: text: The authors propose to modify a convolutional variant of LSTM to include horizontal connections inspired by known interactions in visual cortex.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: TabFact: A Large-scale Dataset for Table-based Fact Verification

Paper content:
The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation.
However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored.
This paper specifically aims to study the fact verification given semi-structured data as evidence.
To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED.
TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning.
To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA).
Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification.
LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification.
Both methods achieve similar accuracy but still lag far behind human performance.
We also perform a comprehensive analysis to demonstrate great future opportunities.

assistant: text: We propose a new dataset to investigate the entailment problem under semi-structured table as premise
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: When is a Convolutional Filter Easy to Learn?

Paper content:
We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function.
Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input.
We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches.
To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions.
Our theory also justifies the two-stage learning rate strategy in deep neural networks.
While our focus is theoretical, we also present experiments that justify our theoretical findings.

assistant: text: We prove randomly initialized (stochastic) gradient descent learns a convolutional filter in polynomial time.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

Paper content:
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients.
Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs).
We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices.
This is done by parametrizing with a skew-symmetric matrix using the Cayley transform.
Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones.
The proposed training scheme involves a straightforward gradient calculation and update step.
In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.

assistant: text: Novel parametrization of RNNs allows representing orthogonal weight matrices relatively easily.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Paper content:
Deep multitask networks, in which one neural network produces multiple predictive outputs, are more scalable and often better regularized than their single-task counterparts.
Such advantages can potentially lead to gains in both speed and performance, but multitask networks are also difficult to train without finding the right balance between tasks.
We present a novel gradient normalization (GradNorm) technique which automatically balances the multitask loss function by directly tuning the gradients to equalize task training rates.
We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting over single networks, static baselines, and other adaptive multitask loss balancing techniques.
GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$.
Thus, what was once a tedious search process which incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks.
Ultimately, we hope to demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

assistant: text: We show how you can boost performance in a multitask network by tuning an adaptive multitask loss function that is learned through directly balancing network gradients.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AN ATTENTION-BASED DEEP NET FOR LEARNING TO RANK

Paper content:
In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query.
Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism.
This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion.
The embeddings are trained with convolutional neural networks or the word2vec model.
We demonstrate the performance of this model with image retrieval and text querying data sets.

assistant: text: learning to rank with several embeddings and attentions
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Empowering Graph Representation Learning with Paired Training and Graph Co-Attention

Paper content:
Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years---mostly on tasks involving node-level predictions.
The setup of prediction tasks over entire graphs (such as property prediction for a molecule, or side-effect prediction for a drug), however, proves to be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction.
Most prior work attempts to predict these graph-level properties while considering only one graph at a time---not allowing the learner to directly leverage structural similarities and motifs across graphs.
Here we propose a setup in which a graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations to easily exchange structural information across them.
We first show that such a setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand to a more generic graph regression setup: enhancing predictions over QM9, a standard molecular prediction benchmark.
Our setup is flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.

assistant: text: We use graph co-attention in a paired graph training system for graph classification and regression.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Reproducibility and Stability Analysis in Metric-Based Few-Shot Learning

Paper content:
We propose a study of the stability of several few-shot learning algorithms subject to variations in the hyper-parameters and optimization schemes while controlling the random seed.
We propose a methodology for testing for statistical differences in model performances under several replications.
To study this specific design, we attempt to reproduce results from three prominent papers: Matching Nets, Prototypical Networks, and TADAM.
We analyze on the miniImagenet dataset on the standard classification task in the 5-ways, 5-shots learning setting at test time.
We find that the selected implementations exhibit stability across random seed, and repeats.
b ∼ Normal(0, σ 2 I) (1) y = Xβ + Zb + α + .(2)Where β ∈ R P is our slope vector, α ∈ R is the intercept, and ∼ Normal(0, I) is the random 127 noise vector.
To model the clusters, we introduce Zb,where Z is the n × q model matrix for the q- and the dependent variable mean is captured by Xβ + α when we marginalize over all the samples.
The random effects component Zb captures variations in the data, it can be interpreted as an individual 134 deviation from the group-level fixed effect.
In our context, we can write the model as follows:136 metric ijk = (A + α 0j + α 1k ) + βExperiment i + i (3) metric ijk = A + βExperiment i + (α 0j + α 1k + i ) (4) DISPLAYFORM0 Where A is the intercept, β is a vector of parameters and Experiment i is a one hot vector of 137 experiments for the observation i.
We can regroup all the random effects, where alpha 0j is a 138 random effect associated with an observation from a random seed j, and alpha 1k is associated to
an observation from a repeat k. Finally, it is possible to regroup all the nuisance parameters in seeds for a given implementation, except for inherent differences due to parallelism on CPU and 250 GPU.
DISPLAYFORM0
Some implementations:
• call a random number generator at an execution point placed before the episodes data 253 generation, hence changing the state of the random number generator,
• generate the episodes data in advance, others generate it for each episode on the fly: different
states of random number generator are involved in the data generation process,
• start training at different states of the random number generator: random number sequences The trend of large-scale compute-intensive ML experiments has caused concern in the community

assistant: text: This paper studies reproducibility for few-shot learning.
user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Approximability of Discriminators Implies Diversity in GANs

Paper content:
While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.
The theoretical work of Arora et al. (2017a) suggests a dilemma about GANs’ statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse.
By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).
For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.
This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.
Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

assistant: text: Analyzes that the Integral Probability Metric can be a good approximation of Wasserstein distance under some mild assumptions.