In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.

Q: Recurrent Neural Networks architectures excel at processing sequences by
 modelling dependencies over different timescales. The recently introduced
 Recurrent Weighted Average (RWA) unit captures long term dependencies
 far better than an LSTM on several challenging tasks. The RWA achieves
 this by applying attention to each input and computing a weighted average
 over the full history of its computations. Unfortunately, the RWA cannot
 change the attention it has assigned to previous timesteps, and so struggles
 with carrying out consecutive tasks or tasks with changing requirements.
 We present the Recurrent Discounted Attention (RDA) unit that builds on
 the RWA by additionally allowing the discounting of the past.
 We empirically compare our model to RWA, LSTM and GRU units on
 several challenging tasks. On tasks with a single output the RWA, RDA and
 GRU units learn much quicker than the LSTM and with better performance.
 On the multiple sequence copy task our RDA unit learns the task three
 times as quickly as the LSTM or GRU units while the RWA fails to learn at
 all. On the Wikipedia character prediction task the LSTM performs best
 but it followed closely by our RDA unit. Overall our RDA unit performs
 well and is sample efficient on a large variety of sequence tasks.

A: We introduce the Recurrent Discounted Unit which applies attention to any length sequence in linear time
****
Q:     State of the art computer vision models have been shown to be vulnerable to small adversarial perturbations of the input. In other words, most images in the data distribution are both correctly classified by the model and are very close to a visually similar misclassified image. Despite substantial research interest, the cause of the phenomenon is still poorly understood and remains unsolved. We hypothesize that this counter intuitive behavior is a naturally occurring result of the high dimensional geometry of the data manifold. As a first step towards exploring this hypothesis, we study a simple synthetic dataset of classifying between two concentric high dimensional spheres. For this dataset we show a fundamental tradeoff between the amount of test error and the average distance to nearest error. In particular, we prove that any model which misclassifies a small constant fraction of a sphere will be vulnerable to adversarial perturbations of size $O(1/\sqrt{d})$. Surprisingly, when we train several different architectures on this dataset, all of their error sets naturally approach this theoretical bound. As a result of the theory, the vulnerability of neural networks to small adversarial perturbations is a logical consequence of the amount of test error observed. We hope that our theoretical analysis of this very simple case will point the way forward to explore how the geometry of complex real-world data sets leads to adversarial examples.

A: We hypothesize that the vulnerability of image models to small adversarial perturbation is a naturally occurring result of the high dimensional geometry of the data manifold. We explore and theoretically prove this hypothesis for a simple synthetic dataset.
****
Q: Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

A:
We suggest a generalization bound that could partly explain the improvement in generalization with over-parametrization.
****