TASK DEFINITION: In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
PROBLEM: Synthesizing user-intended programs from a small number of input-output exam-
 ples is a challenging problem with several important applications like spreadsheet
 manipulation, data wrangling and code refactoring. Existing synthesis systems
 either completely rely on deductive logic techniques that are extensively hand-
 engineered or on purely statistical models that need massive amounts of data, and in
 general fail to provide real-time synthesis on challenging benchmarks. In this work,
 we propose Neural Guided Deductive Search (NGDS), a hybrid synthesis technique
 that combines the best of both symbolic logic techniques and statistical models.
 Thus, it produces programs that satisfy the provided specifications by construction
 and generalize well on unseen examples, similar to data-driven systems. Our
 technique effectively utilizes the deductive search framework to reduce the learning
 problem of the neural component to a simple supervised learning setup. Further,
 this allows us to both train on sparingly available real-world data and still leverage
 powerful recurrent neural network encoders. We demonstrate the effectiveness
 of our method by evaluating on real-world customer scenarios by synthesizing
 accurate programs with up to 12× speed-up compared to state-of-the-art systems.

SOLUTION: We integrate symbolic (deductive) and statistical (neural-based) methods to enable real-time program synthesis with almost perfect generalization from 1 input-output example.

PROBLEM: Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.

SOLUTION: Extracting a finite state machine from a recurrent neural network via quantization for the purpose of interpretability with experiments on Atari.

PROBLEM: Knowledge Distillation (KD) is a common method for transferring the ``knowledge'' learned by one machine learning model (the teacher) into another model (the student), where typically, the teacher has a greater capacity (e.g., more parameters or higher bit-widths). To our knowledge, existing methods overlook the fact that although the student absorbs extra knowledge from the teacher, both models share the same input data -- and this data is the only medium by which the teacher's knowledge can be demonstrated. Due to the difference in model capacities, the student may not benefit fully from the same data points on which the teacher is trained. On the other hand, a human teacher may demonstrate a piece of knowledge with individualized examples adapted to a particular student, for instance, in terms of her cultural background and interests. Inspired by this behavior, we design data augmentation agents with distinct roles to facilitate knowledge distillation. Our data augmentation agents generate distinct training data for the teacher and student, respectively. We focus specifically on KD when the teacher network has greater precision (bit-width) than the student network.

 We find empirically that specially tailored data points enable the teacher's knowledge to be demonstrated more effectively to the student. We compare our approach with existing KD methods on training popular neural architectures and demonstrate that role-wise data augmentation improves the effectiveness of KD over strong prior approaches. The code for reproducing our results will be made publicly available.

SOLUTION:
We study whether and how adaptive data augmentation and knowledge distillation can be leveraged simultaneously in a synergistic manner for better training student networks.