Query: Neural Program Learning under Noisy I/O

title: Image Feature Learning with Genetic Programming
abstract: Learning features from raw data is an important topic in machine learning. This paper presents Genetic Program Feature Learner (GPFL) a novel generative GP feature learner for 2D images. GPFL executes multiple GP runs, each run generates a model that focuses on a particular high level feature of the training images. Then, it combines the models generated by each run into a function that reconstructs the observed images. As a sanity check, we evaluated GPFL on the popular MNIST dataset of handwritten digits, and compared it with the convolutional neural network LeNet5. Our evaluation results show that when considering smaller training sets, GPFL achieves comparable/slightly better classification accuracy than LeNet5. However, GPFL drastically outperforms LeNet5 when considering noisy images as test sets.

title: RobustFill: Neural Program Learning under Noisy I/O
abstract: The problem of automatically generating a computer program from some specification has been studied since the early days of AI. Recently, two competing approaches for automatic program learning have received significant attention: (1) neural program synthesis, where a neural network is conditioned on input/output (I/O) examples and learns to generate a program, and (2) neural program induction, where a neural network generates new outputs directly using a latent program representation. Here, for the first time, we directly compare both approaches on a large scale, real world learning task and we additionally contrast to rule based program synthesis, which uses hand crafted semantics to guide the program generation. Our neural models use a modified attention RNN to allow encoding of variable sized sets of I/O pairs, which achieve 92% accuracy on a real world test set, compared to the 34% accuracy of the previous best neural synthesis approach. The synthesis model also outperforms a comparable induction model on this task, but we more importantly demonstrate that the strength of each approach is highly dependent on the evaluation metric and end user application. Finally, we show that we can train our neural models to remain very robust to the type of noise expected in real world data (e.g. typos) while a highly engineered rule based system fails entirely.

title: Supplementary Material for RobustFill: Neural Program Learning under Noisy I/O
abstract: Where i is the current timestep, hi1 is the previous hidden state, xi is the current input, S s1, sN are the vectors being attended to, and W is a learned parameter matrix. The interpolated context vector ci is concatenated into the input and fed into the LSTM. In the case of double attention, the output of the first attention mechanism C i is concatenated to the input of the second attention,

title: SuperCoder: Program Learning Under Noisy Conditions From Superposition of States
abstract: We propose a new method of program learning in a Domain Specific Language (DSL) which is based on gradient descent with no direct search. The first component of our method is a probabilistic representation of the DSL variables. At each timestep in the program sequence, different DSL functions are applied on the DSL variables with a certain probability, leading to different possible outcomes. Rather than handling all these outputs separately, whose number grows exponentially with each timestep, we collect them into a superposition of variables which captures the information in a single, but fuzzy, state. This state is to be contrasted at the final timestep with the ground truth output, through a loss function. The second component of our method is an attention based recurrent neural network, which provides an appropriate initialization point for the gradient descent that optimizes the probabilistic representation. The method we have developed surpasses the state of the art for synthesising long programs and is able to learn programs under noise.

title: Fault Diagnosis of Rotating Machinery under Noisy Environment Conditions Based on a 1 D Convolutional Autoencoder and 1 D Convolutional Neural Network
abstract: Deep learning methods have been widely used in the field of intelligent fault diagnosis due to their powerful feature learning and classification capabilities. However, it is easy to overfit depth models because of the large number of parameters brought by the multilayer structure. As a result, the methods with excellent performance under experimental conditions may severely degrade under noisy environment conditions, which are ubiquitous in practical industrial applications. In this paper, a novel method combining a one dimensional (1 D) denoising convolutional autoencoder (DCAE) and a 1 D convolutional neural network (CNN) is proposed to address this problem, whereby the former is used for noise reduction of raw vibration signals and the latter for fault diagnosis using the de noised signals. The DCAE model is trained with noisy input for denoising learning. In the CNN model, a global average pooling layer, instead of fully connected layers, is applied as a classifier to reduce the number of parameters and the risk of overfitting. In addition, randomly corrupted signals are adopted as training samples to improve the anti noise diagnosis ability. The proposed method is validated by bearing and gearbox datasets mixed with Gaussian noise. The experimental result shows that the proposed DCAE model is effective in denoising and almost causes no loss of input information, while the using of global average pooling and input corrupt training improves the anti noise ability of the CNN model. As a result, the method combined the DCAE model and the CNN model can realize high accuracy diagnosis even under noisy environment.

title: PLANS: Robust Program Learning from Neurally Inferred Specifications
abstract: Recent years have seen the rise of statistical program learning based on neural models as an alternative to traditional rule based systems for programming by example. Rule based approaches offer correctness guarantees in an unsupervised way as they inherently capture logical rules, while neural models are more realistically scalable to raw, high dimensional input, and provide resistance to noisy I/O specifications. We introduce PLANS (Program LeArning from Neurally inferred Specifications) a hybrid model for program synthesis from visual observations that gets the best of both worlds, relying on (i) a neural architecture trained to extract abstract, high level information from each raw individual input (ii) a rule based system using the extracted information as I/O specifications to synthesize a program capturing the different observations. In order to address the key challenge of making PLANS resistant to noise in the network's output, we introduce a filtering heuristic for I/O specifications based on selective classification techniques. We obtain state of the art performance at program synthesis from diverse demonstration videos in the Karel and ViZDoom environments, while requiring no ground truth program for training. We make our implementation available at this http URL.

title: PLANS: Neuro Symbolic Program Learning from Videos
abstract: Recent years have seen the rise of statistical program learning based on neural models as an alternative to traditional rule based systems for programming by example. Rule based approaches offer correctness guarantees in an unsupervised way as they inherently capture logical rules, while neural models are more realistically scalable to raw, high dimensional input, and provide resistance to noisy I/O specifications. We introduce PLANS (Program LeArning from Neurally inferred Specifications) a hybrid model for program synthesis from visual observations that gets the best of both worlds, relying on (i) a neural architecture trained to extract abstract, high level information from each raw individual input (ii) a rule based system using the extracted information as I/O specifications to synthesize a program capturing the different observations. In order to address the key challenge of making PLANS resistant to noise in the network's output, we introduce a dynamic filtering algorithm for I/O specifications based on selective classification techniques. We obtain state of the art performance at program synthesis from diverse demonstration videos in the Karel and ViZDoom environments, while requiring no ground truth program for training.

title: Neural Program Induction for KBQA Without Gold Programs or Query Annotations
abstract: Neural Program Induction (NPI) is a paradigm for decomposing high level tasks such as complex question answering over knowledge bases (KBQA) into executable programs by employing neural models. Typically, this involves two key phases: i) inferring input program variables from the high level task description, and ii) generating the correct program sequence involving these variables. Here we focus on NPI for Complex KBQA with only the final answer as supervision, and not gold programs. This raises major challenges; namely, i) noisy query annotation in the absence of any supervision can lead to catastrophic forgetting while learning, ii) reward becomes extremely sparse owing to the noise. To deal with these, we propose a noise resilient NPI model, Stable Sparse Reward based Programmer (SSRP) that evades noise induced instability through continual retrospection and its comparison with current learning behavior. On complex KBQA datasets, SSRP performs at par with hand crafted rule based models when provided with gold program input, and in the noisy settings outperforms state of the art models by a significant margin even with a noisier query annotator.

title: Learning from past mistakes: improving automatic speech recognition output via noisy clean phrase context modeling
abstract: Automatic speech recognition (ASR) systems often make unrecoverable errors due to subsystem pruning (acoustic, language and pronunciation models) for example, pruning words due to acoustics using short term context, prior to rescoring with long term context based on linguistics. In this work, we model ASR as a phrase based noisy transformation channel and propose an error correction system that can learn from the aggregate errors of all the independent modules constituting the ASR and attempt to invert those. The proposed system can exploit long term context using a neural network language model and can better choose between existing ASR output possibilities as well as re introduce previously pruned or unseen (Out Of Vocabulary) phrases. It provides corrections under poorly performing ASR conditions without degrading any accurate transcriptions; such corrections are greater on top of out of domain and mismatched data ASR. Our system consistently provides improvements over the baseline ASR, even when baseline is further optimized through Recurrent Neural Network (RNN) language model rescoring. This demonstrates that any ASR improvements can be exploited independently and that our proposed system can potentially still provide benefits on highly optimized ASR. Finally, we present an extensive analysis of the type of errors corrected by our system.

title: Normalized Loss Functions for Deep Learning with Noisy Labels
abstract: Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL) APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state of the art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.