In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
--------
Question: We propose several simple approaches to training deep neural networks on data with noisy labels. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark, showing how additional noisy data can improve state-of-the-art recognition models. 1 Introduction In recent years, deep learning methods have shown impressive results on image classification tasks. However, this achievement is only possible because of large amount of labeled images. Labeling images by hand is a laborious task and takes a lot of time and money. An alternative approach is to generate labels automatically. This includes user tags from social web sites and keywords from image search engines. Considering the abundance of such noisy labels, it is important to find a way to utilize them in deep learning. Unfortunately, those labels are very noisy and unlikely to help training deep networks without additional tricks. Our goal is to study the effect label noise on deep networks, and explore simple ways of improvement. We focus on the robustness of deep networks instead of data cleaning methods, which are well studied and can be used together with robust models directly. Although many noise robust classifiers are proposed so far, there are not many works on training deep networks on noisy labeled data, especially on large scale datasets. Our contribution in this paper is a novel way of modifying deep learning models so they can be effectively trained on data with high level of label noise. The modification is simply done by adding a linear layer on top of the softmax layer, which makes it easy to implement. This additional layer changes the output from the network to give better match to the noisy labels. Also, it is possible to learn the noise distribution directly from the noisy data. Using real-world image classification tasks, we demonstrate that the model actually works very well in practice. We even show that random images without labels (complete noise) can improve the classification performance. 2 Related Work In any classification model, degradation of performance is inevitable when there is noise in training labels [13, 15]. A simple approach to handle noisy labels is a data preprocessing stage, where labels suspected to be incorrect are removed or corrected [1, 3]. However, a weakness of this approach is the difficulty of distinguishing informative hard samples from harmful mislabeled ones [6]. Instead, in this paper, we focus on models robust to presence of label noise. 1 ar X iv :1 40 6. 20 80 v1 [ cs .C V ] 9 J un 2 01 4 The effect of label noise is well studied in common classifiers (e.g., SVMs, kNN, logistic regression), and their label noise robust variants have been proposed. See [5] for comprehensive review. A more recent work [2] proposed a generic unbiased estimator for binary classification with noisy labels. They employ a surrogate cost function that can be expressed by a weighted sum of the original cost functions, and gave theoretical bounds on the performance. In this paper, we will also consider this idea and extend it multiclass. A cost function similar to ours is proposed in [2] to make logistic regression robust to label noise. They also proposed a learning algorithm for noise parameters. However, we consider deep networks, a more powerful and complex classifier than logistic regression, and propose a different learning algorithm for noise parameters that is more suited for back-propagation training. Considering the recent success of deep learning [8, 17, 16], there are very few works about deep learning from noisy labels. In [11, 9], noise modeling is incorporated to neural network in the same way as our proposed model. However, only binary classification is considered in [11], and [9] assumed symmetric label noise (noise is independent of the true label). Therefore, there is only a single noise parameter, which can be tuned by cross-validation. In this paper, we consider multiclass classification and assume more realistic asymmetric label noise, which makes it impossible to use cross-validation to adjust noise parameters (there can be a million parameters). 3 Approach In this paper, we consider two approaches to make an existing classification model, which we call the base model, robust against noisy labels: bottom-up and top-down noise models. In the bottomup model, we add an additional layer to the model that changes the label probabilities output by the base model so it would better match to noisy labels. Top-down model, on other hand, changes given noisy labels before feeding them to the base model. Both models require a noise model for training, so we will give an easy way to estimate noise levels using clean data. Also, it is possible to learn noise distribution from noisy data in the bottom-up model. Although only deep neural networks are used in our experiments, the both approaches can be applied to any classification model with a cross entropy cost. 3.1 Bottom-up Noise Model We assume that label noise is random conditioned on the true class, but independent of the input x (see [10] for more detail about this type of noise). Based on this assumption, we add an additional layer to a deep network (see Figure 1) that changes its output so it would better match to the noisy labels. The weights of this layer corresponds to the probabilities of a certain class being mislabeled to another class. Because those probabilities are often unknown, we will show how estimate them from additional clean data, or from the noisy data itself. Let D be the true data distribution generating correctly labeled samples (x, y∗), where x is an input vector and y∗ is the corresponding label. However, we only observe noisy labeled samples (x, ỹ) that generated from a some noisy distribution D̃. We assume that the label noise is random conditioned on the true labels. Then, the noise distribution can be parameterized by a matrix Q = {qji}: qji := p(ỹ = j|y∗ = i). Q is a probability matrix because its elements are positive and each column sums to one. The probability of input x being labeled as j in D̃ is given by p(ỹ = j|x, θ) = ∑ i p(ỹ = j|y∗ = i)p(y∗ = i|x) = ∑ i qjip(y ∗ = i|x, θ). (1) where p(y∗ = i|x, θ) is the probabilistic output of the base model with parameters θ. If the true noise distribution is known, we can modify this for noisy labeled data. During training, Q will act as an adapter that transforms the model’s output to better match the noisy labels. Deep	  network Learnin from noisy labels in deep neural networks Sainbayar Sukhbaatar Dept. of Computer Science, NYU, 715 Broadway, New Y rk, NY 10003 sainbar@cs.nyu.edu Rob Fergus Courant Institute, NYU, 715 Broadway, New York, NY 10003 fergus@cs.nyu.edu

Answer: Learning from Noisy Labels with Deep Neural Networks


Question: In this paper we formulate the framework of recovering a hidden orthonormal basis given access to a<lb>certain “Basis Encoding Function”. We describe the class of Basis Encoding Functions (BEF), such that<lb>their local maxima on the unit sphere are in one-to-one correspondence with the basis elements. This<lb>description relies on a certain “hidden convexity” property of these functions. A number of theoretical<lb>and practical problems of recent interest can be interpreted as recovering a hidden basis from potentially<lb>noisy observations. Specifically, we show how our simple and general framework applies to Independent<lb>Component Analysis (ICA), tensor decompositions, spectral clustering and Gaussian mixture learning.<lb>We describe a new algorithm, “gradient iteration”, for provable recovery of the hidden basis. We<lb>provide a complete theoretical analysis of Gradient Iteration both for the exact case as well as for the<lb>case when the observed function is a perturbation of the “true” underlying BEF. In both cases we show<lb>convergence and complexity bounds polynomial in dimension and other relevant parameters, such as<lb>perturbation size. Our perturbation results can be considered as a very general non-linear version of<lb>the classical Davis-Kahan theorem for eigenvectors of perturbations of symmetric matrices. In addition<lb>we show that in the exact case the algorithm converges superlinearly and give conditions relating the<lb>degree of convergence to properties of the Basis Encoding Function. Our algorithm can be viewed as a<lb>generalization of the classical power iteration method for eigenanalysis of symmetric matrices as well as<lb>a generalization of power iterations for tensors. Moreover, the Gradient Iteration algorithm can be easily<lb>and efficiently implemented in practice.

Answer: Learning a Hidden Basis Through Imperfect Measurements: An Algorithmic Primitive


Question: A major challenge in paraphrase research is the lack of parallel corpora. In this paper, we present a new method to collect large-scale sentential paraphrases from Twitter by linking tweets through shared URLs. The main advantage of our method is its simplicity, as it gets rid of the classifier or human in the loop needed to select data before annotation and subsequent application of paraphrase identification algorithms in the previous work. We present the largest human-labeled paraphrase corpus to date of 51,524 sentence pairs and the first cross-domain benchmarking for automatic paraphrase identification. In addition, we show that more than 30,000 new sentential paraphrases can be easily and continuously captured every month at ∼70% precision, and demonstrate their utility for downstream NLP tasks through phrasal paraphrase extraction. We make our code and data freely available.1

Answer:
A Continuously Growing Dataset of Sentential Paraphrases