instruction:
In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
question:
We propose several simple approaches to training deep neural networks on data with noisy labels. We introduce an extra noise layer into the network which adapts the network outputs to match the noisy label distribution. The parameters of this noise layer can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks. We demonstrate the approaches on several datasets, including large scale experiments on the ImageNet classification benchmark, showing how additional noisy data can improve state-of-the-art recognition models. 1 Introduction In recent years, deep learning methods have shown impressive results on image classification tasks. However, this achievement is only possible because of large amount of labeled images. Labeling images by hand is a laborious task and takes a lot of time and money. An alternative approach is to generate labels automatically. This includes user tags from social web sites and keywords from image search engines. Considering the abundance of such noisy labels, it is important to find a way to utilize them in deep learning. Unfortunately, those labels are very noisy and unlikely to help training deep networks without additional tricks. Our goal is to study the effect label noise on deep networks, and explore simple ways of improvement. We focus on the robustness of deep networks instead of data cleaning methods, which are well studied and can be used together with robust models directly. Although many noise robust classifiers are proposed so far, there are not many works on training deep networks on noisy labeled data, especially on large scale datasets. Our contribution in this paper is a novel way of modifying deep learning models so they can be effectively trained on data with high level of label noise. The modification is simply done by adding a linear layer on top of the softmax layer, which makes it easy to implement. This additional layer changes the output from the network to give better match to the noisy labels. Also, it is possible to learn the noise distribution directly from the noisy data. Using real-world image classification tasks, we demonstrate that the model actually works very well in practice. We even show that random images without labels (complete noise) can improve the classification performance. 2 Related Work In any classification model, degradation of performance is inevitable when there is noise in training labels [13, 15]. A simple approach to handle noisy labels is a data preprocessing stage, where labels suspected to be incorrect are removed or corrected [1, 3]. However, a weakness of this approach is the difficulty of distinguishing informative hard samples from harmful mislabeled ones [6]. Instead, in this paper, we focus on models robust to presence of label noise. 1 ar X iv :1 40 6. 20 80 v1 [ cs .C V ] 9 J un 2 01 4 The effect of label noise is well studied in common classifiers (e.g., SVMs, kNN, logistic regression), and their label noise robust variants have been proposed. See [5] for comprehensive review. A more recent work [2] proposed a generic unbiased estimator for binary classification with noisy labels. They employ a surrogate cost function that can be expressed by a weighted sum of the original cost functions, and gave theoretical bounds on the performance. In this paper, we will also consider this idea and extend it multiclass. A cost function similar to ours is proposed in [2] to make logistic regression robust to label noise. They also proposed a learning algorithm for noise parameters. However, we consider deep networks, a more powerful and complex classifier than logistic regression, and propose a different learning algorithm for noise parameters that is more suited for back-propagation training. Considering the recent success of deep learning [8, 17, 16], there are very few works about deep learning from noisy labels. In [11, 9], noise modeling is incorporated to neural network in the same way as our proposed model. However, only binary classification is considered in [11], and [9] assumed symmetric label noise (noise is independent of the true label). Therefore, there is only a single noise parameter, which can be tuned by cross-validation. In this paper, we consider multiclass classification and assume more realistic asymmetric label noise, which makes it impossible to use cross-validation to adjust noise parameters (there can be a million parameters). 3 Approach In this paper, we consider two approaches to make an existing classification model, which we call the base model, robust against noisy labels: bottom-up and top-down noise models. In the bottomup model, we add an additional layer to the model that changes the label probabilities output by the base model so it would better match to noisy labels. Top-down model, on other hand, changes given noisy labels before feeding them to the base model. Both models require a noise model for training, so we will give an easy way to estimate noise levels using clean data. Also, it is possible to learn noise distribution from noisy data in the bottom-up model. Although only deep neural networks are used in our experiments, the both approaches can be applied to any classification model with a cross entropy cost. 3.1 Bottom-up Noise Model We assume that label noise is random conditioned on the true class, but independent of the input x (see [10] for more detail about this type of noise). Based on this assumption, we add an additional layer to a deep network (see Figure 1) that changes its output so it would better match to the noisy labels. The weights of this layer corresponds to the probabilities of a certain class being mislabeled to another class. Because those probabilities are often unknown, we will show how estimate them from additional clean data, or from the noisy data itself. Let D be the true data distribution generating correctly labeled samples (x, y∗), where x is an input vector and y∗ is the corresponding label. However, we only observe noisy labeled samples (x, ỹ) that generated from a some noisy distribution D̃. We assume that the label noise is random conditioned on the true labels. Then, the noise distribution can be parameterized by a matrix Q = {qji}: qji := p(ỹ = j|y∗ = i). Q is a probability matrix because its elements are positive and each column sums to one. The probability of input x being labeled as j in D̃ is given by p(ỹ = j|x, θ) = ∑ i p(ỹ = j|y∗ = i)p(y∗ = i|x) = ∑ i qjip(y ∗ = i|x, θ). (1) where p(y∗ = i|x, θ) is the probabilistic output of the base model with parameters θ. If the true noise distribution is known, we can modify this for noisy labeled data. During training, Q will act as an adapter that transforms the model’s output to better match the noisy labels. Deep	  network Learnin from noisy labels in deep neural networks Sainbayar Sukhbaatar Dept. of Computer Science, NYU, 715 Broadway, New Y rk, NY 10003 sainbar@cs.nyu.edu Rob Fergus Courant Institute, NYU, 715 Broadway, New York, NY 10003 fergus@cs.nyu.edu
answer:
Learning from Noisy Labels with Deep Neural Networks


question:
In this paper we examine the benefit of performing named entity recognition (NER) and co-reference resolution to an English and a Greek corpus used for text segmentation. The aim here is to examine whether the combination of text segmentation and information extraction can be beneficial for the identification of the various topics that appear in a document. NER was performed manually in the English corpus and was compared with the output produced by publicly available annotation tools while, an already existing tool was used for the Greek corpus. Produced annotations from both corpora were manually corrected and enriched to cover four types of named entities. Co-reference resolution i.e., substitution of every reference of the same instance with the same named entity identifier was subsequently performed. The evaluation, using five text segmentation algorithms for the English corpus and four for the Greek corpus leads to the conclusion that, the benefit highly depends on the segment’s topic, the number of named entity instances appearing in it, as well as the segment’s length.
answer:
Text Segmentation using Named Entity Recognition and Co-reference Resolution in English and Greek Texts


question:
The sparse, hierarchical and modular processing of natural signals are characteristics that relate to the ability of humans to recognise objects with high accuracy. In this paper, we report a sparse feature processing and encoding method targeted at improving the recognition performance of automated object recognition system. Randomly distributed selection of localised gradient enhanced features followed by the application of aggregate functions represents a modular and hierarchical approach to detect the object features. These object features, in combination with minimum distance classifier, results in object recognition system accuracies of 93% using ALOI, 92% using COIL-100 databases and 69% using PASCAL visual object challenge 2007 database, respectively. Robustness of object recognition performance is tested for variations in noise, object scaling and object shifts. Finally, a comparison with 8 existing object recognition methods indicated an improvement in recognition accuracy of 10% in ALOI, 8% in case of COIL-100 databases and 10% in PASCAL visual object challenge 2007 database.
answer:

Sparse distributed localised gradient fused features of objects