Alleviating Noisy Data in Image Captioning with Cooperative Distillation
Venue: ArXiv
Year: 2020

Image captioning systems have made substantial progress, largely due to the availability of curated datasets like Microsoft COCO or Vizwiz that have accurate descriptions of their corresponding images. Unfortunately, scarce availability of such cleanly labeled data results in trained algorithms producing captions that can be terse and idiosyncratically specific to details in the image. We propose a new technique, cooperative distillation that combines clean curated datasets with the web scale automatically extracted captions of the Google Conceptual Captions dataset (GCC) which can have poor descriptions of images, but is abundant in size and therefore provides a rich vocabulary resulting in more expressive captions. Learning with noisy data is an important and challenging problem in machine learning. In image captioning, the problem of noisy annotations has been addressed by several authors. For example,  uses a number of heuristics (tagging, annotations, word statistics, etc. to filter out low quality captions. Similarly, in  the training captions are denoised by extracting only the noun phrases, which are then treated as a training dataset. In the winning entry of the GCC challenge  authors employ dynamic filtering in the training, which is based on model confidence. Our cooperative distillation (co distill) framework trains a student model on a large noisy dataset. By noise we mean that captions are often grammatically incorrect or do not match the semantic content of images. We also rely on a clean dataset to train a teacher model. We explore whether we can leverage the specific advantages of both types of datasets by training on a rich vocabulary and variety of scene contexts, while alleviating the noisy annotations. Semantic Bridge between Datasets. We use a BERT tokenizer to generate a joint vocabulary across both the noisy and clean datasets at a sub word units level, and we also leverage BERT to represent captions as the embedding of the resulting [CLS] token. Such representation has been shown to capture semantic information that can be used to quantify the semantic similarity between captions (see e.g.  Semantic similarity is a crucial element in our algorithm used for: 1) denoising, by down weighting noisy captions that are not semantically aligned with those generated by the teacher model, and 2) increasing caption diversity, by augmenting the training dataset with generated captions that are aligned with clean ground truth captions.  Transformer networks from [5] are used for both student and teacher models. Captions are generated conditionally on the encoded features via the transformer decoder. The transformer architectures have 2 layers (for both encoder and decoder) embeddings of size 512, and 8 attention heads per layer. Cooperative Distillation. In order to train both the student and the teacher, we alternate between two training streams: denoising and diversity, shown in Figure 1. In the denoising stream, the student minimizes its loss given a fixed teacher model, and in the diversity stream, the teacher minimizes its loss given a fixed student as explained next. Denoising Stream Loss. Given a sample from the noisy student dataset, we embed the ground truth caption through BERT and represent it as the embedding of the resulting [CLS] token. We then push the image through the Teacher Transformer to obtain a softmax distribution, which is decoded using a greedy max approach to produce the teacher's predicted tokens sequence. As with the tokenized ground truth caption, we embed this predicted sequence via BERT. These two embeddings allow us to define a semantic coherence weighting for the denoising stream, which captures the similarity between the noisy ground truth caption and the caption predicted by the teacher model. For a fixed teacher, the student minimizes a per sample loss that combines weighted terms of cross entropy based on ground truth labels and KL divergence between student and teacher predictions. The weight on each term is determined by the semantic coherence calculated through the BERT embedding bridge. If the semantic coherence weight is high, the student can "trust" its ground truth and puts a higher weight on the regular cross entropy training term. If the semantic coherence weight is low, the student instead distills the teacher through a KL term comparing the softlabels from the student and the caption predicted by the 1 ar X iv :2 01 2. 11 69 1v 1 cs .C V 2 1 D ec 2 02 0