Towards diverse generation and reliable classification using neural networks
Keywords: machine learning
generative adversarial networks
generative AI
deep learning
interactive video object segmentation
confidence calibration
weak-supervision
weakly-supervised image segmentation
computer vision
uncertainty

Humans can easily understand their surroundings, analyze the situation, and even figure out the intentions of others, all in a few seconds. However, computers see the surroundings as an array of numbers and we need to write algorithms for them to make a sense of it. It is a huge challenge but significant strides have been made toward the scene understanding problem over the last decade, thanks to an increase in computing infrastructure and datasets. However, several challenges remain in making computers more 'intelligent', some of which are tackled in this thesis. This thesis develops different techniques to create more reliable systems and proposes innovations in loss functions by analyzing the system's prediction distributions.
First, we try to address the well-known problem of mode collapse in Generative Adversarial Networks (GANs) and propose MAD-GAN, an intuitive generalization of the GANs. MAD-GAN is a multi-agent GAN architecture incorporating multiple generators and one discriminator. To enforce that different generators capture diverse high probability modes, the objective function of the discriminator is designed such that along with identifying the real and fake samples, it is also required to identify the generator that generated the sample if it is fake. Intuitively, to succeed in this task, the discriminator must learn to push different generators toward different identifiable modes. Extensive experiments on synthetic and real datasets demonstrate the effectiveness of the MAD-GAN approach.
We then consider the task of semantic segmentation using weak supervision with bounding box annotations. Bounding boxes are noisy labels for the foreground objects. To focus on foreground objects, our approach predicts a per-class attention map that saliently guides the per-pixel cross-entropy loss and refines the segmentation boundaries. This avoids propagating erroneous gradients due to the incorrect foreground labels. Additionally, our approach learns pixel embeddings to simultaneously optimize for high intra-class feature affinity while increasing discrimination between features across different classes. This helps in capturing global context via long-range pairwise interactions. Qualitative and quantitative results along with ablation studies show the benefit of different loss terms on the overall performance.
The widespread usage of neural networks is particularly dependent on their reliability and trustworthiness. We therefore also consider the important task of calibrating deep neural networks – where a model’s confidence should be the same as its correctness. Ideally, we want networks to be accurate, calibrated, and confident. We provide a thorough analysis of the factors causing miscalibration and use the obtained insights to justify the empirically excellent performance of focal loss. We show that, as opposed to the standard cross-entropy loss, focal loss allows us to learn models that are already very well-calibrated. When combined with temperature scaling, whilst preserving accuracy, focal loss yields state-of-the-art calibrated models. To facilitate its use in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. Extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, justify that this approach achieves state-of-the-art calibration without compromising accuracy in almost all cases.
One-shot video object segmentation is an important problem in computer vision with many interesting real-world applications. To tackle this, we finally introduce a similarity learning approach, that can learn to perform dense label transfer from one image to the other. More specifically, the objective is to learn a similarity metric for dense pixel-wise correspondence between two images. This learned model can then be used in a label transfer framework to propagate object annotation from the reference frame to all the subsequent frames in a video. Unlike previous methods, our similarity learning approach works fairly well across various domains, even when no domain adaptation is involved. We demonstrate the effectiveness of our method on two standard datasets with favourable results. This approach not only provides good object segmentation but is also time-efficient. Therefore, using it we also achieved second place in the first DAVIS challenge for interactive video object segmentation, in both quality and speed tracks.