Learning with partial labels: an exploratory study of data scarcity for medical images

The data-driven nature of deep learning models is often associated with the use of large-scale fully labeled datasets. However, the high cost of annotation often makes these datasets unavailable for standard supervised learning. This thesis explores an emerging learning paradigm called partially supervised learning (PSL), which deals with partially labeled examples. Unlike semi-supervised learning, where an example is either fully labeled or unlabeled, PSL addresses a similar but distinct problem. In PSL, each example is labeled for a subset of sub-tasks within the context of multi-task learning. The utilization of multiple small-scale partially labeled datasets in PSL is still a challenging research question. Therefore, in this thesis, I provide a formal definition of PSL for the first time.
Partially labeled datasets are frequently encountered in the medical field due to the high cost of annotation and the specific clinical objectives they serve. In this regard, I leverage partially supervised multi-label classification (PSMLC) on medical images as the task of interest to exemplify the aforementioned concept. This thesis aims to offer empirical insights into enhancing the performance of deep learning models trained on smallscale partially labeled datasets by presenting the following methodological advancements under four difference PSL setups.
This thesis first proposes a practical solution to PSL under data scarcity based on vicinal risk minimization (VRM). While VRM was originally designed as a simple technique to enhance the generalization ability of deep learning models in supervised learning, I demonstrate that, by leveraging principle of maximum entropy, VRM can be successfully applied to PSL. Through experimental evidence, I illustrate that VRM shows advantages over semi-supervised learning methods in both accuracy and computational complexity, particularly in practical and challenging scenarios. This thesis represents the first application of VRM to PSL, offering both theoretical insights and empirical findings. Furthermore, I provide an explanation for the limitations of semi-supervised learning methods in this context.
Fueled by the recent renaissance of self-supervised learning, this thesis aims to explore the potential of utilizing unlabeled data to enhance the performance of deep learning models trained on partially labeled data. While self-supervised learning has played a crucial role in pre-training deep learning models for fully-supervised downstream tasks, its impact on downstream tasks involving only partially labeled data remains unclear. Therefore, this thesis presents the first empirical study to investigate the contributions of self-supervised learning in addressing PSL problems and proposes a novel pretext task based on VRM.
While previous studies have focused on situations where partially labeled datasets are collected in a centralized fashion or can be remotely accessed without communication constraints, this thesis takes a step further to discuss situations where partially labeled datasets are decentralized under data governance constraints. Specifically, the partially labeled datasets are stored separately in different clients within a federated system, and the raw data cannot be exchanged between the clients or uploaded to the central server. This motivates the definition of a new research problem called federated partially supervised learning (FPSL). In this thesis, I theoretically discuss the challenges of FPSL and provide a robust federated solution to address this problem.
Moreover, I observe that in practical scenarios, classes often follow a longtailed distribution, which means that, in addition to common classes, there are also rare classes. Consequently, I discuss a special case of FPSL known as federated partially supervised learning with underrepresented classes. In this case, a few clients store a limited number of partially labeled examples related to the rare classes, while the other clients have a much larger number of partially labeled examples related to common classes. To address this extreme class imbalance, I propose, for the first time, an energy-based federated solution to effectively learn and recognize the rare classes without compromising the performance of common classes. In summary, this thesis explores four unexplored research directions of PSL under data scarcity. For each of four research directions, a novel solution is proposed to address the problem of interest and evaluated with simulated experiments. The empirical results not only provide empirical insights to understand PSL but also pose new research directions on PSL.

Keywords: label-efficient learning
partially supervised learning
Advisor: Voiculescu, I