Constrained deep networks for medical image segmentation

Weakly supervised image segmentation, in the form of partially labeled images, is attracting significant research attention as it can mitigate the need for laborious pixel annotations required by deep learning models. Enforcing high-order, global inequality constraints on the network outputs can leverage unlabeled data by guiding the training with prior knowledge, restricting the search space during training to anatomically feasible solutions. A range of possible values (such as a lower/upper bounds on the size of a organ) can be very valuable to guide training. However, in the context of deep neural networks, standard Lagrangian optimization has been largely avoided, mainly due to the instability and computational complexity ensuing from alternating explicit dual updates and stochastic optimization. Interior point methods, despite their popularity in convex optimization, are not applicable neither, as they require a feasible starting point, which is itself a difficult constrained problem for deep neural networks. In this thesis, we investigate hard inequality constraints in the context of deep networks with both quadratic penalties and more principled log-barrier extensions. We also investigate methods to mitigate class-imbalance in segmentation problems, such as in brain lesions dataset, by constraining the boundary of the predicted segmentation to match the ground-truth boundary. This thesis produced five different publications as first author, and four papers as co-author. Our papers received several awards, and we were invited to publish extended versions of our works in two special issues of Medical Image Analysis (MedIA).In our first contribution, we propose to introduce a differentiable penalty, which enforces inequality constraints directly in the loss function, avoiding expensive Lagrangian dual iterates and proposal generation. From constrained-optimization perspective, our simple penalty-based approach is not optimal as there is no guarantee that the constraints are satisfied. However, surprisingly, it yields substantially better results than Lagrangian-based constrained convolutional neural networks, while reducing the computational demand for training. By annotating only a small fraction of the pixels, our approach reaches performances comparable to full supervision, on three separate tasks. While our experiments focused on basic linear constraints such as the target-region size and image tags, our framework can be easily extended to other non-linear constraints, e.g., invariant shape moments and other region statistics.In our second contribution, we propose log-barrier extensions, which approximate Lagrangian optimization of constrained-CNN problems with a sequence of unconstrained losses. Unlike standard interior-point and log-barrier methods, our formulation does not need an initial feasible solution. We report comprehensive weakly supervised segmentation experiments, with various constraints, showing that our formulation outperforms substantially the existing constrained-CNN methods, both in terms of accuracy, constraint satisfaction and training stability.In our third contribution, we enforce constraints on the boundary of predicted segmentation. Widely used loss functions for CNN segmentation, such as Dice or cross entropy, are based on integrals over the segmentation regions. Unfortunately, for highly unbalanced segmentations, such regional summations have values that differ by several orders of magnitude across classes, which affects training performance and stability. We propose a boundary loss, which takes the form of a distance metric on the space of contours, not regions. This can mitigate the difficulties of highly unbalanced problems because it uses integrals over the interface between regions instead of unbalanced integrals over the regions. Furthermore, a boundary loss complements regional information. Inspired by graph-based optimization techniques for computing activecontour flows, we express a non-symmetric L2 distance on the space of contours as a regional integral, which avoids completely local differential computations involving contour points. This yields a boundary loss expressed with the regional softmax probability outputs of the network, which can be easily combined with standard regional losses and implemented with any existing deep network architecture for N-D segmentation. We report comprehensive evaluations on different unbalanced problems, showing that our boundary loss can yield significant increases in performances while improving training stability.In a fourth contribution, we investigates a curriculum-style strategy for semi-supervised CNN segmentation, which devises a regression network to learn image-level information such as the size of the target region. These regressions are used to effectively regularize the segmentation network, constraining the softmax predictions of the unlabeled images to match the inferred label distributions. Our framework is based on inequality constraints, which tolerate uncertainties in the inferred knowledge, e.g., regressed region size. It can be used for a large variety of region attributes. We evaluated our approach for left ventricle segmentation in magnetic resonance images (MRI), and compared it to standard proposal-based semi-supervision strategies. Our method achieves competitive results, leveraging unlabeled data in a more efficient manner and approaching full-supervision performance.In our fifth and last contribution, we propose a novel weakly supervised framework based on several global constraints derived from box annotations. Particularly, we leverage a classical tightness prior to a deep learning setting via imposing a set of constraints on the network outputs. Such a powerful topological prior prevents solutions from excessive shrinking by enforcing any horizontal or vertical line within the bounding box to contain, at least, one pixel of the target region. Furthermore, we integrate our deep tightness prior with a global background emptiness constraint, guiding training with information outside the bounding box. We demonstrate experimentally that such a global constraint is much more powerful than standard cross-entropy for the background class. The ensuing optimization problem is challenging as it takes the form of a large set of inequality constraints on the network outputs. We solve it with a sequence of unconstrained losses based on our log-barrier extensions. This accommodates standard stochastic gradient descent, while avoiding computationally expensive and unstable Lagrangian dual steps and projections. Extensive experiments over two different public data sets and applications (prostate and brain lesions) demonstrate that the synergy between our global tightness and emptiness priors yield competitive performances, approaching full-supervision performances.All the codes ensuing from this thesis are publicly available, and free to reuse and modify. The functional programming style used makes it easy to integrate new loss functions and constraints, with little-to-no additional coding efforts.