Beyond Gradient Descent for Regularized Segmentation Losses Dmitrii Marin University of Waterloo Canada Meng Tang University of Waterloo Canada

The simplicity of gradient descent (GD) made it the default method for training ever-deeper and complex neural networks. Both loss functions and architectures are often explicitly tuned to be amenable to this basic local optimization. In the context of weakly-supervised CNN segmentation, we demonstrate a well-motivated loss function where an alternative optimizer (ADM) achieves the state-of-the-art while GD performs poorly. Interestingly, GD obtains its best result for a “smoother” tuning of the loss function. The results are consistent across different network architectures. Our loss is motivated by well-understood MRF/CRF regularization models in “shallow” segmentation and their known global solvers. Our work suggests that network design/training should pay more attention to optimization methods. 1. Motivation and Background Weakly supervised training of neural networks is often based on regularized losses combining an empirical loss with some regularization term, which compensates for lack of supervision [38, 14]. Regularized losses are also useful for CNN segmentation [32, 34] where full supervision is often infeasible, particularly in biomedical applications. Such losses are motivated by regularization energies in shallow segmentation, where multi-decade research went into designing robust regularization models based on geometry [24, 7, 5], physics [18, 1], or robust statistics [13]. Such models should represent realistic shape priors compensating for image ambiguities, yet be amenable to efficient solvers. Many robust regularizers commonly used in vision [31, 17] are non-convex and require powerful optimizers to avoid many weak local minima. Basic local optimizers typically fail to produce practically useful results with such models. Effective weakly-supervised CNN methods for vision should incorporate priors compensating for image data ambiguities and lack of supervision just as in shallow vision methods. For example, recent work [38, 34] formulated https://github.com/dmitrii-marin/adm-seg In this paper, “shallow” refers to methods unrelated to deep learning. the problems of semi-supervised classification and weaklysupervised segmentation as minimization of regularized losses. This principled approach outperforms common ‘’proposal generation” methods [23, 20] computing “fake” ground truths to mimic standard fully-supervised training. However, we show that the use of regularization models as losses in deep learning is limited by GD, the backbone optimizer in current training methods. It is well-known that GD leads to poor local minima for many regularizers in shallow segmentation and many stronger algorithms were proposed [4, 6, 21, 31, 15]. Similarly, we show better optimization beyond GD for regularized losses in deep segmentation. One popular general approach applicable to regularized losses is ADMM [3] that splits optimization into two efficiently solvable sub-problems separately focusing on the empirical loss and regularizer. We advocate similar splitting to improve optimization of regularized losses in CNN training. In contrast, ADMM-like splitting of network parameters in different layers was used in [35] to improve parallelism. In our work weakly-supervised CNN segmentation is a context for discussing regularized loss optimization. As a regularizer, we use the common Potts model [6] and consider its nearestand large-neighborhood variants, a.k. a. sparse grid CRF and dense CRF models. We show effectiveness of ADMM-like splitting for grid CRF losses due to availability of powerful sub-problem solvers, e.g. graph cuts [5]. As detailed in [34, Sec.3], an earlier iterative proposal-generation technique by [20] can be related to regularized loss splitting, but their method is limited to dense CRF and its approximate mean-field solver [22]. In fact, given such weak sub-problem solvers, splitting is inferior to basic GD over the regularized loss [34]. More insights on grid and dense CRF are below. 1.1. Pairwise CRF for Shallow Segmentation Robust pairwise Potts model and its binary version (Ising model) are used in many application such as stereo, reconstruction, and segmentation. One can define this model as a cost functional over integer-valued labeling S := (Sp ∈ Z | p ∈ Ω) of image pixels p ∈ Ω as follows