SoCal: Selective Oracle Questioning for Consistency-based Active Learning of Cardiac Signals

1 INTRODUCTION . The success of modern-day deep learning algorithms in the medical domain has been contingent upon the availability of large , labelled datasets ( Poplin et al. , 2018 ; Tomašev et al. , 2019 ; Attia et al. , 2019 ) . Curating such datasets , however , is a challenge due to the time-consuming nature of , and high costs associated with , labelling . This is particularly the case in the medical domain where the input of expert medical professionals is required . One way of overcoming this challenge and exploiting large , unlabelled datasets is via the active learning ( AL ) framework ( Settles , 2009 ) . This framework iterates over three main steps : 1 ) a learner is tasked with acquiring unlabelled instances , usually through an acquisition function , 2 ) an oracle ( e.g . physician ) is tasked with labelling such acquired instances , and 3 ) the learner is trained on the existing and newly-labelled instances . By altering the way in which acquisitions are performed and the degree of involvement of the oracle , the active learning framework aims to improve the performance of a network while minimizing the burden of labelling on the oracle . One principal desideratum for an acquisition function is its ability to reduce the size of the version space , the set of hypotheses ( decision boundaries ) consistent with the labelled training instances . This ability is highly dependent upon the approximation of the version space , a goal that Monte Carlo Dropout ( MCD ) attempts to achieve ( see Fig . 1a ) . For example , state-of-the-art uncertainty-based acquisition functions , such as BALD ( Houlsby et al. , 2011 ) , used alongside MCD acquire instances that lie in a region of uncertainty , a region where there is high disagreement between the hypotheses about a particular instance . In many scenarios , however , estimating this region of uncertainty is nontrivial . Furthermore , existing AL frameworks are overly reliant on the presence of an oracle . Such over-reliance precludes the applicability of AL algorithms to certain environments , such as low-resource healthcare settings , where an oracle is either unavailable or ill-trained for the task at hand . Contributions . In this work , we aim to design an active learning framework that better estimates the region of uncertainty and decreases its reliance on an oracle . Our contributions are as follows : 1 . Consistency-based active learning framework : we propose a novel framework that stochastically perturbs inputs , network parameters , or both to guide the acquisition of unlabelled instances . 2 . Selective oracle questioning : we propose a dynamic strategy which learns , for an acquired unlabelled instance , whether to request a label from an oracle or to generate a pseudo-label instead . 2 RELATED WORK . Active learning methdologies were recently reviewed by Settles ( 2009 ) . In the healthcare domain , Gong et al . ( 2019 ) propose to acquire instances from an electronic health record ( EHR ) database using a Bayesian deep latent Gaussian model to improve mortality prediction . Smailagic et al . ( 2018 ; 2019 ) acquire unannotated medical images by measuring their distance in a latent space to images in the training set . The work of Wang et al . ( 2019 ) is similar to ours in that they focus on the electrocardiogram ( ECG ) . Gal et al . ( 2017 ) adopt BALD ( Houlsby et al. , 2011 ) in the context of Monte Carlo Dropout to acquire datapoints that maximize the Jensen-Shannon divergence ( JSD ) across MC samples . Previous work attempts to learn from multiple or imperfect oracles ( Dekel et al. , 2012 ; Zhang & Chaudhuri , 2015 ; Sinha et al. , 2019 ) . For example , Urner et al . ( 2012 ) propose choosing the oracle that should label a particular instance . Unlike our approach , they do not explore independence from an oracle . Yan et al . ( 2016 ) consider oracle abstention in an AL setting . Instead , we place the decision of abstention under the control of the learner . To the best of our knowledge , previous work , in contrast to ours , has assumed the existence of an oracle and has not explored a dynamic oracle selection strategy . Consistency training in the context of semi-supervised learning helps enforce the smoothness assumption ( Zhu , 2005 ) . For example , Interpolation Consistency Training ( Verma et al. , 2019 ) penalizes networks for not generating a linear combination of outputs in response to a linear combination of inputs . Similarly , Xie et al . ( 2019 ) penalizes networks for generating drastically different outputs in response to perturbed instances . In the process , networks learn perturbation-invariant representations . McCallumzy & Nigamy ( 1998 ) introduce an acquisition function that calculates the average Kullback-Leibler divergence , DKL , between the output of a network and the consensus output across all networks in an ensemble . Unlike ours , their approach does not exploit perturbations . Similar to our work is that of Gao et al . ( 2019 ) which incorporates into the objective function a consistency-loss based on the DKL and actively acquires instances using the variance of the probability assigned to each class by the network in response to perturbed versions of the same instance . Selective classification imbues a network with the ability to abstain from making predictions . Chow ( 1970 ) ; El-Yaniv & Wiener ( 2010 ) introduce the risk-coverage trade-off whereby the empirical risk of a model is inversely related to its rate of abstentions . Wiener & El-Yaniv ( 2011 ) use a support vector machine ( SVM ) to rank and reject instances based on the degree of disagreement between hypotheses . In some frameworks , these are the same instances that active learning views as most informative . Cortes et al . ( 2016 ) outline an objective function that penalizes abstentions that are inappropriate and frequent . Most recently , Liu et al . ( 2019 ) propose the gambler ’ s loss to learn a selection function that determines whether instances are rejected . However , this approach is not implemented in the context of AL . Most similar to our work is SelectiveNet ( Geifman & El-Yaniv , 2019 ) where a multi-head architecture is used alongside an empirical selective risk objective function and a percentile threshold . However , their work assumes the presence of ground-truth labels and , therefore , does not extend to unlabelled instances . 3 BACKGROUND . 3.1 ACTIVE LEARNING . Consider a learner fω : x ∈ Rm → v ∈ Rd , parameterized by ω , that maps an m-dimensional input , x , to a d-dimensional representation , v. Further consider gφ : v ∈ Rd → y ∈ RC that maps a ddimensional representation , v , to a C-dimensional output , y , where C is the number of classes . After training on a pool of labelled data L = ( XL , YL ) for τ epochs , the learner is tasked with querying the unlabelled pool of data U = ( XU , YU ) and acquiring the top b fraction of instances , xb ∼ XU , that it deems to be most informative . The degree of informativeness of an instance is determined by an acquisition function , α , such as BALD ( Houlsby et al. , 2011 ) . Additional acquisition functions can be found in Appendix A . These are typically used in conjunction with Monte Carlo Dropout ( Gal & Ghahramani , 2016 ) to identify instances that lie in the region of uncertainty , a region in which hypotheses disagree the most about instances . 4 METHODS . 4.1 CONSISTENCY-BASED ACTIVE LEARNING . 4.1.1 MONTE CARLO PERTURBATIONS . Unlabelled instances in proximity to the decision boundary are likely to be more informative for training than those further away . To identify such instances , we stochastically perturb them and observe the network ’ s outputs . The intuition is that such outputs will differ significantly across the perturbations for instances close to the decision boundary ( see Fig . 1b ) . We refer to this setup as Monte Carlo Perturbations ( MCP ) and illustrate its derivation in Appendix B . 4.1.2 BAYESIAN ACTIVE LEARNING BY CONSISTENCY . Acquisition functions dependent upon perturbations applied to either the inputs ( MCP ) or the network parameters ( MCD ) alone can fail to identify instances that lie in the region of uncertainty . We illustrate this point with the following example : without loss of generality , let us assume an unlabelled instance is in proximity to some decision boundary A and is classified by the network as belonging to some arbitrary class 3 . Such proximity should deem the instance informative for the training process ( Settles , 2009 ) . In the MCD setting , perturbations are applied to parameters , generating various decision boundaries , which in turn influence the network outputs . In Fig . 2 ( red rectangle ) , we visualize such outputs for three arbitrary classes . If these parameter perturbations happen to be too small in magnitude , for example , then the network will continue to classify the instance as belonging to the same class . At this stage , regardless of whether an uncertainty-based or a consistency-based acquisition function is used , the instance would be deemed uninformative , and thus not acquired . As a result , an instance that should have been acquired ( due to its proximity to the decision boundary ) was erroneously deemed uninformative . A similar argument can be extended to MCP . By applying perturbations to both instances and network parameters , we aim to leverage the smoothness assumption ( Zhu , 2005 ) to better identify instances that lie in the region of uncertainty and thus avoid missing their acquisition . Motivated by this , we propose a framework , entitled Bayesian Active Learning by Consistency ( BALC ) ( see Fig . 1c ) , that consists of three main steps : 1 ) we perturb an instance , x , to generate z , 2 ) we perturb the network parameters , ω , to generate ω′ , and 3 ) we pass both instances through the perturbed network , generating outputs , p ( y|x , ω′ ) and p ( y|z , ω′ ) ∈ RC , respectively . We perform these steps for T stochastic perturbations and generate two matrices of network outputs , G ( x ) , G′ ( z ) ∈ RT×C . We visualize such network outputs in Fig . 2 where T = 3 and C = 3 . To leverage G and G′ , we propose two divergence-based acquisition functions that acquire instances that the network is least robust to . In BALCKLD , we calculate the DKL between two C-dimensional Gaussians that are empirically fit to G and G′ . BALCKLD = DKL ( N ( µ ( x ) , Σ ( x ) ‖ N ( µ ( z ) , Σ ( z ) ) ) ( 1 ) where µ = 1T ∑T i G and Σ = ( G− µ ) T ( G− µ ) represent the empirical mean vector and covariance matrix of the network outputs , respectively . BALCKLD is likely to detect output variations due to input perturbations . We support this claim in Fig . 2 by illustrating two scenarios . In scenario 1 , network output variations are caused solely by input perturbations . In contrast , in scenario 2 , network output variations are caused by both input and parameter perturbations . We show that BALCKLD ≈ 1 and 0 in these two scenarios , respectively . Since the higher the value of an acquisition function , the more informative an instance is , these scenarios illustrate BALCKLD ’ s preference for input perturbations . To detect variations due to both input and parameter perturbations , we introduce BALCJSD whose full derivation can be found in Appendix C. BALCJSD = across parameter perturbations︷ ︸︸ ︷ Ei∈T [ DKL ( Gi ( x ) ‖ G′i ( z ) ) ] − across input perturbations︷ ︸︸ ︷ DKL ( Ei∈T [ G ( x ) ] ‖ Ei∈T [ G′ ( z ) ] ) ( 2 )

Summary: The paper proposes an active learning framework called SoCal that is consistency-based and can decide between whether to make use of the oracle to provide a label or to make use of a pseudo-label generated by the algorithm itself instead. The proposed method hopes to address resource-constrained active learning scenarios where the oracle is not always available or we wish to make use of the oracle as infrequently as possible. Experimental results demonstrate reasonable performance on four publically available physiological datasets.  Experimental results when the oracle is noisy is also reported.