Neural Bootstrapper

1 Introduction . Bootstrapping [ 7 ] or bagging [ 3 ] procedures have been commonly used as a primary tool in quantifying uncertainty lying on statistical inference , e.g . evaluations of standard errors , confidence intervals , and hypothetical null distribution . Despite its success in statistics and machine learning field , the naive use of bootstrap procedures in deep neural network applications has been less practical due to its computational intensity . Bootstrap procedures require evaluating a number of models ; however , training multiple deep neural networks are infeasible in practice in terms of computational cost . To utilize bootstrap for deep neural networks , we propose a novel bootstrapping procedure called Neural Bootstrapper ( NeuBoots ) . The proposed method is mainly motivated by Generative Bootstrap Sampler ( GBS ) [ 38 ] , which trains a bootstrap generator by model parameterization based on Random Weight Bootstrapping ( RWB , [ 37 ] ) framework . For many statistical models , the idea of GBS is more theoretically valid than amortized bootstrap [ 31 ] , which trains an implicit model to approximate the bootstrap distribution over model parameters . However , GBS is hardly scalable to modern deep neural networks containing millions of parameters . Contrary to the previous method , the proposed method is effortlessly scalable and universally applicable to the various architectures . The key idea of NeuBoots is simple ; multiplying bootstrap weights to the final layer of the backbone network and instead of model parameterization . Hence it outputs the bootstrapped predictions of the target without additional parameters and the repetitive ⇤Equal Contribution †Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . computations from scratch . NeuBoots outperforms the previous sampling-based methods [ 13 , 24 , 31 ] on the various uncertainty quantification related tasks with deep convolutional networks [ 17 , 20 , 22 ] . Throughout this paper , we show that NeuBoots has multiple advantages over the existing uncertainty quantification procedures in terms of memory efficiency and computational speed ( see Table 1.1 ) . To verify the empirical power of the proposed method , we apply NeuBoots to a wide range of experiments related to uncertainty quantification and bagging . We apply the NeuBoots to prediction calibration , active learning , out-of-distribution ( OOD ) detection , semantic segmentation , and learning on imbalanced datasets . Notably , we test the proposed method on biomedical data of high-resolution , NIH3T3 data [ 5 ] . In Section 4 , our results show that NeuBoots achieves at least comparable or better performance than the state-of-the-art methods in the considered applications . 2 Preliminaries . As preliminaries , we briefly review the standard bootstrapping [ 7 ] and introduce an idea of Generative Bootstrap Sampler ( GBS , [ 38 ] ) , which is the primary motivation of the proposed method . Let [ m ] : = { 1 , . . . , m } and denote a given training data by D = { ( Xi , yi ) : i 2 [ n ] } , where each feature Xi 2 X ⇢ Rp and its response yi 2 Rd . We denote the class of models f : Rp ! Rd by M. For the standard bootstrapping , we sample B sets of bootstrap data D ( b ) = { ( X ( b ) i , y ( b ) i ) : i 2 [ n ] } with replacement for b 2 [ B ] . For each bootstrap data D ( b ) , we define a loss functional L on f 2M : L ( f , D ( b ) ) : = 1 n nX i=1 ` ( f ( X ( b ) i ) , y ( b ) i ) ( 2.1 ) where ` : Rd⇥Rd ! R is an arbitrary loss function . Then we minimize ( 2.1 ) with respect to f 2M to obtain bootstrapped models : for b 2 [ B ] , bf ( b ) = arg min f2M L ( f , D ( b ) ) . ( 2.2 ) Random Weight Bootstrapping It is well-known that the standard bootstrap uses only ( approximately ) 63 % of observations for each bootstrap evaluation [ 24 ] . To resolve this problem , we use Random Weight Bootstrapping ( RWB , [ 37 ] ) , which reformulates ( 2.2 ) as a sampling of bootstrapping weights for a weighted loss functional . Let W = { w 2 Rn+ : Pn i=1 wi = n } be a dilated standard ( n 1 ) -simplex . For w = ( w1 , . . . , wn ) 2 W and the original training data D , we define the Weighted Bootstrapping Loss ( WBL ) functional on f 2M as follows : L ( f , w , D ) : = 1 n nX i=1 wi ` ( f ( Xi ) , yi ) . ( 2.3 ) Then for any resampled dataset D ( b ) , there exists a unique w 2W such that ( 2.1 ) matches to ( 2.3 ) . This reformulation provides a relaxation method to consider full data set without any omission in bootstrapping . Precisely , as a continuous relaxation of the standard bootstrap , we use Dirichlet distribution [ 32 ] ; PW = n ⇥ Dirichlet ( 1 , . . . , 1 ) , where PW is a probability distribution on the simplex W . Hence RWB fully utilizes the observed data points , since sampled bootstrap weights w ⇠ PW are strictly positive . Also , [ 34 ] showed that RWB achieves the same theoretical properties with these of the standard bootstrap i.e . PW = Multinomial ( n ; 1/n , . . . , 1/n ) in ( 2.3 ) . Bootstrap Distribution Generator Although RWB resolves the data discard problem , training multiple networks bf ( 1 ) , . . . , bf ( B ) remains a computational problem , and one has to store the parameters of every network for prediction . To reduce the computational bottlenecks , GBS [ 38 ] proposes a procedure to train a generator function of bootstrapped estimators for parametric statistical models . The main idea of GBS is to parameterize the model parameter with bootstrap weight w 2 W . When the GBS is applied to bootstrapping neural networks , it considers a bootstrap generator g : Rp ⇥W ! Rd with parameter ✓ ( w ) , where d is the total number of neural net parameters in g , so that g ( X , w ) = g✓ ( w ) ( X ) . Based on ( 2.3 ) , we define a new WBL functional : L ( g , D ) = Ew⇠PW [ L ( g , w , D ) ] , L ( g , w , D ) = 1 n nX i=1 wi ` ( g ( Xi , w ) , yi ) ( 2.4 ) Note that we use the Dirichlet distribution for PW ; hence the functional L ( g , D ) includes RWB procedure itself . Analogous to ( 2.2 ) , we obtain the bootstrap generator bg by optimizing L ( g , D ) : bg = arg min g2M L ( g , D ) ( 2.5 ) Then learned bg can generate bootstrap samples for given target data X⇤ by plugging an arbitrary w 2W into bg ( X⇤ , · ) . We refer to [ 38 , Section 2 ] for detailed theoretical results on GBS . Block Bootstrapping The above bootstrap generator g receives a bootstrap weight vector w of dimension n ; hence its optimization via ( 2.5 ) would be hurdled when the number of data n is large . Hence we utilize a block bootstrapping procedure to reduce the dimension of bootstrap weight vector . We allocate the index set [ n ] to S number of blocks . Let u : [ n ] ! [ S ] denotes the assignment function . Then we impose the same value of weight on all elements in a block such as , wi = ↵s for u ( i ) = s 2 [ S ] , where ↵ = ( ↵1 , . . . , ↵S ) ⇠ S ⇥ Dirichlet ( 1 , . . . , 1 ) . Instead of w , we plug ↵ in g ( X , · ) = g✓ ( · ) ( X ) to generate bootstrap samples and compute the weighted loss function in ( 2.4 ) : L ( g , D ) = E↵⇠S⇥Dirichlet ( 1 , ... ,1 ) '' 1 n nX i=1 ↵u ( i ) ` ( g ( Xi , ↵ ) , yi ) # ( 2.6 ) The above procedure asymptotically converges to the same target distribution where the conventional non-block bootstrap converges . See appendix A for more detailed procedure and proofs . 3 Neural Bootstrapper . Now we propose Neural Bootstrapper ( NeuBoots ) , which reduces computational complexity and memory requirement of the networks in the learning of bootstrapped distribution to being suitable for deep neural networks . How to implement the bootstrap generator g for deep neural networks ? One may consider directly applying GBS to existing deep neural networks by modeling a neural net ✓ ( · ) that outputs the neural net parameters of g. However , this approach is computationally challenging due to the high-dimensionality of the output dimension of ✓ ( · ) Indeed , [ 38 ] proposes an architecture which concatenates bootstrap weight vector to every layer of a given neural network ( Figure 3.1 ( b ) ) and trains it with ( 2.6 ) . However , the bagging performance of GBS gradually degrades as we applied it to the deeper neural networks . This may be because the information of bootstrap weights in the earlier layers less propagate since the target model reduces the parameters of the weights during the training . 3.1 Adaptive Block Bootstrapping . We found that the bootstrap weight in the final layer mainly affects the bootstrap performance of GBS . This fact motivates us to utilize the following adaptive block bootstrapping , which is the key idea of NeuBoots . Take a neural network f✓ 2 M with parameter ✓ . Let M✓1 and F✓2 be the single-layer neural network in the final layer and the feature extractor of f , respectively , with parameter ✓ = ( ✓1 , ✓2 ) , so we can decompose f✓ into M✓1 F✓2 . Set S : = dim ( F✓2 ( X ) ) for the number of blocks for block bootstrapping . Then , we redefine bootstrap generator as follows : g✓ ( X , ↵ ) : = g ( ✓ , ↵ ) ( X ) = M✓1 ( F✓2 ( X ) ↵ ) ( 3.1 ) where denotes an elementwise multiplication . Bootstrap generator ( 3.1 ) can also be trained with ( 2.6 ) ; hence optimized bg✓ ( X , · ) can generate the bootstrapped prediction as we plug ↵ . This modification brings a computational benefit , since we can generate bootstrap samples quickly and memory-efficiently by reusing a priori computed tensor F✓2 ( X ) without repetitive computation from scratch . See Figure 3.1 for the comparison between the previous methods and NeuBoots . In our empirical experience , the bootstrap evaluations over different groupings were consistent for all examined examples in this article . Training and Prediction At every epoch , we update the w↵ = { ↵u ( 1 ) , . . . , ↵u ( n ) } randomly , and the expectation in ( 2.6 ) can be approximated by the average over the sampled weights . Considering the stochastic gradient descent ( SGD ) algorithms to update the parameter ✓ via mini-batch sequence { Dk : Dk ⇢ D } Kk=1 , we plug mini-batch size of bootstrap weight vector { ↵u ( i ) : Xi 2 Dk } in ( 2.6 ) without changing ↵ . Each element of w↵ is not used repeatedly during the epoch , so the sampling and replacement procedures in Algorithm 1 are conducted once at the beginning of epoch . After we obtain the optimized network bg✓ , for the prediction , we use the generator bg⇤ ( · ) = bg✓ ( X⇤ , · ) for a given data point X⇤ . Then we can generate bootstrapped predictions by plugging ↵ ( 1 ) , . . . , ↵ ( B ) in the generator bg⇤ ( · ) , as described in Algorithm 2 . Algorithm 1 : Training step in NeuBoots . Input : Dataset D ; epochs T ; dimension of feature S ; index function u ; learning rate ⇢ . 1 Initialize neural network parameter ( 0 ) and set n : = |D| . 2 for t 2 { 0 , . . . , T 1 } do 3 Sample ↵ ( t ) = { ↵ ( t ) 1 , . . . , ↵ ( t ) S } i.i.d . ⇠ S ⇥ Dirichlet ( 1 , . . . , 1 ) 4 Replace w ( t ) ↵ = { ↵ ( t ) u ( 1 ) , . . . , ↵ ( t ) u ( n ) } 5 Update ✓ ( t+1 ) ✓ ( t ) ⇢r✓L ( g ( ✓ , ↵ ) , w ( t ) ↵ , D ) ✓=✓ ( t ) . Algorithm 2 : Prediction step in NeuBoots . Input : Data point X⇤ 2 Rp ; number of bootstrap sampling B . 1 Compute the feed-forward network bg⇤ ( · ) = bg✓ ( X⇤ , · ) a priori . 2 for b 2 { 1 , . . . , B } do 3 Generate ↵ ( b ) i.i.d.⇠ S ⇥ Dirichlet ( 1 , . . . , 1 ) and evaluate by ( b ) ⇤ = bg⇤ ( ↵ ( b ) ) .

Summary: The work proposes NeuBoots: a new ensembling method for neural networks inspired by bootstrapping estimation. The method is simple and involves training one neural network to simultaneously represent a whole family of models; each such model is scored during training with a loss where different training objects have different weights. These models are obtained from the base model with a single multiplication of weights of training objects (or, more precisely, weights of sets of training objects) and representations in the final part of the network. During testing, models are ensembled for multiple random weightings for training objects. The authors validate the method on computer vision tasks and synthetic data. It is concluded that the method is superior to other bagging methods on various metrics related to uncertainty estimation such as active learning performance or out-of-distribution detection metrics and is more computationally efficient.