DeepReduce: A Sparse-tensor Communication Framework for Federated Deep Learning

1 Introduction . In federated learning [ 43 , 47 , 55 , 72 ] , the training is typically performed by a large number of resource-constrained client devices ( e.g. , smartphones ) , operating on their private data and computing a local model . Periodically , a subset of the devices is polled by a central server that retrieves their gradients , updates the global model and broadcasts it back to the clients . The most constrained resource in client devices is the network , either because the practically sustained bandwidth between remote clients and the server is low ( typically , in the order of 25-50Mbps ) , or , the financial cost ( e.g. , data plans for 4G/5G mobile connections ) is high . On the other hand , deep neural network model sizes have been steadily increasing at a much faster rate than the available bandwidth . Consequently , in federated learning , it is imperative to reduce the communicated data volume . One key observation that can help with reducing this volume is that the data exchanged during the Deep Neural Network ( DNN ) training often correspond to sparse tensors , i.e. , tensors with many zero-value elements . Sparse tensors may be : ( i ) direct artifacts of the training process ; for instance , the gradients of the NCF [ 35 ] and DeepLight [ 18 ] models consist of roughly 40 % and 99 % zero elements , respectively ; or ( ii ) explicitly generated by sparsification [ 6 , 51 , 69 , 71 , 76 , 81 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a commonly used lossy ( e.g. , Top-r or Random-r ) compression approach that selects only a few elements ( with r typically < 1 % ) . Figure 1.a depicts an example tensor containing 8 real values , 4 of which are zero ; its dense representation would require 256 bits . Typically , sparse tensors are represented as a set of hkey , valuei pairs ( see Figure 1.b ) , where key is the index ; notice , however , that the hkey , valuei representation also requires 256 bits , negating the benefit of sparsity . We show an example of our improved approach in Figure 1.c . We consider the indices as an ordered list represented by a Boolean array of 8 bits , such that the ith bit is ‘ 1 ’ if and only if the corresponding gradient element is non-zero . Moreover , we fit a function , V ( i ) = a · i+ b to the gradient values , with parameters , a = 0.6 and b = 4.0 . By transmitting only the bit string and parameters ( a , b ) , we can reconstruct the original tensor while requiring only 72 bits . The above example demonstrates a significant margin for additional compression for sparse tensors . Recent works ( e.g. , SKCompress [ 40 ] ) take advantage of these opportunities , but rely on a tightly coupled index and value compression algorithm that benefits only some scenarios ( see Section 6 ) . In practice , there exist complex trade-offs among data volume , model accuracy , and computational overhead , in conjunction with system aspects , such as the network bandwidth and the communication library ( e.g. , NCCL [ 2 ] , or Gloo [ 29 ] ) . Given that no single solution fits all scenarios , practitioners need the flexibility to adjust how sparse tensors are compressed and transmitted for each particular DNN model and system configuration . In this paper , we propose DeepReduce , a framework for transmitting sparse tensors via a wide-area network , tailored for large-scale federated DNN training . Our contributions include : ( i ) The DeepReduce framework , described in Section 3 , that decomposes the sparse tensor into two sets , indices and values , and allows for independent and combined compression . By decoupling indices from values , our framework enables synergistic combination of a variety of compressors in a way that benefits each particular DNN and training setup . DeepReduce resides between the machine learning framework ( e.g. , Tensorflow , PyTorch ) and the communication library . It exposes an easyto-use API that encapsulates a variety of existing methods , such as Run Length [ 83 ] and Huffman encoding [ 38 ] for index compression ; as well as Deflate [ 20 ] and QSGD [ 7 ] for value compression . DeepReduce also provides an index reordering abstraction , which is useful for combining value with index compressors . ( ii ) Two novel compressors for sparse tensors : a Bloom-filter based index compressor ( Section 4 ) that reduces the size of keys by 50 % , compared to the traditional hkey , valuei sparse representation ; and a curve-fitting based value compressor ( Section 5 ) that reduces the large values array to a small set of parameters . Both of our compressors do not affect the quality of the trained model . ( iii ) An evaluation of DeepReduce on a variety of DNN models and applications ( Section 6 ) . We demonstrate the practical applicability of our framework by realistic federated learning deployments on geographically remote , large-scale cloud infrastructures , and show that DeepReduce compresses already-sparse data by up to 320 % , without affecting the training quality . 2 Background . Notations . By [ d ] we denote the set of d natural numbers { 1 , 2 , · · · , d } . We denote the cardinality and complement of a set X , by |X| and Xc , respectively ; x [ i ] is the ith component of vector x . By xS 2 Rd we denote an |S|-sparse vector , where S ✓ [ d ] is its support . kxk and kxk1 are the ` 2 and ` 1 norm of a vector x , respectively . Class C1 consists of all differentiable functions with continuous derivative and Var [ a , b ] ( f ) denotes the variance of function f 2 C1 [ a , b ] over interval [ a , b ] . Federated Learning ( FL ) trains models collaboratively on data stored in resource-constrained , heterogeneous and geographically remote client devices ( e.g. , smartphones ) . At each round , a fraction of the participating clients are selected . After synchronizing with the server-side global model , the selected clients perform local training1 for some epochs over their private data . The server collects those updates , aggregates and applies them to the global model . Federated averaging ( FedAvg ) [ 55 ] , is a popular instantiation of this process . Other variants exist , such as SCAFFOLD [ 44 ] , FedNova [ 80 ] , parallel SGD [ 85 ] , local SGD [ 68 ] , and FedProx [ 49 ] . Owing to insufficient communication bandwidth in FL , lossy compression is used to reduce the transmitted data volume ; examples include FedBoost [ 31 ] , FedCOM [ 30 ] , and FedPAQ [ 60 ] . Compressor [ 69 ] is a random operator , C ( · ) : Rd ! Rd , that satisfies ECkx C ( x ) k2 ⌦kxk2 , where ⌦ > 0 is the compression factor and the expectation is taken over the randomness of C. If ⌦ = 1 and 2 ( 0 , 1 ] , C is a -compressor , denoted by C . Sparsification . A rank-1 tensor that has mostly zero components is said to be sparse . Many modern DNNs are inherently sparse [ 18 , 35 , 42 ] , and so are their gradients . Sparse gradients can also be generated by a compressor that sparsifies [ 51 , 71 , 76 , 81 ] the gradient g to generate g̃ 2 Rd via : g̃ [ i ] = C ( g [ i ] ) = g [ i ] , if i 2 S , otherwise g̃ [ i ] = 0 . We assume that S ⇢ [ d ] is the support set of g̃ such that g̃ is r-sparse if and only if |S| = r. Two commonly used sparsifiers , Random-r [ 69 ] and Top-r [ 6 , 8 ] , both are -compressors . We refer to further details in Appendix A. Bloom filter [ 12 ] is a probabilistic data structure that represents the elements of a set S. Initially , it is a bit string B of m bits , all set to 0 . To insert an element yj 2 S in the Bloom filter , we apply k independent hash functions hi on yj . Each hi yields a bit location in B , and changes that bit to 1 . To query if yj 2 S , we apply all k hash functions on it . If hi ( yj ) = 0 for any i 2 [ k ] , then the element surely does not belong to S ( i.e. , no false negatives ) . In contrast , if hi ( yj ) = 1 for all i 2 [ k ] , then yj may or may not belong to S ( i.e. , possible false positives ) . The false positive rate ( FPR ) [ 12 ] of B is : ✏ ⇡ ( 1 e k|S|/m ) k ; see Lemma 3 in Appendix A.2 . 3 System Architecture . DeepReduce ( see Figure 12 in the Appendix ) resides between the machine learning framework ( e.g. , TensorFlow , PyTorch ) and the communication library , and is optimized for federated DNN training . It offers a simple API whose functions can be overridden to implement , with minimal effort , a wide variety of index and value compression methods for sparse tensors . At the transmitting side , the input to DeepReduce is a sparse tensor directly from the ML framework , for the case of inherently sparse models ; or , is generated by an explicit sparsification process . In our implementation , we employ the GRACE [ 84 ] library for the sparsification operation , since it includes many popular sparsifiers ; other libraries can also be used . Sparse tensors are typically represented as hkey , valuei tuples . DeepReduce decouples the keys from the values and constructs two separate data structures . Let g̃ 2 Rd be the sparse gradient , where d is the number of model parameters and kg̃k0 = r , is the number of nonzero gradient elements . Let S be the set of r indices corresponding to those elements . DeepReduce implements two equivalent representations of S : ( i ) an array of r integers ; and ( ii ) a bit string B with d bits , where 8i 2 [ 1 , d ] , B [ i ] = 1 if and only if g̃ [ i ] 6= 0 . These two representations are useful for supporting a variety of index compressors ; e.g. , the bit string representation is used in [ 14 ] . The Index Compression module encapsulates the two representations and implements several algorithms for index compression . It supports both lossy compressors ( e.g. , our Bloom-filter based proposal ) , as well as lossless ones , such as the existing Run Length ( RLE ) [ 83 ] and Huffman [ 38 , 27 ] encoders ( see Appendix A.1 ) ; there is also an option to bypass index compression . The Value Compression module receives the sparse gradient values and compresses them independently . Several compressors , such as Deflate [ 20 ] , QSGD [ 7 ] , and our own curve-fitting based methods , are implemented . Again , there is an option to bypass value compression . Some compressors ( e.g. , our own proposals ) , require reordering of the gradient elements , which is handled by the Index reorder module . DeepReduce then combines , in one container , the compressed index and value 1In contrast to FL , conventional data-parallel , distributed training [ 87 ] synchronizes the updates from all workers at each iteration ( see Appendix A ) . structures , the reordering information and any required metadata . Then , the container is passed to the communication library . The receiving side mirrors the structure of the transmitter , but implements the inverse functions , that is , index and value decompression , and index reordering . The reconstructed sparse gradient is routed to GRACE for de-sparsification , or passed directly to the ML framework . It is worth mentioning that DeepReduce is general enough to represent popular existing methods that employ proprietary combined value and index compression . For example , SKCompress [ 40 ] can be implemented in DeepReduce as follows : SketchML [ 39 ] plus Huffman for values , no index reordering , and delta encoding plus Huffman for indices .

Summary: Problem: Communication overheads in the current FL setup, especially while transferring deep model gradients from clients to server and vice-versa.  Solution: The proposed DeepReduce framework offers to communicate sparse tensors in federated setup. Supports both the existing approaches for sparse communication as well as proposes two novel methods. Results: A significant reduction in the transfer of data over the existing techniques.