DeepReduce: A Sparse-tensor Communication Framework for Federated Deep Learning

1 Introduction . In federated learning [ 43 , 47 , 55 , 72 ] , the training is typically performed by a large number of resource-constrained client devices ( e.g. , smartphones ) , operating on their private data and computing a local model . Periodically , a subset of the devices is polled by a central server that retrieves their gradients , updates the global model and broadcasts it back to the clients . The most constrained resource in client devices is the network , either because the practically sustained bandwidth between remote clients and the server is low ( typically , in the order of 25-50Mbps ) , or , the financial cost ( e.g. , data plans for 4G/5G mobile connections ) is high . On the other hand , deep neural network model sizes have been steadily increasing at a much faster rate than the available bandwidth . Consequently , in federated learning , it is imperative to reduce the communicated data volume . One key observation that can help with reducing this volume is that the data exchanged during the Deep Neural Network ( DNN ) training often correspond to sparse tensors , i.e. , tensors with many zero-value elements . Sparse tensors may be : ( i ) direct artifacts of the training process ; for instance , the gradients of the NCF [ 35 ] and DeepLight [ 18 ] models consist of roughly 40 % and 99 % zero elements , respectively ; or ( ii ) explicitly generated by sparsification [ 6 , 51 , 69 , 71 , 76 , 81 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a commonly used lossy ( e.g. , Top-r or Random-r ) compression approach that selects only a few elements ( with r typically < 1 % ) . Figure 1.a depicts an example tensor containing 8 real values , 4 of which are zero ; its dense representation would require 256 bits . Typically , sparse tensors are represented as a set of hkey , valuei pairs ( see Figure 1.b ) , where key is the index ; notice , however , that the hkey , valuei representation also requires 256 bits , negating the benefit of sparsity . We show an example of our improved approach in Figure 1.c . We consider the indices as an ordered list represented by a Boolean array of 8 bits , such that the ith bit is ‘ 1 ’ if and only if the corresponding gradient element is non-zero . Moreover , we fit a function , V ( i ) = a · i+ b to the gradient values , with parameters , a = 0.6 and b = 4.0 . By transmitting only the bit string and parameters ( a , b ) , we can reconstruct the original tensor while requiring only 72 bits . The above example demonstrates a significant margin for additional compression for sparse tensors . Recent works ( e.g. , SKCompress [ 40 ] ) take advantage of these opportunities , but rely on a tightly coupled index and value compression algorithm that benefits only some scenarios ( see Section 6 ) . In practice , there exist complex trade-offs among data volume , model accuracy , and computational overhead , in conjunction with system aspects , such as the network bandwidth and the communication library ( e.g. , NCCL [ 2 ] , or Gloo [ 29 ] ) . Given that no single solution fits all scenarios , practitioners need the flexibility to adjust how sparse tensors are compressed and transmitted for each particular DNN model and system configuration . In this paper , we propose DeepReduce , a framework for transmitting sparse tensors via a wide-area network , tailored for large-scale federated DNN training . Our contributions include : ( i ) The DeepReduce framework , described in Section 3 , that decomposes the sparse tensor into two sets , indices and values , and allows for independent and combined compression . By decoupling indices from values , our framework enables synergistic combination of a variety of compressors in a way that benefits each particular DNN and training setup . DeepReduce resides between the machine learning framework ( e.g. , Tensorflow , PyTorch ) and the communication library . It exposes an easyto-use API that encapsulates a variety of existing methods , such as Run Length [ 83 ] and Huffman encoding [ 38 ] for index compression ; as well as Deflate [ 20 ] and QSGD [ 7 ] for value compression . DeepReduce also provides an index reordering abstraction , which is useful for combining value with index compressors . ( ii ) Two novel compressors for sparse tensors : a Bloom-filter based index compressor ( Section 4 ) that reduces the size of keys by 50 % , compared to the traditional hkey , valuei sparse representation ; and a curve-fitting based value compressor ( Section 5 ) that reduces the large values array to a small set of parameters . Both of our compressors do not affect the quality of the trained model . ( iii ) An evaluation of DeepReduce on a variety of DNN models and applications ( Section 6 ) . We demonstrate the practical applicability of our framework by realistic federated learning deployments on geographically remote , large-scale cloud infrastructures , and show that DeepReduce compresses already-sparse data by up to 320 % , without affecting the training quality . 2 Background . Notations . By [ d ] we denote the set of d natural numbers { 1 , 2 , · · · , d } . We denote the cardinality and complement of a set X , by |X| and Xc , respectively ; x [ i ] is the ith component of vector x . By xS 2 Rd we denote an |S|-sparse vector , where S ✓ [ d ] is its support . kxk and kxk1 are the ` 2 and ` 1 norm of a vector x , respectively . Class C1 consists of all differentiable functions with continuous derivative and Var [ a , b ] ( f ) denotes the variance of function f 2 C1 [ a , b ] over interval [ a , b ] . Federated Learning ( FL ) trains models collaboratively on data stored in resource-constrained , heterogeneous and geographically remote client devices ( e.g. , smartphones ) . At each round , a fraction of the participating clients are selected . After synchronizing with the server-side global model , the selected clients perform local training1 for some epochs over their private data . The server collects those updates , aggregates and applies them to the global model . Federated averaging ( FedAvg ) [ 55 ] , is a popular instantiation of this process . Other variants exist , such as SCAFFOLD [ 44 ] , FedNova [ 80 ] , parallel SGD [ 85 ] , local SGD [ 68 ] , and FedProx [ 49 ] . Owing to insufficient communication bandwidth in FL , lossy compression is used to reduce the transmitted data volume ; examples include FedBoost [ 31 ] , FedCOM [ 30 ] , and FedPAQ [ 60 ] . Compressor [ 69 ] is a random operator , C ( · ) : Rd ! Rd , that satisfies ECkx C ( x ) k2 ⌦kxk2 , where ⌦ > 0 is the compression factor and the expectation is taken over the randomness of C. If ⌦ = 1 and 2 ( 0 , 1 ] , C is a -compressor , denoted by C . Sparsification . A rank-1 tensor that has mostly zero components is said to be sparse . Many modern DNNs are inherently sparse [ 18 , 35 , 42 ] , and so are their gradients . Sparse gradients can also be generated by a compressor that sparsifies [ 51 , 71 , 76 , 81 ] the gradient g to generate g̃ 2 Rd via : g̃ [ i ] = C ( g [ i ] ) = g [ i ] , if i 2 S , otherwise g̃ [ i ] = 0 . We assume that S ⇢ [ d ] is the support set of g̃ such that g̃ is r-sparse if and only if |S| = r. Two commonly used sparsifiers , Random-r [ 69 ] and Top-r [ 6 , 8 ] , both are -compressors . We refer to further details in Appendix A. Bloom filter [ 12 ] is a probabilistic data structure that represents the elements of a set S. Initially , it is a bit string B of m bits , all set to 0 . To insert an element yj 2 S in the Bloom filter , we apply k independent hash functions hi on yj . Each hi yields a bit location in B , and changes that bit to 1 . To query if yj 2 S , we apply all k hash functions on it . If hi ( yj ) = 0 for any i 2 [ k ] , then the element surely does not belong to S ( i.e. , no false negatives ) . In contrast , if hi ( yj ) = 1 for all i 2 [ k ] , then yj may or may not belong to S ( i.e. , possible false positives ) . The false positive rate ( FPR ) [ 12 ] of B is : ✏ ⇡ ( 1 e k|S|/m ) k ; see Lemma 3 in Appendix A.2 . 3 System Architecture . DeepReduce ( see Figure 12 in the Appendix ) resides between the machine learning framework ( e.g. , TensorFlow , PyTorch ) and the communication library , and is optimized for federated DNN training . It offers a simple API whose functions can be overridden to implement , with minimal effort , a wide variety of index and value compression methods for sparse tensors . At the transmitting side , the input to DeepReduce is a sparse tensor directly from the ML framework , for the case of inherently sparse models ; or , is generated by an explicit sparsification process . In our implementation , we employ the GRACE [ 84 ] library for the sparsification operation , since it includes many popular sparsifiers ; other libraries can also be used . Sparse tensors are typically represented as hkey , valuei tuples . DeepReduce decouples the keys from the values and constructs two separate data structures . Let g̃ 2 Rd be the sparse gradient , where d is the number of model parameters and kg̃k0 = r , is the number of nonzero gradient elements . Let S be the set of r indices corresponding to those elements . DeepReduce implements two equivalent representations of S : ( i ) an array of r integers ; and ( ii ) a bit string B with d bits , where 8i 2 [ 1 , d ] , B [ i ] = 1 if and only if g̃ [ i ] 6= 0 . These two representations are useful for supporting a variety of index compressors ; e.g. , the bit string representation is used in [ 14 ] . The Index Compression module encapsulates the two representations and implements several algorithms for index compression . It supports both lossy compressors ( e.g. , our Bloom-filter based proposal ) , as well as lossless ones , such as the existing Run Length ( RLE ) [ 83 ] and Huffman [ 38 , 27 ] encoders ( see Appendix A.1 ) ; there is also an option to bypass index compression . The Value Compression module receives the sparse gradient values and compresses them independently . Several compressors , such as Deflate [ 20 ] , QSGD [ 7 ] , and our own curve-fitting based methods , are implemented . Again , there is an option to bypass value compression . Some compressors ( e.g. , our own proposals ) , require reordering of the gradient elements , which is handled by the Index reorder module . DeepReduce then combines , in one container , the compressed index and value 1In contrast to FL , conventional data-parallel , distributed training [ 87 ] synchronizes the updates from all workers at each iteration ( see Appendix A ) . structures , the reordering information and any required metadata . Then , the container is passed to the communication library . The receiving side mirrors the structure of the transmitter , but implements the inverse functions , that is , index and value decompression , and index reordering . The reconstructed sparse gradient is routed to GRACE for de-sparsification , or passed directly to the ML framework . It is worth mentioning that DeepReduce is general enough to represent popular existing methods that employ proprietary combined value and index compression . For example , SKCompress [ 40 ] can be implemented in DeepReduce as follows : SketchML [ 39 ] plus Huffman for values , no index reordering , and delta encoding plus Huffman for indices .

Summary: This paper proposed a framework to compress sparse tensors in the context of federated learning, in order to further save the amount of transmitted data between the server and clients. The proposed method could be incorporated into the existing pipeline of distributed training with sparsification. The main idea is to compress/decompress the index and value separately with different compression methods. The paper comes up with two compression methods, i.e., Bloom-filter based index compressor and curve-fitting based value compressor. The paper shows some synthesized and realistic experiments to justify the proposed methods.