# QoE Modeling on Split Features with Distributed Deep Learning
## 5 Distributed Learning Approaches on Split Feature Scenarios
### Vertical Federated Learning (vFL)  
In order to introduce greater flexibility beyond \(DEL\), we present _vFL_ as a rather more appealing approach in this Section. In a _vFL_ setting, the worker nodes train on different attributes that belong to the same samples across all nodes in parallel. The neural network is divided into multiple partitions, and each partition is handled at a different worker computation node. As in any Stochastic Gradient Descent (SGD) based machine learning algorithm, it consists of a forward propagation and a backwards propagation phase. Each collaborating worker does a forward pass, i.e., multiplying the input parameters with a linear transformation followed by a nonlinear function Sigmoid until the last layer of the local neural network, i.e., cut-layer. The other workers can do similar transformation on the features until the cut-layer. Next the inferred values from all workers' last layers are sent over via a communication channel to the master node, where they are concatenated. The master node then performs a forward-pass until its last layer where the error is computed on the ground-truth labels that belong to the corresponding samples. Based on the computed error, the master node performs a backwards-propagation and updates the weights of its neurons up until its cut-layer, and then splits the gradients of its last layer among the workers and transmits them to the corresponding workers. The workers then perform backwards-propagation continued from the received gradients from their cut-layers until their first layer, and updates all the weights of their neurons at each layer. This completes one round. This modular structure (e.g., cascaded layers) of neural networks allows to design its architecture based on use case requirements in a flexible way. In _vFL_, there were 32 neurons at the first layers, and varying number of neurons (ranging from 2 to 32) at the cut-layers (as refer to the layer interfacing to the master node). The neuron count at the cut-layer of the master node was the sum of the neurons of the cut-layers of the worker nodes. The master node had one hidden layer with 32 neurons, and a final layer with 2 neurons. The non-linear activation functions were identical, i.e., ReLu on all hidden layers, and Sigmoid on the last layer. Signaling flows between the participating nodes via open-source message broker, RabbitMQ [39]. In this prototype _vFL_ solution is implemented in a Kubernetes cluster where the computation nodes (workers and master) are spawned as  
Figure 6: Kernel Density Estimate plots of all features are given in the case of content-based split. independent individual pods that communicate with each other via RabbitMQ. During the run time, we monitor the accuracy, the training time, and the amount of bytes being sent and received at every computation pod. In a real-life scenario where multiple collaborating entities federate, the computation nodes are expected to run on different Kubernetes clusters. Kubernetes supports multiple clusters, therefore a communication link between multiple Kubernetes clusters would enable \(vFL\) in a real-life setting. Transmitting the inferred values from the worker nodes to the master, and the computed gradients from master to the worker nodes has two advantages as compared to \(hFL\). First, the matrices sent and received from workers to the master node are \(B\times M\) 2D matrices. Here \(B\) is the batch size, and \(M\) is the number of neurons only on the cut-layer of the worker models. Therefore, the trained model weights from all layers of the local models are not communicated unlike the case of \(hFL\). Especially, if the last interface layer (cut-layer) neurons can be kept small size, the transmitted data can further be reduced. Second, the worker nodes only receive the gradients in response to the subset of its own neurons in its cut-layer.