# A Fairness-Aware Peer-to-Peer Decentralized Learning Framework with Heterogeneous Devices
## 5 Performance Evaluation
### Methodology  
**Deep learning models and datasets:** Two deep learning models are adopted to evaluate the proposed approach. We first consider a 5-layer Convolutional Neural Network (CNN) model (model 1) that consists of a couple \(3\times 3\) convolutional layers (32 channels for layer 1 and 64 channels for layer 2). A \(2\times 2\) maximum pooling layer is added after each convolutional layer, along with a fully-connected 512-unit layer and the _Relu_ activation function, and finally a softmax output layer (1,663,370 parameters) [35]. We then have a 2-layer CNN (model 2). To be specific, we adopt: (i) two \(3\times 3\) 64-channel convolutional layers that are followed by a \(2\times 2\) maximum pooling layer, (ii) two \(3\times 3\) 128-channel convolutional layers that are followed by a \(2\times 2\) maximum pooling layer, and (iii) three \(3\times 3\) 256-channel convolutional layers which are followed by a \(2\times 2\) maximum pooling layer. After that, the two fully connected layers with 512 units and _Relu_ activation are adopted [16].  
Two public datasets are used in our study, E-MNIST [36] and CIFAR-10 [37]. Specifically, **E-MNIST** contains 280,000 characters over 10 balanced classes, including 240,000 training samples and 40,000 testing samples, in which each image has 28 \(\times\) 28 pixels. **CIFAR-10** consists of 60,000 10-class images. We set 50,000 images for training and 10,000 images for testing. Both datasets provide rich data space so that our experimental setup can sample local datasets with diverse conditions (case 1 to case 4). In addition, E-MNIST dataset and CIFAR-10 have been widely used in existing related works such as [16]. All samples come from 10 balanced classes such as airplanes, cars, and others. Each sample is a 32 \(\times\) 32 colorful image with the RGB channels with the size 3072 bytes, including 1024 bytes on the red, green, and blue channel values. Note that we train model 1 and model 2 over E-MNIST and CIFAR-10, respectively.  
**Baseline approaches:** Two baseline approaches are considered for comparison: (i) _decentralized learning with full averaging aggregation (DL-Avg)_ in which each learning pair aggregates the received model with the weight averaging algorithm [16] and (ii) _decentralized federated learning with segmented aggregation (DFL-Seg-Avg)_ in which it adopts a model segment level decentralized federated learning that selects a set of model parameters to aggregate each segment [38].  
**Performance metrics and scenarios:** The following performance metrics are used to evaluate the efficacy of our approach: (i) _Average local accuracy_ in which each learning model over its local testing dataset is validated for the local accuracy and (ii) _accuracy distribution_ in which the histograms of learning performance (i.e., local accuracy) are measured for the fairness of decentralized learning process. Note that the local learners determine the learning objective with its own preference.  
Our experiments consider the various experimental scenarios, including balanced-i.i.d., unbalanced-i.i.d., and unbalanced-non-i.i.d. environments for local learners: (i) _i.i.d. data_--each local learner directly collects the local data samples from E-MNIST (CIFAR-10) dataset; (ii) _non-i.i.d. data_--each local learner collects \(\zeta\) (\(\zeta=2,3,4\)) classes from E-MNIST (CIFAR-10) dataset; (iii) _balanced data_--the size of local dataset is 800 (1000) images from E-MNIST (CIFAR-10) dataset; (iv) _unbalanced data_: the size of local dataset ranges from 400 to 1600 images. Moreover, we assign 2 nodes with the large dataset (10,000 samples) in the unbalanced-non.i.i.d. scenario. Table 2 lists the key parameters in our evaluation.