## 2 Preliminaries and Related Work
### Preliminaries
#### 2.1.3 Non-IID Data in Federated Learning  
The data used to train the local model on FL could be non-IID data. This is due to the heterogeneity of datasets among devices, since the local datasets depend on the device's performance and usage, resulting in non-IID data [33; 34]. Each data sample has features and labels that identify them. Assuming that we can represent all features as \(F\), containing several features \((f_{1},\ f_{2},\ f_{3},\ \ldots)\), each feature \(f_{i}\) has a domain of different values that identify the sample. The data label can be represented as \(L\) that contains a value from its different values (classes). So, we can say the device \((i)\) has samples where each sample can be represented as \((F,\ L)\). The device samples follow a local distribution \(P_{i}(F,\ L)=P_{i}(F|L)\ P_{i}(L)=P_{i}(L|F)\ P_{i}(F)\). For that, when we say the data are non-IID, we mean that \(P\) is different from device to device, with the difference being caused by differences in the features, labels, or both [33; 35]. We can classify the types of non-IID data as shown in Figure 3.  
* Feature skew: Feature skew indicates that the features differ among devices; this can be described as the \(P_{i}(F)\) being different while \(P_{i}(L|F)\) is the same. The features can be non-overlapped between devices, partially overlapped, or fully overlapped. In  
Figure 3: Classification of non-IID data. non-overlapping feature skew, the different devices have different features; this case is similar to vertical federated learning; images with different angles are an example. While in partial overlapping feature skew, some features can be overlapped. Full overlapping feature skew is a case similar to horizontal federated learning; an example is the case of having two datasets for the same numbers (digits), one is written in a bold line while the other is written with a thin line [33; 34; 35]. * Label distribution skew: Label distribution skew indicates that the devices have different labels; this can be described as the \(P_{i}(L)\) being different while \(P_{i}(F|L)\) is the same. This skew can happen when the device tends to have local data with the same labels (for example, it can be caused by location variations between devices) or labels from some classes more than others. Label skew is defined in different ways in two studies. The study in [21] introduces the quantity label imbalance, and the study in [36] introduces the distribution label imbalance. Generally, the amount of data belonging to the same class is not equal and varies between devices. -
* Distribution label imbalance: In this skew, each device has a proportion of the samples from each label class that follows Dirichlet distribution \(Dir()\). The portion of the data that belongs to a specific class (\(c\)) is distributed on device (\(i\)) with a probability \(p_{c}\sim Dir_{i}(\beta)\), where \(\beta\) is the concentration parameter that determines the imbalance level; a higher value indicates a high imbalance partition [33; 35; 36]. * Same feature, different labels: The case of the same feature with different labels implies that the distribution of \(P_{i}(L|F)\) is different but \(P_{i}(F)\) is the same. In this case, the same features indicate different classes (labels) on different devices; the data label for the same feature can be \(c_{1}\) on the first device and \(c_{2}\) on the other device. This could depend on the user preference; for example, in the same weather condition, some people may refer to a rainy day as good weather, while other people refer to the same rainy weather as a bad day [33; 34]. * Same label, different features: The case of the same label with different features implies that the distribution of \(P_{i}(F|L)\) is different but \(P_{i}(L)\) is the same. Different features on different devices could belong to the same class.