# An Understanding of the Vulnerability of Datasets to Disparate Membership Inference Attacks
## 2 Methodology
### Data  
In order to create the vulnerability metric, 105 different datasets from the UCI Machine Learning Repository and Kaggle dataset repository were utilized [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]. In order to focus on the datasets themselves and remove the effects of the utilized classification algorithms and attack models, combinations of classification models and attack models were used for each dataset as described in Table 1. More information on the attack method is provided in _Methodology: Membership Inference Attack_. All classification models created from the datasets--henceforth, referred to as _victim models_ given that these are the attacked models--were developed using the default settings for each function as defined in the scikit-learn Python library [64]. Several metrics for the attack and victim models were collected, including the accuracy, F1-score, precision, and recall, and were then averaged over all 13 combinations of attacks to develop a singular observation for each dataset. This averaging of metrics allowed for the capture of the dataset response to a variety of attacks and a separation of the vulnerability metric from the type of victim/shadow/attack model combination utilized. When combined with the features developed for the dataset (as discussed in _Methodology: Feature Selection_), this provided a vulnerability metric dataset of 118 features and 877 instances, since each class within a dataset is an individual observation. Of these 877 instances, 110 (12.5%) were held out as a test set while insuring that original datasets remained entirely in the training or testing set in order to prevent data leakage. Before selecting a dataset to include in the study, each dataset was inspected to ensure minimal missing data. If a dataset included a feature with greater than 25% missing data, that feature was removed. However, if it was determined that removal of too many features was necessary for proper classification, the dataset was not included. If the number of missing data entries was small enough to allow for dropping observations with missing data while maintaining the dataset utility, this method was utilized to remove missing data. If not, but the feature had less than 25% missing values, then the missing values were imputed using the feature average. All categorical variables were one-hot encoded. Any binary features remained as such. Finally, prior to utilization, all datasets whose values were outside a zero-to-one range were standardized using the default settings of the MinMaxScaler function within the scikit-learn library. By maintaining consistency in data preparation across all utilized datasets, control was maintained in the process. With the exception of scaling, the same preprocessing steps were completed both before development of the victim model and before the development of the dataset features for vulnerability classification as discussed in _Methodology: Development of Dataset Features_. It should also be noted that any dataset that was too small--less than roughly 100 observations in the macro dataset--was difficult to attack using the methodology discussed later (depending on the number of classes in which the dataset was divided) and was not used in the study. This was a result of the neural-net-attack methodology needing to divide the dataset into subgroups for training attack models on individual classes as described below.