# SMARTEN--A Sample-Based Approach towards Privacy-Friendly Data Refinement
## 4 Smarten
### Data Preparation Phase  
Due to the IoT and decreasing storage costs, data are constantly being captured and stored persistently. Data processors and especially data stewards are therefore faced with a flood of raw data that they cannot handle manually. Therefore, there are approaches towards data pre-processing powered by artificial intelligence. However, it cannot substitute human knowledge in data preparation. Human data wrangling experts, such as data stewards, achieve better results in data preparation because they can rely on broad domain knowledge and their experience. In the data preparation phase, it must be ensured to keep the human in the loop [68]. In our approach, we therefore adopt a dynamic sampling approach to meet this premise. Here, the data steward operates on a representative yet manageable sample which is extracted from the bulk of raw data. In doing so, the data steward defines data preparation rules that can be generalized and applied to the entire base data. While such an approach is primarily intended to relieve the experts, it also represents a privacy preserving measure [69]. Since the data steward never has access to the entire dataset, he or she can only gain as much insight as required for the fulfillment of his or her task. In addition, particularly sensitive data can be concealed entirely. However, by applying afterwards the defined rules to the entire base data, a comprehensive data preparation can still be ensured. Figure 3 illustrates the workflow that is followed in our approach. \(\dagger\) Initially, the _data sampler_ picks a representative data sample from the stock of raw data. Descriptive analyses are performed on the base data to identify data items that require special attention, e.g., null values, extreme values, and outliers. Metadata provided by the data producers can further facilitate the sample selection, for instance, to identify attributes  
Figure 3: Workflow of the sample-based approach for data preparation applied in SMARTEN. that are either particularly relevant for data preparation or have little or no influence on data refinement. 2 If a reasonable sample is found, the data sampler forwards it to the _privacy assessor_3. The privacy assessor evaluates whether the sample meets the necessary privacy requirements. These are specified by the data producer by means of a _privacy threshold_. For more information on setting these thresholds, see Section 4.3. In addition to \(\epsilon\)-differential privacy (see Section 3), other metrics can also be used for this purpose. For instance, \(k\)-_anonymity_ describes how well an individual can disappear in a crowd. The \(k\) indicates that there are at least \(k-1\) other individuals (respectively, data items) with similar attribute values, so that they cannot be distinguished unambiguously. Thus, information obtained from the data cannot be associated with a single individual, but only with a group of size \(k\). Another metric is the _entropy_ which evaluates the information content of a dataset. In simple terms, it assesses how much a data item deviates from the expected norm. For instance, if it is known that the value of an attribute \(\mathbb{A}\) is almost in all cases \(x\), then a data item where \(\mathbb{A}\) has the value \(y\) has a particularly high information content. For more information on these and other technical privacy metrics that can be used to evaluate the samples, please refer to literature, e.g., the work by [PERSON] and [PERSON] [70]. With regard to the quality of the data refinement, however, the principle of fairness must also be considered in the sampling. For instance, no bias must be introduced due to the sampling [71]. For instance, if the value of the attribute \(\mathbb{A}\) is predominantly \(x\) in the sample, while the attribute values \(x\) and \(y\) are equally distributed in the base data, the data steward might estimate the significance \(\mathbb{A}\) incorrectly. This would have a negative impact on the quality of the outcome of the data preparation phase.