## 5 Results
### Data Preparation
Validity:  
* Structure not suitable for software tools;
* Structure too complex (e.g., consisting of one large matrix);
* Big number of attributes;
* Attributes in rows instead of columns;
* Composed (non-atomic) attributes;
* Categorical attributes not usable in data-mining methods;
* Records from non-working days misrepresent results;
* Characters not manageable by the process;
* Words illegible where using specific characters;
* More versions of spelling. Uniqueness:  
Redundancy due to usage of multiple sources. Structure of the data should be already decided in the 'Data Understanding' phase, before first export is made; these exports are usually manageable by a data scientist, if they set up the export themselves, or if they cooperate with the person responsible for the exports. It is important to get with the structure as close to ideal as possible. The crucial criterion is manageability by software tools used in the 'Data Preparation' phase (and following phases). During this step, problems such as a too-complex structure (consisting of large matrices) or a big number of attributes are addressed. Ideal structure for most of the data-mining projects is with attributes presented in columns (with the first row as a header holding attributes' names) and individual records (distinguished by timestamp, ID of product, etc.) in rows. This structure is acceptable as an input for data-mining methods in software tools such as RapidMiner, Statistica, etc. When designing required structure, one must think of the way the data should be later examined. For example, 'Are we going to analyze every machine/account/year separately? ', 'Is our focus on errors solely, or on the process values? ', or 'What structure are the other datasets in, will they be compatible enough to be joined together?' Most structure adjustments can be achieved via scripts, either in scripting languages, such as Python, or even via VBA (Visual Basic for Applications--programming language for Microsoft Excel). RapidMiner also provides various preprocessing options, and alongside Python, it was our most valuable tool. It is expected that all the attributes will be measured at once. Most exports provide this, but some are only able to provide one attribute (process value) at a time, along with timestamp (and some additional information, such as machine identifier). This proves to be a viability problem for future analysis. It can be addressed in the following way: First, all the data are recorded in one file, one row per record. One column is for timestamp, another is for the parameter identifier, and the third one is for value. Second, time units (years,..., milliseconds) are calculated from the timestamp (number of units depending on accuracy of said timestamp) for each row. Third, suitable time unit for aggregation is decided--that means the most specific time unit which simultaneously keeps an adequate proportion of each attribute (with minimal information loss). The time unit should ideally cover all of the attributes at once. Four, the attributes are aggregated by timestamp and value (ideally averages, or average+mode in the case of categorical value) and grouped by chosen time unit (and the ones preceding it). Some of the sources, especially the ones recorded for a longtime period, may undergo major structure changes over time. This needs to be considered and resolved before joining with other sources. This may involve changing frequency of recording. If the data are to be used together, the more frequently changing data have to be adjusted to match frequency of the other part. Similar to the problem mentioned in previous subparagraph, this may be performed by aggregating values of the more frequent part and grouping them via the smallest time unit presented in the less frequent part. Joining two (and cumulatively more) datasets is ideally performed with ID (e.g., of a product) as a primary key.