# Pattern Discovery in White Etching Crack Experimental Data Using Machine Learning Techniques
## 2 Data and Methods
### Artificial Neural Networks  
Artificial neural network (ANN) models were trained to classify the WEC risk (high or low) of an oil, taking the identities of its constituting chemical compounds and their respective percentages as input. The available dataset of oil compositions and their risk classification were used to train and test the ANN models. Similar to the random forest model, developing an ANN model involved selecting and tuning hyperparameter values to improve model accuracy. Eleven neural networks were developed, gradually increasing the 10-fold cross validation classification accuracy on unseen test oilsto 99.8% by tuning the hyperparameters of the networks [28]. Changing the following hyperparameters proved most influential on model performance: the number of hidden layers, the number of nodes per layer, the types of activation functions, the type and parameters of regularization, the type of loss function, and the parameters of the optimizer function.  
The network delivering the highest accuracy of 99.8% contained 3 hidden layers with L2 regularization applied only after the first hidden layer to help prevent overfitting. [PERSON] provides more details on L2 regularization [36]. The 3 hidden layers contained 19, 15, and 9 nodes, respectively. The activation function used after every hidden layer was the leaky rectified linear unit (leaky ReLU) [37] as a countermeasure against the vanishing gradient problem. The adaptive moment estimation (Adamax) optimization function [38] was used to optimize the neural network during training with the exponential decay rates for the first and second moment estimates set to 0.93 and 0.98, respectively, and the learning rate set to 0.0018. Categorical cross entropy was used as the loss function. Finally, the output layer consisted of two nodes corresponding to high or low risk with respect to WECs. Softmax [39] was used as the activation function following the output layer in order to facilitate the determination of the target classification, high or low risk, of a given oil composition.  
ANN models, through the process of training, approximate a desired function taking in the available input and producing the desired output. In this case, the input was the composition of each oil under investigation with respect to the 21 possible compounds in the dataset; i.e., there were 21 input variables. The output was the WEC risk level of the lubricant, high or low risk. More complex problems require more complex neural networks, and the aforementioned hyperparameters allowed for modularity in constructing the ANN model to meet the required complexity of the problem at hand. The process of tuning the hyperparameters involved iterations of trial and testing guided by previous experience and domain knowledge. [PERSON] provides a more detailed overview of artificial neural networks and deep learning [40].  
The neural networks were tested further to identify the most influential oil compounds in terms of risk of WECs. Twenty-one compound identities were present in the available data, so 10-fold cross validation was performed 21 times on each network architecture. Each time, a network with the previously selected hyperparameters was trained on correct data but tested on data with one of the compound's information shuffled. If the average 10-fold cross validation classification accuracy did not significantly decrease (\(>\)97%) as a result of distorting the data of a compound, then it was concluded that this compound was not influential in classifying WEC risk of an oil. As far as we know, this method of feature selection was developed during this investigation, inspired by the fundamental idea of the Boruta analysis. Additional work will be done to test its capabilities with different datasets before the release of a more detailed publication concerning the method. For ease of reference, this method is named Neural Network-based Feature Selection (hereafter referred to as \"NN-based FS\").