# Deep Learning Based Inversion of Locally Anisotropic Weld Properties from Ultrasonic Array Data
## 2 Methods
### Deep Neural Networks  
Neural networks are mathematical mappings that emulate the relationship between two parameter spaces, which learn (or are trained) by processing examples of input-output data pairs. Here, we train a DNN to approximate the inverse of \(f\) in Equation (2), mapping the weld grain orientation models \(\mathbf{m}_{\theta}\) from their corresponding time of flight data \(\mathbf{T_{m}}\). This can be written  
\[\text{DNN}(\mathbf{T_{m}})=\mathbf{m}_{\theta}^{\text{pred}}, \tag{3}\]  
where the superscript 'pred' denotes a predicted set of grain orientations. DNNs are formed by a series of hidden layers, with each layer containing a set of nodes. The network consists of connections that connect the output of a node in one layer to the input of a node in the following layer. In this work we use a fully connected DNN, where all nodes in a given layer are connected to all nodes in the following layer. The output of a node is the weighted sum from the previous layer connections, with an additional bias term, passed through a non-linear activation function (see Table 1 for details). The weights and biases are adjusted during the training process to map between the input and output parameter spaces. Here, the Adam optimisation algorithm [39] is used to update these weights and biases, and was chosen as as it provides fast convergence and simple implementation. The algorithm adjusts weights and biases to minimise a loss function, which we define as the mean absolute error (MAE), given by:  
\[\text{MAE}=\frac{\sum_{i=1}^{N}\mathbf{m}_{\theta}^{\text{true}}(i)-\mathbf{m }_{\theta}^{\text{pred}}(i)|}{N}, \tag{4}\]  
where \(i\) denotes the model index within the training data set, \(N\) is the number of models (so \(N=8000\) for model training and \(N=100\) for testing the trained network), the superscript 'true' denotes the known set of grain orientations that are used to generate the input ToF data. For each of the transducer array configurations shown in Figure 3, there are 32 transmitting elements and 32 receiving elements, therefore there are \(32\times 32=1024\) ToF measurements and 1024 nodes in the input layer of the DNN. We follow the approach of [25; 40], where a separate network is trained for each output model sub-region, and so a total of 91 networks are trained to predict the grain orientations of the sub-regions within the weld. An additional set of constant hyperparameters need to be assigned before the learning process can begin: such as the learning rate and decay rate used for the Adam optimisation, the number of hidden layers, the number of nodes in each layer and the non-linear activation function. To select the hyperparamters we use _hyperopt_, a stochastic optimisa  
Figure 5: The calculated travel times at all receiving elements for the anisotropic multi-stencil fast marching method (AMSFMM) plotted against the travel times taken from the finite element analysis (FEA) models, plotted for three transmitting elements \(s_{i}\), where \(i\) is the transmitter index. The data are fit with a third degree polynomial \(G\) (red curves), which is for calibrating the FEA generated data so it can be used with the DNN trained on AMSFMM data. tion library [41], which samples many different network hyperparameters to find the best performing network architecture, using a separate network architecture for each of the transducer array configurations. The outputs of these hyperparameter optimisations are shown in Table 1.  
For network training, both the input travel times and the output orientations are scaled to have zero mean and unit variance. Of the 10,000 models in the full training data set, a validation data set is taken using 2000 models. To avoid over-fitting the network to the training data, the loss function in Equation (4) is evaluated over the validation data set at each iteration, and an early stopping criterion is implemented so that the training process stops once the loss function stops decreasing (with a patience of 20 iterations).