# Topology Optimisation under Uncertainties with Neural Networks
## 3 Neural Network Architectures
### Topology Convolutional Neural Networks (TCNN)
#### 3.1.2 Data Preparation
Therefore, we build on the special RNN concept of a Long Short-Term Memory (LSTM) in the context of our problem, which is more robust against the vanishing gradient issue and provides promising results, especially in the analysis of time series. For a background on time series analysis and a review of the different methods, we refer to the survey article [21]. In practice, time series are usually stored as one-dimensional sequences in vector format. Consequently, there is no out-of-the-box LSTM layer implementation for structures such as the input tensor we require in Equation (14). Nevertheless, we still do not wish to abandon the mechanism of convolution within the NN in order to keep the structural information of \(\varphi\) and \(u\). An LSTM layer with a convolutional structure can be constructed by replacing the matrix vector multiplication within a standard LSTM layer by convolutional layers. The unique advantage of an LSTM according to [20] is its cell-gate architecture, which mitigates the vanishing gradient problem. More precisely, it consists of a \"memory cell\" \(c_{t}:\mathbb{R}^{4\times d_{\text{in}}^{t}}\to\mathbb{R}^{d_{\text{out}}^{t}}\) that serves as an accumulator of the current state \(t\leq T,\ t\in\mathbb{N}\), in the processed sequence. The information capacity of the last status \(c_{t-1}\) within \(c_{t}\) is controlled by the activation of the so-called \"forget gate\" \(f_{t}:\mathbb{R}^{3\times d_{\text{in}}^{t}}\to\mathbb{R}^{d_{\text{out}}^{t}}\). The information capacity of the input state \(x_{t}\in\mathbb{R}^{d_{\text{in}}^{t}}\) is controlled by the activation of the input gate \(i_{t}\). Which information (or whether any at all) gets transferred from memory cell \(c_{t}\) to state \(h_{t}:\mathbb{R}^{2\times d_{\text{in}}^{t}}\to\mathbb{R}^{d_{\text{out}}^{t}}\) is in turn controlled by the activation of the output gate \(o_{t}\). From a technical point of view, the gates can be understood as learning forward layers. ### TLSTM Architecture

An ordinary LSTM layer to generate complex sequences with long-range structure as presented in [22] corresponds to the described logic above and can be formulated numerically for a sequence of one-dimensional input state \(x_{t}\in\mathbb{R}^{d_{\text{in}}^{t}}\) and output vector \(h_{t}\in\mathbb{R}^{d_{\text{out}}^{t}}\) as an equation system

\[ \begin{split} i_{t}(x_{t},h_{t-1},c_{t-1})&=\sigma( \mathcal{W}_{xi}x_{t}+\mathcal{W}_{hi}h_{t-1}+\mathcal{W}_{ci}\odot c_{t-1}+b_ {i}),\\ f_{t}(x_{t},h_{t-1},c_{t-1})&=\sigma(\mathcal{W}_{ xf}x_{t}+\mathcal{W}_{hf}h_{t-1}+\mathcal{W}_{cf}\odot c_{t-1}+b_{f}),\\ c_{t}(f_{t},c_{t-1},i_{t},x_{t},h_{t-1})&=f_{t} \odot c_{t-1}+i_{t}\odot\tanh\big{(}\mathcal{W}_{xc}x_{t}+\mathcal{W}_{hc}h_{ t-1}+b_{c}),\\ o_{t}(x_{t},h_{t-1},c_{t})&=\sigma(\mathcal{W}_{ xo}x_{t}+\mathcal{W}_{ho}h_{t-1}+\mathcal{W}_{co}\odot c_{t}+b_{o}),\\ h_{t}(o_{t},c_{t})&=o_{t}\odot\tanh\left(c_{t} \right),\end{split}  \tag{24}\]

where \(\sigma:\mathbb{R}\to\mathbb{R}\) with \(\sigma(x)=\frac{1}{1+\mathrm{e}^{x}}\) and \(\tanh\) are evaluated element-wise. The operation \(\odot\) denotes the Hadamard product and the subscripts of the weight matrices \(\mathcal{W}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}^{t}}\) describe the affiliation to the gates. For example, \(\mathcal{W}_{xi}\) is the weight matrix to input \(x_{t}\) of gate \(i_{t}\). This illustrates how the weights of the LSTMs are transferred to the weights of convolution LSTMs in the following. We want to reformulate Equation (24) by replacing all matrix-vector multiplications (i.e., the forward layer) with a convolution layer from Definition 2. This is inspired by [23], which has already provided the theoretic architecture of a convolutional LSTM layer with the approach on precipitation forecasting.