Custom Layers in Tensorflow. Enhance the Tensorflow Experience... | by Marco Sanguineti | Towards Data Science

In my studies of artificial intelligence algorithms, apart from the steep initial learning curve, I was struck by the immediacy with which such algorithms can be developed and used by those with the skills, given the huge availability of open-source and freely accessible resources. In particular, TensorFlow makes it possible to develop in a very practical and simple way complex algorithms that would require months of implementation, even by an experienced programmer, to achieve vaguely comparable results.
The abundance of documentation makes it easy to understand how to apply the numerous pre-implemented models, just as detailed tutorials make it possible to tackle various problems in a practical way (Natural Language processing, image classification, supervised and supervised unsupervised learning...).
Like various fields of scientific learning, there comes a time when it becomes necessary to go deeper into the topics covered to achieve better and better results.
To this end, the use of the Keras Function API, combined with Tensorflow’s Eager execution and the possibility of generating completely customised layers, loss functions, models, metrics and callbacks, makes it possible to tackle artificial intelligence problems in a much deeper and more detailed way. Moreover, applying the basic concepts of Tensor, variable, constant and the application of the GradientTape can give life to completely original algorithms and adapted to the specific need, with a view to custom training.
In this article, we will use a custom layer, developed by subclassing the Layer object in Tensorflow. We will develop a quadratic layer, as opposed to a classical Dense layer characterised by a linear pre-activation + application of an activation function (typically non-linear). We will create a very basic neural network model using the functional API of Keras and we will test our model on a representative dataset of a complex engineering problem, obtaining however nice results.
The noise generated by an airfoil is an important environmental parameter for the aerospace industry. A great part of the total airframe noise is the airfoil self-noise, related to the interaction between the airfoil itself and the turbulence effects generated by its boundary layer. The self-noise data set was developed by NASA. It was obtained from a series of aerodynamic and acoustic tests of two and three-dimensional airfoil blade sections conducted in an anechoic wind tunnel, at various wind tunnel speeds and angles of attack.
Model Features:
Frequency [Hz]
The angle of attack [deg]
Chord length [m]
Free-stream velocity [m/s]
Suction side displacement thickness [m]
Model Labels:
Scaled sound pressure level [dB]
I made a video for the generation of a basic Dense neural network for facing this aerodynamic problem, let’s take a look if interested. You can also find the full code here.
Let us now see how to generate a quadratic layer. To see how the whole problem was tackled, you can find the complete code here, or you can watch the video at the beginning of this article. To understand the working principle of a standard Dense layer, we can start from the concept of perceptron function. This basic model of artificial intelligence involves the execution of the sum of the product of a set of input parameters in the neuron itself and the relative weights, generating a pre-activation value. Once the pre-activation is calculated, it is necessary to apply to it an activation function to assign non-linearity to the output of the generated surrogate model.
Based on these concepts, a Dense layer will be made up of “N” neurons working on the same principle. In our application case, we generate a customised layer whose relationship between input and pre-activation of the neuron itself is quadratic, and not linear.
To generate our layer within the Tensorflow working environment, we inherit the properties of the layer object and then initialise its attributes. We define two new attributes: the number of neurons in the layer and the activation function.
In the build method, we define the trainable parameters of our layer. We need to specify the size of each of these tensors, the data format and thus the fact that these parameters are actually trainable. The first two parameters are randomly initialised, while the bias term is initialised to zero. The call method allows us to have the response of our layer, evaluated in the forward propagation phase: we implement the quadratic expression above.
We generate our simple model, with only one hidden layer: the custom one we have created.
We compile our model, define the loss function and choose Adam as the optimiser. We then train the model for 500 epochs and check the results. Within our script, we also show a comparison between the value of the parameters of our custom layer before and after the training, to check that it works correctly.
We then run validation plots to verify the performance of the model on the training and validation set. The results are discrete, considering how simple the model is and that no optimisation of its hyperparameters has been performed. This confirms, however, the correct functioning of our custom layer.
Let’s check some numerical results:
Y0 - mae-train - 0.05067Y0 - mse-train - 0.00441Y0 - r2-train - 0.87064Y0 - pearson-train - 0.93566Y0 - mae-test - 0.04748Y0 - mse-test - 0.00402Y0 - r2-test - 0.87471Y0 - pearson-test - 0.93871
We have seen how to tackle a complex aerodynamic problem with a relatively simple model, using Colab, Tensorflow and Python. We created a model with only one hidden layer, using a layer we built ourselves instead of those pre-implemented within the Tensorflow environment. We checked its correct functioning and verified the results. I hope you enjoyed this article, any feedback is welcome.
Until next time, Marco
3-minutes machine learning is a series of tutorials, videos and articles related to the world of AI, Deep Learning, and Data Science. You can find the complete videos collection on YouTube. The GitHub repository contains all the Google Colab notebooks shown in the articles and videos. I hope these contents will be useful or simply of interest to you. Any feedback is welcome.
Check out the other episodes:
Images Generation with Neural Style Transfer and Tensorflow
Cats VS Dogs Convolutional Classifier
Plot a TensorFlow Model with Keras Functional API
Neural Network for Predicting the Energy Performance of a Building