# Higher-order HDL: Applied to MLP neural network hardware implementation

Samuel Garcia, Mingjun Zhang

► To cite this version:

Samuel Garcia, Mingjun Zhang. Higher-order HDL: Applied to MLP neural network hardware implementation. 6th International Conference on Multidisciplinary Design Optimization and Applications (MDOA 2024), Oct 2024, Paris, France. pp.02004, 10.1051/e3sconf/202563102004. hal-05297834

HAL Id: hal-05297834

https://hal.science/hal-05297834v1

Submitted on 17 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Higher-order HDL: Applied to MLP neural network hardware implementation

Samuel GARCIA<fnref n="1" />, a) Mingjun ZHANG<fnref n="1" />,<fnref n="2" /> b)

**Abstract.** In this article, we describe a methodology for the rapid implementation of a hardware architecture using a higher-order approach. This methodology uses a combination of TCL and VHDL for higher-order coding (i.e. code produced by code) and is supported by industry- standard HDL development tools. To explore this method- ology, we used an FPGA implementation of an artificial neural network (ANN) as a guinea pig application. This enabled us to produce a fully generic multilayer perceptron model where the number of layers, the size of each layer, the types of synaptic signals and the activation function are easily customizable. Not only does this approach make the development of such an application faster, but the high degree of genericity of the model cannot be achieved with conventional VHDL methodology. This article presents feedback from our first steps with this methodology and its application to MLP hardware architecture.

*Index Terms—*VHDL, TCL, Artificial Neural Net- works, Multilayer Perceptron, higher-order programming, Methodology*

## 1. INTRODUCTION

Higher-order programming has been around since the beginning of computing. The basic idea is that, instead of writing the code for a program, an application developer would write a program which, in turn, would generate the application code. This considerably optimizes development effort, particularly for complex applications with a regular structure in the code. Applying this principle to the design of hardware architectures is as simple as generating HDL description code from a custom computer program. This makes a lot of sense, as most hardware architectures have a complex structure but a high degree of structural regularity. However, these techniques seem to be little used (a few classic exceptions will be discussed later in this article). This is why we want to explore this concept and experiment with both a computational and a methodological approach.

The chosen guinea pig application is an FPGA implementation of an artificial neural network (ANN). Specifically, we want to produce a generic design for a multilayer perceptron of any dimension. In MLB, the number of layers and the number of neurons in each layer can greatly affect the learning time of the system, and these parameters can be studied and adapted to the final application of the neural network. These parameters can be studied and adapted to the final application of the neural network, but the principle of the structure remains unchanged. This structure will be used later in this document. The MLP architecture presents a good balance between complexity and regularity, making it a perfect candidate for studying higher-order HDLs.

For this experiment, we used VHDL as the hardware description language (generated code) and TCL as the higher-order programming language (to write the custom code generator). The choice of TCL in this context seems obvious, as it is highly optimized for manipulating text, and a TCL interpreter is already included in the vast majority of hardware design tools, removing the need for a new design tool. The script generating the final code can simply be included in the build script for simulation and implementation workflows.

The remaining of this article will be structured as follow: section II will make a brief abstract about MLP and present its architecture in more detail. In section III we will discuss general principle of higher-order programing and set up some terminological definitions. Section IV will relate our higher-order implementation of generic MLP with a discussion about both the code and the methodological approach. Section V will address one of the main curb of higher-order programing which is the debugging difficulty. Conclusion and further perspectives will be drawn in section V.

<footnote n="1">¹ CNAM – CEDERIC – LAETITIA Paris, France</footnote>

<footnote n="2">²DGUT – CNAM Institute, Dongguan university of Technology, Dongguan, China</footnote>

<footnote>a) Corresponding author: samuel.garcia@lecnam.net; b) ming-jun.zhang@lecnam.net;</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. MLP general architecture</figcaption></figure>

## II. THE MULTILAYER PERCEptron (MLP)

The Multilayer perceptron is one of the most widely used artificial neural networks. It was first presented in [1], [2]. The general architecture of an MLP is shown in figure 1. It is structured in layers. Each layer comprises a fixed number of neurons. The first layer is the input layer, with each input entering a neuron. The last layer is the output layer, composed of one neuron for each output. Between the input and output layers, one or more hidden layers are inserted. Each layer comprises a given number of neurons. Each neuron of a given layer performs a pondered sum of the previous layer's outputs and then perform a thresholding of this sum. In order to ease the learning algorithm this thresholding function must be derivable around zero. To perform this a linear sigmoid function is generally used instead of a simple binary thresholding (see figure 2). This thresholding function is called the neuron's activation function. The output of a neuron is the output of this activation function applied to the pondered sum of the inputs.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Activation function</figcaption></figure>

The signals transmitting output from one layer of neurons to the next are called synapses. Between each layer, we have $Ln \times Ln-1$ synapses, where Ln is the number of neurons in layer n. Each synapse has a coefficient associated with it, called the synaptic weight. This is the weight used for the above-mentioned weighted sum at the neuron input.

The overall function of the MLP is tuned by adjusting the synaptic weights. This is usually done via a learning procedure in which we present cases on the input while comparing the output with the expected result for each case. From this, we determine an error factor. This error factor will then be used to adjust the synaptic weights, starting with the output layer and working up to the input layer, so that each is adjusted according to its relative contribution to the error. This procedure is known as error backpropagation [3].

It should be noted that this learning procedure is only described to complete the general description of the MLP, but will not be discussed further in this article, as we focus solely on the implementation of a running MLP that simply takes the pre-learned synaptic weights as parameters.
Several hardware implementations of MLPs have been presented in [4], [5], [6], [7], [8]. We won't claim that our particular implementation is new or better. In this article, we only use it as a guinea pig application to test our new methodology.

# III. HIGHER-ORDER HDL : THE THDL METHODOLOGY

## A. Classical VHDL methodology

The classic VHDL methodology is shown in figure 3. Here we show the simulation flow based on the MODELSIM tool suite [9], but it can be generalized to most existing tool suites for simulation and synthesis. The inputs are one or more VHDL files, providing a description of the hardware architecture we want to simulate and possibly a test bench for it, and a tcl script that provides instructions for the tool suite (i.e. what to compile in what order, what to simulate/implement, ...). MODELSIM executes this script, which in turn calls the VHDL files in order to compile them (VCOM instruction), build the simulation engine (VSIM instruction) and execute it (run instruction). This command script is very often automatically generated by another script part of the IDE used (e.g. ISE, VIVADO, Quartus, HDL designer, ...).

<figure><img src="image_4.png" /><figcaption>Fig. 3. Classical VHDL flow</figcaption></figure>

For a typical application developer, the methodology is therefore mainly based on the use of the description mechanism provided by the VHDL language. There are two caveats to this. Firstly, not all possible constructs allowed by the VHDL standard can be synthesized. Secondly, even within the synthesizable subset of VHDL, not all constructs are supported by all tools. This is particularly the case for constructs added to the latest versions of the VHDL standard (e.g., generic type/type polymorphism).

*Case of generate VHDL statement:* As this article is dedicated to the description of higher-order hardware, we couldn't avoid talking about VHDL instructions.

The VHDL language provides two very useful constructs for performing a kind of higher-order description. These are the IF GENERATE and FOR GENERATE instructions. [10]. They enable a conditional or repetitive parametric description based on a generic parameter.

Although this is a very powerful construction in some cases, their use limits this usefulness to certain specific cases.

The first limitation is that the generic parameter used to loop must either be hard-coded, or the result of simple operations. By simple operations, we mean the standard operations provided by the base language, to the exclusion of higher-level functions. A very annoying example of this is that you can't use the log function. The case of the generic multiplexer is a good illustration of this limitation.

The second limitation is that this iterative description can only be applied in the body of the architecture. This means you can't use this very useful description to build loops to define signals and/or ports. This limits the practical
use of FOR GENERATE to the construction of rectangular arrays where the table type of normal size can be defined in advance.

The idea behind the methodology we present in this article is to be able to extend this concept to any structure that can be simply described using standard coding statements (i.e. if, for, while, case, ...), and not to be limited to the body of the architecture.

## B. Terminological definitions

Before presenting our higher-order hardware description methodology, we will settle some basic definition about higher-order programing in general.

The complete tool chain consist of a frontend and a backend. The frontend will take the developer's code as input to generate an intermediate description that will then be passed as input of the backend to generate the desired final output. The frontend, in our case, is a TCL interpreter [11]. It run the input source code that will generate the intermediate VHDL files. The backend is any chosen VHDL tool chain (The example presented in this article will use MODELSIM). It's input is a set of VHDL files. The output is either a set of netlist and finally a bitstream (synthesis flow) or a discrete event simulation (simulation flow). Figure 4 show the complete chain and the global flow.

Before presenting our methodology for describing higher-order hardware, let's establish some basic definitions of higher-order programming in general.

The complete tool chain consists of a frontend and a backend. The frontend takes the developer's code as input to generate an intermediate description, which is then passed on to the backend to generate the desired final output. The frontend, in our case, is a TCL interpreter. It executes the input source code that will generate the intermediate VHDL files. The backend is any VHDL tool chain (the example in this article uses MODELSIM). Its input is a set of VHDL files. The output is either a set of netlists and ultimately a bitstream (synthesis stream), or a simulation of discrete events (simulation stream). Figure 4 shows the complete chain and overall flow.

Note that the front end can be used recursively. The TCL language is particularly well suited to this purpose. This could then be used as an application of the lambda calculus paradigm [12], [13] where the output lambda operator is a hardware architecture.

<figure><img src="image_5.png" /><figcaption>Fig. 4. general flow for higher-order hardware description</figcaption></figure>
### C. Introducing the THDL methodology

In this section, we present the THDL methodology. As already mentioned, it consists in describing a hardware architecture by developing a TCL script, which in turn generates the VHDL description of this architecture. The name THDL comes from the contraction of TCL and HDL.

This general principle can be broken down into two types of methodology:

The first is a symbiotic approach in which the application developer writes VHDL code into which TCL script parts are inserted. Interpretation of these TCL script parts will result in a pure VHDL intermediate file customized on the fly. This approach is quite similar to the development of a typical web application where html code is used both as the general structure of the input file and as the intermediate code, but where another scripting language (e.g. PHP, JS, ruby, ...) is used to customize certain parts.

The second is much more advanced, but also much more powerful. It consists in masking the intermediate language with high-level functions so that the general logic of the description can be adapted to the application. A good example of this is the erroneous order description (see behind).

Figure 5 shows the practical adaptation of the above-mentioned classic design flow to the THDL methodology. It is very similar, but instead of having the command script (i.e. the do file in MODELSIM) directly compile the input VHDL files, it first interprets the TCL script of the thdl input file, which generates the VHLD files that will then be compiled.

<figure><img src="image_6.png" /><figcaption>Fig. 5. THDL flow</figcaption></figure>

We have not explicitly shown the testbench in this figure. It can be performed in the classical way using pure VHDL, it can also be performed using the THDL method, or it can also be performed directly in the form of a pure TCL script, with MODELSIM providing a specific set of TCL functions for this purpose.

1) *out of order description:* The VHDL description of an architecture is divided into several parts. Input/output ports are located in the entity. The declaration of internal signals is in the architecture header. The functional description is in the body of the architecture. And eventually, some global type definitions and component declarations are found in a package.

This means that for a functional part of the architecture, code must potentially be added in each of these parts of the code. When writing code by hand, the most common practice is to go back and forth between all these parts of the code in order to have the code required for each functional part according to the application/architecture chosen. In other words, code is almost never written line by line in order.

However, when a text file is written to the computer's hard disk, it must be done line by line in order. To add a line in the middle of a source code, you would have to read the previous version of the code, analyze it to find the exact line where it can be added, then copy the lines from the previous version into the new version of the file, write
the code to be added, then copy the rest of the previous version. This would be far too slow and tedious, especially considering the complexity of the VHDL analyzer.

Generating the source code on the fly, line by line, directly in order would be a huge methodological mis- take. The whole idea is to be able to describe the overall structure of the architecture as a high lever loop/condition combination and then for each instance generate the appropriate code taking into account the resulting parameters. Considering a simple FOR loop as general structure (i.e. 1D grid architecture), generating the code in order would require up to 4 separate for loop, one for each part of the VHDL. That would not be much of a gain over classical approaches.

To get around this problem, we took advantage of the fact that in VHDL, instructions are concurrent. This means that the order in which lines are written doesn't matter as long as they're in the right part of the code. In this way, an entity can be seen as a simple list of port declarations. The header of an architecture is simply a list of types, signals or component declarations. The same applies to a package, minus the signals. And the body of an architecture is just a list of concurrent declarations.

This is exactly what we propose in the THDL method. Instead of writing directly to the output VHDL code files, the description script simply adds entries to one of these lists using high-level functions. Then, a set of standard functions is called at the end of the script to write the VHDL files with all the elements of the corresponding list and the corresponding VHDL block code. Line 10 of the listing 1 shows an example of a global array data type declaration in a package. It uses a tcl function called mkArray to generate the required VHDL code, then this line of code is added to the package declaration list. Line 19 is the call to the mkPkg function, which will generate the package code and output it as a .vhd file.

## IV. HIGHER-ORDER IMPLEMENTATION OF MLP

The listing 1 shows an extract from the THDL description of the perceptron. Line 1 is the call to the THDL lib, which defines standard functions for writing VHDL code files from the various description lists, as well as high-level tcl functions for automatically generating special forms of VHDL syntax. Line 3 defines the number of layers and the number of neurons for each layer of the neural network we want to implement. In general terms, we're defining our generic parameters. Lines 9 to 17 are the two nested loops for iterating on each layer first, then on each neuron. The other parts presented here have already been discussed in the previous section.

This implementation is very simple and in fact very similar to what a purely TCL implementation of this MLP would have been. The only difference is that, instead of calling a TCL function to perform the neuron's computation, a function that instantiates a VHDL component of the neuron is called.

Testing of this description is still in progress. The full code will be available after publication of this article, along with some summary results.

## V. RETURN ON DEBUGGING HIGHER-ORDER

The back-end compiler (in our case VCOM) only gives feedback on the basis of its own input. These inputs are the VHDL files, which for us are intermediate files. The back-end compiler warns us of an error and tells us the line where that error occurred, so that the application developer knows where to look for the error. But in higher-order HDL, this information becomes useless as it points to a file that the application developer didn't write directly. In order to provide the application developer with useful debugging feedback, we need to be able to link error messages from the back-end compiler to the exact file and line in the front-end that generated the faulty code.

Such feedback is essential if the THDL methodology is to be fully usable. But this is no easy task. It should be possible to split the interpretation of the THDL script and trace the line that will lead to the generation of a code (for
example, by adding something to one of the VHDL declaration lists), and link it to the corresponding output code file/line. An additional file would then be generated with a correspondence list between THDL and VHDL lines. The call to the VCOM compiler could then be made by redirecting its standard output to a special script which would then translate the faulty VHDL file/line mentioned into the THDL line number, using the previously created dictionary file.

CONCLUSION AND PERSPECTIVES

In this article, we presented a first experiment in higher-order hardware description. We discussed a methodology called THDL that combines TCL scripting and VHDL to achieve a higher-order description using almost any existing tool. In this first experiment, we applied the THDL methodology to the description of a multilayer perceptron. This experiment validated the feasibility of the method. It resulted in a small, concise source code that can generate a large amount of customized intermediate VHDL code. This description is also much more generic than can be achieved in practice using pure VHDL source code. This genericity is essential for design space exploration and code reuse. It also revealed the need for an unordered description approach in which lines of code are not generated directly in order, but in rhythm with the structure of the application. We have developed a set of standard TCL functions for automatically writing intermediate VHDL files, as well as high-level functions for some common VHDL code constructs. A first perspective will be to consolidate and extend this set of standard functions.

This experiment also highlighted the difficulty of debugging. This should be at least partially overcome by the solution proposed in the previous section. Another problem that arose was that higher-order coding proved to be much more intellectually demanding than direct coding. The application developer must have in mind both the desired final architecture, the intermediate VHDL structure and the structure of the higher-order code. This limits the application approach to sufficient complexity that the leverage of the higher-order description overcomes the additional complexity of the method itself. The use of an out-of-order description and a high-level function has partially alleviated this problem, as it abstracts much of the VHDL. Further development of the methodology and training should also help to alleviate this drawback. The libTHDL tcl library and MLP source code will be made available and opened after final publication.

```mlp
source $libPath/libTHDL.tcl

set Layers {64 100 150 100 3}
(...)
set archBody ""
set archHd ""
set glob_types ""

foreach layer L $Layers {
    lappend glob_types [mkArray "synL${L}Array" $layer "Tsyn"]
    lappend signals "SIGNAL TsynsL$L: synL${L}Array;"
    (...)
    for {set N 0} {$N < [lindex $Layers $L]} {incr N} {
        lappend archBody [neuronInst $L $N]
        (...)
    }
}

(...)
mkPkg mlp.pkg $glob_types pkgFDesc
```

<caption>Listing 1: MLP description loop</caption>
REFERENCES

[1] F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. 01 1962.

[2] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1," ch. Learning Internal Representations by Error Propagation, pp. 318–362, Cambridge, MA, USA: MIT Press, 1986.

[3] M. J. Zhang, S. Garcia, and M. Terre, "Fast learning architecture for neural networks," pp. 1611–1615, 2022.

[4] C. Morales Morales, U. Flores, M. Adam Medina, M. Diaz Salazar, J. Abiel Caballero, D. Criado Cruz, and S. Pavoni Oliver, "Digital artificial neural network implementation on a fpga for data classification," IEEE Latin America Transactions, vol. 13, pp. 3216–3220, Oct 2015.

[5] J. Skodzik, V. Altmann, B. Wagner, P. Danielis, and D. Tim mermann, "A highly integrable fpga-based
runtime-configurable multilayer perceptron," in 2013 IEEE 27th International Conference on Advanced
Information Networking and Applications (AINA), pp. 429–436, March 2013.

[6] R. C. Garcia, W. I. Suemitsu, J. O. P. Pinto, and A. M. Soares, "Implementation of quadric perceptron with hardlims activation function in a fpga for nonlinear pattern classification," in IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society, pp. 2432–2437, Nov 2013.

[7] Y. Taright and M. Hubin, "Fpga implementation of a multilayer perceptron neural network using vhdl," in ICSP '98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344), vol. 2, pp. 1311 4 vol.2, Oct 1998.

[8] M. J. Zhang, S. Garcia, and M. Terre, "Real-time fast learning hardware implementation," International Journal for Simulation and Multidisciplinary Design Optimization, vol. 14, p. 1, 2023.

[9] M. Inc., "Modelsim user's manual."

[10] "Ieee standard vhdl language reference manual," IEEE Std 1076-2008 (Revision of IEEE Std 1076-2002),
2009.

[11] J. K. Ousterhout, "Tcl: An embeddable command language," 1990.

[12] A. Church, "A set of postulates for the foundation of logic," Annals of Mathematics, vol. 33, no. 2, pp. 346–366, 1932.

[13] A. M. Turing, "Computability and lambda-definability," Journal of Symbolic Logic, vol. 2, no. 4, p. 153–163, 1937.