Journal of Robotics, Networking and Artificial Life
Vol. 7(1); June (2020), pp. 58–62
DOI: https://doi.org/10.2991/jrnal.k.200512.012; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

A Hardware-Oriented Echo State Network and its
FPGA Implementation
Kentaro Honda, Hakaru Tamukoh*
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu,Fukuoka 808-0196, Japan

ARTICLE INFOABSTRACTArticle History

This paper proposes implementation of an Echo State Network (ESN) to Field Programmable Gate Array (FPGA). The proposed
method is able to reduce hardware resources by using fixed-point operation, quantization of weights, which includes accumulate
operations and efficient dataflow modules. The performance of the designed circuit is verified via experiments including
prediction of sine and cosine waves. Experimental result shows that the proposed circuit supports to 200 MHz of operation
frequency and facilitates faster computing of the ESN algorithm compared with a central processing unit.

Received 11 November 2019
Accepted 04 March 2020
Keywords
Reservoir computing
echo state network
field programmable gate array© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Neural networks are highly expected to be applied into embedded
systems such as robots and automobiles. However, Deep NeuralNetworks (DNNs) [1] require high computational power because
a lot of accumulate operations are being processed using them.
Generally, graphics processing units are used to accelerate thesecomputations; however, as their power consumption is high, implementing embedded systems using them is difficult due to a power
limit. To mitigate this problem, we have implemented DNNs intohardware such as Field Programmable Gate Arrays (FPGAs), realizing high-speed calculation with low power consumption.
In this paper, we have implemented an Echo State Network (ESN)[2], a kind of Reservoir Computing (RC) into an FPGA. An RC is a
Recurrent Neural Network (RNN) model in which only the weights
of an output layer are defined in the training step. ESNs are able tolearn time-series data faster than general RNNs such as Long Shortterm Memory (LSTM). In ESNs, a lot of accumulate operations ofinput data and weights are executed, however, there are limitationsof FPGA resources such as Loot Up Table (LUT), Flip Flop (FF) andDigital Signal Processor (DSP). As a result, we have modified thealgorithms and architectures of ESNs. Furthermore, we implement
the proposed hardware-oriented algorithms into the FPGA andshow the effectiveness of the proposed methods by comparing theproposed circuit with other.

2. ECHO STATE NETWORKThe ESN is a type of RC which consists of three layers: an inputlayer, a reservoir layer and an output layer, shown in Figure 1, inCorresponding author. Email: tamukoh@brain.kyutech.ac.jp*

which the neurons of the reservoir layer are randomly connectedto each other.
ESN is described by Equations (1) and (2),x(t ) = f ((1 - d )x(t - 1) + d (winu(t ) + wres x(t - 1)))(1)z (t ) = wout × x(t ) (2)where x(t) and z(t) are output of the reservoir and output layer,
respectively, time t, u(t) is an input data, d is the leak rate, which is
the rate of the term x(t − 1) that affects x(t). win, wres, and wout are theweights of the input, reservoir and output layer, respectively. The
activation function f is defined as the hyperbolic tangent function.
The reservoir layer follows the Echo State Property (ESP) [3] and its
weights are initialized by the following steps:
1. All weights of the reservoir layer are generated from a normaldistribution.
2. A spectral radius (the maximum eigenvalue of the weights) iscalculated and all the generated weights are divided by it.
3. All weights are multiplied by a constant value.
In standard RNNs, all weights are updated following the backpropagation through time algorithm [4]. On the other hand, in the ESN,
only the weights of the output layer are updated in one-shot learning
through ridge regression as follows:
wout = ( X T X + l I )-1 X TY (3)where X is the matrix of x(t) for all time-series, Y is the matrix of thesupervised signal for all time-series, and l is the regularization term.K. Honda and H. Tamukoh / Journal of Robotics, Networking and Artificial Life 7(1) 58–62

uk = m + s vk 59(5)where uk and yk are input and output at time k. a, b, g, d, m, s are
hyper parameters which we set to value (0.3, 0.05, 1.5, 0.1, 1, 0.5).
vk is the random number from 0 to 1. Training data contains 4000
time steps and test data contains 300 time steps, but we used only
the last 200 time steps data.
Figure 3 shows the prediction of 100–200 time steps of the quantized model where the number of reservoir neurons was 1000.
The black line represents the supervised signal and the blue line
represents output of the quantized model. The quantized model
was able to reproduce NARMA10. Figure 4 shows the MSE of thesupervised signal and outputs of each model, with varying in thenumber of neurons in the reservoir. The accuracy of the quantizedmodel was similar to conventional model.

Figure 1 | Echo state network.

3.2. Fixed Point
Generally, computations are conducted with floating point numbers, which are an exponential representation, and can represent
a wide range of numbers. A circuit using floating point numbers
is more complex as it requires many FPGA resources. In contrast, although the fixed-point representation can only represent
a narrow range of numbers, the circuit resources is less complex
compared with that using the floating point.

Figure 2 | Circuit of neuron.

3. HARDWARE-ORIENTED ESN
There are certain limitations of FPGA’s resources, therefore, wehave to modify the algorithms to suitable for FPGA implementation. In this paper, we design a circuit for the ESN by following
three methods.

3.3. Sequence Product–Sumof Output Layer
As shown in Figure 5, general product–sum operations can be represented by a tree structure. Using this representation, the numberof adders and multipliers increases with the number of neurons.

3.1. Quantization
One way to reduce the complexity of a circuit is using quantizedvalues that are able to simplify the computation while maintaining
its accuracy [5].
Therefore, we calculated the outputs of the reservoir layer[Equation (1)] using quantized weights. Generally, the weightsof input and reservoir layers are real numbers resulting inseveral DSPs to compute real number multiplications. Therefore,
we transformed the real valued weights to ternary values: 0 or ±1.
Furthermore, the accuracy by using this quantization for bothtraining and prediction mode are maintained.

Figure 3 | Output of quantized model.

The circuit of the neuron is shown in Figure 2. Where n is thenumber of reservoir’s neurons. un and wn are inputs and weights ofinput and reservoir layers, respectively, and m is the bit width of theinput data. Furthermore, the circuit is able to calculate accumulate
operations using only AND and OR operations.
We have verified the accuracy of a quantized model and the conventional model. The task carried out to evaluate their performance
was NARMA10 [6] with equations as follows:yk +1 = a yk + b yk +9åyi =0k -i + g uk uk -9 + d (4)Figure 4 | MSE of each model.

K. Honda and H. Tamukoh / Journal of Robotics, Networking and Artificial Life 7(1) 58–62

60

Figure 7 | Circuits architecture of conventional model.

Figure 5 | Tree structure of the product–sum circuit.

Figure 6 | Sequence structure of the product–sum circuit.

Therefore, in this research, we sequentially calculated the product–
sum of the output layer. Figure 6 illustrates the product–sum operation by the proposed method, where Ai is an intermediate variable
that temporarily stores the accumulate value. As this method consists of only a single adder, multiplier, and register per neuron in
the output layer, the complexity of the circuit is reduced.

Figure 8 | Circuits architecture of proposed model.

Table 1 | Details of each circuit

4. FPGA IMPLEMENTATIONAs shown in Figure 7, the conventional model used two pipeline. In
process 1 and 2, the reservoir module calculates the state of a single
neuron in reservoir layer and stores it in memory. In process 3, thereservoir module repeats the process 1 and 2 for the rest of reservoir
neurons. In process 4, an output module calculates the output of asingle neuron in output layer by using tree structure. In process 5,
the output module repeats process 4 for the rest of output neurons.
Figure 8 illustrates the circuits architecture of the proposed model.
We implement the sequential structure of the product–sum circuit (as in Figure 6) in parallel for the output layer. Therefore, theproposed circuit is able to calculate a single neuron of reservoir
layer and output layer simultaneously in process 4. As a result, theproposed model processes more efficiency than the conventionalmodel. Table 1 shows the comparison between the conventionalmodel and the proposed model.

5. EXPERIMENTIn the experiment, we have created two types of circuits in orderto verify the effectiveness of the proposed circuit and comparedits calculation speed with those of the other devices. The task toevaluate their performance is the prediction problem of sine andcosine waves. The number of neurons in the input, reservoir andthe output layers were 2, 100, and 2, respectively, and the prediction
was computed in an FPGA. The target device is a Zynq UltraScale+
MPSoC ZCU102. Furthermore, the experiment was conductedwith an operating frequency of 200 MHz and a data width of 32-bits
operations [7]. Table 2 shows experimental conditions.

Operation
Conventional Floating point
Proposed
Fixed point

Weights of inputand reservoirlayers

Module of outputlayerReal values
Ternary values

Tree structure
Sequence structure

Table 2 | Experimental conditions
ToolSDSoC 2018.3

Target device
Clock frequency

Zynq UltraScale+ MPSoC ZCU102
200 MHz

6. RESULTS
Figure 9 shows the prediction of the conventional and proposedcircuits. The black, blue, and red lines represent the supervised
signal, prediction of the conventional circuit and prediction of theproposed circuit, respectively. Both circuits were able to reproduce sine and cosine waves. Tables 3 and 4 shows the utilizationof resources for the conventional and proposed circuit, respectively. The proposed method was able to reduce the overall useof resources approximately 50%. Table 5 shows the comparison
between electric energy of conventional circuit and proposed circuit. The proposed method reduced the electric energy consumption by approximately 80% compared with the conventional one.
Table 6 shows a comparison between the computation speed of theFPGA and other devices. The proposed circuit was approximately
25 and 340 times faster than a desktop CPU and embedded CPU,respectively.

K. Honda and H. Tamukoh / Journal of Robotics, Networking and Artificial Life 7(1) 58–62

61Table 6 | Computation speed of devices
Platform
CPU 3.2GHz (i7-8700)
Embedded CPU 1.2GHz (Quad Arm
Cortex-A53)
FPGA 200MHz (XCZU9EG-2FFVBG1156E)

Latency (ms)5.215
68.1230.200

possible while the circuit resources were reduced. To achieve this,
fixed-point computation, quantification of weights, and sequential
product–sum computation techniques were used.
In the future, it is expected to apply the proposed circuit and methods to embedded systems such as automobiles and robots.

CONFLICTS OF INTEREST

Figure 9 | Output of circuits.

The authors declare they have no conflicts of interest.

Table 3 | Utilization of resources for the conventional circuit

BRAM_18k
DSP_48E
LUT
FF

Used

Total

Utilization

106519
60557
96556

912
2520
274,080
548,160

11.57
20.48
22.07
17.55

This research is supported by JSPS KAKENHI grant number
17H01798.

REFERENCES

Table 4 | Utilization of resources for the proposed circuit

BRAM_18k
DSP_48E
LUT
FF

ACKNOWLEDGMENT

Used

Total

Utilization

48
2028933
44021

912
2520
274,080
548,160

5.26
0.79
10.56
8.03

[1]
[2]

[3]
Table 5 | Electric energy consumption of each circuits

Conventional
Proposed[4]Latency (ms)Power (w)Electric energy(w*ms)

0.43
0.20

1.46
0.67

0.63
0.13

[5]

[6]

7. CONCLUSION
We were able to successfully adapt the circuit to enhance ESN computation in the FPGA. As a result, high-speed computation was[7]

G.E. Hinton, S. Osindero, Y.W. Teh, A fast learning algorithm fordeep belief nets, Neural Comput. 18 (2006), 1527–1554.
H. Jaeger, The “echo state” approach to analysing and training
recurrent neural networks – with an Erratum note, German
National Research Center for Information Technology GMD,Bonn, Germany, Technical Report, 148 (2001), 13.
I.B. Yildiz, H. Jaeger, S.J. Kiebel, Re-visiting the echo state property, Neural Netw. 35 (2012), 1–9.
P.J. Werbos, Backpropagation through time: what it does and how
to do it, Proc. IEEE 78 (1990), 1550–1560.
Y. Aratani, Y.Y. Jye, A. Suzuki, D. Shuto, T. Morie, H. Tamukoh,
Multi-valued quantization neural networks toward hardware
implementation, IEEE International Conference on Artificial Lifeand Robotics (ICAROB), 22 (2017), 132–135.
A.F. Atiya, A.G. Parlos, New results on recurrent network training:
unifying the algorithms and accelerating convergence, IEEETrans. Neural Netw. 11 (2000), 697–709.
XILINX, Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit,available from: https://www.xilinx.com/products/boards-andkits/ek-u1-zcu102-g.html (accessed December 1, 2019).

K. Honda and H. Tamukoh / Journal of Robotics, Networking and Artificial Life 7(1) 58–62

62

AUTHORS INTRODUCTION
Mr. Kentaro Honda
He received Master of Engineering degreefrom Kyushu Institute of Technology in2019. His research interests include neuralnetwork and digital circuit.

Associate Prof. Hakaru Tamukoh
He received the B.Eng. degree from Miyazaki
University, Japan, in 2001. He received theM.Eng. and the PhD degree from Kyushu
Institute of Technology, Japan, in 2003 and2006, respectively. He was a postdoctoral
research fellow of 21st century center ofexcellent program at Kyushu Institute ofTechnology, from April 2006 to September
2007. He was an Assistant Professor of Tokyo University ofAgriculture and Technology, from October 2007 to January
2013. He is currently an Associate Professor in the Graduate
School of Life Science and System Engineering, Kyushu
Institute of Technology, Japan. His research interest includes
hardware/software complex system, digital hardware design,neural networks, soft-computing and home service robots. He
is a member of IEICE, SOFT, JNNS, IEEE, JSAI and RSJ.