Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________Synchronization of Pulse-Coupled Phase Oscillators over Multi-FPGA Communication Links
Dinda Pramanta, Takashi Morie, Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu 808-0196, Japan
E-mail: dinda-pramanta@edu.brain.kyutech.ac.jp, morie@brain.kyutech.ac.jp, tamukoh@brain.kyutech.ac.jp

Abstract
This study proposes an implementation of pulse-coupled phase oscillators over multiple
field-programmable-gate-array (FPGA) communication links. Two FPGAs are connected by a gigabit transceiver
and a First-In First-Out interface. To verify the effect of communication delay between FPGAs on oscillator
synchronization, we implement four oscillators on the multi-FPGA platform. We have successfully observed
synchronization over two FPGAs correctly, despite of a 0.1 μs communication delay. The measurement results
show that first spike synchronization requires 12.47 μs with a 3.2 Gbps communication throughput.
Keywords: pulse-coupled oscillators, synchronization, Winfree model, FIFO, GTX, multi-FPGA.

VLSI, we implemented a region-based coupled Markov
random field model using pulse-coupled phaseoscillators and applied it to image region segmentation.
Measurement results showed that the proposed device
achieved high efficiency in terms of performance per
power consumption.9 In the FPGA Implementation, we
introduced a parameterized design that enabled us toemulate various sizes and interconnections ofpulse-coupled oscillator networks.11 From this previouswork, we have confirmed that spike-based computation
is suitable for realizing brain-like VLSI. However, alarge-scale network cannot be implemented on a single
chip owing to limited circuit resources.
A multi-FPGA-based design would overcome theresource limitations of a single FPGA12 and enable us toimplement large-scale pulse-coupled phase oscillator
networks. However, communication delays between
FPGAs might affect synchronization between oscillators,
which is the most important function of pulse-coupledphase oscillator systems. Therefore, high-speed
communication is required to implement oscillator
systems over multiple FPGAs, and the synchronization
state over a multi-FPGA system should be verified.

1. IntroductionPulse-coupled phase oscillator networks are a typeof spiking neural network and a promising model for
realizing brain-like very large-scale integration (VLSI).
Mathematical models of coupled oscillators have been
proposed and intensively studied.1-6 In these models,mutual interactions between oscillators are represented
by a simple phase sensitivity function, and the couplings
between oscillators are determined by the timings of
spike pulses output by each oscillator. Based on thisprinciple, nonlinear processing is performed by a single
transition in spike timing. In addition, only one-bit
signal lines are required between oscillators for spike
pulse communication. These features allow us toachieve efficient hardware implementation, andhigh-performance
andlow-power
intelligent
information processing VLSI systems can beconstructed based on spike-based computation.
To implement spike-based computation anddemonstrate its effectiveness, we have proposed analog
VLSI circuits7-9 and digital circuits using field
programmable gate arrays (FPGAs).10,11 For the analog

Copyright © 2017, the Authors. Published by Atlantis Press.
This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
91

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________In this study, we propose an implementation ofmulti-FPGA communication links for pulse-coupledphase oscillators. First, we design a digital circuit ofpulse-coupled oscillators based on the Winfree model.2
Second, we propose a method for communication
between two FPGAs using these digital oscillators,
First-In First-Out (FIFO) interface, and a serial
connection using a gigabit transceiver (GTX).13 We
employed two FPGA boards for the multi-FPGA
platform and connected them by a GTX serial
connection. Experimental results show that afour-oscillator network reached a synchronization state,
and first spike synchronization over two FPGAs took
12.47 μs, with a datastream bitrate of 3.2 Gbps. This
study is an extension of our previous work.15

𝑆𝑆𝑆𝑗 (𝑡) = �= 𝜔𝑖 + 𝑍(𝜙𝑖 )𝑆𝑆𝑆(𝑡),(4)2.2. Pulse-coupled phase oscillator dynamics
Conceptually, the oscillator operation is based on
the connections between each oscillator and its
neighbors. This model is determined by only local
interactions but can generate various phenomena in alarge oscillator network based on the coupling formulas
shown in Eqs. (1) and (2).

The Winfree model2 provides an efficient way ofdesigning pulse-coupled oscillator circuits. The
fundamental relation for pulse-coupled oscillators canbe expressed as follows:
𝑑𝑑𝑖𝑖 𝜙𝑗 (𝑡) = 𝜙𝑡ℎ𝑜𝑜ℎ𝑒𝑒𝑒𝑒𝑒𝑒

The phase value at the next time step is calculated byadding the current phase value, natural angularfrequency, and the sum of products of the phasesensitivity function and input pulses. The naturalangular frequency 𝜔𝑖 is assumed to be constant. Theoscillator outputs a spike pulse when the phase variable
reaches threshold value 𝜙th and then resets; 𝜙𝑖 = 0.
Using the above discretization, the pulse-coupled phaseoscillator model can be implemented with simple digital
logic circuits.11

2. Pulse-coupled Oscillator Model𝑑∅𝑖1,0,(1)

where 𝜙i is the i-th phase variable (2π periodic), 𝜔𝑖 is
the i-th natural angular frequency, and 𝑍(𝜙𝑖 ) is a phasesensitivity function, which gives the response of the i-thoscillator. Inputs from other oscillators, Spk(t), areassumed here to give the following pulse input:
𝑆𝑆𝑆(𝑡) =𝐾0𝑁∞∑N
j=1 ∑n=1 𝛿�𝑡 − 𝑡𝑗𝑗 �,(2)Where 𝐾0 is the coupling strength, N is the number ofoscillators, and 𝑡𝑗𝑗 is the firing time. Mathematically,δ is a Dirac delta function that represents the timing of
input spikes without a pulse width. However, in
physical hardware, spike pulses have a definite width ∆t,during which 𝜙i is updated according to the value of𝑍(𝜙𝑖 ).
2.1. Discretized model

To implement Eqs. (1) and (2) in digital hardware,
the model of pulse-coupled phase oscillators is
discretized as follows.
𝜙𝑖 (𝑡 + 1) = 𝜙𝑖 (𝑡) + 𝜔𝑖 +K0
N∑Nj=1 Z(𝜙𝑖 )𝑆𝑆𝑆𝑗 (t)Fig. 1 Pulse-coupled phase oscillators. (a) Coupled network
schematic. (b) Positive and negative updates.

(3)

92

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________𝑐𝑙𝑜𝑐𝑆
𝑢𝑆𝑑𝑎𝑡𝑒CNT

SPK_GEN

2

Spki

sign

𝑜𝑇𝑜𝐹𝐼𝐹𝑂

clock

𝑐𝑀𝑖𝑑0

Spkj

𝑐𝑀𝑆𝐵

𝑐𝑀𝑖𝑑1

Oscillator
CircuitupdateUpdatecircuitcMSB, cMid0, cMid1

Functiongenerator

SpkiZpZn

oToFIFO

Fig. 2 Pulse-coupled phase oscillator circuit design.

FPGA-1iDSoDS

oToFIFO

WrEn

RdEn

iEnable

Full

FIFO

Pulse-coupled
Oscillator
ModuleoSpikeGTX

To FPGA-2oEmp

Fig. 3 Multi-FPGA communication system.

Figure 1 shows a schematic of two pulse-coupled phaseoscillators and a timing diagram that explains their
updates. The two oscillators are coupled by the spikes
𝑆𝑆𝑆𝑖 and 𝑆𝑆𝑆𝑗 . We assume that the phase sensitivity
function is 𝑍(𝜙𝑖 ) = −sin(𝜙𝑖 ). Synchronization occursby updating the pulse timing. There are two update
phases for each oscillator: positive updating andnegative updating. Positive updating occurs whenever
there is spike input (𝑆𝑆𝑆𝑖 ) to another oscillator 𝜙𝑗 andthe leading condition triggers the function Z( 𝜙𝑗 ),making the timing of the maximum value of 𝜙𝑗 earlier.
Negative updating occurs when there is spike input( 𝑆𝑆𝑆𝑗 ) to another oscillator 𝜙𝑖 and the lagging
condition triggers the function Z(𝜙𝑖 ), making the timingof the maximum value of 𝜙𝑖 later.
3.

variable 𝜙i and counts clock inputs to implement 𝜔𝑖in Eq. (3).
In this design, each time step t in the discretizedmodel corresponds to a clock cycle. The signals cMSB,cMid0, and cMid1 determine the shape of the function
Z(𝜙𝑖 ) and are used in the function generator circuit. Thefunction generator circuit combines these signals tooutput Zp and Zn. The update circuit receives Zp and Zn,
and Spkj that is the output received from the otheroscillator, and outputs an update signal.
3.2. Multi-FPGA communication system
Figure 3 shows a block diagram of the multi-FPGAcommunication system. A FIFO interface is inserted
between the pulse-coupled oscillator module and theGTX.
The pulse-coupled oscillator circuit module has
three main channels, as shown in Fig. 3. For every
spike-data output (oSpike = 1) from the oscillator
module, a data valid signal (oToFIFO = 1) is sent at thesame time to the FIFO interface.
The FIFO module is used as an interface between
two clock domains, i.e., the oscillator and GTX clocks.
When the FIFO receives a spike (iDS = 1) with a write
enable signal (WrEn = 1), the FIFO writes the spike into
its internal memory. When the Full flag is high (Full =Hardware Architecture Design

3.1. Digital design of pulse-coupled oscillators
The hardware architecture of the pulse-coupledphase oscillator is shown in Fig. 2. It comprises anoscillator circuit, a function generator circuit, and an
update circuit.11 The oscillator circuit contains an n-bit
counter (CNT), a spike generator (SPKGEN), andcombinational circuits. The CNT represents a phase93

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________1), the FIFO’s internal memory is full and it will not
accept any further data for writing (iEnable = 0).

hardware connections between the two FPGAs areshown in Fig. 4. A SubMiniature version A (SMA)
cable connection was used for the physical environment
hardware between the two FPGAs’ GTX channels. The
ML605 boards provide a differential signal connection
for each Tx and receiver (Rx). Therefore, four SMA
cables are required to connect to the two FPGA boards.

The GTX is a configurable transceiver (Tx) and istightly integrated with the programmable logic
resources of the FPGA. The FIFO interface is connectedto the GTX and transfers spikes from the FPGA-1 side
to the FPGA-2 side. To configure the GTX, weemployed a free and open high-speed communication
protocol called Aurora.13 It controls FIFO’s interfaces(RdEn, oDS, and oEMP) to read data from the FIFO andprovides serial communication between the FPGAs atup to 10 Gbps.14

4. Experimental Results
In general, physical connections between FPGAs
incur communication delays. If the communication
delay affects the synchronization states of thepulse-coupled phase oscillator networks, we cannot
implement a large-scale network using the multi-FPGA
platform. Therefore, to show the feasibility of themulti-FPGAplatform,
weexaminedbasicsynchronization phenomena in pulse-coupled phaseoscillator networks.

3.3. Multi-FPGA platform using two FPGAs
We employed two Virtex-6 XC6VLX240T ML605
Xilinx FPGA boards as the multi-FPGA platform. TheFPGA 1

TX

4.1. Performance of multi-FPGA communication
To evaluate the performance of the serial connection,
we measured the throughput and latency of themulti-FPGA communication. We implemented a 16-bit
incremental data generator (counter) in FPGA-1.
FPGA-2 received the incremental data and looped themback to FPGA-1. The clock frequency of the FPGAs
was set to 200 MHz. Figure 5 shows the measured
throughput results for the multi-FPGA platform.
“Counter” represents the number of clock cycles for
FPGA-1. It counted up to 200 million every second
because the clock frequency is 200 MHz. “Throughput”
represents the number of valid 16-bit data values thatwere received in a given time.
From these results, we can calculate the data rate as
16 bit * 199719549/s = 3195 Mbps ≈ 3.2 Gbps. Inaddition, we measured the difference in the incremental
data between the Tx and Rx modules to find the latencyof the multi-FPGA communication. The results showthat the total latency from Tx to Rx was 20 counters or
0.1 μs.

GTXRXFPGA 2

Fig. 4 Multi-FPGA platform using two
Virtex6-XCVL240T ML605 boards.
△X

1sec

△Y

Counter

XXX

…

XXX

Throughput

XXX

…

XXX

△Y
Counter

200000000

200000001…

XXX

Throughput

199719549

199719550…

XXX

Fig. 5 Throughput measurement results using a 200 MHz clock of the Virtex6-XCVL240T ML605.

94

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________Spk1

1

Spk1
Spk2
Spk3

2

Spk2

Spk3

30 2 4 6 8
3.08 μsec

10 12 14 16 18[μsec]

20.01μsec

(b)

(a)

Fig. 6. Pulse-coupled spiking oscillator with three in-phase oscillators. (a) Three oscillators with rotary connections, and(b) synchronization results from ChipScope.
Table 1. Device utilization.

4.2. Synchronization inside one FPGA

Resource
Slice Register
Look-up
TablesIOBsTo demonstrate the basic operation ofpulse-coupled oscillator networks in one FPGA, we
implemented a simple rotary network, as shown in Fig.
6(a). Here, we employed ChipScope, which is anin-circuit debugger provided by Xilinx11 to observe
spike pulses. Figure 6(b) shows the state of the FPGA,
starting from the beginning at time zero (0 μs). In thiscase, the first spike occurred at 3.08 μs and it reached asynchronization state at 20.01 μs.

FPG A 1 - Sid eSpk4( Au ro ra )GTXGTXRX-sid e ( A u r o r a )

FI FO

FI FO

Spk3

∅4( Au ro ra )

FI FO

∅1

GTX

( Au ro ra )

Spk1

∅2

Spk2

Spk3

∅3150720

101

600

4.4. FPGA design summary
Table 1 shows synthesized results for the total
resources used on one FPGA. Two oscillators were
implemented at the hardware level with the FPGA
synthesizer in Xilinx Tools. Using ISE Design Suite
software information, we can report that the proposedmodel circuit was implemented with a maximum
frequency of 434.972 MHz.

TX-sid eFig. 7. Implementation of four oscillators over twoFPGAs.

Spk1
Spk3
Spk4348

Figure 7 shows an implementation of four
oscillators on the multi-FPGA platform. Figure 8 shows
the measurement results of synchronization in FPGA-1
using ChipScope. These results show that in-phasesynchronization was correctly observed at 12.47 μs. We
have therefore confirmed that our pulse-coupled phaseoscillator networks were able to reach a synchronization
state over the multi-FPGA platform, even thoughcommunication between the FPGAs included a 0.1 μs
latency.

FPG A 2 - Sid eGTXFI FO

Spk1

Available301440

4.3. Synchronization over two FPGAsRX-sid e

GTX Clo ck
D o m a inTX-sid e

Used4160 0.034 μsec

12.5

[μsec]

Fig. 8. Measurement results for four pulse-coupled phase oscillators over two FPGAs using ChipScope. The signals Spk1,Spk3, and Spk4 correspond to those in Fig. 7.

95

Journal of Robotics, Networking and Artificial Life, Vol. 4, No. 1 (June 2017) 91–96___________________________________________________________________________________________________________

5. Conclusion

7. D. Atuti, N. Kato, K. Nakada and T. Morie, CMOS
circuit implementation of a coupled phase oscillator
system using pulse modulation approach, in European
Conference on Circuit Theory and Design (ECCTD),(Seville, Spain, 2007), pp. 827–830.
8. K. Matsuzaka, T. Tohara, K. Nakada and T. Morie,Analog CMOS circuit implementation of a pulse-coupledphase oscillator systemand observation
ofsynchronization phenomena, Nonlinear Theory and Its
Applications, IEICE 3(2) (2012) 180–190.
9. K. Matsuzaka, H. Tanaka, S. Ohkubo and T. Morie,
VLSI implementation of coupled MRF model using
pulse-coupled phase oscillators, Electron. Lett. 51(1)(2015) 46–48.
10. Y. Suedomi, H. Tamukoh, M. Tanaka, K. Matsuzaka andT. Morie, Parameterized digital hardware design ofpulse-coupled phase oscillator model toward spike-based
computing, in 20th International Conference on Neural
Information Processing (ICONIP2013), vol. III, (2013),
pp. 17–24.
11. Y. Suedomi, H. Tamukoh, K. Matsuzaka, M. Tanaka andT. Morie, Parameterized digital hardware design ofpulse-coupledphase
oscillator
networks,
Neurocomputing 165 (2015) 54–62.
12. J. Li, Y. Atsumari, H. Kubo, Y. Ogishima, S. Yokota, H.
Tamukoh and M. Sekine, A multidimensional
configurable processor array - Vocalise, IEICE Trans. on
Information and Systems E98-D(2) (2015) 313–324.
13. Xilinx Inc, Aurora 8B/10B Protocol Specification, (2010),
http://www.xilinx.com/support/documentation/ipdocumentation/au-rora8b10bprotocolspecsp002.pdf.
14. A. Athavale and C. Christensen, High-Speed Serial I/O
Made Simple. Xilinx, Inc., 1.0 edition, (2005),
http://www.xilinx.com/publications/archives/book-s/seria
lio.pdf.
15. D. Pramanta, T. Morie and H. Tamukoh, Implementationof multi-FPGA communication using pulse-coupledphase oscillators, Proc. 2017 Int. Conf. on Artificial Life
And Robotics (ICAROB2017), (2017), pp. 128–131.

In this study, we proposed an implementation ofmulti-FPGA communication links for pulse-coupledphase oscillators. In our experiments, two FPGA boards
were used to implement a four-oscillator network, andfirst spike synchronization over two FPGAs required
12.47 μs with a datastream bitrate of 3.2 Gbps. From
our experimental results, we verified that thepulse-coupled phase oscillators synchronized over twoFPGAs via high-speed serial communication with a 0.1
μs delay.
In future work, we will increase the number ofFPGAs and implement large-scale pulse-coupled phaseoscillator networks, and then apply them to engineering
applications such as image processing for service
robots.
Acknowledgments
This research was supported by JSPS KAKENHI
Grant Numbers 17H01798, 26330279, and15H01706.
References
1. Y. Kuramoto, Chemical Oscillation, Waves, andTurbulence (Springer, Berlin, 1984).
2. A. T. Winfree, The Geometry of Biological Time
(Springer, New York, 1980).
3. R. Mirollo and S. Strogatz, Synchronization ofpulse-coupled biological oscillators, SIAM J. Appl. Math.
50(6) (1990) 1645–1662.
4. Y.
Kuramoto,
Collective
synchronization
ofpulse-coupled oscillators and excitable units, Physica D
50(1) (1991) 15–30.
5. F. C. Hoppensteadt and E. M. Izhikevich, Weakly
Connected Neural Networks (Springer, New York, 1997).
6. E. M. Izhikevich, Dynamical Systems in Neuroscience:
The Geometry of Excitability and Bursting (MIT Press,Cambridge, MA, 2007).

96