FDL, a Deterministic
100 Mbytes/sec Data Link
J. Bovier, J-F. Gilot
Creative Electronic Systems

Abstract
Data transfers in real time applications needs more than just a speed increase to meet the new
constraints of data acquisition and control systems. Data gathering from multiple sources, labelling,transfer scheduling and data scattering to multiple destinations are of primary importance.

Over the past few years the power of processing units used in real time data acquisition and control systems has increased bya factor of 100 (from 1 mips to 100 mips) thanks to the availability of RISC-based microcontrollers with clock rates well
above 100 MHz. Instructions are now executed in less than 10 ns if cache memory is used. In order to make the best use ofthis power, external agents are needed to move data to and from processors’ memories.
Intelligent I/O boards have been provided to relieve the processing units of direct instrument control and data transfer
protocol management. They pack data in buffers ready to be grabbed for processing. The missing piece in this scheme is anautonomous system able to gather data from I/O boards and scatter it to processors’ memories at speeds matching those ofthe processors.
The Fas t Data Li nk (FDL) i s a s et of hardware and s oftware tool s des i gned to trans fer data
"i ntel l i gentl y" between mul ti pl e s ources and des ti nati ons .
The backbone of the FDL is a multimaster/multidrop (up to 15 nodes) copper link synchronised at 50 MHz. The physicalsupport is a cable composed of 25 twisted pairs (16 data - 2 parity - 7 control signals) extending over a maximum distanceof 30 meters.
SOURCEGATHERINGLABELLING

BRIDGETRANSFERSOURCESCHEDULINGROUTING

SOURCEDESTINATIONSCATTERINGDESTINATIONDESTINATION

Figure 1. FDL General Concept
The client/server approach has been chosen in designing FDL interfaces. Upon request of a client or occurrence of an
external event, the interface gathers data from data acquisition busses, dual ported memories or registers and stores it locallyin an intermediate buffer. Information is then added to the data packet before transfer over the link so that each data packet
contains routing parameters. In the destination nodes data packets are routed to their final destinations.
A point-to-point fibre optic link has been designed to interconnect at full speed local FDL networks over long distances. Up
to 15 optical connections can be implemented from a local network.
Informati on i s trans mi tted over the FDL i n uni ts of "cel l s ". A cel l i s a s equence of 18 words (of 16bi ts ) emi tted by the current mas ter every 20 ns .

The link is source-synchronised. The current master drives the clock signal and sends one data word every clock cycle. The
cell synchronisation signal indicates to the slave nodes the beginning of an 18 word cell. The first two words of every cell
contain routing information (destination node identifier, cell identifier, destination buffer identifier, etc.) They are generated
by the FDL master port and decoded by the FDL slave port. The payload of a cell is 32 bytes.

D15
word #0
word #1

D0

bridge_id
prio XX

node_id
XtypeX

sclcX

X

cell_idbuffer_id

word #2
payload 32 bytes
word #17

Figure 2. Cell S tructure

data #N

data #idata #0

header32 >= N >= 0

data from source

destination parameters

Figure 3. Data Packet S tructure

For data transfer, cells are organised in packets of up to 1 kbyte (32 cells maximum) preceded by a header cell containing
parameters for the final destination. Thus routing and administrative information imposes overhead of 14%.
Single cells are used for FDL management. The FDL offers two levels of priority for cell transmission to allow
management cells to take precedence over data cells. When no node has cells to transmit, the current FDL master transmits
empty cells to maintain link synchronisation.
The FDL i s a determi ni s ti c l i nk. The arbi trati on mechani s m guarantees a defi ned data trans fer rate
and the maxi mum l atency for bus mas ters hi p attri buti on.
To offer fair access to the link, bus arbitration is assumed by the current master. This implements a rotating arbitration. A
requesting node asserts the Bus Request signal at the beginning of a cell; the current master grants the bus at the beginningof the next cell. This allows the next requesting node in the arbitration daisy chain to own link mastership before the end of
the second cell following the Bus Request assertion. This mechanism fixes the latency for a change of master at 720 ns +time of flight between current master and requester (40-150 ns).
l atency: < 1 µ s

In addition the link shares ownership fairly amongst requesting nodes, thus guaranteeing minimum latency for data transfer
completion. For a fully loaded link the overhead for arbitration is one time of flight every two cells. Depending on thephysical length of the link the average arbitration overhead is then between 6 and 20%.
The nominal transfer rate of the link is 100 Mbyte/sec x (1.0-Overhead fraction). The average transfer rate is the nominaltransfer rate divided by the average number of requesting nodes.
DATA
FDL
PORT

F_BMA

BUFRAM
256k

V_BMA

VME
MASTER
PORT

VME
SLAVE
PORT
V
M
E
B
U
S

trigger

R3052
R_CPU

SYSRAM
512k

VME
SLAVE
PORT

CLIENTVME
MASTERPORT

FDL8050

F
F
D
DL
L
L
LI
NI
N
K
K

DATA
FDL
PORT

F_BMA

BUFRAM
256k

V_BMA

VME
MASTER
PORT

VME
SLAVE
PORTV
M
E
B
US

triggerR3052
R_CPU

SYSRAM
512k

VME
SLAVE
PORT

FDL8050

Figure 4. FDL8050 VME interface
The FDL8050 i nterface i s a VME s erver des i gned to trans fer data between VME crates at rates up to70 Mbyte/ s ec. A cl i ent i s any VME mas ter abl e to acces s the FDL8050's VME s l ave port.
The central part of the VME/FDL interface is a fast dual ported memory (BUFRAM) accessed by two block movers at 200
Mbytes/sec. The first one (V_BMA) gathers/scatters data from/to the VME bus. It packs the data which it reads into blocks

up to 1 kbyte in size. The second one moves data packets between the BUFRAM and the link. It handles link protocol andpacket fragmentation into cells.
The two block movers are controlled by a RISC microprocessor (R_CPU (mips R3052)) managing data transfer andcommunication with clients. The system memory (SYSRAM) is accessible from both the V_BMA and the VME bus. Thisfeature is used to implement a 64 kbyte mirror memory. External trigger lines are connected to the microprocessor. They
are used for event generation.
The trans mi s s i on mechani s m us es reques t fi fos (of 256 words ) to keep both BMAs bus y wi th data
packets for trans fer.
To trigger a V_BMA data transfer, the R_CPU prepares a transfer descriptor in the SYSRAM and then posts the descriptor
index in the V_BMA request fifo. The V_BMA is built around an R3051 microprocessor in order to handle complex VME
transfer descriptors. It checks for block boundaries and recovers from bus errors in less than 1 µs. At the end of the transfer
the V_BMA updates the descriptor with the transfer status and writes the descriptor index into the V_BMA status fifo. If
required, control information such as sequence number, time stamp, etc. is added to the data packet.
The R_CPU builds the header cell with final routing parameters in the HEADRAM, and posts the cell index in theF_BMA request fifo. The F_BMA then transfers the header cell followed by the associated data cells over the link. At theend of the transfer the cell index is written in the F_BMA status fifo.
The destination node catches cells containing its own node identifier and stores them in the BUFRAM. The HEADRAM
holds the header cell and the F_BMA status fifo the cell index. The R_CPU reads the header cell and builds a transfer
descriptor for the V_BMA. It then triggers the V_BMA to transfer the data packet from BUFRAM to VME. A single cell
containing an acknowledge from the data packet transfer is sent back to the transmitter for flow control.
A 12-s tage trans mi s s i on pi pel i ne has been i mpl emented to maxi mi s e data throughput whi l econtrol l i ng arri val of data packets . Up to 8 VME trans fers can be concurrentl y acti ve.
High priority single cells (service cells) are transmitted over the link for system management; they take precedence over datacells for link access A dedicated F_BMA request fifo is used to post these cells, the HEADRAM contains room for 256 ofthem, which are directly handled by the R_CPU. These service cells are used for data transfer acknowledge, remote request,
event generation and time synchronisation.
The FDL supports time stamping of data packets and transfer synchronisation (isochronous mode). To this end every FDLinterface has a hardware clock running at 1 MHz. The interface maintains a calendar time accessible by VME clients. One
node in an FDL system keeps the global time reference and every 10 ms it broadcasts this reference value to other nodes.
Time synchronisation service cells are directly handled by the interface hardware.
S peci al ti mers (25 MHz) on FDL i nterfaces al l ow a ti me s ynchroni s ati on of al l nodes wi th anaccuracy of 1 µ s .
The FDL supports two scheduling modes for data transfers. In the asynchronous mode they are triggered by an external event(client request, VME interrupt, front panel trigger, etc.) and are handled by the server on a first-in first-out basis. In theisochronous mode data transfers are activated periodically at a user-chosen frequency. The two modes of data transfers can bemixed.
When the isochronous phase occurs, all asynchronous data transfers are suspended and a pre-defined transfer descriptor isgiven to the R_CPU for processing. At the end of the isochronous phase (the duration is programmable) asynchronous datatransfers are resumed. A special F_BMA request fifo is used to post isochronous data transfers in order to allow instantswitching between isochronous and asynchronous modes.
The isochronous mode has been implemented to allow all FDL interfaces to activate special data transfers periodically at thesame time. This ensures the time coherence of the data collected by the system.

SYSRAMCONTROLBLOCK
REQUEST BLOCK(64 bytes)
statuscommandmode

req_blk #0

REQUEST
TABLE

req_blk #iparameters

req_blk #127

VM
E

REGISTERSRESETFREE_REQPEND_REQ

Figure 5. FDL8050 programmer' s interface
The VME clients interact with the FDL interface through a shared data structure located in the SYSRAM. Two hardware
registers (FREE_REQ and PEND_REQ) are used to control access to that structure. Remote reset of the interface isactivated by writing in the reset register. Up to 127 connection channels can be opened at a time to these VME clients.
Each communication channel is controlled by a 64 byte request block in which the client posts the transfer parameters.
The FDL firmware supports direct and indirect data transfers. For direct data transfers the client provides all routing
parameters in the request block. In the indirect mode, the client provides the reference of a transfer descriptor. Transfer
descriptors are linked lists of elementary transfer parameters (chained list of buffers, direct rings, indirect rings, etc.). Thesetransfer descriptors can be pre-defined through a resource creation mechanism, or can be read by the FDL at transfer time.
The FDL firmware supports the creation of remote transfer descriptors in order to handle indirect transfers in destinations.
The server supports 8 levels of priority for client requests and handles programmable time-out. Broadcast to multiple VME
destinations has been emulated in order to implement protocols such as TCP-IP.
Device drivers for the most popular operating systems have been written to control the FDL8050 interface. If the driver islinked with the TCP-IP layer, a workstation can be used as a gateway to pass TCP-IP packets over the FDL. This allows aVME multiprocessor system to be controlled from a local area network while fast data transfers are going on between VME
crates.
Interfaces to other busses such PCI, CAMAC, FASTBUS, etc. are under development in order to build complete dataacquisition and control systems. We feel that PCI will be a standard in future workstations. The PCI/FDL interface will beused to push data directly into workstation memories under control of the operating system, so the data will be directly
available to data processing software.
We will also provide embedded FDL interfaces to be directly implemented in instruments and detectors in which a register ora dual-ported memory will be used to post data for transfer over the link.
The performance of a real time system depends mainly on three parameters: the processing power, the interrupt
response and the data transfer rate. The only way to maximise all these parameters simultaneously is to make them
independent by dedicating specialised hardware to each of them. The FDL has been designed to maximise data transfer
without compromising processing power or interrupt response.