# Architecture and performance of the KM3NeT front-end firmware

Sebastiano Aiello, Arnauld Albert, Sergio Alves Garre, Zineb Aly, Fabrizio Ameli, Michel Andre, Giorgos Androulakis, Marco Anghinolfi, Mancia Anguita, Gisela Anton, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sebastiano Aiello, Arnauld Albert, Sergio Alves Garre, Zineb Aly, Fabrizio Ameli, et al. Architecture and performance of the KM3NeT front-end firmware. Journal of Astronomical Telescopes Instruments and Systems, 2021, 7 (1), pp.016001. 10.1117/1.JATIS.7.1.016001. hal-03210384

HAL Id: hal-03210384

https://hal.science/hal-03210384v1

Submitted on 8 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Architecture and performance of the KM3NeT
front-end firmware

Sebastiano Aiello,ª Arnauld Albert,ᵇ,c Sergio Alves Garrep,ᵈ Zineb Alyq,ᵉ

Fabrizio Ameli,f Michel Andre,g Giorgos Androulakis,h Marco Anghinolfi,i

Mancia Anguita,j Gisela Anton,k Miquel Ardid,l Julien Aublin,m

Christos Bagatelas,h Giancarlo Barbarino,⁰,p Bruny Baret,⁰

Suzan Basegmez du Pree,p Anastasios Belias,h Meriem Bendahman,q

Edward Berbee,p Ad M. van den Berg,r Vincent Bertinp,ᵉ

Vincent van Beverenp, p Simone Biagip, s Andrea Biagioni, f

Matthias Bissingerp, k Markus Boettcher,t Jihad Boumaazap, q

Mohammed Boutap, u Mieke Bouwhuisp, p Cristiano Bozza,v

Horea Brânzaş,w Ronald Bruijnp,x Jurgen Brunner,e Ernst-Jan Buis,y

Raffaele Buompanep,⁰,z Jose Busto,e Barbara Caiffip, i David Calvop, d,\*

Antonio Capone,f,aa Victor Carreterop, d Paolo Castaldip,ab,ac

Silvia Celli,f,aa,ad Mohamed Chababp, ae Nhan Chaup, m Andrew Chenp,af

Silvio Cherubinip, s,ag Vitaliano Chiarella,ah Tommaso Chiarusip, ab

Marco Circellap,ª Rosanna Cocimano,s Joao A. B. Coelho,m Alexis Coleirop, m

Marta Colomer Molla,d,m Stephane Colongesp, m Rosa Coniglione,s

Imanol Corredoirap, d Paschal Coyle,e Alexandre Creusotp, m Giacomo

Cuttone,s Antonio D' Amicop,p Antonio D'Onofrio,n,z Richard Dallierp,aj

Mauro De Palmap,ai,ak Irene Di Palmap, f,aa Antonio F. Dízaj, j

Dídac Diego-Tortosp, l Carla Distefano,s Alba Domip, e,i,k,al Roberto Donà,ab,am

Corinne Donzaud,m Damien Dornic,e Manuel Doerr,an Doriane Drouhin,b,c

Thomas Eberlp, k Ahmed Eddyamoui,q Thijs van Eedenp, p Daan van Eijk,p

Imad El Bojaddainip, u Dominik Elsaesser,an Alexander Enzenhoefer,e

Victor Espinosa Rosellp, l Paolo Fermanip, f,aa Giovanna Ferrarp, s,ag

Miroslav D. Filipović,ao Francesco Filippinip,ab,am Luigi Antonio Fuscep, e

Omar Gabellap, ap Tamas Galle, q Alfonso Andres Garcia Sotop, p

Fabio Garufip, n,o Yoann Gateletp, m Nicole Geißelbrecht,k

Lucio Gialanellap, n,z Emidio Giorgop, s Sara R. Gozzinip, q Rodrigo Gracia,p

Kay Grafp, k Dario Grasop, q Giuseppe Grelap, ar Daniel Guderian,as

Carlo Guidip, i,al Steffen Hallmann,k Hassane Hamdaouip, q Hans van Haren,at

Aart Heijboerp, p Amar Hekalop, an Juan Jose Hernández-Rey,d

Jannik Hofestädtp, k Feifei Huang,au Walid Idrissi Ibnsalihp, n,z Alin Ilioni,m

Giulia Illuminati,d Clancy W. Jamesp,av Peter Jansweijerp, p

Maarten de Jong,p Paul de Jong,p,x Bouke Jisse Jungp, p Matthias Kadler,an

Piotr Kalaczyński,aw Oleg Kalekin,k Uli F. Katz,k

Nafis Rezwan Khan Chowdhuryp, d Giorgi Kistaurip,ax Frits van der Knaap,y

Els N. Koffeman,p,x Paul Kooijman,x,ay Antoine Kouchner,m,az

Michael Kreterp, t Vladimir Kulikovskiy,i Robert Lahmann,k

Giuseppina Larosp,s Remy Le Breton,m Ornella Leonardip, s

Francesco Leone,s,ag Emanuele Leonorp,ª Jean Lesrelp, m

Giuseppe Levip,ab,am Massimiliano Lincetto,e Miles Lindsey Clark,m

Thomas Lipreap, aj Alessandro Lonardo,f Fabio Longhitanop, a

Daniel Lopez-Cotop,ba Lukas Madererp, m Jerzy Mikolaj Mańczak,d

Karl Mannheim,an Annarita Margiottop, ab,am Antonio Marinellip, n
Christos Markou,ᵇ Lilian Martin,ᵃⱼ Juan Antonio Martínez-Mora,<fnref n="1" />

Agnese Martini,ᵃʰ Fabio Marzaioli,ⁿ,zion Stefano Mastroianni,ⁿ

Safaa Mazzou,ᵃᵉ Karel W. Melis,ᵈ Gennaro Miele,ⁿ,o Pasquale Migliozzi,ⁿ

Emilio Migneco,⁵ Piotr Mijakowski,ᵃ Luis Salvador Miranda Palacios,ᵇᵇ

Carlos Maximiliano Mollo,ⁿ Mauro Morganti,ᵃq,bc Michael Moser,ⱼ

Abdelilah Moussa,ᵗ Rasa Muller,ᵈ David Muñoz Pérez,ᵈ Paolo Musico,ᵢ

Mario Musumeci,⁵ Lodewijk Nauta,ᵈ Sergio Navas,ᵇᵃ Carlo A. Nicolau,ᵇ

Brian Ó Fearraigh,ₚ,ₓ Mitchell O'Sullivan,ᵃMENT MUKHARBEK ORGANOKOV,ᵃu

Angelo Orlando,⁵ Juan Palacios González,ᵈ Gogita Papalashvili,ᶜ,ax

Riccardo Papaleo,⁵ Cosimo Pastore,ᵃᵢ Alice M. Păun,ᶜ,w

Gabriela Emilia Păvălaş, w Giuliano Pellegrini,ᵃᵇ Carmelo Pellegrino,ᶜ,am,bd

Mathieu Perrin-Terrin,ᶜ Paolo Piattelli,ᶜ, s Camiel Pieterse,ᵈ

Konstantinos Pikounis,ᶜ,h Ofelia Pisanti,ᶜ,ⁿ,o Chiara Poirè,<fnref n="1" /> Vlad Popa, w

Thierry Pradier,ᵃu Gerd Pühlhofer,ᵇe Sara Pulvirenti,⁵ Omphile Rabyang,ᵗ

Fabrizio Raffaelli,ᶜ,aq Nunzio Randazzo,ᵃ Soebur Razzaque,ᵇᵇ

Diego Real,ᶜ,d,\* Stefan Reck,ᵏ Giorgio Riccobene,⁵ Marc Richer,ᵃu

Stephane Rivoire,ᵃp Alberto Rovelli,⁵ Francisco Salesa Greus,ᶜ, d

Dorothea Franziska Elisabeth Samtleben,ₚ,bf Agustin Sánchez Losa,ᶜ,ai

Matteo Sanguineti,ⱼ,al Andrea E. Santangelo,ᵇe Domenico Santonocito,⁵

Piera Sapienza,⁵ Jan-Willem Schmelling,ₚ Jutta Schnabel,ᵏ

Johannes Schumann,ᶜ,⁵ Jordan Seneca,ₚ Irene Sgura,ᵃᵢ Rezo Shanidze,ᵃx

Ankur Sharma,ᶜ,ᵇg Francesco Simeone, Anna Sinopoulou, h

Bernardino Spisso,ᶜ,ⁿ,ar Maurizio Spurio,ᶜ,ab,am Dimitris Stavropoulos,ʰ

Jos Steijger,ₚ Simona Maria Stellacci,ⁿ,ar Mauro Taiuti,ᵢ,al Yahya Tayalati,q

Enrique Tenllado,ᵇᵃ Tarak Thakore,ᵈ Steven Tingay,ᶜ,av

Ekaterini Tzamariudaki,ʰ Dimitrios Tzanetatos,ʰ

Veronique Van Elewyck,ᵃm,az Georges Vasileiadis,ᵃp Federico Versari,ab,am

Salvo Viola,⁵ Daniele Vivolo,ⁿ,o Gwenhael de Wasseige,ᵃ Joern Wilms,ᵇᵇ

Rafa Wojaczyński,ᵃw Els de Wolf,ₚ,ₓ Dmitry Zaborov,ᵇi

Sandra Zavatarelli,ᵢ Angela Zegarelli,ᵢ,aa Daniele Zito,⁵

Juan de Dios Zornoza,ᵈ Juan Zúñiga,ᵈ and Natalia Zywucka,ᶜ, t

<footnote>ᵃIFN, Sezione di Catania, Catania, Italy</footnote>

<footnote>ᵇUniversité de Strasbourg, Université de Haute Alsace, GRPHE, Colmar, France</footnote>

<footnote>ᶜIN2P3, IPHC, Strasbourg, France</footnote>

<footnote>ᵈIFIC—Instituto de Física Corpuscular (CSIC—Universitat de València), Paterna,
Valencia, Spain</footnote>

<footnote>ᶜAix Marseille Université, CNRS/IN2P3, CPPM, Marseille, France</footnote>

<footnote>ᵢIFN, Sezione di Roma, Roma, Italy</footnote>

<footnote>ᵇUniversitat Politècnica de Catalunya, Laboratori d'Aplicacions Bioacústiques,
Centre Tecnològic de Vilanova i la Geltrú, Vilanova i la Geltrú, Spain</footnote>

<footnote>ʰNCSR Demokritos, Institute of Nuclear and Particle Physics, Athens, Greece</footnote>

<footnote>ᵢIFN, Sezione di Genova, Genova, Italy</footnote>

<footnote>jUniversity of Granada, Department of Computer Architecture and Technology/CITIC,
Granada, Spain</footnote>

<footnote>kFriedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen Centre for Astroparticle
Physics, Erlangen, Germany</footnote>

<footnote n="1">¹Universitat Politècnica de València, Instituto de Investigación para la Gestión Integrada de las Zonas Costeras, Gandia, Spain</footnote>

<footnote>mUniversité de Paris, CNRS, Astroparticule et Cosmologie, Paris, France</footnote>

<footnote>ⁿIFN, Sezione di Napoli, Complesso Universitario di Monte S. Angelo, Napoli, Italy</footnote>
<sup>o</sup>Università di Napoli “Federico II,” Dip. Scienze Fisiche “E. Pancini,” Complesso Universitario di Monte S. Angelo, Napoli, Italy

<sup>p</sup>Nikhet, National Institute for Subatomic Physics, Amsterdam, The Netherlands

<sup>q</sup>University Mohammed V in Rabat, Faculty of Sciences, Rabat, Morocco

<sup>r</sup>KVI-CART University of Groningen, Groningen, The Netherlands

<sup>s</sup>INFN, Laboratori Nazionali del Sud, Catania, Italy

<sup>t</sup>North-West University, Centre for Space Research, Potchefstroom, South Africa

<sup>u</sup>University Mohammed I, Faculty of Sciences, BV Mohammed VI, Oujda, Morocco

<sup>v</sup>Università di Salerno e INFN Gruppo Collegato di Salerno, Dipartimento di Matematica, Fisciano, Italy

<sup>w</sup>ISS, Măgurele, Romania

<sup>x</sup>University of Amsterdam, Institute of Physics/IHEF, Amsterdam, The Netherlands

<sup>y</sup>TNO, Technical Sciences, Delft, The Netherlands

<sup>z</sup>Università degli Studi della Campania “Luigi Vanvitelli,” Dipartimento di Matematica e Fisica, Caserta, Italy

<sup>aa</sup>Università La Sapienza, Dipartimento di Fisica, Roma, Italy

<sup>ab</sup>INFN, Sezione di Bologna, Bologna, Italy

<sup>ac</sup>Università di Bologna, Dipartimento di Ingegneria dell'Energia Elettrica e dell'Informazione "Guglielmo Marconi," Bologna, Italy

<sup>ad</sup>Gran Sasso Science Institute, GSSI, L'Aquila, Italy

<sup>ae</sup>Cadi Ayyad University, Physics Department, Faculty of Science Semlalia, Marrakech, Morocco

<sup>af</sup>University of the Witwatersrand, School of Physics, Johannesburg, South Africa

<sup>ag</sup>Università di Catania, Dipartimento di Fisica e Astronomia "Ettore Majorana," Catania, Italy

<sup>ah</sup>INFN, LNF, Via Enrico Fermi, Frascati, Italy

<sup>ai</sup>INFN, Sezione di Bari, Bari, Italy

<sup>aj</sup>Université de Nantes, Subatech, IMT Atlantique, IN2P3-CNRS, Nantes, France

<sup>ak</sup>University of Bari, Bari, Italy

<sup>al</sup>Università di Genova, Genova, Italy

<sup>am</sup>Università di Bologna, Dipartimento di Fisica e Astronomia, Bologna, Italy

<sup>an</sup>University Würzburg, Würzburg, Germany

<sup>ao</sup>Western Sydney University, School of Computing, Engineering and Mathematics, Penrith, Australia

<sup>ap</sup>Laboratoire Univers et Particules de Montpellier, Montpellier Cédex, France

<sup>aq</sup>INFN, Sezione di Pisa, Pisa, Italy

<sup>ar</sup>Università di Salerno e INFN Gruppo Collegato di Salerno, Dipartimento di Fisica, Fisciano, Italy

<sup>as</sup>University of Münster, Institut für Kernphysik, Münster, Germany

<sup>at</sup>NIOZ (Royal Netherlands Institute for Sea Research) and Utrecht University, Texel, The Netherlands

<sup>au</sup>Université de Strasbourg, CNRS IPHC UMR 7178, Strasbourg, France

<sup>av</sup>Curtin University, International Centre for Radio Astronomy Research, Bentley, Western Australia, Australia

<sup>aw</sup>National Centre for Nuclear Research, Warsaw, Poland

<sup>ax</sup>Tbilisi State University, Department of Physics, Tbilisi, Georgia

<sup>ay</sup>Utrecht University, Department of Physics and Astronomy, Utrecht, The Netherlands

<sup>az</sup>Institut Universitaire de France, Paris, France

<sup>ba</sup>University of Granada, Dpto. de Física Teórica y del Cosmos & C.A.F.P.E., Granada, Spain

<sup>bb</sup>University of Johannesburg, Department Physics, Auckland Park, South Africa

<sup>bc</sup>Accademia Navale di Livorno, Livorno, Italy

<sup>bd</sup>INFN, CNAF, Bologna, Italy

<sup>be</sup>Eberhard Karls Universität Tübingen, Institut für Astronomie und Astrophysik, Tübingen, Germany

<sup>bf</sup>Leiden University, Leiden Institute of Physics, Leiden, The Netherlands
<sup>bg</sup>Università di Pisa, Dipartimento di Fisica, Pisa, Italy

<sup>bh</sup>Friedrich-Alexander-Universität Erlangen-Nürnberg, Remeis Sternwarte, Bamberg, Germany

<sup>bi</sup>NRC “Kurchatov Institute,” A.I. Alikhanov Institute for Theoretical and Experimental Physics, Moscow, Russia

**Abstract.** The KM3NeT infrastructure consists of two deep-sea neutrino telescopes being deployed in the Mediterranean Sea. The telescopes will detect extraterrestrial and atmospheric neutrinos by means of the incident photons induced by the passage of relativistic charged particles through the seawater as a consequence of a neutrino interaction. The telescopes are configured in a three-dimensional grid of digital optical modules, each hosting 31 photomultipliers. The photomultiplier signals produced by the incident Cherenkov photons are converted into digital information consisting of the integrated pulse duration and the time at which it surpasses a chosen threshold. The digitization is done by means of time to digital converters (TDCs) embedded in the field programmable gate array of the central logic board. Subsequently, a state machine formats the acquired data for its transmission to shore. We present the architecture and performance of the front-end firmware consisting of the TDCs and the state machine. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. [DOI: 10.1117/1.JATIS.7.1.016001]

**Keywords:** neutrino telescope; acquisition firmware; time to digital converters; KM3NeT.

Paper 20103 received Jul. 20, 2020; accepted for publication Dec. 16, 2020; published online Jan. 7, 2021.

# 1 Introduction

The KM3NeT neutrino telescopes constitute a deep-sea research infrastructure¹,² being deployed in the Mediterranean Sea, composed of two detectors placed in two different sites but sharing the same technology. Astroparticle Research with Cosmics in the Abyss (ARCA),³ located 100 km away from Capo Passero, the southern tip of Sicily, Italy, at a depth of 3450 m will be mainly dedicated to high-energy neutrino astrophysics. Oscillation Research with Cosmics in the Abyss (ORCA)⁴ is situated 40 km off the coast near Toulon, France, at a depth of about 2450 m and has been optimized for the study of atmospheric neutrino oscillations.

The telescopes have been designed to detect the Cherenkov photons induced by relativistic charged particles produced in neutrino interactions with the detector surroundings. A three-dimensional (3-D) array of digital optical modules (DOMs) detects the Cherenkov photons, allowing for the reconstruction of the trajectory and the energy of the incoming neutrino.⁵ The DOM⁶⁻⁸ [Fig. 1(a)] consists of a 17-in.-diameter pressure-resistant glass sphere housing 31 3-in. photomultiplier tubes (PMTs) together with the front-end and readout electronics.⁸,⁹ Eighteen DOMs, mounted in a vertical structure, form a detection unit (DU). Each DU is anchored to the seabed and stands on it due to the buoyancy of the DOMs and a buoy at its top [Fig. 1(b)].

When a photon impinges on the PMT cathode, a photoelectron can be produced with a probability given by the quantum efficiency of the PMT (usually ~30%). Subsequently, a cascade of electrons is generated up to the PMT anode. If the electrical signal at the anode crosses the threshold of a discriminator, it is preprocessed by a dedicated electronic board attached to the PMT. A low voltage differential signal (LVDS) is then generated, with its starting time equal to the threshold crossing time and its duration equal to the time that the waveform is above the configurable threshold. This time duration is called time over threshold (ToT). The LVDS signals generated by the PMTs are collected by the signal collection board and routed to the central logic board (CLB), where the readout acquisition and digitization of the PMT data is performed.

The front-end firmware is embedded in a 160T Kintex-7 field programmable gate array (FPGA) over 160,000 logic cells¹⁰ (see Fig. 2). All the firmware modules are configured and

<footnote>*Address all correspondence to David Calvo, dacaldia@ific.uv.es; Diego Real, real@ific.uv.es</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 1** (a) A KM3NeT DOM. (b) Artist representation of the 3-D grid of the underwater telescope showing the vertical strings holding the DOMs.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 2** Detail of the FPGA mounted on the CLB. The FPGA is a Xilinx Kintex-7 160T of commercial grade. The FPGA package selected for KM3NeT is the FBG676.</figcaption></figure>

controlled by an embedded LatticeMico32 (LM32) microprocessor and its Wishbone bus. The FPGA has a specific hardware logic resource with serial-to-parallel and parallel-to-serial converters,¹¹ where the time to digital converters (TDCs) are implemented. The 31 TDCs, one for each PMT in the DOM, are coded in hardware description language (HDL) in the FPGA. They digitize the LVDS signals to obtain both the arrival time of the pulse and its ToT. The PMT signal digitized by the TDCs is called a “hit.” Once a hit is obtained, a state machine (SM) organizes the hits generated by the TDCs and encodes these into user datagram protocol (UDP) Jumbo frames to be sent to the shore station via the CLB optical link. To reduce the complexity of the hardware and firmware of the DOMs, the concept “all-data-to-shore” is applied, where all the readout information is sent to the shore station without any data filtering. The readout is organized and sent in time intervals of 100 ms, called a “time slice.” The TDCs restart their counters at the start of each time slice, being the arrival time of the hits relative to the start of the time slice. To combine and analyze the data provided by all the DOMs, the White Rabbit protocol¹² is used to synchronize the clocks of all the CLBs of the detector with 1-ns resolution. The power consumption of the described front-end firmware is 1.4 W, which represents the 21% of the DOM power consumption. All the devices within the DOM as well as the front-end firmware are shown in Fig. 3.
<figure><img src="image_5.png" /><figcaption>**Fig. 3** Block diagram of the DOM. Optic, acoustic, instrumentation, front-end firmware, and all the interfaces are represented.</figcaption></figure>

The requirements of the data acquisition (DAQ) architecture are presented in Sec. 2, the TDC requirements in Sec. 2.1, and the SM requirements in Sec. 2.2. The TDCs are presented, together with several of the qualification tests performed, in Sec. 3, and the SM is described in Sec. 4. Section 5 describes the main test setups used for the validation of the acquisition firmware, and Sec. 6 presents some examples of data acquired in the first DUs deployed. Finally, a summary is presented in Sec. 7.

## 2 KM3NeT Front-End Firmware Requirements

The main requirements established by the KM3NeT collaboration for the front-end firmware refer to both the digitization of the PMT signals by TDCs and the firmware of the SM that controls the acquisition and generation of the data output packets sent to the shore station. The most important element to reconstruct the particle trajectory from the Cherenkov light is the arrival time of the light on the PMTs. For the KM3NeT requirements, the relative arrival times should be known with an accuracy of 1 ns, which provides an angular resolution of 0.1° for astrophysical events.

### 2.1 TDC Requirements

Each TDC channel has to be able to deal with pulse rates up to 200 kHz, with 7 kHz being the expected average rate of the PMT signals.¹³ Table 1 summarizes the data throughput expected for different stages of the KM3NeT infrastructure.

<table><caption>**Table 1** TDC expected average throughput for the DOM and DU.</caption><thead><tr><td>Case</td><td>Expected rate (7 kHz)</td></tr></thead><tbody><tr><td>DOM (31 TDC channels)</td><td>11 Mbps</td></tr><tr><td>DUs (18 DOMs)</td><td>200 Mbps</td></tr></tbody></table>
<table><caption>**Table 2** Format of the hit provided by the TDCs. One byte is reserved for the identification number of the PMT, 4 bytes are reserved for the arrival time of the hit, counted as nano-seconds passed since the start of the time slice, and one byte is reserved for the length of the pulse or ToT. Therefore, a total of 6 bytes are needed to codify a hit.</caption><thead><tr><td>Info</td><td>Codification</td></tr></thead><tbody><tr><td>PMT Id</td><td>1 byte</td></tr><tr><td>Hit arrival time</td><td>4 bytes</td></tr><tr><td>Hit length or ToT</td><td>1 byte</td></tr><tr><td>Total hit coding</td><td>6 bytes</td></tr></tbody></table>

The TDCs acquire the LVDS signals generated by the PMTs base, detecting its rise time and duration (ToT) with 1 ns precision. The hit arrival time is coded with four bytes. The duration of the signal, which corresponds to the hit ToT, is also measured with 1 ns precision. The minimum detectable ToT is 1 ns, whereas the maximum ToT allowed by the sampling architecture is 255 ns (coded with one byte). It is worth mentioning that the TDC range of threshold has been extended by means of the so-called “multihit” functionality, described in Sec. 3.6. The minimum time between two consecutive LVDS, or dead time, is 5 ns due to the internal configuration of the TDCs. Each hit (composed of a PMT identification, a ToT, and a time stamp) is encoded with 6 bytes, using the format summarized in Table 2. Each TDC channel has associated one first-in first-out (FIFO) memory with a capacity of 1024 hits.

All the control of the TDC intellectual property (IP) core is done via the Wishbone bus,¹⁴ the bus chosen to interconnect the different firmware IP cores. There is a dedicated 32-bit CPU register that enables the IP core and another one that enables each of the TDC channels independently.

## 2.2 State Machine Requirements

The SM has to organize the optical acquisition in time slices, whose lengths are configurable between 10 and 100 ms. In addition, as in the case of the TDCs, the SM is also integrated as a Wishbone slave with one register to define the payload of the UDP packets sent to the shore station, another register to define the duration of the time slice and six more registers to control the full flags and to send interrupt requests (IRQs) to the LM32 microprocessor. The main requirements for the TDC and the SM are summarized in Table 3.

## 3 Time to Digital Converters

TDCs, which convert a pulse time duration into a numeric value, are used in various applications where an accurate measurement of time is needed. TDCs can be implemented in both application specific integrated circuits (ASICs) and FPGAs. ASICs can provide better accuracy when a high time resolution is required. However, FPGAs provide a faster development time and the flexibility to adapt the logic to operating requirements. Moreover, it is possible to use the FPGA logic resources to process the TDC data and interface with the rest of the DAQ system. In the case of KM3NeT, it is not necessary to include extracomponents as in other more accurate architectures such as Vernier, to achieve the desired resolution, which results in a higher-reliability system. For these reasons, KM3NeT collaboration has chosen to implement the TDCs in FPGAs.

Multitapped delay-line architectures were not considered because 31 delay-lines should be implemented, and it would be more problematic to maintain with the rest of the systems implemented in the FPGA. Moreover, the level of resolution and range required by KM3NeT can be achieved by means of synchronous architectures¹⁵,¹⁶ more appropriate and simpler for an FPGA implementation.
<table><caption>**Table 3** Front-end firmware requirements. This table summarizes both the requirements of the KM3NeT collaboration for the TDCs and the SM.</caption><thead><tr><th colspan="2">TDC requirements</th></tr></thead><tbody><tr><td>Number of channels</td><td>31</td></tr><tr><td>Maximum expected frequency per channel</td><td>7 kHz</td></tr><tr><td>Time stamp resolution</td><td>1 ns</td></tr><tr><td>Pulse length resolution</td><td>1 ns</td></tr><tr><td>Maximum pulse length</td><td>255 ns</td></tr><tr><td>Minimum pulse length</td><td>3 ns</td></tr><tr><td>TDC dead time</td><td>5 ns</td></tr><tr><td>Number of FIFOs per channel</td><td>1</td></tr><tr><td>FIFO capacity</td><td>1024 hits (6 bytes per hit)</td></tr><tr><td>Pulse length</td><td>1 byte</td></tr><tr><td>Time stamp</td><td>4 bytes</td></tr><tr><td>Slow control</td><td>Wishbone slave</td></tr><tr><th colspan="2">State machine requirements</th></tr><tr><td>TDCs time slice length</td><td>Between 10 and 100 ms</td></tr><tr><td>Slow control</td><td>Wishbone slave</td></tr><tr><td>TDC time slice duration</td><td>1 register</td></tr><tr><td>Jumbo frame size</td><td>1 register</td></tr><tr><td>Internet protocol source address</td><td>1 register</td></tr><tr><td>Internet protocol destination address</td><td>1 register</td></tr><tr><td>TDC UDP server port CLB port number</td><td>1 register</td></tr></tbody></table>

The main synchronous architecture for the implementation of TDCs in an FPGA is the 4x-oversampling technique (see Fig. 4), which is reliable and can be easily coded with HDL. The four clocks with the required different phases can be directly implemented in the phase locked loops (PLLs) of the FPGA digital clock managers. As a preventive action, the distribution of the clocks in the FPGA resources was carefully designed, since any asymmetry in the high-speed lines distributing the clocks would result in nonlinearities that will degrade the performance of the TDCs. The FPGA allows for specifying the time constraints of the clock distribution. Moreover, the specific hardware provided by most of the FPGAs to serialize and deserialze can implement the 4x-oversampling technique in an even more precise way, as the very fast shift registers present in the serializer-deserIALIZer (SERDES) can be used by the TDCs. In Fig. 5, a schematic view of the 4x-oversampling technique is shown.

The technique finally chosen was the 4x-oversampling technique: a PLL inside the FPGA generates two clocks of equal frequency but different phases (CLK₀ and CLK₉₀). These two phases are routed to a deserializer primitive inside the input/output blocks of the FPGA. The deserializer primitive allows the oversampling of an incoming data stream on both rising and falling edges of the generated clocks, CLK₀ and CLK₉₀, resulting in four times the sampling frequency with respect to the original clock (CLK₀, CLK₉₀, CLK₁₈₀, CLK₂₇₀). If a better resolution is needed, it is possible to either duplicate and shift the sampling clocks by 45 deg in phase, or double the frequency of the clocks.
<figure><img src="image_6.png" /><figcaption>**Fig. 4** Oversampling architecture. The input signal is connected with no delay to the flip-flops performing the acquisition. The clocks of the flip-flops are driven by equal-frequency clocks shifted with equidistant phases, therefore increasing the sampling rate by the number of phases being used.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 5** Scheme of the 4x-oversampling technique. The sampling quadruples the clock frequency using four phases of the original clock, shifted by 90 deg each, thus obtaining a sampling frequency of 1 GHz when using a clock with a period of 4 ns.</figcaption></figure>

The most important parameters for the characterization of the TDC are: the time range of measurement; the resolution in terms of the least significant bit (LSB); the precision or standard uncertainty of the measurement; the nonlinearities [both the differential nonlinearity (DNL) and the integral nonlinearity (INL)]; the dead time or the shortest time between two consecutive hits; and the maximum readout speed. These are discussed in the following sections.

## 3.1 TDC Architecture

The implementation of the KM3NeT TDCs uses the dedicated input hardware available in the Kintex 7 family of Xilinx FPGA: the input buffer for differential signals (IBUFDS) and the Xilinx input-output serializer-deserIALIZer (IOSERDES), in particular the ISERDESE2 primitive. It includes a firmware data recovery unit (DRU), the logic that takes care of the acquired hits and stores them into the TDC FIFOs. The LVDS signals arriving from the PMT bases enter the FPGA through the IBUFDS input while the 4x-oversampling technique is implemented in the IOSERDES. The IOSERDES block is configured in oversampling mode, with two clock inputs shifted by 90 deg. Internally, the IOSERDES captures the input data, both on the rising and falling edges of the two input clocks, acquiring the input signal exactly as the 4x-oversampling technique requires, i.e., in four equidistant (90 deg) phases derived from the original clock of
<figure><img src="image_8.png" /><figcaption>**Fig. 6** The architecture of the KM3NeT TDCs. Three different subsystems can be identified in the TDC. The first one, running at 1 ns, contains the differential input to the FPGA, the IBUFDS, together with the IOSERDES, where the acquisition is performed. The second one contains the DRU, with the logic to adapt the hits to the format required and to store them in the FIFOs. It works at 4 ns. The third one consists of the logic to read out the FIFOs and provides the obtained hits to the next acquisition level, the SM, running in this case at 16 ns. The interface between the second and the third subsystem is done by means of the FIFOs.</figcaption></figure>

250 MHz. A scheme of the 4x-oversampling technique implemented in the Kintex-7 160T is shown in Fig. 6.

The TDCs have three clock domains; the first one, of 1 GHz, includes the IOSERDES oversampling, where the hit is acquired; the second one, of 250 MHz, is used by the DRU; and the third one, of 62.5 MHz, is the system clock domain of the CLB and the one used by the SM to read out the TDC FIFOs. The FIFOs are able to store 1024 hits (each hit has the size and the structure as shown in Table 2), with an almost-full threshold of 1012, to deal with momentary bursts of data, like those due to bioluminescence. The FIFO allows for the readout while other hits are being processed by the SM.

In Fig. 7, a schematic view of the FIFOs is shown. The occupancy of the FIFO is also used to control the acquisition of the TDC channel. When the almost-full-threshold signal of the TDC FIFO is triggered (set to 1012 hits by default, but configurable), the acquisition is stopped until the next time slice, provided that the almost-full signal has been deasserted. Thus, the effective size of the TDC FIFO is 1012 hits, reserving the remaining 12 positions for time slice markers. The time slice markers are specific identifiers used to tag the transition between two consecutive time slices.

## 3.2 TDC Implementation and Resources

The TDCs have been coded entirely in the FPGA using HDL. The resources used per channel include one IBUFDS and one IOSERDES for the TDC readout firmware, 463 registers and 483 look-up tables (mainly used by the DRU) and 3 memory blocks of 36 Kb, used by the TDC FIFOs. Table 4 summarizes the resources used by the TDCs both for one channel only and in total.

The TDC IP core has 31 inputs for the LVDS PMT signals as well as two inputs for the two clocks needed to oversample the input signal (CLK₀ and CLK₉₀). Moreover, the TDC IP core has one bus output, from where the SM can read out the values of the 31 TDC FIFOs. The bus also includes the control and monitor signals to read out the hits stored in the FIFOs, such as the value of the FIFO full and almost-full flags. In addition, the IP core includes the control and monitor signals of a Wishbone slave, which allows for interaction with the embedded software.
<figure><img src="image_9.png" /><figcaption>**Fig. 7** Schematic view of the FIFOs. The structure of a complete hit is shown, as well as the time slice marker, which has the same length as a hit and indicates the transition between time slices. The FIFO almost-full flag is also shown. The flag becomes active only if the occupancy of the FIFO is 1012 elements.</figcaption></figure>

<table><caption>**Table 4** Detailed TDC resources for both 1 channel and 31 channels.</caption><thead><tr><th>Resources</th><th>Available</th><th>Used (1 ch.)</th><th>% Used (1 ch.)</th><th>Used (31 ch.)</th><th>% Used (31 ch.)</th></tr></thead><tbody><tr><td>Look-up table</td><td>101,400</td><td>483</td><td>0.5</td><td>12,568</td><td>12</td></tr><tr><td>Registers</td><td>202,800</td><td>463</td><td>0.2</td><td>6917</td><td>3</td></tr><tr><td>Memory (36Kb)</td><td>325</td><td>3</td><td>1</td><td>93</td><td>28</td></tr><tr><td>Input/output ports</td><td>400</td><td>34</td><td>8.5</td><td>92</td><td>23</td></tr><tr><td>Clocking</td><td>32</td><td>5</td><td>15</td><td>5</td><td>15</td></tr></tbody></table>

The control registers of the Wishbone slave include one register with one control bit to enable the complete TDC IP core, as well as another register to enable each of the 31 TDC channels. The high rate veto (HRV) and multihit capabilities, explained in Secs. 3.5 and 3.6, are also configured and managed by four different registers. Table 5 lists the Wishbone registers used by the TDC channels.

## 3.3 Resolution

The clock frequency and the number of clock phases determine the resolution of the TDC. In the KM3NeT case, the frequency of the IOSERDES clocks is 250 MHz, and the number of phases is four. Therefore, the resolution obtained by the TDCs is 1 ns (1 GHz). It would be possible to increase the resolution of the TDCs by increasing the number of phases, the frequency of the
<table><caption>**Table 5** Detailed TDC Wishbone control registers and LM32 memory addresses. Five registers have been included to enable or disable both the whole TDC core and the individual channels and to control the HRV and multihit functions.</caption><thead><tr><td>LM32 base address</td><td>Register</td><td>Name</td></tr></thead><tbody><tr><td>x00020100</td><td>0x00</td><td>Enable core (1 bit). Reserved (31 bits)</td></tr><tr><td>0x00020104</td><td>0x01</td><td>Enable channels</td></tr><tr><td>0x00020108</td><td>0x02</td><td>HRV value</td></tr><tr><td>0x0002010C</td><td>0x03</td><td>Enable HRV on channels</td></tr><tr><td>0x00020110</td><td>0x04</td><td>Enable multihit on channels</td></tr></tbody></table>

clock or both. In a Kintex-7 160T, it is feasible to achieve clock frequencies up to 500 MHz, whereas the number of phases could also be increased up to eight with an 8×-oversampling technique. With these two modifications, the resolution could be 250 ps without any modification of the hardware.

## 3.4 Precision

Some factors contributing to the degradation of the TDC precision ($\sigma$) are jitter of the clock, electronic noise, and variations in temperature and power. But, since the resolution of the implemented TDCs is not excessively high, these are dominated by quantization errors, as shown in Fig. 8. The arrival time of the hits is asynchronous with the TDC clock since they are not correlated. For this reason, the time interval between the hit starting time and the sampling edge of the TDC is uniformly distributed. The maximum quantization error of a simple measurement is $\pm 1$ ns in both rising edge and falling edge. The precision ($\sigma$), characterized by the standard deviation of the distribution of repeated measurements, is defined as

<figure><img src="image_10.png" /><figcaption>**Fig. 8** TDC quantization error. Two examples of hits are given to explain the intrinsic quantization error of the oversampling TDCs. The first one represents a hit with a ToT of 6.8 ns, which is acquired in six clock samples due to the relative position of the hit with respect to the sampling clock. The quantization error is -0.8 ns. In the second example, the hit has a ToT of 7.2 ns. In this case, due to the relative position of the hit, the number of clock samples is eight, giving a quantization error of 0.8 ns. The maximum values for the quantization error would be $\pm 1$ ns.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 9 TDC precision. TDC quantization error as a function of the fractional part of the ratio $ToT/t_0$. The model representing an ideal TDC is shown as a continuous line and the measurements with a real TDC as red points.</figcaption></figure>

$$ \sigma = t_0 \sqrt{\eta(1 - \eta)}, \qquad (1) $$

where $t_0$ is the resolution of the TDC and $\eta$ is the decimal part of the ratio $ToT/t_0$, denoted by $\text{Frac}$ (see Fig. 9):

$$ \eta = \text{Frac} \left[ \frac{ToT}{t_0} \right]. \qquad (2) $$

The average value of the precision ($\sigma_{Av}$) is calculated by integrating $\sigma$ over $\eta$ from Eq. (1) between zero and one, which results in

$$ \sigma_{Av} = \int_{0}^{1} t_{0} \sqrt{\eta(1 - \eta)} d\eta = \frac{\pi t_{0}}{8} \approx 0.393 t_{0} = 0.393 \text{ ns.} \qquad (3) $$

## 3.5 High Rate Veto

Neutrino telescopes in seawater are affected by several sources of external noise. Studies performed at active sites have shown that bioluminescence, in particular bioluminescence bursts, is an important source of optical noise for electronics with rates that, in extreme cases, could saturate the DAQ. To avoid the saturation of the data taking, the TDCs in KM3NeT incorporate an HRV system to stop the acquisition when an unexpected rate increase is detected until the high rate has ended. The HRV works on a channel and time slice basis. The number of hits detected since the start of the time slice is counted in each of the 31 TDC channels. When this number exceeds some predefined threshold, the acquisition for that channel is blocked until the start of the next time slice. The unaffected TDC channels continue acquiring data. Figure 10 shows a schematic diagram of the HRV operation. The HRV operation prevents the readout of the hits generated while a burst of bioluminescence is happening, preventing the saturation of the communication bandwidth.

## 3.6 Multihit

The multihit option allows for acquiring hits with a duration longer than the limit of 255 ns. Large pulses could be generated in different situations; photons separated few ps in a Cherenkov yield of a muon track; ionized particles along muon trajectory or even exotic particle such as monopoles, which can actually provoke long pulses. If the multihit option is disabled, any hit with a duration longer than 255 ns is digitized as only one hit of 255 ns. On the other hand, if the multihit option is enabled, the same hit is digitized as several consecutive hits, all with a duration of 255 ns except the last one whose duration is the remaining time to complete the real duration of the physical event. Once the data arrive at the shore station, the online trigger and data acquisition system¹⁷ merges the consecutive hits into one hit whose duration equals the real duration of
<figure><img src="image_12.png" /><figcaption>**Fig. 10** HRV. Channels 0 and 30 are represented with the HRV is active and set to 4 hits. In channel 1, the number of hits in the time slice evaluated is 3, so the HRV is not triggered. The number of hits in channel 30 is 6, so the HRV is triggered, the events are not stored in FIFO anymore and the acquisition is stopped until the beginning of the next time slice.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 11** Multihit option. The figure represents one hit of 835 ns analyzed in two different scenarios. In the first one, the multihit option is enabled, so the original hit is acquired as four consecutive hits, the first three of 255 ns length, and the fourth one of 70 ns. In the second scenario, the multihit option is disabled, therefore the original hit is acquired as a single hit of 255 ns.</figcaption></figure>

the physical event. The particles that generate these pulses are not included in the standard phys-
ics analyses of KM3NeT; however, they could provide useful information to understand unex-
pected physics phenomena or anomalous behavior of the detector. The use of the multihit can
increase the data rate during those moments. However such situations are not expected to occur
frequently, therefore multihit will not have a significant impact on the DAQ system performance.
Figure 11 shows the operation of the multihit option.

**3.7 Dead Time**

As already mentioned in Sec. 2.1, the dead time refers to the minimum time between two con-
secutive hits. The TDC dead time is 5 ns. The value is intrinsic to the architecture of the TDCs.
The multihit option has no influence upon the dead time. If a pattern of hits separated <5 ns is
applied to the TDCs, sometimes some hits could be either merged or discarded depending the
instant within the IOSERDES position when they arrive. On the other hand, if the time difference is 5 ns, then, the hits are detected as separate entities.

## 3.8 Nonlinearity

Two of the parameters defining the quality of the acquisition are the DNL and INL. A statistical-code-density test¹⁸⁻²⁰ is needed to determine the TDC nonlinearity characteristics. In these tests, more than 6 million asynchronous pulses are measured and evaluated to assess the uncertainties of the TDC measurements. The hits generated in the seawater are not correlated in time with the TDC clock, and they can be considered as a uniformly random train of pulses at the input of the TDCs. The number of acquisitions per IOSERDES cycle of the TDC, n (pulses acquired) should be large enough to reduce the statistical uncertainty, which can be approximated as 1/√N, where N is the total number of generated pulses.

### 3.8.1 Differential nonlinearity

The DNL can be defined as the deviation of a single quantization step from the ideal LSB. In the case of the oversampling technique, the TDC usually has a reduced DNL, as the feature is intrinsic to the architecture, where the quantization step is always an integer fraction of the clock period. The skew of the clocks, related to the accuracy of the FPGA PLL to generate four clock signals, is the main contributors to the appearance of the DNL. The DNL is evaluated by comparing the number of pulses per IOSERDES cycle (nᵢ) with the mean value, which in this case results in n̄ = N/4 since the N generated pulses will be detected randomly by the four cycles of the IOSERDES. For each IOSERDES cycle, the DNL is defined as

$$ \text{DNL}_i = \frac{n_i - \bar{n}}{\bar{n}}. \qquad (4) $$

In Fig. 12, an example is shown of the DNL measurements performed in the laboratory for one of the 31 TDC channels. The DNLs are calculated for the four IOSERDES cycles. Five tests have been carried out, each one with 50,000 measurements. The error bars show the standard deviation, which is defined as

<figure><img src="image_14.png" /><figcaption>**Fig. 12** Measured differential nonlinearities. Five tests have been performed with 50,000 events each. The error bars show the standard deviation of these five measurements. The DNLs have been measured for each IOSERDES cycle.</figcaption></figure>
$$ \sigma_i = \sqrt{\frac{1}{M} \sum_{j=1}^{M} D_j^2}, \qquad (5) $$

where M is the number of tests (five in this case) and $D_j$ is the DNL value for each of the tests performed. These tests show the maximum error produced by the DNL is lower than 40 ps, which is negligible for the TDCs performance.

### 3.8.2 Integral nonlinearity

The INL refers to the maximum deviation of the TDC transfer function from the ideal straight line. It can be calculated as

$$ \text{INL} = \frac{T - T_{\text{in}}}{t_{\text{bin}}}, \qquad (6) $$

where $T_{\text{in}}$ is the width of the input, $T$ is the mean of the pulse width measurements, and $t_{\text{bin}}$ is the size of the bin, which in this case is 1 ns. INL measurements are shown in Fig. 13 for different time widths starting at 5 ns up to the end of the TDC range, 255 ns, in steps of 5 ns. Five tests have been performed, each one with 10,000 measurements. The error bars take into account the dispersion of the INL measured after the five tests, where in the worst case the INL was 0.118 LSB.

## 3.9 Temperature Effects

The effects of temperature on the TDC performance have been evaluated using a climatic chamber and the test setup explained in Sec. 5, where the CLB has been operated at temperatures ranging from -35°C to 60°C. The room temperature and the temperature of the FPGA die have been registered. For each of 20 temperature values, a repetitive pattern has been applied to the TDCs. A Virtex6 ML605 evaluation board placed outside the climatic chamber has been used as pattern generator. The pattern generated has been measured with an oscilloscope before being applied to the TDC. For each temperature, the same pattern has been applied to one

<figure><img src="image_15.png" /><figcaption>Fig. 13 INL for step sizes of 5 ns between 5 and 255 ns. The error bars show the dispersion of the five tests performed, each one with 10,000 measurements.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Fig. 14 Precision tests at different temperatures. At each temperature, the standard deviation of the measured ToT values with respect to the real ToT is represented.</figcaption></figure>

TDC channel. Each pattern consisted of 10,000 identical pulses with a duration of 5 ns, spaced by 50 ns. The precision ($\sigma_{LSB}$) is estimated as

$$ \sigma_{LSB} = \sqrt{\frac{\sum_{i=1}^{N} (\text{ToT}_i - \text{ToT}_{\text{real}})^2}{N}}, \qquad (7) $$

where $\text{ToT}_i$ corresponds to each of the 10,000 ToT measurements performed by the TDCs, $N$ is the total number of pulses, and $\text{ToT}_{\text{real}}$ is the real value of the pulse width applied, 5 ns in this case. As can be seen in Fig. 14, the precision is quite satisfactory for the entire TDC range varying from 0.5 ps at -35°C to 2.2 ps at 60°C. The value of the $\sigma_{LSB}$ is ~2 ps at 25°C and does not degrade significantly with temperature.

# 4 Data Processing: The State Machine

The firmware block called SM implements a seven-state mealy finite state machine (FSM)²¹ responsible for downloading the TDC data stored in the FIFO memories, enclosing the data in a UDP packet and sending it to the buffer stream selector called buffer stream IP multiplexer (IPMux). The IPMux sends these data to the shore station through the optical link using UDP frames. As already mentioned previously, the data flow is handled in time slices with a duration of 100 ms. Each time slice is identified by its start time in coordinated universal time (UTC), allowing for its unequivocal identification. The time stamps of the digitized data in the time slice are relative to the time slice start UTC time.

## 4.1 Digital Data Formatting

There are three communication channels: one for the TDC data, another for the acoustic data codec using the Audio Engineering Society (AES) protocol, and a channel for the monitoring data (explained in Sec. 4.2). The SM organizes the TDC data packets in two different segments: the header (see Table 6) and the TDC data. The first double word (32 bits) of the header is formed by the identifier of the communication channel sending the data. The second double word refers to the run number. The run number refers to a global identifier for a determined time-span,
<table><caption>**Table 6** TDC format header.</caption><thead><tr><td>Field</td><td>Number of bits</td></tr></thead><tbody><tr><td>Type ("TTDC")</td><td>32</td></tr><tr><td>Run number</td><td>32</td></tr><tr><td>Frame index</td><td>32</td></tr><tr><td>UTC time</td><td>64</td></tr><tr><td>Trailer (1 bit) &amp; FIFO almost-full status (31 bits)</td><td>32</td></tr><tr><td>Time valid (1 bit) &amp; HRV status (31 bits)</td><td>32</td></tr></tbody></table>

usually lasting for several hours, during which time data are taken with a fixed set of input parameter values to control the detector and the readout. The frame index field contains the identification of the UDP packet within the current time slice. The next field is the start time (in UTC) of the time slice, which is coded with two double words (64 bits). The penultimate double word of the header is reserved for a status register, where the status of the “almost-full” FIFO flag is coded for each TDC channel. The first bit of this word indicates whether it is the last packet of the time slice or a trailer packet. The first bit of the last double word shows whether the CLB is synchronized and the time is valid, and the remaining 31 bits indicate the status of the HRV of each TDC channel. Table 6 summarizes the different fields used in the header of the UDP packets. The UDP packets are completed with TDC hits, whose format is described in Table 2.

## 4.2 Firmware Architecture of the State Machine

The SM waits for the first UTC time provided by the White Rabbit Precision Time Protocol (Core, called SuperTime (ST), to start to operate. Once the first UTC time is received, the header is created. The header of the UDP packet consists of seven double words (224 bits). After creating the header, the data download process starts. A round-robin procedure has been implemented to download the information from all the channels. The procedure tries to download events from one channel as much as possible to reduce the dead time during the channel transitions. The maximum hit rate the firmware can handle is 600 kHz per channel, which means a throughput of 0.9 Gbps. All the data are sent to the shore station through an optical network based on switches with a maximum throughput of 1 Gbps. Nine DOMs will be connected per switch, so the maximum hit rate allowed by the switches is 74 kHz per channel.

The TDC data are structured in blocks of 48 bits. The SM splits the information in units of 16 bits. The download process continues until a special marker is detected in the FIFO memory. When it is detected, the data download for that channel is blocked until the beginning of the next time slice. If the maximum number of bytes for the UDP payload is achieved, the SM ends the current UDP packet and starts another one with the same header increasing the index number by one. When the time slice marker is detected in all the TDC channels, indicating that there are no more hits for that time slice, the packet is ended, a special trailer packet is sent and a new packet, belonging to the next time slice, is started.

Once the download is complete, TDC data corresponding to a certain time slice, a special packet is sent to the shore station, the trailer packet. This packet contains only a header with the trailer packet bit set. The trailer packet does not incorporate any TDC data. If all the FIFOs are emptied before the end of the time slice, the SM will stay waiting for new data. A 32-bit register contains the empty status of all the TDC memories, thus by checking this register, the SM will start downloading the data again. The process is shown in Fig. 15.

Another important block managed by the SM is the monitoring channel, where monitoring data are forwarded to the IPMux. Each time slice, a packet containing the UTC time, DOM identifier, the UTC FIFO status, and the TDC hit counters (one per TDC channel) is transferred. The memory information, which resides in the LM32 dual-port memory address space, is also
<figure><img src="image_17.png" /><figcaption>**Fig. 15** Diagram of the mealy FSM implemented in the firmware readout. The diagram describes the operation of the TDC SM.</figcaption></figure>

<table><caption>**Table 7** The 12-Wishbone registers used to control and monitor the SM and the IRQ for both TDC and acoustic channel</caption><thead><tr><td>LM32 base Addr</td><td>Register</td><td>Name</td></tr></thead><tbody><tr><td>x00020B00</td><td>0x00</td><td>Number of payload bytes per UDP packet</td></tr><tr><td>0x00020B04</td><td>0x01</td><td>DOM ID</td></tr><tr><td>0x00020B08</td><td>0x02</td><td>Run number</td></tr><tr><td>0x00020B0C</td><td>0x03</td><td>Time slice duration (16 ns clock ticks)</td></tr><tr><td>0x000020B10</td><td>0x04</td><td>Control status</td></tr><tr><td>0x00020B14</td><td>0x05</td><td>Monitor memory base</td></tr><tr><td>0x00020B18</td><td>0x06</td><td>Monitor memory words</td></tr><tr><td>0x00020B1C</td><td>0x07</td><td>TDC full IRQ</td></tr><tr><td>0x00020B20</td><td>0x08</td><td>TDC full IRQ mask</td></tr><tr><td>0x00020B24</td><td>0x09</td><td>AES full IRQ</td></tr><tr><td>0x00020B28</td><td>0x0A</td><td>AES full IRQ mask</td></tr><tr><td>0x00020B2C</td><td>0x0B</td><td>TDC - AES ST</td></tr></tbody></table>

transferred. The memory base address is a double word aligned and can be set via the monitoring memory base address register. The amount of double words to be read out by the monitoring SM can be set via the monitoring memory words register (see SM registers in Table 7). The amount of words that can be read is limited to 256 to fit in a standard Ethernet packet (1500 bytes) to reduce the data rate.

The percentage of resources used by the TDC SM IP is negligible with respect to the total resources of the Kintex-7 160T FPGA (see Table 8).

## 5 Test Setup

The first test setup used for evaluating the TDC firmware consisted of a KC705 Xilinx evaluation board, where a first version of the TDCs was implemented, and a ML605 Xilinx evaluation board
<table><caption>**Table 8** Logic, register, memory, and clock resources used in the implementation of the state machine IP for TDC, acoustic, and monitoring channel.</caption><thead><tr><td>Resources</td><td>Used</td><td>Total available</td><td>Used fraction (%)</td></tr></thead><tbody><tr><td>Look-up table</td><td>3442</td><td>101,400</td><td>3.4</td></tr><tr><td>Registers</td><td>3366</td><td>202,800</td><td>1.7</td></tr><tr><td>Memory (36 Kb)</td><td>2</td><td>325</td><td>0.6</td></tr><tr><td>Clocking</td><td>2</td><td>32</td><td>6.3</td></tr></tbody></table>

<figure><img src="image_18.png" /><figcaption>**Fig. 16** Test bench setups. (a) The evaluation KC705 board (circuit on the right) where the first TDCs were implemented, connected to a ML605 board (on the left) acting as pattern generator. (b) CLB being stimulated for testing the TDC with a ML605 board.</figcaption></figure>

used for generating the pattern supplied as inputs to the TDCs. In a further step, the KC705 board was replaced by a CLB. Both test setups are shown in Fig. 16. For temperature tests, a climatic chamber from DYCOMETAL was used. For all the tests, the data were sent from the ML605 through a CAT-5 cable using LVDS signals.

## 6 Data from Deployed DUs

Several DUs have already been deployed at the ARCA and ORCA sites, whose acquired data are available for analysis and allow for validating the real operation of the acquisition firmware. The bioluminescence observed for a single TDC channel is shown in Fig. 17, where the drop generated by the activation of the HRV in the TDCs is observed. The maximum rate measured is 20 kHz, corresponding to an HRV set to 2000 hits for a time slice of 100 ms. An example of the data obtained from the TDCs when the multihit option is disabled is shown in Fig. 18, where the ToT distribution for a given PMT installed in a DOM of a deployed DU is presented. The maximum ToT value is 255 ns, which is the maximum range of the TDCs when the multihit option is disabled. As previously explained, the hits with a duration longer than 255 ns are truncated to this maximum value, producing an accumulation of hits at 255 ns. Figure 19 shows the first hits obtained when the multihit option is enabled. The consecutive hits with a duration of 255 ns, resulting from the segmentation of one physical event longer than 255 ns, are properly merged.

The DNL has also been obtained from a collection of detector data. In Fig. 20, the DNL measurements performed for deployed DUs are represented. All the 31 TDC channels, with over 6 million of hits, have been evaluated. The DNLs have been calculated for the four cycles of the IOSERDES but only the highest value is represented.
<figure><img src="image_19.png" /><figcaption>**Fig. 17** Distribution of the detection rate during one physics run of 6 h. Data correspond to one unique channel of two deployed DOMs. TDC HRV is active and limits the maximum rate to 2000 hits per time slice per channel.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>**Fig. 18** Distribution of the ToT hits recorded by a given PMT of a deployed DOM when the multihit option is disabled. The ToT range ends at 255 ns, where an accumulation of hits is produced and expected when this option is enabled.</figcaption></figure>

# 7 Summary

The architecture and performance of the front-end firmware for the KM3NeT have been presented. The front-end firmware has been developed to manage 31 TDC channels implemented in a Xilinx Kintex-7 160T FPGA occupying very low resources and providing the required resolution (1 ns) with a high precision and low DNL and INL. The implementation in an FPGA provides the flexibility to modify the front-end logic and to integrate other systems in the same device; therefore, it is not necessary to include additional components to digitize events, which increases reliability. The TDCs have a dead time of 5 ns. Two features have been implemented
<figure><img src="image_21.png" /><figcaption>**Fig. 19** ToT values of the first hits recorded in the 31 channels of a DOM at the beginning of the run. The multihit option is enabled and working in channels 12, 13, and 17.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>**Fig. 20** DNL values obtained from one DOM of the deployed DUs data. More than 6 million events were measured. The DNL were computed for the four bins in the 31 channels, but only the maximum absolute value of each channel is shown in the figure.</figcaption></figure>

to improve their performance: the multihit, which allows for the recording of long duration hits, and the HRV, which dynamically limits the maximum DAS rate. The performance of the acquisition firmware is almost independent of the temperature. The test bench setup, where the front-end firmware has been evaluated, has also been described and the results presented. Finally, the data obtained from the first deployed DUs have been analyzed showing agreement with the results obtained in the laboratory. The DAS front-end and readout firmware have been validated and are now successfully running at the two sites of KM3NeT.

## Acknowledgments

The authors acknowledge the financial support of the funding agencies: Agence Nationale de la Recherche (contract ANR-15-CE31-0020), Centre National de la Recherche Scientifique (CNRS), Commission Européenne (FEDER fund and Marie Curie Program), Institut Universitaire de France (IUF), LabEx UnivEarthS (ANR-10-LABX-0023 and ANR-18-IDEX-0001), Paris Île-de-France Region, France; Shota Rustaveli National Science Foundation of Georgia (SRNSFG, FR-18-1268), Georgia; Deutsche Forschungsgemeinschaft (DFG), Germany; The
General Secretariat of Research and Technology (GSRT), Greece; Istituto Nazionale di Fisica Nucleare (INFN), Ministero dell'Istruzione, dell'Università e della Ricerca (MIUR), PRIN 2017 program (Grant NAT-NET 2017W4HA7S) Italy; Ministry of Higher Education Scientific Research and Professional Training, ICTP through Grant AF-13, Morocco; Nederlandse organisatie voor Wetenschappelijk Onderzoek (NWO), the Netherlands; The National Science Centre, Poland (2015/18/E/ST2/00758); National Authority for Scientific Research (ANCS), Romania; Ministerio de Ciencia, Innovación, Investigación y Universidades (MCIU): Programa Estatal de Generación de Conocimiento (refs. PGC2018-096663-B-C41, -A-C42, -B-C43, -B-C44) (MCIU/FEDER), Severo Ochoa Centre of Excellence and MultiDark Consolider (MCIU), Junta de Andalucía (ref. SOMM17/6104/UGR), Generalitat Valenciana: Grisolía (ref. GRISOLIA/2018/119) and GenT (ref. CIDEGENT/2018/034) programs, La Caixa Foundation (ref. LCF/BQ/IN17/11620019), EU: MSC program (ref. 713673), Spain.

## References

1. KM3NeT Collaboration, "KM3NeT: technical design report for a deep-sea research infrastructure in the Mediterranean Sea incorporating a very large volume neutrino telescope," 2009, http://www.km3net.org.

2. R. Lahmann, "The neutrino telescope of the KM3NeT deep-sea research infrastructure," *Phys. Procedia* **37**, 1209-1216 (2012).

3. R. Coniglione, "KM3NeT-ARCA project status and plan," *EPJ Web Conf.* **116**, 11003 (2016).

4. J. Brunner, "KM3NeT—ORCA," in *17th Int. Workshop Neutrino Telesc. Proc.*, Venice, Italy (2018).

5. T. Chiarusi and M. Spurio, "High-energy astrophysics with neutrino telescopes," *Eur. Phys. J. C* **65**, 649-701 (2010).

6. M. Circella, "The digital optical module (DOM) for the KM3NeT detector," in *Proc. 33nd ICRC*, Rio de Janeiro (2013).

7. E. Leonora and V. Giordano, "Design and production of the digital optical module of the KM3NeT project," *EPJ Web Conf.* **136**, 4008 (2017).

8. D. Real and D. Calvo, "Digital optical module electronics of KM3NeT," *Phys. Part. Nucl.* **47**, 918-925 (2016).

9. D. Real et al., "KM3NeT front-end and readout electronics system: hardware, firmware, and software," *J. Astron. Telesc. Instrum. Syst.* **5**(4), 046001 (2019).

10. Xilinx, "7 series FPGAs data sheet: overview," 2018, https://www.xilinx.com/support/documentation/data_sheets/ds180_7Series_Overview.pdf (accessed 6 June 2020).

11. Xilinx, "7 series FPGAs select IO resources," 2018, https://www.xilinx.com/support/documentation/user_guides/ug471_7Series>SelectIO.pdf (accessed 6 June 2020).

12. M. Lipiński et al., "White rabbit: a PTP application for robust sub-nanosecond synchronization," in *IEEE Int. Symp. Precision Clock Synchron. Meas., Control and Commun.*, pp. 25-30 (2011).

13. M. Ageron et al, "Dependence of atmospheric muon flux on seawater depth measured with the first KM3NeT detection units," *Eur. Phys. J.* **80**, 99 (2020).

14. R. Herveille, "Wishbone system-on-chip (SoC) interconnection architecture for portable IP cores," 2012, https://wishbone-interconnect.readthedocs.io/en/latest/ (accessed 16 March 2020).

15. M. Fries and J. Williams, "High-precision TDC in an FPGA using a 192 MHz quadrature clock," in *IEEE Nucl. Science Symp. Conf. Record*, Vol. 1, pp. 580-584 (2002).

16. M. Beretta et al., "The characterization and application of a low resource FPGA-based time to digital converter" *Nucl. Instrum. Methods Phys. Res., Sect. A* **739**, 75-82 (2014).

17. C. Pellegrino and T. Chiarusi, "The Trigger and Data Acquisition System for the KM3NeT neutrino telescope," in *EPJ Web of Conf.*, Vol. **116**, p. 05005 (2016).

18. J. Kalisz, M. Pawlowski, and R. Pelka, "Error analysis and design of the Nutt time-interval digitizer with picosecond resolution," *J. Phys. E Sci. Instrum.* **20**, 1330 (1987).

19. M. Correa and F. Pérez, "Characterization and optimization of an optical and electronic architecture for photon counting," *J. Phys. Conf. Ser.* **1002**, 012002 (2018).
20. J. Kalisz, "Review of methods for time interval measurements with picosecond resolution," *Metrologia* **41**, 17–32 (2004).

21. Xilinx, “Finite state machines,” 2015, https://www.xilinx.com/support/documentation/university/Vivado-Teaching/HDL-Design/2015x/VHDL/docs-pdf/lab10.pdf (accessed 6 June 2020).

**David Calvo** is a PhD candidate and research engineer at the Instituto de Física Corpuscular of Valencia. He received his MS degree in computing from the University Jaume I in 2006, his MS degree in electronics from the University of Valencia in 2009 and his MS degree in electronic systems design from the Polytechnic University of Valencia in 2012. His research interests are focused on digital electronics, synchronization, and readout acquisition systems. He is the author of several publications on electronics.

**Diego Real** is a PhD in physics and research engineer at the Instituto de Física Corpuscular. He received his BS degree in electronics in 1997 and his MS degree in control and electronics in 2000, both from the Polytechnic University of Valencia. He is the author of several publications on electronics. His current research interests include acquisition and synchronization systems for particle physics. He has been the electronics project leader of the KM3NeT telescope and member of the technical advisory board of the GVD-Baikal telescope since 2013.

Biographies of the other authors are not available.