The Karabo Control System
S. Hauf*, N. Anakkappalla, J. H. Bin Taufik, V. Bondar, R. Costa,W. Ehsan, S. Esenov, G. Flucke, A. Garcia-Tabares,G. Giovanetti, D. Goeries, D. Hickin, I. Karpics, A. Klimovskaia, A.
Parenti, A. Samadli, H. Santos, A. Silenzi, M. A. Smith, F. Sohn,
M. Staffehl, C. Youngman*steffen.hauf@xfel.eu

19TH INTERNATIONAL CONFERENCE ON
ACCELERATOR AND LARGE EXPERIMENTAL
PHYSICS CONTROL SYSTEMS (ICALEPCS 2023)

Tswana: the answer

The Karabo Control System
S. Hauf*, N. Anakkappalla, J. H. Bin Taufik, V. Bondar, R. Costa,W. Ehsan, S. Esenov, G. Flucke, A. Garcia-Tabares,G. Giovanetti, D. Goeries, D. Hickin, I. Karpics, A. Klimovskaia, A.
Parenti, A. Samadli, H. Santos, A. Silenzi, M. A. Smith, F. Sohn,
M. Staffehl, C. Youngman*steffen.hauf@xfel.eu

19TH INTERNATIONAL CONFERENCE ON
ACCELERATOR AND LARGE EXPERIMENTAL
PHYSICS CONTROL SYSTEMS (ICALEPCS 2023)

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

The European XFEL*The development of light source facilities
has been faster than the increase incomputer processing capacity (i.e.,Moore’s Law)*

We see this in the amount of data
generated. For EuXFEL this can be
multiple PetaByte/week. The Data
Acquisition System is implemented inKarabo, as are the starting points of theonline preview systems which support
near-realtime processing of >3kHz
Mpixel images .

3

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

4Beamline layout and experiment stations

KaraboDOOCS*Diagnostic data from theaccelerator is importantfor experiment analysis*Bridges between DOOCS
and Karabo are collaboratively
maintained with DESY andfacilitate transfer between
the two systems (see alsoTUMBMO3)

1 MB/s - 100 MB/s
S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

The Karabo Control System

10 Hz burst

5

Karabo connects it all – Example: MIDAGIPD
Time server*Key compoenents to look out
for in all topics:* Timeserver: a central
communication point for
timing information* Bespoke MHz imagingdetectors* Commercial cameras* Fast digitizers* Large Beckhoff loops,often interconnected via
middlelayer devices andinterlock conditions* Processing pipelines,
e.g. detector calibration

15 GB/s
10 Hz burst

Beckhoff
1 B/s - 1 MB/sevent driven

1 MB/s - 1 GB/s
10 Hz burst

The Karabo Control System

Karabo - Architecture*

Central Message Broker (Control and slow data)* Currently: OpenMQ* Soon RabbitMQ.

*

Event driven:* Data propagates through the system whenvalues change – push not poll

*Message driven:* Signal – Slot paradigm* Asynchronous core, synchronousconvenience in middleware

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

6

The Karabo Control System

7S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo - Architecture*Central Message Broker (Control and slow data)* Currently: OpenMQ* Soon RabbitMQ.

*

Event driven:* Data propagates through the system whenvalues change – push not poll

*Message driven:* Signal – Slot paradigm* Asynchronous core, synchronous
convenience in middleware

Lesson learned: Operators accustomed
to a control system that polls data, may
need time to adjust to, and trust event
driven data. Is everything working, even if
it does not update once per second?

GG(1The Karabo Control System

Karabo - Architecture*pipeline (p2p) connections (scientific/large) data
* Scatter/Gather/Copy/Distribute* Block/Drop on congestion* TCP* Also GUI Server – GUI client* Capable of saturating a 10G line

*Dynamic, discoverable topology* No central database instance*

GUI Server:* Gateway to the Control system

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

8

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo - Architecture*pipeline (p2p) connections (scientific/large) data
* Scatter/Gather/Copy/Distribute* Block/Drop on congestion* TCP* Also GUI Server – GUI client* Capable of saturating a 10G line

*Dynamic, discoverable topology* No central database instance*GUI Server:* Gateway to the Control system

Lesson learned: Fortunate design
choice, when remote access was still far
at the horizon. Trusted, and restricted
protocol to remote clients. Currently
implementing authentication. Server isfully under our control and can enforce
e.g. read-only access.

9The Karabo Control SystemS. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

10Metrics in Influx: > 240 Billion
Increase per month: ~ 10 Billion

Karabo – Data Logging using the Influx Time Series Database**Datalogging vs. Data Acquisition* Datalogging is continous for slow (broker) data
► It is done by default► For all devices► Internal data product for maintenence* Data Acquisition is „run“ based► Explicitely started► Includes large and fast data
► Subselection of slow data
► Data product for facility users► Multiple PB per week per instrument
Data Operation
Karabo dataloggers* Proprietary text-based format* Influx Time-series based

Center (DOC) main
Grafana panel. Most
data here is from theKarabo Influx loggers

Trendlinein Karabo
showingtime evolutionof device
state.

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

11Metrics in Influx: > 240 Billion
Increase per month: ~ 10 Billion

Karabo – Data Logging using the Influx Time Series Database**Datalogging vs. Data Acquisition* Datalogging is continous for slow (broker) data
► It is done by default► For all devices► Internal data product for maintenence* Data Acquisition is „run“ based► Explicitely started► Includes large and fast data
► Subselection of slow data
► Data product for facility users
Lesson learned: The event driven design► Multiple PB per week per instrument
Data Operationof Karabo
very well to InfluxDB’s
Center(DOC) matchesmainGrafanapanel.
Most
design of ingesting non-regular spaced
Karabo dataloggers
data
is fromthe However the irregulartimehereseriesdata.
KaraboInfluxloggers* Proprietary text-based format
grid can be challenging for some analysis* Influx Time-series basedtasks that use this data.

Trendlinein Karabo
showingtime evolutionof devicestate.

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo – C++, Karabo Bound, and Karabo Middlelayer APIs
GeneralC++ API

Python Bound API

Middlelayer API•••
•

•
•
•
•
•
•

• Exposes C++ API via
Boost.Python• Devices are separate
processes• Was aimed at p2p
heavy devices whiche.g. need numpy• Not always pythonic• Python asyncio• Decorators annotate
Karabo structures• Emphasis on
interaction with otherdevices• Pythonic

Event driven
Asynchronous
Self-descriptiveCommon, hierarchicaldata container
supporting attributes
on leafs: Karabo Hash• Binary and XMLserialization• Extensible: core +“Devices“C++14 and Boost
Smart pointers
Template-heavy
Boost.asio
Eventloop based
Devices are threads
on a single server• Aimed at highperformance devices

12

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

13Common Example: Concurrent Motion
From Bluesky documentation
Other systems:- Bliss e.g. motor_group- Sardana: moveMultiple- ….
- Karabacon: MotorGroupKarabo Middlelayer

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Common Example: Concurrent Anything…
MotionPower Supplies

Instantiation

14

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Common Example: Concurrent Anything…
MotionKarabo Middlelayer
prefers
using Python ”primitives” where possible over domain specific extensions.
PowerSupplies
 Learn once – use often

Instantiation

15

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

The Karabo Ecosystem – Usage of the three APIs
*Middlelayer API frequently has themost expressive syntax coupledwith shortest “time-to-market”.

*

C++ and Python Bound are actively
maintained and new devices are still
being implemented, especially inhigh-performance fields.

16

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

The Karabo Ecosystem – Usage of the three APIs*

Middlelayer API frequently has themost expressive syntax coupledwith shortest “time-to-market”.

*

C++ and Python Bound are actively
maintained and new devices are still
being implemented, especially inhigh-performance fields.

Lesson learned: the concise andexpressive syntax is usually preferred bydevelopers in the EuXFEL Controls group,and MDL is also the API that most device
developers are most accustomed to.

17

The Karabo Control System

Karabo - The Karabo GUI*

Separate Python Package, wellmatched to the framework

*

PyQt5

*

Connects to Karabo via the GUIserver (tcp, p2p)*

Extensible via „gui-extensions“*

Distinguishing features:* GUI scene builder* Projects to logically groupdevices, scenes and macros

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

18

The Karabo Control SystemKarabo - The Karabo GUI*

Separate Python Package, wellmatched to the framework

*

PyQt5

*

Connects to Karabo via the GUIserver (tcp, p2p)*

Extensible via „gui-extensions“*Distinguishing features:* GUI scene builder* Projects to logically groupdevices, scenes and macros

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

19

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo - The Karabo GUI*

Separate Python Package, wellmatched to the framework

*

PyQt5

*Connects to Karabo via the GUIserver (tcp, p2p)*

Extensible via „gui-extensions“*Distinguishing features:* GUI scene builder* Projects to logically groupdevices, scenes and macros

Lesson learned: A coding-free option ofdefining synoptic views enables operators
to create a control environment well suited
to their tasks. The downside is that scenes
are quickly created, and then forgotten,
leading to maintenance overhead.

20

The Karabo Control SystemS. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo - The Karabo GUI*

Separate Python Package, wellmatched to the framework*PyQt5*Connects to Karabo via the GUIserver (tcp, p2p)

*Extensible via „gui-extensions“*Distinguishing features:* GUI scene builder* Projects to logically groupdevices, scenes and macros

Lesson learned: A coding-free option ofdefining synoptic views enables operators
to create a control environment well suited
to their tasks. The downside is that scenes(and projects) are quickly created, andthen forgotten, leading to maintenanceoverhead.

21

The Karabo Control System

22

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo Hardware Devices – an incomplete list of what is supportedCore Services•
•••
•
•
•

Data Logging
Data Acquisition
Gui Server
Alarms & Notification
Recovery Portal
Motor Configurator
Scan ToolDigitizers••••

SP devices ADQ 412
SP devices ADQ 7
SP devices ADQ 14
FastADCMotion•••••

Beckhoff MC2*Smaract
Hexapod
NanoCube…

Power Supplies•••

Wiener MPOD
Keithley
Agilant

Commercial Cameras

X-ray Detectors

••
•
•
•
•

•
•
•
•
•
•

Basler via Aravis
GenicamGreatEyes
PI MTE3
Andor Zylar, …
Shimadzu HPvx

Jungfrau •Gotthard •EpixAGIPD
LPD
DSSC

Scopes and …meters
Timepix3…

•••••

OceanOptics
GENTEC
Tetronix
LeCroyMCS Beam Stab.
…

Vacuum Components*Chillers & Thermo Contr.*Bridging••••••••••••••

AdixonPfeiffer
Infinicon
Agilant

Huber
K2
JulaboKeithleyLakeshore
…

SCPI
DOOCS
EPICS (in progress)
Tango (in progress)* via Beckhoff PLC integration

The Karabo Control System

MO2BCO04 S. Huynh: Applying Standardised Software
THPDP023 S. Samadli: Evolution of Control System and PLC
S. Hauf for the ControlsArchitecturalGroup - ICALEPCS2023, 12.10.23
Concepts
to Design Robust and Adaptable Plc 23
Integration at the European XFEL
Solutions
TUSDSC03 N. Mashayekh: Integrating Tools to Aid the Automationof PLC Development Within the TwinCat Environment
THPDP021 N. Coppola: Equipment life-cycle Management at
EuXFEL

Karabo Hardware Devices – an incomplete list of what is supportedCore Services

MotionCommercial Cameras

MO2AO03 A. Garcia-Tabares: The Solid Sample Scanning
Workflow at the European XFEL• Basler via Aravis• Data Logging• Beckhoff MC2*
THPDP024F.
Sohn:AutomaticConfigurationof Motors at the• Genicam• Data Acquisition• Smaract
EuropeanXFEL• GreatEyes• Gui Server• Hexapod• PI MTE3• Alarms & Notification• NanoCube• Andor Zylar, …
• Recovery Portal• …
• Shimadzu HPvx
• Motor Configurator
WE1BCO02 J. Malka: Data Management Infrastructure for•
ScanTool
European XFEL

Digitizers••••

SP devices ADQ 412
SP devices ADQ 7
SP devices ADQ 14
FastADCPower Supplies•••

Wiener MPOD
Keithley
Agilant

MO4AO06 B. Fernandes: Overview and Outlook of Fpga
Based Hardware Solutions for Data Synchronization,Acquisition and Processing at the Euxfel

Vacuum Components*

Scopes and …meters

X-ray Detectors••••••

Jungfrau •Gotthard •Epix
AGIPD
LPD
DSSC

Timepix3…

••
•
•
•

Chillers & Thermo Contr.*

OceanOptics
GENTEC
Tetronix
LeCroyMCS Beam Stab.
…

Bridging• SCPI• Huber• Adixon• DOOCS•K2
THPDP022• PfeifferB. Rio: Adaptable Control System for the Photon
Beamlines
atEuropean
XFEL:
Integrating
NewDevices
and• EPICS (in progress)• Julabo• InfiniconTechnologies
forAdvancedResearch• Tango (in progress)• Keithley• Agilant• Lakeshore
TUPDP033• … M. Smith: Applying Model Predictive Control toRegulate Thermal Stability for a Hard X-ray Monochromator
Using the Karabo Software Control Framework* via Beckhoff PLC integration

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

24

Where is this going, and how can I get it?
Move to RabbitMQ and AMQP
WebUIRDMA – 80% IB link speed

https://github.com/European-XFEL/Karabo

Elog Integration

MDL on PyPi
Authenticating & Authorizing Web API{REST:API}*GUI side Authentication and Authorization

2024

ELK Stack for System Logs

2025The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

25

Where is this going, and how can I get it?
Move to RabbitMQ and AMQP
WebUIRDMA – 80Gb/s++

https://github.com/European-XFEL/Karabo

Elog Integration

MDL on PyPi
Authenticating & Authorizing Web API{REST:API}*GUI side Authentication and Authorization

2024

ELK Stack for System Logs

2025

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Karabo and Large Language Model: towards AI assistants*LLMs as code documentation assistants****

System prompt to ask AI to add or update documentation tocode using a diff format that doesn’t require counting lines
Works well to batch-document code lacking mostdocumentation
Karabo agnostic, available at https://github.com/EuropeanXFEL soon

LLMs as coding assistants****

GPT4 was not trained on Karabo code at the time of tests
System prompts describing the MDL API and the scene model
suffice
Iterative approach, feeding exceptions back into the model
https://syncandshare.xfel.eu/index.php/s/kt6NbSjJfMg7Pf526

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

27

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

Conclusion*Karabo is a flexible SCADA framework with a modern event-driven and asynchronous core.
* 3 APIs with individual strengths* State of the art condition logging using the Influx Time Series database* Highly scalable, and no central database authority required*The Karabo GUI is closely matched to the framework* Can flexibly visualize all data types the framework provides* Synoptic views (scenes) without coding* Extensible*

LLMs can document and code in Karabo reasonable well and offer opportunities in improving
development workflows.

*

Karabo (and some devices) have recently been open sourced:

https://github.com/European-XFEL/Karabo28

The Karabo Control System

S. Hauf for the Controls Group - ICALEPCS 2023, 12.10.23

29

Survey about Knowledge Sharing
among Software Developers and Scientists
Study objectives:*

Analyze the knowledge sharing practices
in scientific software development*Derive tool needs for developers and scientists

Access the survey 1:

Survey:*10-20 min*For everyone developing or using
control or data analysis systems at scientific facilities*

Win one of five 50€ vouchers
for an online shop of your choice
1 https://umfragen.uni-hamburg.de/index.php/975348?lang=en