JACoW : Parallel processing for the high frame rate upgrade of the LHC synchrotron radiation telescope

The Beam Synchrotron Radiation Telescope (BSRT) is routinely used for estimating the transverse beam size, profile and emittance in the LHC; quantities playing a crucial role in the optimisation of the luminosity levels required by the experiments. During the 2017 LHC run, the intensified analog cameras used by this system to image the beam have been replaced by GigE digital cameras coupled to image intensifiers. Preliminary tests revealed that the typically used sub-image rectangles of 128 × 128 pixels can be acquired at rates of up to 400 frames per second, more than 10 times faster than the previous acquisition rate. To address the increase in CPU workload for the image processing, new VME CPU cards (Intel 4 core/2.5GHz/8GB RAM) are envisaged to be installed (replacing the previous Intel Core 2 Duo/1.5GHz/1GB RAM). This paper focuses on the software changes proposed in order to take advantage of the multi-core capabilities of the new CPU for parallel computations. It will describe how beam profile calculations can be pipe-lined through a pool of threads while ensuring that the CPU keeps up with the increased data rate. To conclude, an analysis of the system performance will be presented. THE BSRT INSTRUMENT IN THE LHC The BSRT (Beam Synchrotron Radiation Telescope) is a very important instrument for the operation of the LHC and is currently the only system providing non-invasive measurements of the transverse beam size for calculation of emittance on a per-bunch basis. It does so by imaging the light intensity of visible and ultra-violet synchrotron radiation produced by the particles in each bunch. Synchrotron radiation is generated whenever the trajectory of a moving, charged particle is bent due to a magnetic field. This radiation is emitted in the forward direction, in a narrow cone tangentially to the trajectory. The emitted radiation power is proportional to the fourth power of the beam energy and inversely proportional to the square of the bending radius. In the particular case of the LHC, where the beam energy currently lies between 450GeV and 6.5TeV , one can see that the BSRT system is required to handle a change of more than 4 orders of magnitude in terms of radiation power. For this reason, the BSRT setup comprises a set of optical filters placed between the light source and the image intensifier, to keep the light intensity in a range where it will not damage either the camera or the image intensifier. A simplified diagram of the instrument setup is shown in Fig. 1. In this diagram, most of the complexity of the overall setup includ∗ diogo.alves@cern.ch ing the details of the optical telescope path, light splitters, lenses etc have been omitted. For more details see [1]. HARDWARE OVERVIEW The hardware setup comprises various items. On the instrument side itself, a subset of the components requiring real-time control are: • Digital camera Basler Ace area scan CMOS-based 1936 × 1216 resolution camera • Relays controls camera on/off state • Image intensifier enhances/controls light intensity seen by the camera • Relays controls on/off state of image intensifier • Pneumatic device control of combinations of optical filters • 3x step motors x,y,z position control of the camera • 2x stepmotors x,y position control of the image steering mirror The major change for the BSRT between the 2016 and 2017 LHC run was the replacement of the original analog cameras with digital cameras. The new cameras, Basler ACE acA1920-50gm [2], are CMOS-based cameras with 1936 × 1216 pixel resolution and a 1Gbit/s ethernet interface. Preliminary tests have shown that, when configured in free running mode and in steady state conditions, 128 × 128 pixels (typically used in LHC operations) can be streamed over ethernet at rates close to 400 frames/sec. Another crucial component of the overall system is the image intensifier which consists of a photo-cathode, a multi-channel-plate electron amplification stage and a fluorescent screen for enhancing the intensity of the light seen by the cameras. The connection between these components and their power supplies is controlled via relays using a serial interface. Such devices are also used to control the different optical filter combinations required to obtain the desired light attenuation. Finally, various stepping motors are used in real-time to control both the position of the camera and the image steering mirror to ensure the beam spot is always centred. The VME-based [3] hardware used for the control, data acquisition and real-time processing of the BSRT instrument comprises the following modules: • CPU1 runs the BSRT main controller as well as the controllers for the pneumatic actuators and relays • CPU2 runs the stepping motor controllers 16th Int. Conf. on Accelerator and Large Experimental Control Systems ICALEPCS2017, Barcelona, Spain JACoW Publishing ISBN: 978-3-95450-193-9 doi:10.18429/JACoW-ICALEPCS2017-TUPHA032 TUPHA032 442 Co nt en tf ro m th is w or k m ay be us ed un de rt he te rm so ft he CC BY 3. 0 lic en ce (© 20 17 ). A ny di str ib ut io n of th is w or k m us tm ai nt ai n at tri bu tio n to th e au th or (s ), tit le of th e w or k, pu bl ish er ,a nd D O I. Control Systems Upgrades • BOBR timing card that receives the LHC turn and bunch clock required for synchronising the system with the various bunches in the machine • STTY serial interface card • DAB general purpose FPGA motherboard used for triggering the bunch gating and the camera acquisition • VMOD general purpose board supporting mezzanines for analog and digital I/O • STEPCTRL control of the stepping motors Two VME crates are presently used for the online BSRT system. The VME compatible MEN-A20 CPU boards [4] employ an Intel Core 2 Duo (dual core) processor with a clock frequency of 1.5GHz per core and 1GB of RAM. The (Beam Synchronous Timing Receiver) BOBR card [5] is used to receive the synchronous LHC turn and bunch clock required for gating the camera images on each bunch. The Data Acquisition Board (DAB) [6] is a custom made CERN generic card designed by TRIUMF (Canada) with on-board digital signal processing capabilities. This is used for sending the gating signal to the image intensifier and for providing the hardware trigger for the camera. The VMOD, a general purpose card supporting mezzanines for analog and digital I/O, is mainly used in this system for driving the gain of the image intensifier (analog) and configuring the camera triggering mode (TTL). As depicted in Fig. 1, the synchrotron radiation emitted is channeled through an optical telescope assembly towards a mirror that steers the light through an appropriate set of attenuation filters to the image intensifier from which light is generated for the CMOS sensor of the camera. Both the mirror and the camera are mounted on moveable stages which are controlled in real-time via stepping motors using the in-house developed STEPCTRL card. CPU1, which runs the main BSRT controller, directly uses the VME bus to control the bunch gating, camera triggering and intensifier gain via the various VME cards installed in the same crate. It also controls the camera on/off status and optical filter combinations via a serial interface (STTY card). As for CPU2, it runs the controller for the various stepping motors. The camera configuration and control is done via ethernet with the frames also streamed from the camera to the main BSRT controller application via ethernet. CONTROL AND ACQUISITION SOFTWARE OVERVIEW The control and data acquisition software running on the CPUs described in the previous section has been developed using the CERN Front-End Software Architecture (FESA) [7] C++ framework for real-time systems. This framework provides not only highly customisable multithreaded scheduling facilities but also a tight integration the CERN control infrastructure including interfacing with S T T Y