Multi-threading for ESO Pipelines
Lander de Bilbao1, 2, 3, *, Lars Kr. Lundin1, Pascal Ballester1, Klaus Banse1, Carlo Izzo1, César E. García-Dabó4, Ralf Palsa1
1: European Southern Observatory – Software Development Division, Karl-Schwarzschild-Str. 2 D-85748 Garching bei München, Germany
2: Instituto de Física de Cantabria – CSIC, Avda. de los Castros s/n E-39005 Santander, Spain
3: Fundación Española para la Ciencia y la Tecnología, Rosario Pino 14-16 E-28020 Madrid, Spain
4: FRACTAL SLNE, Tulipán 2, portal 13, 1. A E-28231 Las Rozas de Madrid, Spain

Abstract. The second generation of instruments for ESO's VLT at Paranal Observatory will increase dramatically the volume of raw data per night. This inturn leads to very high computational needs that can be, up to some level, addressed by existing multi-core, shared-memory computers. To fully utilize these
multi-processor systems we need to implement a programming environment supporting multi-threaded execution of applications. Such parallel execution
needs to be introduced at the level of pipeline recipes as well as within the Common Pipeline Library (CPL)[1] on which all operational VLT pipelines of ESO
are based. We describe our approach to providing such a new, multi-threading pipeline setup and evaluate possible implementation solutions with some
performance measurements.

1. Introduction

4. VIRCAM / VISTA

ESO currently maintains a number of instrument pipelines for the VLT. They consist of anumber of sequential recipes, which are built on top of a common, thread-unsafe library(CPL). Also the second generation instrument pipelines, currently in development, are basedon the CPL. They vary significantly with respect to their algorithms.

VIRCAM is the most demanding pipeline [4] in terms of computational resources at the timeof writing. It is a 16-chip instrument and therefore it offers a great potential for parallelization.
This makes it an excellent testbed to assess the potential of OpenMP.
From a set of 2048 x 2048 flat images, the recipe
vircam_twilight_flat_combine removes any image whichis saturated or underexposed, linearizes the remaining
ones, removes the dark current from them, combinesthem with rejection and normalizes the result by its
median.

Initial attempts to parallelize recipe execution have been done by enabling, via batchscheduling, the concurrent execution of independent data reduction processes, showing
limited potential. Therefore the option of multi-threading recipes is considered as well.
Our aim is to enable the implementation of multi-threading recipes on top of a thread-safe
CPL, while still coping with the maintenance of the legacy sequential code of the existing
pipelines. We do not intend to develop a new thread-safe library but rather to modify the CPL
where necessary. For this purpose we have chosen OpenMP [2].

VISTA Focal Plane, 16 detectors on
their mount. Credit: VISTA

2. CPLParallelism at the FITS extension level is the obviousstrategy. However, it is not the most efficient one due to
its lack of data locality (repeated reading of data fromthe cache reducing the high-latency access to mainmemory).

A combined strategy, introducing parallelism a) at the FITS extension level and b) at theimage level, is applied.

The CPL includes a number of global variables which make the library inherently threadunsafe. In some cases, global variables cannot simply be made local. The necessary API
modifications would impose an unfeasible amount of code changes in the CPL. This is thecase with the error handling modules.
A solution to this problem is to provide private copies of global variables for each thread. Thisis easily enabled using simple OpenMP pragmas:
static cpl_boolean cpl_error_status = CPL_FALSE;
static cpl_boolean cpl_error_read_only = CPL_FALSE;At the first level (a) the introduction of the following pragmas is needed:#pragma omp threadprivate ([global variable list])#pragma omp parallel for ordered\private([private variable list]) \shared([shared variable list])\schedule(dynamic)\num_threads([Nth])for ( ; ; ) {/* Code to process each extension */#pragma omp threadprivate(cpl_error_status, cpl_error_read_only)This approach implies new semantics for error handling in a multi-threading environment.

#pragma omp orderedsave_ext();}3. VISIR

The ordered construct ensures that the products are appended to the result FITS file in theproper order.

A first candidate for parallelization are the VISIR[3] science recipes, for which an evennumber of 256 x 256 (nodding + chopping) images are loaded into memory (512kB RAMeach), normalized with respect to DIT (Detector Integration Time) and pairwise added. The
final step consists of destriping the added images, using an iterative filtering.

At the second level (b) several pragmas of the type already used in VISIR are added on topof every loop traversing the pixel buffer. In order to avoid false sharing (undesired writing to asingle cache line by more than one thread) at this level, an alternative memory allocator,
assuring cache line alignment, HOARD[5], has been used.

Parallelization can be achieved in this case by adding, on top of the loop over the images, asingle pragma of the type:Fig. 3 shows that scalability is rather poor. Images are large; the recipe execution is rather
memory-intensive. The 8 cores available share 2 L2 caches only, while during the sequentialexecution, a single process has a whole L2 cache at its disposal. Applying nested levels ofparallelism improves the data locality and shows better results. Nevertheless, this multithreaded solution pays off in comparison with the parallelization via batch scheduling (in thiscase, launching a separate data reduction process for each chip), which offered a speedup,
in regard to the sequential execution, only close to 2. Here the maximum speedup is 4.69
(59% efficient) on 16 threads, with 4 threads at the extension level, then each one launchingother 4 at the image level.

#pragma omp parallel for private([private variable list]) schedule(static, 2)for ( ; ; ) {/* Code for each iteration */}The schedule construct is added due to the need to process the frames pairwise.

Results with VISIR

Figure 3:
Speedup
inVISTA
execution, using different
parallelization strategies(see 6 for hardware
specification). There are
4 variants: single threadat the image level (red), 2
threads at the image level(purple), 4 threads at theimage level (blue) and 8
threads at the image level(green).

SpeedupSpeedup

Figure 1:
Speedup in VISIR execution (see 6for hardware specification).
It shows a good scalability up to 4threads, with a maximum of 5.89
on 8 threads (74% efficient). On 6threads, the speedup is lower thanexpected, because this amount ofthreads is not optimal for themachine in use.

Results with VISTA

Total number of threads

Total number of threads

5. Conclusions and future work

6. Data sheet

Given a suitable parallelization strategy, multi-threaded recipes bring substantial advantage
in performance. Nevertheless, their scalability is limited by the nature of the reduction task
itself. The optimal number of threads might not necessarily coincide with the number ofavailable cores. For future machines, a combined solution where several independent data
reduction processes are launched, each process in turn launching in a number of threads, isexpected to be the most efficient one.

The machine used for these tests is a workstation with 8 cores, the Intel Xeon E5420 (Dualdie @ 2.50Ghz, 4-cores + 6MB cache each). The multi-threaded recipes have been built on
top of an internal development version of CPL 5.1 with CFITSIO 3.181 (reentrant).

The parallelization strategy presented with the VISIR pipeline will be very likely adopted by
the pipeline for the second generation instrument MUSE in order to reduce data cubes.

7. References[1] http://eso.org/cpl[2] http://openmp.org[3] http://eso.org/sci/facilities/paranal/instruments/visir[4] http://eso.org/sci/facilities/paranal/instruments/vista[5] http://www.hoard.org/

With regard to future work, the consequences of the new semantics for error handling in CPL
must be analyzed, especially the case where one might need to propagate an error fromwithin a parallel region. Implementing multi-threaded versions of highly computationdemanding CPL functions is also a possibility.
(*) Lander de Bilbao’s position at ESO is funded by Spain as part of the in-kind contribution to ESO. Contact: lbilbao@eso.org