ALMAWebQL v2

Technical Architecture

JVO Demonstration:
ALMAWebQL v2
Christopher ZAPART, Yuji SHIRASAKI
Japanese Virtual Observatory @ NAOJ

December 2016

1 / 10ALMAWebQL v2

Technical Architecture

ALMAWebQL v2Features:
an interactive preview of ALMA datasetsa rich Internet application built on AJAX,
HTML5 and SVG
a custom web server built on top of the GNUlibmicrohttpd C libraryreal-time image zooming
near real-time frequency spectrum updates
partial FITS downloads
HiDPI display support with automatic
image/font rescaling (4K display ready)2 / 10ALMAWebQL v2

Technical Architecture

future releases
reduce the graphics size (slow to access fromabroad?)
real-time spectrum updates over slow
networks
fully integrate ALMA OT / Splatalogue spectral
lines
polarisation support
handle >1TB FITS files
automatic source velocity resolution (wheredo you get reliable redshifts from?)
what would YOU like to see?
3 / 10ALMAWebQL v2

Technical Architecture

Modern client-server architecture

4 / 10ALMAWebQL v2

Technical Architecture

server: NVMe PCI-E SSD
10GB FITS files load in a few seconds

NVMe PCI Express SSDIntel DC P3600 1.6TBx2, RAID0
ALMAWebQL v2 FITS cache2.5" SATA3 SSD
1TBx8, RAID5
main FITS storage

5 / 10ALMAWebQL v2

Technical Architecture

under the hood: mmaped FITS files
custom multi-threaded reading of FITS files
from SSD into RAM (cfitsio library too slow
with 10GB+ FITS)
on-the-fly ENDIAN and FLOAT32 to FLOAT16(half-float) conversion whilst reading FITS
FITS files mmapped to Linux kernel user
space memory instead of read()/write() calls
efficiency gains: mmap avoids extra memorybuffers
Linux kernel device drivers use mmap too
6 / 10ALMAWebQL v2

Technical Architecture

under the hood: multi-threading
OpenMP multi-threading (16-core server, dualIntel Xeon E5-2640 v3 @ 2.60GHz CPU)Intel C/C++ compiler for the main C code
efficient AVX2 SIMD vectorisation with thelesser-known Intel SPMD Program Compiler

https://ispc.github.io/
Intel SPMD compiler accelerates frequency
spectrum calculation
image creation routines also employ Intel
SPMD compiler
7 / 10ALMAWebQL v2

Technical Architecture

under the hood: memory allocation
multi-threaded jemalloc memory allocator

http://www.canonware.com/jemalloc/
faster than glibc malloc/freeavoids memory fragmentation when dealing
with large data (i.e. arrays > 10GB)
FITS files stored in a half-float (16-bit FLOAT)
little-endian binary format in a custom
ALMAWebQL v2 FITS cache
50% memory consumption reduction(processing a 24GB FITS file consumes
12GB RAM server-side)
8 / 10ALMAWebQL v2

Technical Architecture

multi-threaded PNG image creationa custom C code 10x faster than libpng
parallel lossless PNG zlib compression (NOdependency on slow sequential libpng)
a single image split into multiple chunks to becompressed separately in parallel zlib streams
independent adler32 checksum calculation ineach zlib stream
adler32 zlib checksums are combined at theend of the processparallel checksumming of PNG IDAT chunks
9 / 10ALMAWebQL v2

Technical Architecture

LZ4-compressed FITS headers
text compresses very well (typically 5x)
FITS headers contain a lot of duplicatedstrings
compressed FITS headers sent from theserver to the web browser asbase64-encoded JSON strings
client-side JavaScript LZ4 decompression (ina browser)

10 / 10